DEBUG: Filtered offsite request to

在做爬虫项目时，出现了一个问题，解析一个网站二次爬取时没有获取到数据，就写了一个测试程序试了下，测试程序如下

import scrapy
from scrapy.linkextractors import LinkExtractor
from scrapy.spiders import CrawlSpider, Rule

class ZhenaiSpider(CrawlSpider):
name = 'zhenai'
allowed_domains = ['www.zhenai.com']
start_urls = ['http://www.zhenai.com/zhenghun/beijing/1']

rules = (
Rule(LinkExtractor(allow=r'http://www.zhenai.com/zhenghun/beijing/\d+'), callback='parse_item', follow=False),
)

def parse_item(self, response):
a_list = response.xpath('//div[@class="content"]/table//tr/th/a')
for a in a_list:
item = {}
title = a.xpath('./text()').extract_first()
item['title'] = title
detail_url = a.xpath('./@href').extract_first()
yield scrapy.Request(url=detail_url, meta={'item': item}, callback=self.parse_info)

def parse_info(self, response):
print('ok')
item = response.meta['item']

yield item

结果无法打印ok字符，也没有错误日志出现

打开调试模式，发现 [scrapy.spidermiddlewares.offsite] DEBUG: Filtered offsite request to 'album.zhenai.com': <GET http://album.zhenai.com/u/109625287>，原来是二次解析的域名被过滤掉了，解决办法

解决办法一：yield scrapy.Request(url=detail_url, meta={'item': item}, callback=self.parse_info, dont_filter=True)

原理：忽略allowed_domains的过滤

解决办法二：将allowed_domains = ['www.zhenai.com']更改为allowed_domains = ['zhenai.com'] 即更换为对应的一级域名

DEBUG: Filtered offsite request to相关推荐

爬虫中遇到的问题Crawled (404)，[scrapy.spidermiddlewares.offsite] DEBUG: Filtered offsite request to
1.错误1:url地址有误 Crawled (200) <GET http://www.itcast.cn/robots.txt> (referer: None) DEBUG: Crawl ...
[scrapy] DEBUG: Filtered offsite request to
[scrapy 常见问题整理] DEBUG: Filtered offsite request to 使用scrapy爬取豆瓣TOP250电影信息在进行自动翻页爬取的时候,出现了一个问题,解析自动翻页 ...
2018-12-06 18:09:55 [scrapy.spidermiddlewares.offsite] DEBUG: Filtered offsite request to 'weixin.so
2018-12-06 18:09:55 [scrapy.spidermiddlewares.offsite] DEBUG: Filtered offsite request to 'weixin.so ...
[scrapy.spidermiddlewares.offsite] DEBUG: Filtered offsite request to ‘XXX‘
一.问题描述 scrapy-redis中使用RedisCrawlSpider类爬虫,以Rule规则来匹配地址,运行爬虫时出现错误: [scrapy.spidermiddlewares.offsite] ...
scrapy运行时提示DEBUG: Filtered offsite request to
用scrapy爬取数据时,使用yield scrapy.Requeste()不能把数据传到下一个解析函数时,提示Filtered offsite request to [域名] 说明此时你请求的域名l ...
python爬虫学习日记(1) scrapy爬取时，报错Filtered offsite request
使用scrapy框架抓取某妹子图网时,只能抓取第一页,第二页报错 [scrapy.spidermiddlewares.offsite] DEBUG: Filtered offsite request ...
用scrapy写爬虫显示 Filtered offsite request to 错误.
爬电源网公司信息,无法爬取. 查看控制台发现报如下错误: DEBUG: Filtered offsite request to 'product.dianyuan.com': 果断上网百度 ,找到答案 ...
scrapy 爬网站显示 Filtered offsite request to 错误.
爬取zol 网站图片,无法抓取. 在 setting.py 文件中设置日志记录等级 LOG_LEVEL= 'DEBUG' LOG_FILE ='log.txt' 查看日志发现报 2015-11 ...
scrapy 爬网站显示 Filtered offsite request to 错误
查看日志发现报 2018-09-12 00:27:58 [scrapy.spidermiddlewares.offsite] DEBUG: Filtered offsite request to ' ...

DEBUG: Filtered offsite request to

DEBUG: Filtered offsite request to相关推荐

最新文章

热门文章