[scrapy] DEBUG: Filtered offsite request to

[scrapy 常见问题整理] DEBUG: Filtered offsite request to

使用scrapy爬取豆瓣TOP250电影信息在进行自动翻页爬取的时候，出现了一个问题，解析自动翻页之后网页爬取时没有获取到数据。

测试代码:

# -*- coding: utf-8 -*-
import scrapy
from douDanMovie.items import DoudanmovieItem
from scrapy import Requestclass DoubanSpiderSpider(scrapy.Spider):name = "douban_spider"allowed_domains = ["www.douban.com"]start_urls = ('https://movie.douban.com/top250',)headers = {'User-Agent': 'Mozilla/5.0 (Windows NT 6.1; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/53.0.2785.143 Safari/537.36',}def start_requests(self):url = 'https://movie.douban.com/top250'yield Request(url, headers=self.headers)def parse(self, response):item = DoudanmovieItem()movies = response.xpath('//ol[@class="grid_view"]/li')for movie in movies:item['ranking'] = movie.xpath('.//div[@class="pic"]/em/text()').extract()[0]item['movie_name'] = movie.xpath('.//div[@class="hd"]/a/span[1]/text()').extract()[0]item['score'] = movie.xpath('.//div[@class="star"]/span[@class="rating_num"]/text()').extract()[0]item['score_num'] = movie.xpath('.//div[@class="star"]/span/text()').re(r'(\d+)人评价')[0]yield itemnext_url = response.xpath('//span[@class="next"]/a/@href').extract()#此处解析的 next_url数据正常if next_url:next_url = 'https://movie.douban.com/top250' + next_url[0]yield Request(url = next_url,headers=self.headers)

错误信息：

2018-11-24 12:06:01 [scrapy] DEBUG: Filtered offsite request to 'movie.douban.com': <GET https://movie.douban.com/top250?start=25&filter=>
2018-11-24 12:06:01 [scrapy] INFO: Closing spider (finished)
2018-11-24 12:06:01 [scrapy] INFO: Dumping Scrapy stats:
{'downloader/request_bytes': 301,

问题分析：
由于在allowed_domains中定义了“www.douban.com”，在进行翻页二次解析的时候域名“https://movie.douban.com/top250?start=25&filter=>
”和allowed_domains中定义的不一致，因此将该域名给过滤掉了

问题解决：

将allowed_domains = [‘www.douban.com’]更改为allowed_domains = [‘豆瓣.com’] 即更换为对应的一级域名
在进行二次request的时候，通过将dont_filter设置为True，不样将request给过滤掉。
如下为Request的定义：
class scrapy.http.Request(url[, callback, method=‘GET’, headers, body, cookies, meta, encoding=‘utf-8’, priority=0, dont_filter=False, errback])
dont_filter 参数说明，其默认为False
dont_filter (boolean) – indicates that this request should not be filtered by the scheduler. This is used when you want to perform an identical request multiple times, to ignore the duplicates filter. Use it with care, or you will get into crawling loops. Default to False.

[scrapy] DEBUG: Filtered offsite request to相关推荐

爬虫中遇到的问题Crawled (404)，[scrapy.spidermiddlewares.offsite] DEBUG: Filtered offsite request to
1.错误1:url地址有误 Crawled (200) <GET http://www.itcast.cn/robots.txt> (referer: None) DEBUG: Crawl ...
2018-12-06 18:09:55 [scrapy.spidermiddlewares.offsite] DEBUG: Filtered offsite request to 'weixin.so
2018-12-06 18:09:55 [scrapy.spidermiddlewares.offsite] DEBUG: Filtered offsite request to 'weixin.so ...
[scrapy.spidermiddlewares.offsite] DEBUG: Filtered offsite request to ‘XXX‘
一.问题描述 scrapy-redis中使用RedisCrawlSpider类爬虫,以Rule规则来匹配地址,运行爬虫时出现错误: [scrapy.spidermiddlewares.offsite] ...
DEBUG: Filtered offsite request to
在做爬虫项目时,出现了一个问题,解析一个网站二次爬取时没有获取到数据,就写了一个测试程序试了下,测试程序如下 import scrapy from scrapy.linkextractors impo ...
scrapy运行时提示DEBUG: Filtered offsite request to
用scrapy爬取数据时,使用yield scrapy.Requeste()不能把数据传到下一个解析函数时,提示Filtered offsite request to [域名] 说明此时你请求的域名l ...
python爬虫学习日记(1) scrapy爬取时，报错Filtered offsite request
使用scrapy框架抓取某妹子图网时,只能抓取第一页,第二页报错 [scrapy.spidermiddlewares.offsite] DEBUG: Filtered offsite request ...
scrapy 爬网站显示 Filtered offsite request to 错误
查看日志发现报 2018-09-12 00:27:58 [scrapy.spidermiddlewares.offsite] DEBUG: Filtered offsite request to ' ...
用scrapy写爬虫显示 Filtered offsite request to 错误.
爬电源网公司信息,无法爬取. 查看控制台发现报如下错误: DEBUG: Filtered offsite request to 'product.dianyuan.com': 果断上网百度 ,找到答案 ...
scrapy 爬网站显示 Filtered offsite request to 错误.
爬取zol 网站图片,无法抓取. 在 setting.py 文件中设置日志记录等级 LOG_LEVEL= 'DEBUG' LOG_FILE ='log.txt' 查看日志发现报 2015-11 ...

[scrapy] DEBUG: Filtered offsite request to

[scrapy] DEBUG: Filtered offsite request to相关推荐

最新文章

热门文章