一、中间件的使用

1-1 具体方法详解

1-1-1 process_request - 正常请求调用

1-1-2 process_response - 正常返回调用

1-1-3 process_exception - 捕获错误调用

二、Proxy 相关官方中间件

2-1 HttpProxyMiddleware

2-2 RetryMiddleware

2-2-1 源码分析

三、实现代理 IP池

3-1 代理 IP 设置

3-2 代理 ip 爬取 + mongodb 存储

3-3 代理池的随机代理设置

3-4 Rertry 更换代理并删除无效 ip

四、IPProxyPool - IP 池 Python 脚本

五、scrapy-proxies-tool - Scrapy 的 IP池 Python库

python爬虫scrapy之downloader_middleware设置proxy代理

scrapy代理的配置方法

一、中间件的使用

官方 - 下载中间件 -文档

from scrapy import signalsclass MyscrapyDownloaderMiddleware(object):# Not all methods need to be defined. If a method is not defined,# scrapy acts as if the downloader middleware does not modify the# passed objects.@classmethoddef from_crawler(cls, crawler):# This method is used by Scrapy to create your spiders.s = cls()crawler.signals.connect(s.spider_opened, signal=signals.spider_opened)return sdef process_request(self, request, spider):# Called for each request that goes through the downloader# middleware.# Must either:# - return None: continue processing this request# - or return a Response object# - or return a Request object# - or raise IgnoreRequest: process_exception() methods of#   installed downloader middleware will be calledreturn Nonedef process_response(self, request, response, spider):# Called with the response returned from the downloader.# Must either;# - return a Response object# - return a Request object# - or raise IgnoreRequestreturn responsedef process_exception(self, request, exception, spider):# Called when a download handler or a process_request()# (from other downloader middleware) raises an exception.# Must either:# - return None: continue processing this exception# - return a Response object: stops process_exception() chain# - return a Request object: stops process_exception() chainpassdef spider_opened(self, spider):spider.logger.info('Spider opened: %s' % spider.name)

1-1 具体方法详解

1-1-1 process_request - 正常请求调用

1-1-2 process_response - 正常返回调用

1-1-3 process_exception - 捕获错误调用

二、Proxy 相关官方中间件

2-1 HttpProxyMiddleware

官方 - HttpProxy 中间件 - 文档

2-2 RetryMiddleware

官方 - RetryMiddleware - 文档

注意！！不支持使用空来进行配置，只能使用 ‘1/0’和‘true/flase’进行配置

2-2-1 源码分析

import logging
from twisted.internet import defer
from twisted.internet.error import TimeoutError, DNSLookupError, \ConnectionRefusedError, ConnectionDone, ConnectError, \ConnectionLost, TCPTimedOutError
from twisted.web.client import ResponseFailedfrom scrapy.exceptions import NotConfigured
from scrapy.utils.response import response_status_message
from scrapy.core.downloader.handlers.http11 import TunnelError
from scrapy.utils.python import global_object_namelogger = logging.getLogger(__name__)class RetryMiddleware(object):# IOError is raised by the HttpCompression middleware when trying to# decompress an empty responseEXCEPTIONS_TO_RETRY = (defer.TimeoutError, TimeoutError, DNSLookupError,ConnectionRefusedError, ConnectionDone, ConnectError,ConnectionLost, TCPTimedOutError, ResponseFailed,IOError, TunnelError)def __init__(self, settings):if not settings.getbool('RETRY_ENABLED'):raise NotConfiguredself.max_retry_times = settings.getint('RETRY_TIMES')self.retry_http_codes = set(int(x) for x in settings.getlist('RETRY_HTTP_CODES'))self.priority_adjust = settings.getint('RETRY_PRIORITY_ADJUST')@classmethoddef from_crawler(cls, crawler):return cls(crawler.settings)def process_response(self, request, response, spider):if request.meta.get('dont_retry', False):return responseif response.status in self.retry_http_codes:reason = response_status_message(response.status)return self._retry(request, reason, spider) or responsereturn response# 捕获错误信息触发执行def process_exception(self, request, exception, spider):# 如果错误属于捕获列表内的错误，并且，请求元信息内设置的配置为设置重启 - 处理错误请求if isinstance(exception, self.EXCEPTIONS_TO_RETRY) \and not request.meta.get('dont_retry', False):return self._retry(request, exception, spider)def _retry(self, request, reason, spider):# 重试次数获取retries = request.meta.get('retry_times', 0) + 1# 最多重试次数retry_times = self.max_retry_timesif 'max_retry_times' in request.meta:retry_times = request.meta['max_retry_times']stats = spider.crawler.stats# 若重试次数未达到最多重试次数if retries <= retry_times:logger.debug("Retrying %(request)s (failed %(retries)d times): %(reason)s",{'request': request, 'retries': retries, 'reason': reason},extra={'spider': spider})retryreq = request.copy()retryreq.meta['retry_times'] = retriesretryreq.dont_filter = Trueretryreq.priority = request.priority + self.priority_adjustif isinstance(reason, Exception):reason = global_object_name(reason.__class__)stats.inc_value('retry/count')stats.inc_value('retry/reason_count/%s' % reason)# 返回request对象 （属于 process_exception 返回）return retryreq# 否则不进行重试else:stats.inc_value('retry/max_reached')logger.debug("Gave up retrying %(request)s (failed %(retries)d times): %(reason)s",{'request': request, 'retries': retries, 'reason': reason},extra={'spider': spider})

三、实现代理 IP池

3-1 代理 IP 设置

在 process_request 方法内，每次经过该中间件的该方法，就会自动包裹代理ip

验证代理是否有效

3-2 代理 ip 爬取 + mongodb 存储

https://www.kuaidaili.com/free/inha/1/ - 代理ip获取页

import scrapy
from bs4 import BeautifulSoupfrom myscrapy.items import IpSpiderItemclass IpSpider(scrapy.Spider):name = 'ipspider'# 替代start_urlsdef start_requests(self):# dont_filter=False 开启去重功能（爬过的url不再爬取），需要 setting 内的配置# 根据爬取页规则，指定爬取页数for page in range(1, 5):yield scrapy.Request('https://www.kuaidaili.com/free/inha/%s/' % page, self.parse, dont_filter=True, )def parse(self, response):body = str(response.body, encoding='utf-8')ippool = BeautifulSoup(body, 'lxml')iptable = ippool.find('table', class_='table table-bordered table-striped')ip_trs = iptable.tbody.find_all('tr')for tr in ip_trs:ip = tr.find('td', attrs={'data-title': 'IP'}).textport = tr.find('td', attrs={'data-title': 'PORT'}).textprint('http://%s:%s' % (ip, port))item = IpSpiderItem()item['ip'] = 'http://%s:%s' % (ip, port)yield item

3-3 代理池的随机代理设置

import pymongo
import random
from scrapy.downloadermiddlewares.httpproxy import HttpProxyMiddlewareclass MyProxyMiddleware(HttpProxyMiddleware):def process_request(self, request, spider):conn = pymongo.MongoClient(spider.settings.get('MONGO_URI'))db = conn[spider.settings.get('MONGO_DATABASE')]collection = db['ips']ip = random.choice([obj for obj in collection.find()])request.meta['proxy'] = ip['ip']

3-4 Rertry 更换代理并删除无效 ip

更换代理思路总结

由于，代理配置 process_request 方法内，所以每次请求都会是新的 ip 包裹，则使用 process_exception 或 process_response 方法，业务处理后（删除库中IP等），返回 request 对象，重新发送请求。（详情参考 1-1 方法返回值解释）

注意！不要忘记在 settings 内开启 DOWNLOADER_MIDDLEWARES 配置

class MyProxyMiddleware(object):# 中间件权值小于默认543，优先原有的retrydef mongo_conn(self, spider):conn = pymongo.MongoClient(spider.settings.get('MONGO_URI'))db = conn[spider.settings.get('MONGO_DATABASE')]collection = db['ips']# ip = random.choice([obj for obj in collection.find()])# print(ip['ip'])return collection# 每次访问页面都会通过该方法进行代理包裹def process_request(self, request, spider):collection = self.mongo_conn(spider)ip = random.choice([obj for obj in collection.find()])request.meta['proxy'] = ip['ip']print('=========', request.meta['proxy'])# 非错误情况下的状态码处理def process_response(self, request, response, spider):if response.status == 503:request.replace(url='http://2019.ip138.com/ic.asp')return requestreturn responsedef process_exception(self, request, exception, spider):# 出现异常时（超时）使用代理print("error retry ……")# 在库中删除无效ipcollection = self.mongo_conn(spider)del_ip = {"ip": request.meta['proxy']}collection.delete_one(del_ip)# 返回request，重新发送请求return request

四、IPProxyPool - IP 池 Python 脚本

官方 github - IPProxyPool

作者 - 博客 - 详细使用方法

五、scrapy-proxies-tool - Scrapy 的 IP池 Python库

Random proxy middleware for Scrapy

官方 - scrapy-proxies - github

Scrapy - IP 池相关推荐

Scrapy ip代理池
一.概述在众多的网站防爬措施中,有一种是根据ip的访问频率进行限制,即在某一时间段内,当某个ip的访问次数达到一定的阀值时,该ip就会被拉黑.在一段时间内禁止访问. 应对的方法有两种: 1. 降低爬 ...
python—简单数据抓取七（采取蘑菇API代理设置scrapy的代理IP池并利用redis形成队列依次使用，利用ip池访问网页并将scrapy爬取转移到items的数据存入到数据库）
学习目标: Python学习二十七-简单数据抓取七学习内容: 1.采取蘑菇API代理设置scrapy的代理IP池并利用redis形成队列依次使用 2.利用ip池访问网页并将scrapy爬取转移到it ...
pythonscrapy爬虫ip代理池_在Scrapy中使用IP池或用户代理（python3）
一.创建Scrapy工程 1 scrapy startproject 工程名二.进入工程目录,根据爬虫模板生成爬虫文件 1 scrapy genspider -l # 查看可用模板2 scrapy ...
scrapy中添加ip池的方法
scrapy中添加ip池的方法我使用的是scrapy2.2 setting 中写下ip池 IPPOOL = [{'ipaddr':'221.230.72.165:80'}, {'ipaddr':'1 ...
[爬虫架构] 如何在分布式爬虫架构中动态维护一个代理IP池（付费代理）
前言: 当分布式爬虫使用代理IP技术时,通过直接在爬虫程序中添加平台api接口的方式已经不能满足我们了,因为分布式的爬虫架构每秒的代理IP-URL的请求数会远远大于平台限制的请求频率,当然,对于没有限 ...
Scrapy ip代理无响应
Scrapy ip代理长时间无响应原因分析在setting.py中添加可用ip代理池: PROXIES=['http://182.149.82.74:9999','http://121.237.25 ...
requests模块代理IP池搭建视频爬取
requests模块&代理IP池搭建一 requests模块使用 1.1 get请求 1.2 url编码和解码 1.3 携带请求头 1.4 携带cookie 1.5 发送post请求 1.6 ...
爬虫代理ip池构建及使用
ip有效性检测 1.对于免费代理的检测 #免费代理或不用密码的代理 url = 'http://httpbin.org/get'proxy = '127.0.0.0:8000'proxies = {' ...
构建一个给爬虫使用的代理IP池
做网络爬虫时,一般对代理IP的需求量比较大.因为在爬取网站信息的过程中,很多网站做了反爬虫策略,可能会对每个IP做频次控制.这样我们在爬取网站时就需要很多代理IP. 代理IP的获取,可以从以下几个途径 ...

Scrapy - IP 池

一、中间件的使用

1-1 具体方法详解

1-1-1 process_request - 正常请求调用

1-1-2 process_response - 正常返回调用

1-1-3 process_exception - 捕获错误调用

二、Proxy 相关官方中间件

2-1 HttpProxyMiddleware

2-2 RetryMiddleware

2-2-1 源码分析

三、实现代理 IP池

3-1 代理 IP 设置

3-2 代理 ip 爬取 + mongodb 存储

3-3 代理池的随机代理设置

3-4 Rertry 更换代理并删除无效 ip

四、IPProxyPool - IP 池 Python 脚本

五、scrapy-proxies-tool - Scrapy 的 IP池 Python库

Scrapy - IP 池相关推荐

最新文章

热门文章