scrapy爬虫之中间件Middleware

设置随机请求头

spider

class MiddlewSpider(scrapy.Spider):name = 'middlew'allowed_domains = ['exercise.kingname.info']start_urls = ['http://exercise.kingname.info/exercise_middleware_ua']count = 1def parse(self, response):print("*"*40)print(response.xpath("//body").extract())print("*" * 40)time.sleep(1)self.count +=1yield scrapy.Request(url=self.start_urls[0]+"/"+str(self.count),callback=self.parse)item ={}yield itempass

middleware

import random
class UAMiddleware(object):def process_request(self,request,spider):USER = ["Mozilla/5.0 (Windows NT 10.0; WOW64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/45.0.2454.101 Safari/537.36","Dalvik/1.6.0 (Linux; U; Android 4.2.1; 2013022 MIUI/JHACNBL30.0)","Mozilla/5.0 (Linux; U; Android 4.4.2; zh-cn; HUAWEI MT7-TL00 Build/HuaweiMT7-TL00) AppleWebKit/533.1 (KHTML, like Gecko) Version/4.0 Mobile Safari/533.1","AndroidDownloadManager","Apache-HttpClient/UNAVAILABLE (java 1.4)","Dalvik/1.6.0 (Linux; U; Android 4.3; SM-N7508V Build/JLS36C)","Android50-AndroidPhone-8000-76-0-Statistics-wifi","Dalvik/1.6.0 (Linux; U; Android 4.4.4; MI 3 MIUI/V7.2.1.0.KXCCNDA)","Dalvik/1.6.0 (Linux; U; Android 4.4.2; Lenovo A3800-d Build/LenovoA3800-d)","Lite 1.0 ( http://litesuits.com )","Mozilla/4.0 (compatible; MSIE 8.0; Windows NT 5.1; Trident/4.0; .NET4.0C; .NET4.0E; .NET CLR 2.0.50727)","Mozilla/5.0 (Windows NT 6.1) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/38.0.2125.122 Safari/537.36 SE 2.X MetaSr 1.0","Mozilla/5.0 (Linux; U; Android 4.1.1; zh-cn; HTC T528t Build/JRO03H) AppleWebKit/534.30 (KHTML, like Gecko) Version/4.0 Mobile Safari/534.30; 360browser(securitypay,securityinstalled); 360(android,uppayplugin); 360 Aphone Browser (2.0.4)",]ua = random.choice(USER)request.headers['User-Agent'] = ua

settings

scrapy爬虫之中间件Middleware相关推荐

框架下载_25. Scrapy 框架-下载中间件Middleware
1. Spider 下载中间件(Middleware) Spider 中间件(Middleware) 下载器中间件是介入到 Scrapy 的 spider 处理机制的钩子框架,您可以添加代码来处理发送 ...
Scrapy 下载器中间件(Downloader Middleware)
Scrapy 下载器中间件官方文档:https://scrapy-chs.readthedocs.io/zh_CN/1.0/topics/downloader-middleware.html 官方英 ...
Scrapy爬虫框架第五讲（linux环境)【download middleware用法】
DOWNLOAD MIDDLEWRE用法详解通过上面的Scrapy工作架构我们对其功能进行下总结: (1).在Scheduler调度出队列时的Request送给downloader下载前对其进行修改 ...
Scrapy 2.6 Downloader Middleware 下载器中间件使用指南
Python3 的 Scrapy 爬虫框架中数据爬取过程中的下载器中间件是一个挂钩 Scrapy 的请求/响应处理的框架.是一个轻量级的低级系统并且应用于全局更改 Scrapy 的请求和响应. 其主 ...
三十八、Scrapy 下载中间件Middleware
@Author:Runsen 文章目录 1. Spider 下载中间件(Middleware) 2. 激活一个下载DOWNLOADER_MIDDLEWARES 3. 编写你自己的下载中间件 4 使用代 ...
scrapy 爬虫学习二[中间件的学习]
scrapy源码解析参考连接:https://www.jianshu.com/p/d492adf17312 ,直接看大佬的就行了,这里便就不多说了. 今天要学习的是:Scrapy框架中的downloa ...
Python 网络爬虫笔记9 -- Scrapy爬虫框架
Python 网络爬虫笔记9 – Scrapy爬虫框架 Python 网络爬虫系列笔记是笔者在学习嵩天老师的<Python网络爬虫与信息提取>课程及笔者实践网络爬虫的笔记. 课程链接:Py ...
一、scrapy爬虫框架——概念作用和工作流程 scrapy的入门使用
scrapy的概念和流程学习目标: 了解 scrapy的概念了解 scrapy框架的作用掌握 scrapy框架的运行流程掌握 scrapy中每个模块的作用 1. scrapy的概念 Scrap ...
python scrapy框架爬虫_Python Scrapy爬虫框架
Scrapy爬虫框架结构: 数据流的3个路径: 一: 1.Engine从Spider处获得爬取请求(Request) 2.Engine将爬取请求转发给Scheduler,用于调度二: 3.Engin ...

scrapy爬虫之中间件Middleware

scrapy爬虫之中间件Middleware相关推荐

最新文章

热门文章