python（七）爬虫框架

常见的爬虫框架

Scrapy框架：Scrapy框架是一套比较成熟的Python爬虫框架，是使用Python开发快速，高层次的信息爬取框架，可以高效的爬取web页面并提取出结构化的数据
PySpider框架：PySpider是以Python脚本为驱动的抓取环模型爬虫框架
Crawley框架：Crawley也是python开发的框架，该框架致力于改变人们从互联网中提取数据的方式
Portia框架：Portia框架是一款允许没有任何变成基础的用户可视化地爬取网页的爬虫框架
Newspaper：Newspaper框架是一款用来提取新闻、文章以及内容分析的Python爬虫框架

Scrapy框架的工作原理

Scrapy Engin（引擎）：Scrapy框架的核心部分。负责在Spider和Item Pipeline、Downloader、Scheduler中间通信、传递数据的
Spider（爬虫）：发送需要爬取的连接给引擎，最后引擎把其他模块请求回来的数据再发给爬虫，爬虫就去解析想要的数据
Scheduler（调度器）：负责接收引擎发送过来的请求，并按照一定的方式进行排列和整理，负责调度请求的顺序等。
Downloader（下载器）：负责接收引擎发送过来的请求，然后去网络上下载对应的数据再交给引擎。
Item Pipeline（管道）：负责将爬虫传递过来的数据进行保存
Downloader Middlewares(下载中间件):可以扩展下载器和引擎之间通信功能的中间件
Spider Middlewares(爬虫中间件):可以扩展引擎和爬虫之间通信功能的中间件

scrapy项目

创建项目：语法：scrapy startproject 项目名称 (通过命令创建)

scrapy框架的使用

爬取豆瓣top250图书

项目结构：

修改配置文件settings

BOT_NAME = 'douban'SPIDER_MODULES = ['douban.spiders']
NEWSPIDER_MODULE = 'douban.spiders'
ROBOTSTXT_OBEY = FalseDEFAULT_REQUEST_HEADERS = {'Accept': 'text/html,application/xhtml+xml,application/xml;q=0.9,*/*;q=0.8','Accept-Language': 'en','User-Agent': 'Mozilla/5.0 (Windows NT 10.0; WOW64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/86.0.4240.111 Safari/537.36'
}ITEM_PIPELINES = {'douban.pipelines.DoubanPipeline': 300,
}

编写items文件

import scrapy
class DoubanItem(scrapy.Item):# define the fields for your item here like:# name = scrapy.Field()title = scrapy.Field()publish = scrapy.Field()score = scrapy.Field()

编写爬虫文件 top250book

import scrapy
from bs4 import BeautifulSoup
from ..items import DoubanItem
class DoubanSpider(scrapy.Spider):name = 'douban' #定义爬虫的名称，启动程序时使用addowed_domains=['book.douban.com/top250'] # 定义允许爬虫爬取的域名start_urls = ['https://book.douban.com/top250'] # 定义的起始的网址,告诉爬虫应该从哪个网址开始爬取# parse是scrapy里默认的处理response（响应）的一个方法def parse(self, response, **kwargs):bs = BeautifulSoup(response.text,'html.parser')tr_tags = bs.find_all('tr',class_='item')for tr in tr_tags:item = DoubanItem() # 这个类的对象title = tr.find_all('a')[1]['title'] # 提取书的名称publish = tr.find('p',class_='pl').text # 提取出版信息score = tr.find('span',class_='rating_nums').text # 提取评分信息item['title'] = titleitem['publish'] = publishitem['score'] = score# print([title,publish,score])# 数据封装完毕后，需要提交给引擎yield item

编写pipelines文件

from itemadapter import ItemAdapterimport openpyxl
class DoubanPipeline:def __init__(self):self.wb = openpyxl.Workbook()self.ws = self.wb.active# 添加表头self.ws.append(['名称','出版信息','评分'])def process_item(self, item, spider):# 数据存储操作line = [item['title'],item['publish'],item['score']]self.ws.append(line) # 将列表中的数据存储到工作表中return item# 定义一个关闭的方法def close_spider(self,spider):self.wb.save('book.xlsx')self.wb.close()

编写并运行start文件

from scrapy import cmdline
# cmdline.execute('scrapy crawl douban'.split(' '))
# print('scrapy crawl douban'.split(' '))
cmdline.execute(['scrapy', 'crawl', 'douban'])

运行完之后数据会保存到book.xlsx中。如图：

爬取笔趣阁小说

分析URL：提取每页中章节小说的URL
创建项目
修改配置文件settings.py


BOT_NAME = 'story'SPIDER_MODULES = ['story.spiders']
NEWSPIDER_MODULE = 'story.spiders'USER_AGENT = 'Mozilla/5.0 (Windows NT 10.0; WOW64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/86.0.4240.111 Safari/537.36'DOWNLOAD_DELAY = 1 #下载延迟
DEFAULT_REQUEST_HEADERS = {'Accept': 'text/html,application/xhtml+xml,application/xml;q=0.9,*/*;q=0.8','Accept-Language': 'en','Cookie': 'fontFamily=null; fontColor=null; fontSize=null; bg=null; UM_distinctid=175b634de922cf-0b54db46c3c024-3e604000-144000-175b634de9343e; Hm_lvt_0684ece53fc1331e366f282d44f9c519=1605079720; ASP.NET_SessionId=j4uj1auhycpqkfe1da5tmfta; bookid=9118; CNZZDATA1278951146=472983357-1605078139-%7C1605083129; booklist=%257B%2522BookId%2522%253A9118%252C%2522ChapterId%2522%253A3329140%252C%2522ChapterName%2522%253A%2522%25u65B0%25u4E66%25u53D1%25u5E03%2522%257D; fikker-KqHJ-Hc8Q=jJJW0OhJR3jSHeBJ0JTUBqnMOKIJDiKW; fikker-KqHJ-Hc8Q=jJJW0OhJR3jSHeBJ0JTUBqnMOKIJDiKW; Hm_lpvt_0684ece53fc1331e366f282d44f9c519=1605087647'
}ITEM_PIPELINES = {'story.pipelines.StoryPipeline': 300,
}

编写爬虫文件 qustory

import scrapy
class QustorySpider(scrapy.Spider):name = 'qustory'allowed_domains = ['qu.la']# start_urls = ['https://www.qu.la/book/9118/3324988.html']start_urls = ['https://www.qu.la/book/9118/3325320.html']# start_urls = ['https://www.qu.la/book/9118/3329139.html']def parse(self, response):# 解析提取数据title = response.xpath('//h1[@class="title"]/text()').extract()content = response.xpath('string(//div[@id="content"])').extract_first().strip().replace('    ','')\.replace('@@','').replace('sjwzl("乾坤剑神");','').replace('　　　　','\n')next_url = 'https://www.qu.la/'+response.xpath('//div[@class="section-opt"]/a[3]/@href').extract_first()# print(next_url)yield {'title':title,'content':content}if next_url != 'https://www.qu.la/javascript:void(0);':yield scrapy.Request(next_url,callback=self.parse)else:print('小说爬取完毕')

编写pipelines

from itemadapter import ItemAdapter
class StoryPipeline:def open_spider(self,spider):self.file = open('乾坤剑神.txt','w+',encoding='utf-8')def process_item(self, item, spider):info = ''.join(item['title'])+"\t"+item['content']+"\n"self.file.write(info)return itemdef close_spider(self,spider):self.file.close()

scrapy框架提取数据的方法

方法/函数	描述
xpath()	它返回选择器列表，代表由xpath表达式参数选择的节点
css()	它返回选择器列表，代表由css表达式参数选择的节点
re()	返回unicode字符串列表，当正则表达式作为参数时提取
extract()	返回一个unicode字符串以及所选择数据
extract_first()	返回第一个unicode字符串及所选数据

CrawlSpider爬虫

继承Spider爬虫，在之前的基础上增加了新的功能，可以定义爬取的url的规则，以后scrapy碰到满足条件的url都进行爬取，而不用手动yield Request
语法:scrapy genspider -t crawl 爬虫名称域名

LinkExtractors链接提取器

使用LinkExtractors可以不用程序提取自己想要的URL，然后发送请求。这些工作都可以交给LinkExtractors，它会在所有爬的页面中找到满足规则的URL，进行自动爬取
主要参数：

allow：允许的url，所有满足这个正则表达式的url都会被提取
deny：禁止的url，所有满足这个正则表达式的url都不会被提取
allow_domains:允许的域名，只有在这个里面指定的域名的url才会被提取
deny_domains:禁止的域名，所有在这个里面指定的域名的url都不会被提取
restrict_xpaths:严格的xpath，和allow共同过滤链接

Rule规则类

主要参数：

link_extractor:一个Link_Extractor对象，用于定义爬虫规则
callback:满足这个规则的url，应该要执行哪个回调函数
follow:指定根据该规则从response中提取的链接是否需要跟进
process_links:从link_extractor中获取到链接后会传递给这个参数，用来过滤不需要爬取的链接

实现猎云网文章数据爬取，异步保存到mysql数据库

同步和异步是相对的
同步就是客户端发送请求给服务器，等待服务器相应的时候，客户端不能做其他事情
异步就是客户端发送请求给服务器，等待响应的同时，客户端可以做其他事情

编写配置文件

BOT_NAME = 'lieyun'SPIDER_MODULES = ['lieyun.spiders']
NEWSPIDER_MODULE = 'lieyun.spiders'
ROBOTSTXT_OBEY = FalseITEM_PIPELINES = {'lieyun.pipelines.LieyunPipeline': 300,
}MYSQL_DB_CONFIG={'DRIVER':'mysql.connector','HOST':'127.0.0.1','USER':'admin','PASSWORD':'admin123','DATABASE':'lywdb','auth_plugin':'mysql_native_password'
}

读取配置文件

from itemadapter import ItemAdapter
from twisted.enterprise import adbapiclass LieyunPipeline:def __init__(self,mysql_config):# 创建连接池对象self.dbpool = adbapi.ConnectionPool(mysql_config['DRIVER'],host=mysql_config['HOST'],user=mysql_config['USER'],password = mysql_config['PASSWORD'],database=mysql_config['DATABASE'],auth_plugin = mysql_config['auth_plugin'],charset = 'utf8')@classmethoddef from_crawler(cls,crawler): #只要重写该方法，那么以后创建对象时，就会调用该方法获取pipeline对象mysql_config = crawler.settings['MYSQL_DB_CONFIG']return cls(mysql_config)def process_item(self, item, spider):result = self.dbpool.runInteraction(self.insert_item,item)result.addErrback(self.insert_error)return item# 执行sql语句的方法def insert_item(self,cursor,item):sql = 'insert into article (title,author,pub_time,content,article_url) values (%s,%s,%s,%s,%s);'args = (item['title'],item['author'],item['publish_time'],item['content'],item['article_url'])cursor.execute(sql,args)def insert_error(self,failure):print('=============================')print(failure)print('=============================')

编写items

import scrapy
class LieyunItem(scrapy.Item):# define the fields for your item here like:# name = scrapy.Field()title = scrapy.Field()author = scrapy.Field()publish_time = scrapy.Field()content = scrapy.Field()article_url = scrapy.Field()

编写爬虫文件

from scrapy.linkextractors import LinkExtractor
from scrapy.spiders import CrawlSpider, Rule
from ..items import LieyunItemclass LywSpider(CrawlSpider):name = 'lyw'allowed_domains = ['lieyunwang.com']start_urls = ['https://www.lieyunwang.com/latest/p1.html']rules = (Rule(LinkExtractor(allow=r'/latest/p\d+.html'), follow=True),Rule(LinkExtractor(allow=r'/archives/\d+'), callback='parse_item',follow=False),)def parse_item(self, response):item = LieyunItem()title_list = response.xpath('//h1[@class="lyw-article-title-inner"]/text()').getall()title = ''.join(title_list).strip()time = response.xpath('//h1[@class="lyw-article-title-inner"]/span/text()').get() # 发布时间author_name = response.xpath('//a[contains(@class,"author-name open_reporter_box")]/text()').get() # 作者content = response.xpath('//div[@class="main-text"]//text()').getall()content = ''.join(content).strip()article_url = response.urlitem['title'] = titleitem['publish_time'] = timeitem['author'] = author_nameitem['content'] = contentitem['article_url'] = article_urlyield item

scrapy模拟登陆

使用FormRequest

import scrapy
class QdloginSpider(scrapy.Spider):name = 'qdlogin'allowed_domains = ['qu.la/']# start_urls = ['http://my.qidian.com//']def start_requests(self):url = 'https://www.qu.la/Login.php'formdata = {'action': 'login','username': '18','password': 'w1','usecookie': '720'}yield scrapy.FormRequest(url,formdata=formdata,callback=self.parse)def parse(self, response):yield scrapy.Request('https://www.qu.la/book/9118/',callback=self.parse_article)def parse_article(self,response):print(response.text)

scrapy下载图片文件

Scrapy提供了一个item pipeline，来下载属于某个特定项目的图片

避免重新下载最近已经下载过的文件
可以修改指定文件路径
可以将下载的图片转换成通用的图片格式（JPG等）
可以方便的生成缩略图
可以方便的检测图片的宽和高，确保他们满足最小限制
异步下载，效率非常高

settings文件

BOT_NAME = 'zcool'
SPIDER_MODULES = ['zcool.spiders']
NEWSPIDER_MODULE = 'zcool.spiders'
ROBOTSTXT_OBEY = False
from scrapy.pipelines.images import ImagesPipeline
ITEM_PIPELINES = {'zcool.pipelines.ZcoolPipeline': 300,# 'scrapy.pipelines.images.ImagesPipeline': 300,
}
IMAGES_STORE='d:/pic'

items文件

import scrapy
class ZcoolItem(scrapy.Item):image_urls = scrapy.Field()title = scrapy.Field()images  = scrapy.Field()

spider文件

from scrapy.linkextractors import LinkExtractor
from scrapy.spiders import CrawlSpider, Rule
from ..items import ZcoolItemclass ZcoolspiderSpider(CrawlSpider):name = 'zcoolspider'allowed_domains = ['zcool.com.cn']start_urls = ['https://www.zcool.com.cn/?p=1#tab_anchor']rules = (Rule(LinkExtractor(allow=r'/?p=\d+#tab_anchor'), follow=True),Rule(LinkExtractor(allow=r'/work/.+=.html'), callback='parse_item', follow=False),)def parse_item(self, response):img_urls = response.xpath('//div[@class="work-show-box mt-40 js-work-content"]//img/@src').getall()title_list = response.xpath('//h2/text()').getall()title = ''.join(title_list).strip()# yield 到pipelines中item = ZcoolItem(title=title,image_urls=img_urls)yield item

pipelines文件

import hashlib
import os
from itemadapter import ItemAdapter
from scrapy import Request
from scrapy.pipelines.images import ImagesPipeline
from scrapy.utils.python import to_bytes
from zcool import settings
class ZcoolPipeline(ImagesPipeline): # 自定义的用于下载图片的pipelinedef get_media_requests(self, item, info):image_requests = super().get_media_requests(item,info)# 获取图片链接请求的列表for img_req in image_requests:img_req.item = item # 对每个图片链接的请求都添加一个item属性return image_requests# 改变存储路径def file_path(self, request, response=None, info=None, *, item=None):old_path = super().file_path(request,response,info)title = request.item['title'] # 获取图片标题save_path =os.path.join(settings.IMAGES_STORE,title)# 原路径中提取文件名image_name = old_path.replace('full/','')return os.path.join(save_path,image_name)# print(os.path.join(save_path,image_name))

下载器中间件

编写下载器中间件，需要下载器中实现两个方法
process_request(self,request,spider),在请求发送之前会执行
process_response(self,request,response,spider),数据下载到引擎之前执行
process_request方法详解：
返回值为None:Scrapy将继续处理request,执行其他中间件的相应方法，知道合适的下载器处理函数被调用
返回Response对象：Scrapy将不会调用任何其他的process_request方法，将直接返回这个response对象，已经激活的中间件process_response()方法则会在每个response返回时被调用。
返回Request：不再使用之前的request对象去下载数据，而是根据现在返回的request对象返回数据
如果该方法抛出了异常，则会调用process_exception方法

process_response方法详解：
返回Response：会将这个新的response对象传给其他中间件，最终传给爬虫
返回Request：下载器链被切断，返回的request会重新被下载器调度下载
如果该方法中抛出了异常，那么调用requests的errback方法，如果没有指定这个方法，那么会抛出一个异常

scrapy 动态设置ua

导入fake_useragent模块中的类UserAgent

from scrapy import signals
from itemadapter import is_item, ItemAdapter
from fake_useragent import UserAgent
class HttpuaDownloaderMiddleware:def process_request(self, request, spider):request.headers['User-Agent'] = UserAgent().randomreturn Nonedef process_response(self, request, response, spider):return responsedef process_exception(self, request, exception, spider):passdef spider_opened(self, spider):spider.logger.info('Spider opened: %s' % spider.name)

selenium与scrapy结合获取百度主页代码

settings文件

BOT_NAME = 'ss'SPIDER_MODULES = ['ss.spiders']
NEWSPIDER_MODULE = 'ss.spiders'USER_AGENT = "Mozilla/5.0 (Windows NT 10.0; WOW64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/86.0.4240.111 Safari/537.36"ROBOTSTXT_OBEY = False
DOWNLOADER_MIDDLEWARES = {'ss.middlewares.SsDownloaderMiddleware': 300,
}

spider文件

import scrapy
from scrapy import Spider
from scrapy import signals
from selenium.webdriver import Chrome
class BaiduSpider(scrapy.Spider):name = 'baidu'allowed_domains = ['baidu.com']start_urls = ['http://baidu.com/']@classmethoddef from_crawler(cls, crawler, *args, **kwargs):spider = super().from_crawler(crawler,*args,**kwargs)spider.driver = Chrome() # 创建完爬虫对象之后，添加一个浏览器对象crawler.signals.connect(spider.spider_close,signals=signals.spider_closed)def spider_close(self,spider):spider.driver.close()def parse(self, response):print(response.text)

下载器中间件

# Define here the models for your spider middleware
#
# See documentation in:
# https://docs.scrapy.org/en/latest/topics/spider-middleware.htmlfrom scrapy import signals# useful for handling different item types with a single interface
from itemadapter import is_item, ItemAdapter
from selenium.webdriver import Chrome
from scrapy.http import HtmlResponseclass SsDownloaderMiddleware:# def __init__(self):#     self.driver = Chrome()def process_request(self, request, spider):spider.driver.get(request.url)html = spider.driver.page_sourcereturn HtmlResponse(url=request.url,request=request,body=html,encoding='utf-8')def process_response(self, request, response, spider):return responsedef process_exception(self, request, exception, spider):passdef spider_opened(self, spider):spider.logger.info('Spider opened: %s' % spider.name)

将链家租房信息保存到MongoDB中

settings

BOT_NAME = 'lianjia'SPIDER_MODULES = ['lianjia.spiders']
NEWSPIDER_MODULE = 'lianjia.spiders'ROBOTSTXT_OBEY = FalseDOWNLOADER_MIDDLEWARES = {'lianjia.middlewares.LianjiaDownloaderMiddleware': 300,
}
ITEM_PIPELINES = {'lianjia.pipelines.LianjiaPipeline': 300,
}
LOG_ENABLED = False

spider

import scrapy
class ZufangSpider(scrapy.Spider):name = 'zufang'allowed_domains = ['lianjia.com']start_urls = [f'https://bj.lianjia.com/zufang/pg{i}/#contentList' for i in range(1,3)]def parse(self, response):full_urls=[f'https://bj.lianjia.com{i}' for i in response.xpath('//div[@class="content__list--item--main"]/p[@class="content__list--item--title"]/a/@href').getall()]for item in full_urls:yield scrapy.Request(url=item,callback=self.parse_info)def parse_info(self,response):title = response.xpath('//p[@class="content__title"]/text()').get() # 标题total_price = response.xpath('//div[@class="content__aside--title"]/span/text()|//div[@class ="content__aside--title"]/text()').getall()price = ''.join(total_price).replace('\n            ','').strip() # 价格mode = response.xpath('//ul[@class="content__aside__list"]/li[1]/text()').get() # 租赁方式type = response.xpath('//ul[@class="content__aside__list"]/li[2]/text()').get() # 房屋类型direction = response.xpath('//ul[@class="content__aside__list"]/li[3]/span[2]/text()').get() # 朝向楼层elevator = response.xpath('//div[@class="content__article__info"]/ul/li[9]/text()').get() # 电梯parking = response.xpath('//div[@class="content__article__info"]/ul/li[11]/text()').get() # 车位water = response.xpath('//div[@class="content__article__info"]/ul/li[12]/text()').get() # 用水electric = response.xpath('//div[@class="content__article__info"]/ul/li[14]/text()').get() # 用电gas = response.xpath('//div[@class="content__article__info"]/ul/li[15]/text()').get() # 燃气heating = response.xpath('//div[@class="content__article__info"]/ul/li[17]/text()').get() # 采暖# print(title)yield {'title':title,'price':price,'mode':mode,'type':type,'direction':direction,'elevator':elevator,'parking':parking,'water':water,'electric':electric,'gas':gas,'heating':heating}

下载器中间件

from fake_useragent import UserAgent
from scrapy import signals
from itemadapter import is_item, ItemAdapterclass LianjiaDownloaderMiddleware:def process_request(self, request, spider):request.headers['User-Agent'] = UserAgent().randomreturn Nonedef process_response(self, request, response, spider):return responsedef process_exception(self, request, exception, spider):passdef spider_opened(self, spider):spider.logger.info('Spider opened: %s' % spider.name)

pipelines

from itemadapter import ItemAdapter
from pymongo import MongoClientclass LianjiaPipeline:def open_spider(self, spider):self.client = MongoClient() # 创建MongoDB服务器self.lianjia = self.client.zufang.lianjia # 创建数据库和文档def close_spider(self, spider):self.client.close()def process_item(self, item, spider):self.lianjia.insert(dict(item))