【Python爬虫】：使用高性能异步多进程爬虫获取豆瓣电影Top250

在本篇博文当中，将会教会大家如何使用高性能爬虫，快速爬取并解析页面当中的信息。一般情况下，如果我们请求网页的次数太多，每次都要发出一次请求，进行串行执行的话，那么请求将会占用我们大量的时间，这样得不偿失。因此我们可以i使用高性能爬虫，也就是采用多进程，异步的方式对数据进行爬取和解析，这样就可以在更快的时间内得到我们想要的结果。本篇博文给出有关爬取豆瓣电影的例子，以此来教会大家如何使用高性能爬虫。

一.网页分析

首先我们来分析豆瓣电影的网页代码，在本次的案例当中。我们需要爬取豆瓣电影top250当中的标题title和星数star。

发现，豆瓣电影当中的所有有关电影的信息全部都隐藏在< ol class="grid view">这个标签，当中，因此我们在编写xpath的时候，可以利用对它做一个循环。然后又发现，对于电影的title而言，有两个地方出现，一个地方是在图片上，另一个地方是在span标签下的class = title处，但是在span标签下具有多个标题，为了以免引起混，因此我们使用图片当中所暗含的标题title文字，使用xpath进行定位即可。

对于star而言，就更加简单了。我们发现每次一个star的分数出现，就会有又一个<div class="star">的标签在前面，然后再出现了与span有关的标签，因此我们编写xpath表达式为：//ol[@class="grid_view"]//div[@class="star"]/span[@class="rating_num"]/text()

这样就可以得到一整个页面的star的数值了。当然这样我们只能获取第一页的我们想要得到的数据，怎么得到第二页的数据呢？

二.翻页处理

翻页处理对于豆瓣电影这个网站还是比较简单的。我们分别查看第一，二，三页的url，就会惊奇的发现它的网址如下：

https://movie.douban.com/top250?start=0&filter=
https://movie.douban.com/top250?start=25&filter=
https://movie.douban.com/top250?start=50&filter=

十分明显，这个网址后面有问号说明想要获取页面内容肯定需要发起get请求，都没有做有关post请求的加密，这样看来这也太简单了吧！

同样的我们发现里面的参数start在不断的变化，而filter却保持不变。因此我们只需要得到start参数的规律就知道该怎么编写爬虫了。

对于start而言，每跳转一页，就会增加25的数值，因为每一个页面里面均仅有25部电影。这样我们就找到了start参数的规律，开始编写爬虫。

三.爬虫代码的编写

在编写的代码时候，我们导入了多进程的库，使用这个库进行爬虫，也就只需要在原本代码的基础之上多添加两行代码即可，如下所示：

pool=Pool(4)
pool.map(get_information,number_ls)

这两行代码当中，第一个参数的4表示了我们使用4个进程的进程池进行数据的抓取。数值越大，爬取的效率就越高，这取决于你CPU的数量，数值不能超过CPU核心数的数量，因为一个一个CPU核心同时只能够运行单个进程。

第二行代码使用了map函数，第一个参数填写我们进行爬虫的函数，第二个参数填写爬虫函数所需要的参数。把这两个东西放到map函数里，就可以开始高性能爬虫了。

Remark：
在进行进程池爬虫的时候，我们放入的参数number_ls一定是一个列表，同时我们在get_imformation函数里使用得到的参数时，每次系统会调用这个参数列表当中的任意一个数值，而不是对整个列表进行调用。

由于整个原因，因此我们编写整个的代码·如下所示：

import requests
from lxml import etree
from multiprocessing.dummy import Pool
cookie='bid=N3Zqe_FFUKc; douban-fav-remind=1; viewed="27093751"; _vwo_uuid_v2=D401F17C96234AE149C4E04B78C3C8066|6fcc3cefe576bff2b89cdf28c4c5f597; __gads=ID=21cdec44606b00df-2250ba4d7ac4009b:T=1604034713:RT=1604034713:S=ALNI_Mb6iYJKYfbUjLxlisTQX5HCODTGKg; gr_user_id=fb6ac40c-94c3-400e-b170-47e126a9b78a; _gid=GA1.2.1520341169.1612004212; _ga=GA1.2.645228582.1602221486; ll="108288"; UM_distinctid=17752f076e4530-0b6eef25ebabba-f7b1332-1fa400-17752f076e57f0; Hm_lvt_19fc7b106453f97b6a84d64302f21a04=1612004228; Hm_lpvt_19fc7b106453f97b6a84d64302f21a04=1612004253; ap_v=0,6.0; _pk_ref.100001.4cf6=%5B%22%22%2C%22%22%2C1612004299%2C%22https%3A%2F%2Fwww.google.com%2F%22%5D; _pk_ses.100001.4cf6=*; __utma=30149280.645228582.1602221486.1611225800.1612004300.9; __utmb=30149280.0.10.1612004300; __utmc=30149280; __utmz=30149280.1612004300.9.9.utmcsr=google|utmccn=(organic)|utmcmd=organic|utmctr=(not%20provided); __utma=223695111.645228582.1602221486.1612004300.1612004300.1; __utmb=223695111.0.10.1612004300; __utmc=223695111; __utmz=223695111.1612004300.1.1.utmcsr=google|utmccn=(organic)|utmcmd=organic|utmctr=(not%20provided); _pk_id.100001.4cf6=9a1bb1df4597b334.1612004299.1.1612005471.1612004299.'headers={'User-Agent':'Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/85.0.4183.83 Safari/537.36',
}url='https://movie.douban.com/top250'number_ls=[]
for i in range(0,251,25):number_ls.append(i)print(number_ls)def get_information(number_ls):param={'start':number_ls,'filter' :''}page_content=requests.get(url=url,headers=headers,params=param).textwith open('douban.html','w',encoding='utf-8') as fp:fp.write(page_content)tree=etree.HTML(page_content)vedio_title=tree.xpath('//ol[@class="grid_view"]//div[@class="pic"]//a/img/@alt')star=tree.xpath('//ol[@class="grid_view"]//div[@class="star"]/span[@class="rating_num"]/text()')vedio_title_ls=[]star_ls=[]for i in vedio_title:vedio_title_ls.append(i)for i in star:star_ls.append(i)j=0while j<len(star_ls):print("the movie is ",vedio_title_ls[j])print("the star is ",star_ls[j])print()j+=1pool=Pool(4)
pool.map(get_information,number_ls)

四.输出的结果

输出的结果十分完美，一共有250份电影，如下图所示：

[0, 25, 50, 75, 100, 125, 150, 175, 200, 225, 250]
the movie is  搏击俱乐部
the star is  9.0the movie is  教父2
the star is  9.2the movie is  狮子王
the star is  9.0the movie is  指环王2：双塔奇兵
the star is  9.1the movie is  死亡诗社
the star is  9.1the movie is  钢琴家
the star is  9.2the movie is  黑客帝国
the star is  9.0the movie is  指环王1：魔戒再现
the star is  9.0the movie is  饮食男女
the star is  9.1the movie is  窃听风暴
the star is  9.1the movie is  美丽心灵
the star is  9.0the movie is  让子弹飞
the star is  8.8the movie is  绿皮书
the star is  8.9the movie is  两杆大烟枪
the star is  9.1the movie is  本杰明·巴顿奇事
the star is  8.9the movie is  海蒂和爷爷
the star is  9.2the movie is  飞越疯人院
the star is  9.1the movie is  看不见的客人
the star is  8.8the movie is  西西里的美丽传说
the star is  8.9the movie is  拯救大兵瑞恩
the star is  9.0the movie is  穿条纹睡衣的男孩
the star is  9.1the movie is  小鞋子
the star is  9.2the movie is  音乐之声
the star is  9.0the movie is  情书
the star is  8.9the movie is  海豚湾
the star is  9.3the movie is  美国往事
the star is  9.2the movie is  致命魔术
the star is  8.9the movie is  沉默的羔羊
the star is  8.9the movie is  低俗小说
the star is  8.9the movie is  禁闭岛
the star is  8.8the movie is  蝴蝶效应
the star is  8.8the movie is  七宗罪
the star is  8.8the movie is  心灵捕手
the star is  8.9the movie is  布达佩斯大饭店
the star is  8.9the movie is  春光乍泄
the star is  8.9the movie is  摩登时代
the star is  9.3the movie is  被嫌弃的松子的一生
the star is  8.9the movie is  哈利·波特与死亡圣器(下)
the star is  8.9the movie is  阿凡达
the star is  8.7the movie is  喜剧之王
the star is  8.8the movie is  致命ID
the star is  8.8the movie is  剪刀手爱德华
the star is  8.7the movie is  勇敢的心
the star is  8.9the movie is  加勒比海盗
the star is  8.8the movie is  杀人回忆
the star is  8.9the movie is  狩猎
the star is  9.1the movie is  请以你的名字呼唤我
the star is  8.9the movie is  天使爱美丽
the star is  8.7the movie is  断背山
the star is  8.8the movie is  红辣椒
the star is  9.0the movie is  触不可及
the star is  9.2the movie is  蝙蝠侠：黑暗骑士
the star is  9.2the movie is  末代皇帝
the star is  9.3the movie is  活着
the star is  9.3the movie is  寻梦环游记
the star is  9.1the movie is  乱世佳人
the star is  9.3the movie is  何以为家
the star is  9.1the movie is  指环王3：王者无敌
the star is  9.2the movie is  飞屋环游记
the star is  9.0the movie is  摔跤吧！爸爸
the star is  9.0the movie is  哈利·波特与魔法石
the star is  9.1the movie is  素媛
the star is  9.3the movie is  少年派的奇幻漂流
the star is  9.1the movie is  十二怒汉
the star is  9.4the movie is  哈尔的移动城堡
the star is  9.1the movie is  鬼子来了
the star is  9.3the movie is  天空之城
the star is  9.1the movie is  大话西游之月光宝盒
the star is  9.0the movie is  我不是药神
the star is  9.0the movie is  闻香识女人
the star is  9.1the movie is  罗马假日
the star is  9.0the movie is  天堂电影院
the star is  9.2the movie is  辩护人
the star is  9.2the movie is  猫鼠游戏
the star is  9.0the movie is  大闹天宫
the star is  9.4the movie is  肖申克的救赎
the star is  9.7the movie is  霸王别姬
the star is  9.6the movie is  阿甘正传
the star is  9.5the movie is  这个杀手不太冷
the star is  9.4the movie is  泰坦尼克号
the star is  9.4the movie is  美丽人生
the star is  9.5the movie is  千与千寻
the star is  9.4the movie is  辛德勒的名单
the star is  9.5the movie is  盗梦空间
the star is  9.3the movie is  忠犬八公的故事
the star is  9.4the movie is  星际穿越
the star is  9.3the movie is  海上钢琴师
the star is  9.3the movie is  楚门的世界
the star is  9.3the movie is  三傻大闹宝莱坞
the star is  9.2the movie is  机器人总动员
the star is  9.3the movie is  放牛班的春天
the star is  9.3the movie is  大话西游之大圣娶亲
the star is  9.2the movie is  疯狂动物城
the star is  9.2the movie is  无间道
the star is  9.2the movie is  熔炉
the star is  9.3the movie is  教父
the star is  9.3the movie is  当幸福来敲门
the star is  9.1the movie is  龙猫
the star is  9.2the movie is  怦然心动
the star is  9.1the movie is  控方证人
the star is  9.6the movie is  7号房的礼物
the star is  8.9the movie is  幽灵公主
the star is  8.9the movie is  小森林 夏秋篇
the star is  9.0the movie is  阳光灿烂的日子
the star is  8.8the movie is  第六感
the star is  8.9the movie is  重庆森林
the star is  8.8the movie is  入殓师
the star is  8.9the movie is  唐伯虎点秋香
the star is  8.7the movie is  小森林 冬春篇
the star is  9.0the movie is  爱在黎明破晓前
the star is  8.8the movie is  超脱
the star is  8.9the movie is  消失的爱人
the star is  8.7the movie is  一一
the star is  9.0the movie is  菊次郎的夏天
the star is  8.8the movie is  蝙蝠侠：黑暗骑士崛起
the star is  8.8the movie is  侧耳倾听
the star is  8.9the movie is  倩女幽魂
the star is  8.7the movie is  功夫
the star is  8.6the movie is  超能陆战队
the star is  8.7the movie is  无人知晓
the star is  9.1the movie is  人生果实
the star is  9.5the movie is  萤火之森
the star is  8.9the movie is  甜蜜蜜
the star is  8.8the movie is  借东西的小人阿莉埃蒂
the star is  8.8the movie is  玛丽和马克思
the star is  8.9the movie is  爱在日落黄昏时
the star is  8.8the movie is  驯龙高手
the star is  8.7the movie is  完美的世界
the star is  9.1the movie is  幸福终点站
the star is  8.8the movie is  告白
the star is  8.7the movie is  大鱼
the star is  8.8the movie is  阳光姐妹淘
the star is  8.8the movie is  射雕英雄传之东成西就
the star is  8.7the movie is  哈利·波特与阿兹卡班的囚徒
the star is  8.8the movie is  恐怖直播
the star is  8.8the movie is  天书奇谭
the star is  9.2the movie is  怪兽电力公司
the star is  8.7the movie is  神偷奶爸
the star is  8.6the movie is  玩具总动员3
the star is  8.8the movie is  傲慢与偏见
the star is  8.6the movie is  时空恋旅人
the star is  8.8the movie is  哈利·波特与密室
the star is  8.7the movie is  教父3
the star is  8.9the movie is  釜山行
the star is  8.6the movie is  血战钢锯岭
the star is  8.7the movie is  哪吒闹海
the star is  9.1the movie is  被解救的姜戈
the star is  8.7the movie is  七武士
the star is  9.3the movie is  喜宴
the star is  8.9the movie is  电锯惊魂
the star is  8.7the movie is  爆裂鼓手
the star is  8.7the movie is  贫民窟的百万富翁
the star is  8.6the movie is  萤火虫之墓
the star is  8.7the movie is  东邪西毒
the star is  8.6the movie is  海街日记
the star is  8.8the movie is  黑天鹅
the star is  8.6the movie is  惊魂记
the star is  9.0the movie is  无敌破坏王
the star is  8.7the movie is  你看起来好像很好吃
the star is  8.9the movie is  冰川时代
the star is  8.6the movie is  雨人
the star is  8.7the movie is  小偷家族
the star is  8.7the movie is  绿里奇迹
the star is  8.9the movie is  恋恋笔记本
the star is  8.5the movie is  爱在午夜降临前
the star is  8.8the movie is  疯狂的石头
the star is  8.5the movie is  哈利·波特与火焰杯
the star is  8.6the movie is  寄生虫
the star is  8.7the movie is  恐怖游轮
the star is  8.5the movie is  奇迹男孩
the star is  8.6the movie is  雨中曲
the star is  9.0the movie is  魔女宅急便
the star is  8.7the movie is  二十二
the star is  8.7the movie is  海边的曼彻斯特
the star is  8.6the movie is  房间
the star is  8.8the movie is  风之谷
the star is  8.9the movie is  一个叫欧维的男人决定去死
the star is  8.9the movie is  我是山姆
the star is  8.9the movie is  头号玩家
the star is  8.7the movie is  英雄本色
the star is  8.7the movie is  上帝之城
the star is  9.0the movie is  谍影重重3
the star is  8.8the movie is  疯狂原始人
the star is  8.7the movie is  未麻的部屋
the star is  9.0the movie is  岁月神偷
the star is  8.7the movie is  卢旺达饭店
the star is  8.9the movie is  纵横四海
the star is  8.8the movie is  三块广告牌
the star is  8.7the movie is  达拉斯买家俱乐部
the star is  8.8the movie is  花样年华
the star is  8.7the movie is  心迷宫
the star is  8.7the movie is  记忆碎片
the star is  8.6the movie is  模仿游戏
the star is  8.7the movie is  黑客帝国3：矩阵革命
the star is  8.8the movie is  新世界
the star is  8.8the movie is  头脑特工队
the star is  8.7the movie is  荒蛮故事
the star is  8.8the movie is  你的名字。
the star is  8.4the movie is  真爱至上
the star is  8.6the movie is  忠犬八公物语
the star is  9.2the movie is  谍影重重2
the star is  8.7the movie is  阿飞正传
the star is  8.5the movie is  地球上的星星
the star is  8.9the movie is  彗星来的那一夜
the star is  8.5the movie is  完美陌生人
the star is  8.5the movie is  战争之王
the star is  8.7the movie is  谍影重重
the star is  8.6the movie is  香水
the star is  8.5the movie is  东京教父
the star is  9.0the movie is  东京物语
the star is  9.2the movie is  朗读者
the star is  8.6the movie is  千钧一发
the star is  8.8the movie is  再次出发之纽约遇见你
the star is  8.6the movie is  驴得水
the star is  8.3the movie is  猜火车
the star is  8.5the movie is  黑客帝国2：重装上阵
the star is  8.6the movie is  无间道2
the star is  8.6the movie is  我爱你
the star is  9.1the movie is  浪潮
the star is  8.7the movie is  崖上的波妞
the star is  8.5the movie is  聚焦
the star is  8.8the movie is  小萝莉的猴神大叔
the star is  8.4the movie is  追随
the star is  8.9the movie is  黑鹰坠落
the star is  8.7the movie is  网络谜踪
the star is  8.6the movie is  虎口脱险
the star is  8.9the movie is  人工智能
the star is  8.7the movie is  九品芝麻官
the star is  8.6the movie is  2001太空漫游
the star is  8.8the movie is  可可西里
the star is  8.8the movie is  罗生门
the star is  8.8the movie is  色，戒
the star is  8.5the movie is  终结者2：审判日
the star is  8.7the movie is  城市之光
the star is  9.3the movie is  初恋这件小事
the star is  8.4the movie is  魂断蓝桥
the star is  8.8the movie is  牯岭街少年杀人事件
the star is  8.9the movie is  遗愿清单
the star is  8.7the movie is  大佛普拉斯
the star is  8.7the movie is  新龙门客栈
the star is  8.6the movie is  波西米亚狂想曲
the star is  8.7the movie is  源代码
the star is  8.5the movie is  青蛇
the star is  8.6the movie is  海洋
the star is  9.1the movie is  燃情岁月
the star is  8.8the movie is  无耻混蛋
the star is  8.6the movie is  疯狂的麦克斯4：狂暴之路
the star is  8.6the movie is  血钻
the star is  8.7the movie is  穿越时空的少女
the star is  8.6the movie is  步履不停
the star is  8.8