Python爬取豆瓣电影短评

豆瓣是比较难爬取的网站之一，主要因为豆瓣默认如果不登录账号的话只能爬取10页的评论。所以我就带着cookie去爬取，而且设置了一个用户代理池，尽可能的伪装成浏览器。然而当我爬了三四次，一共几十页评论之后的第二天，我的豆瓣账号就被封了。。。

还是老规矩，F12审查元素，点击network并刷新页面，然后惊人的发现json文件中没有找到有关评论的信息。豆瓣的影评没有存到json文件里，而是直接放在HTML静态页面上，那我们就直接打开HTML文件的头部信息，拿到URL和cookie，然后直接上代码。

通用信息中的URL：

请求头中的cookie：

# -*-coding:utf-8-*-
import urllib.request
from bs4 import BeautifulSoup
import random
import timedef getHtml(url):"""获取url页面"""user_agents = ['Mozilla/5.0 (Windows NT 6.1; WOW64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/39.0.2171.95 Safari/537.36 OPR/26.0.1656.60','Opera/8.0 (Windows NT 5.1; U; en)','Mozilla/5.0 (Windows NT 5.1; U; en; rv:1.8.1) Gecko/20061208 Firefox/2.0.0 Opera 9.50','Mozilla/4.0 (compatible; MSIE 6.0; Windows NT 5.1; en) Opera 9.50','Mozilla/5.0 (Windows NT 6.1; WOW64; rv:34.0) Gecko/20100101 Firefox/34.0','Mozilla/5.0 (X11; U; Linux x86_64; zh-CN; rv:1.9.2.10) Gecko/20100922 Ubuntu/10.10 (maverick) Firefox/3.6.10','Mozilla/5.0 (Windows NT 6.1; WOW64) AppleWebKit/534.57.2 (KHTML, like Gecko) Version/5.1.7 Safari/534.57.2 ','Mozilla/5.0 (Windows NT 6.1; WOW64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/39.0.2171.71 Safari/537.36','Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/537.11 (KHTML, like Gecko) Chrome/23.0.1271.64 Safari/537.11','Mozilla/5.0 (Windows; U; Windows NT 6.1; en-US) AppleWebKit/534.16 (KHTML, like Gecko) Chrome/10.0.648.133 Safari/534.16','Mozilla/5.0 (Windows NT 6.1; WOW64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/30.0.1599.101 Safari/537.36','Mozilla/5.0 (Windows NT 6.1; WOW64; Trident/7.0; rv:11.0) like Gecko','Mozilla/5.0 (Windows NT 6.1; WOW64) AppleWebKit/536.11 (KHTML, like Gecko) Chrome/20.0.1132.11 TaoBrowser/2.0 Safari/536.11','Mozilla/5.0 (Windows NT 6.1; WOW64) AppleWebKit/537.1 (KHTML, like Gecko) Chrome/21.0.1180.71 Safari/537.1 LBBROWSER','Mozilla/4.0 (compatible; MSIE 6.0; Windows NT 5.1; SV1; QQDownload 732; .NET4.0C; .NET4.0E)','Mozilla/5.0 (Windows NT 5.1) AppleWebKit/535.11 (KHTML, like Gecko) Chrome/17.0.963.84 Safari/535.11 SE 2.X MetaSr 1.0','Mozilla/4.0 (compatible; MSIE 7.0; Windows NT 5.1; Trident/4.0; SV1; QQDownload 732; .NET4.0C; .NET4.0E; SE 2.X MetaSr 1.0) ',"Mozilla/4.0 (compatible; MSIE 6.0; Windows NT 5.1; SV1; AcooBrowser; .NET CLR 1.1.4322; .NET CLR 2.0.50727)","Mozilla/4.0 (compatible; MSIE 7.0; Windows NT 6.0; Acoo Browser; SLCC1; .NET CLR 2.0.50727; Media Center PC 5.0; .NET CLR 3.0.04506)","Mozilla/4.0 (compatible; MSIE 7.0; AOL 9.5; AOLBuild 4337.35; Windows NT 5.1; .NET CLR 1.1.4322; .NET CLR 2.0.50727)","Mozilla/5.0 (Windows; U; MSIE 9.0; Windows NT 9.0; en-US)","Mozilla/5.0 (compatible; MSIE 9.0; Windows NT 6.1; Win64; x64; Trident/5.0; .NET CLR 3.5.30729; .NET CLR 3.0.30729; .NET CLR 2.0.50727; Media Center PC 6.0)","Mozilla/5.0 (compatible; MSIE 8.0; Windows NT 6.0; Trident/4.0; WOW64; Trident/4.0; SLCC2; .NET CLR 2.0.50727; .NET CLR 3.5.30729; .NET CLR 3.0.30729; .NET CLR 1.0.3705; .NET CLR 1.1.4322)","Mozilla/4.0 (compatible; MSIE 7.0b; Windows NT 5.2; .NET CLR 1.1.4322; .NET CLR 2.0.50727; InfoPath.2; .NET CLR 3.0.04506.30)","Mozilla/5.0 (Windows; U; Windows NT 5.1; zh-CN) AppleWebKit/523.15 (KHTML, like Gecko, Safari/419.3) Arora/0.3 (Change: 287 c9dfb30)","Mozilla/5.0 (X11; U; Linux; en-US) AppleWebKit/527+ (KHTML, like Gecko, Safari/419.3) Arora/0.6","Mozilla/5.0 (Windows; U; Windows NT 5.1; en-US; rv:1.8.1.2pre) Gecko/20070215 K-Ninja/2.1.1","Mozilla/5.0 (Windows; U; Windows NT 5.1; zh-CN; rv:1.9) Gecko/20080705 Firefox/3.0 Kapiko/3.0","Mozilla/5.0 (X11; Linux i686; U;) Gecko/20070322 Kazehakase/0.4.5","Mozilla/5.0 (X11; U; Linux i686; en-US; rv:1.9.0.8) Gecko Fedora/1.9.0.8-1.fc10 Kazehakase/0.5.6","Mozilla/5.0 (Windows NT 6.1; WOW64) AppleWebKit/535.11 (KHTML, like Gecko) Chrome/17.0.963.56 Safari/535.11","Mozilla/5.0 (Macintosh; Intel Mac OS X 10_7_3) AppleWebKit/535.20 (KHTML, like Gecko) Chrome/19.0.1036.7 Safari/535.20","Opera/9.80 (Macintosh; Intel Mac OS X 10.6.8; U; fr) Presto/2.9.168 Version/11.52"]headers = {'Cookie': 'douban-profile-remind=1; trc_cookie_storage=taboola%2520global%253Auser-id%3De1c60947-a390-4686-856c-f46c4a32e761-tuct3ee994a; __utmv=30149280.16715; _vwo_uuid_v2=D86904745E1B8CFBCE3B86C645D26D73E|da7b9bc06f5d01692e863e3ef93a79aa; ll="118106"; bid=ritmPjTqVZ0; __yadk_uid=w5qthAeH1m9Q5sKuM5jcaIJc5jtoQbk9; __gads=ID=4c78ae3f2ae997a9:T=1580449446:S=ALNI_MblBbT9dgASYCycz7yeVP4uMK6hpA; push_doumail_num=0; dbcl2="167155432:fpzNKjDvwrU"; push_noty_num=0; ct=y; ck=tMAk; _pk_ref.100001.4cf6=%5B%22%22%2C%22%22%2C1583240130%2C%22https%3A%2F%2Fcn.bing.com%2F%22%5D; _pk_ses.100001.4cf6=*; ap_v=0,6.0; __utma=30149280.1642366477.1541387570.1581926084.1583240131.43; __utmb=30149280.0.10.1583240131; __utmc=30149280; __utmz=30149280.1583240131.43.3.utmcsr=cn.bing.com|utmccn=(referral)|utmcmd=referral|utmcct=/; __utma=223695111.778911602.1541387570.1581926084.1583240131.40; __utmb=223695111.0.10.1583240131; __utmc=223695111; __utmz=223695111.1583240131.40.1.utmcsr=cn.bing.com|utmccn=(referral)|utmcmd=referral|utmcct=/; _pk_id.100001.4cf6=dd777e506416d181.1541387571.39.1583240394.1581926693.','User-Agent': str(random.choice(user_agents)),'Referer': 'https: // movie.douban.com / subject / 26752088 / comments?status = P','Connection': 'keep-alive'}req = urllib.request.Request(url,headers=headers)req = urllib.request.urlopen(req)content = req.read().decode('utf-8')return contentdef getComment(url):"""解析HTML页面"""html = getHtml(url)soupComment = BeautifulSoup(html, 'html.parser')comments = soupComment.findAll('span', 'short')onePageComments = []for comment in comments:# print(comment.getText()+'\n')onePageComments.append(comment.getText()+'\n')return onePageCommentsif __name__ == '__main__':f = open('我不是药神.txt', 'a', encoding='utf-8')j = 0for page in range(15):  # 豆瓣爬取多页评论需要验证。url = 'https://movie.douban.com/subject/26752088/comments?start=' + str(20*page) + '&limit=20&sort=new_score&status=P'print('第%s页的评论:' % (page))print(url + '\n')for i in getComment(url):f.write(i)print(j,i)j += 1time.sleep(10)print('\n')

这里分析页面使用了bs4库中的html.paeser，这是个解析HTML的工具。在页面中我们不难找到评论信息存储在属性为‘short’的span标签中：

Python爬取豆瓣电影短评相关推荐

python爬取豆瓣电影评论_python 爬取豆瓣电影评论，并进行词云展示及出现的问题解决办法...
def getHtml(url): """获取url页面""" headers = {'User-Agent':'Mozilla/5.0 ( ...
利用python爬取豆瓣电影top250
利用python爬取豆瓣电影top250: 注:本内容只是作为个人学习记录 1.业务分析进入网页https://movie.douban.com/top250 可以看见每部电影都呈现在眼前,点击电影 ...
Python爬取豆瓣电影top250的电影信息
Python爬取豆瓣电影top250的电影信息前言一.简介二.实例源码展示小结前言相信很多小伙伴在学习网络爬虫时,老师们会举一些实例案例讲解爬虫知识,本文介绍的就是经典爬虫实际案例–爬取豆 ...
爬取豆瓣电影短评并使用词云简单分析top50
先使用进程池爬取豆瓣电影短评 import requests import re import random import time import pandas as pd from pymongo ...
源码大公开！Python爬取豆瓣电影Top250源代码，赶紧收藏！
哈喽~大家好,我是恰恰.不知道是不是有很多小伙伴跟我一样,很喜欢看电影,尤其是那种别人推荐的豆瓣高分电影,所以学Python就有一个好处,用Python爬取豆瓣电影那是分分钟的事,再也不用因为有些电影 ...
python 爬取豆瓣电影排名
python 爬取豆瓣电影排名任务要求获取豆瓣电影排名前250信息打包进文件夹代码展示代码如下 import requests#导入requests库 import redef GetHtm ...
python爬取豆瓣电影top250_Python爬虫——爬取豆瓣电影Top250代码实例
利用python爬取豆瓣电影Top250的相关信息,包括电影详情链接,图片链接,影片中文名,影片外国名,评分,评价数,概况,导演,主演,年份,地区,类别这12项内容,然后将爬取的信息写入Excel表中 ...
python爬取豆瓣电影top250的代码_Python爬虫——爬取豆瓣电影Top250代码实例
利用python爬取豆瓣电影Top250的相关信息,包括电影详情链接,图片链接,影片中文名,影片外国名,评分,评价数,概况,导演,主演,年份,地区,类别这12项内容,然后将爬取的信息写入Excel表中 ...
python爬取豆瓣电影榜单
python爬取豆瓣电影榜单 python爬取豆瓣电影榜单并保存到本地excel中,以后就不愁没片看了. 目标确定我们想要抓取的电影的相关内容. 抓取豆瓣top250电影的排名.电影名.评价(总结很 ...

Python爬取豆瓣电影短评

Python爬取豆瓣电影短评相关推荐

最新文章

热门文章