最近在做毕业设计，需要收集用户的评分数据做协同过滤算法，同时收集评论数据做情感分析

坑点

豆瓣图书可以没有评分，或者用户评论了但没给评分。而且豆瓣图书的编码方式很无奈呀，热门书籍附近总是冷门书籍，无评分、无评论那种，所以经常输出failed
不能爬得太快了，每分钟只能40-50张页面，一个requests只能访问一千次，否则就报状态码403

fake_useragent的用法

在这次爬虫中使用了fake_useragent来伪造请求头，因为听说豆瓣的反爬机制比较好
fake_useragent的用法简单如下，random是随机产生一个请求头

from fake_useragent import UserAgent
import requests
ua=UserAgent()
url="https://www.baidu.com"    #请求的网址
headers={"User-Agent":ua.random}   #请求头
response=requests.get(url=url,headers=headers)   #请求网址
print(headers)
print(response.status_code)   #响应状态信息
text = response.headers
for line in text.items():print(line)

爬取豆瓣读书的图书信息和评论信息

首先需要观察的是这些的链接
https://book.douban.com/subject/26953606/ 图书信息页面
https://book.douban.com/subject/26953606/comments/ 第一页评论页面
https://book.douban.com/subject/26953606/comments/hot?p=2 第二页评论页面
可以看到前面都是相同的https://book.douban.com/subject/再加一个图书id，评论页面后面接一个/comments/，第二页评论后面接一个hot?p=2，由此递推低3页是hot?p=3
其中一些写入文本的操作，因为我是要收集数据的
第二天又修改了一下，热门图书的分布实在是太稀疏了，所以在程序里先判断评论总数是否超过一千，如果超过一千条就继续爬取，否则continue
又改bug了，是数字的，写入文件一定要将其转换成str

#coding=utf-8
#下载豆瓣图书的评分、评论，需要建立四张表。auther:wuyou
#表一：图书ID，图书名，平均分
#表二：用户ID，用户名
#表三：图书ID，热门评论
#表四：用户ID，图书ID,评分，评分时间
import requests
import time
import random
from bs4 import BeautifulSoup
from fake_useragent import UserAgent
ua = UserAgent()
header = {'User-Agent': ua.random
}
def get_score(book_id,text):                   #获取(图书ID，图书名，图书评分)soup = BeautifulSoup(text,'lxml')try:book_name = soup.select("#wrapper > h1 > span")   #返回书名的列表name = book_name[0].stringbook_score = soup.select("#interest_sectl > div > div.rating_self.clearfix > strong")  #返回分数的列表score = book_score[0].string#print("book name is " + str(name)+" and score is "+str(score))  打印书名和分数line = str(book_id) + "," + name + "," + str(score) + "\n"   #拼接图书信息with open("BookInfo.txt","a",encoding="utf-8") as file:    #表一：图书ID，图书名，平均分file.write(line)file.close()except:print("book " + str(book_id) + "get score is  failed!")def write_txt(soup,book_id):    #参与为url，图书id，和网页页码try:     #为了防止报错，因为有些人可以不打分，那么在user_info下只有一个spancomment_list = soup.find_all("span","short")     #找到评论所在的区域comments = ""flag = 0for line in comment_list:       #把逗号全部替换成分号bc = line.stringbc = bc.replace(",","。")     #将英文逗号替换成句号bc = bc.replace("，","。")    #将中文逗号替换成句号bc = bc .replace(";","。")    #将分号替换成句号if flag == 0:     #如果是第一条评论flag += 1else:comments += ";"      #评论之间用分号间隔comments += bcwith open("BookComments.txt","a",encoding="utf-8") as file:    #表三：图书ID，热门评论BookComments = str(book_id) + "," +comments + "\n"file.write(BookComments)file.close()user_list = soup.find_all("span", "comment-info")   #找到用户和评分的所在区域user_info_txt = open("UserInfo.txt","a",encoding="utf-8")user_score_txt = open("UserScore.txt","a",encoding="utf-8")for user_info in user_list:user_name = user_info.find("a").string         #用户姓名所在的<a></a>user_url = user_info.find("a").attrs["href"]   #提取出超链接user_id = user_url.split("/")[-2]              #提取出用户idscore = user_info.find_all("span")[0].attrs["title"]   #找到用户评分的区域,得到分数time_info = user_info.find_all("span")[1].string   #提取出评分的时间time_info = time_info.split("-")score_year = time_info[0]       #截取出评论时间的年份user_info_txt.write(user_id + "," +user_name + "\n")            #表二：用户ID，用户名user_score_txt.write(user_id + "," + str(book_id) + "," + score + "," + str(score_year) + "\n")  #表四：用户ID，图书ID,评分，评分时间#print("book_id is " + book_id +" user name is " + user_name + ",id is " + user_id + ",score is " + score_info + " " + time_info)   打印出一系列信息user_info_txt.close()user_score_txt.close()except:print("cannot find!")def get_comments(soup, comment_url, book_id, page):         #获取(图书ID，图书评论),(图书ID，用户ID，用户评分),(用户ID，用户名)while page <= 2:           #爬取的页数if int(page) == 1:     #如果是第一页write_txt(soup, book_id)          #传入超链接page += 1           #页数加一else:comment_url += "hot?p=" + str(page)   #拼合链接time.sleep(random.uniform(3,6))html = requests.get(url=comment_url,headers=header)if html.status_code == 200:comment_text = html.textsoup = BeautifulSoup(comment_text,"lxml")write_txt(soup, book_id)          #传入网页内容page += 1           #页数加一#https://book.douban.com/subject/1007305/
if __name__ == '__main__':url="https://book.douban.com/subject/"startID=1007304  #起始的图书IDst = 0   #循环的起点lens=20000   #len=20000时，需要爬取的总书籍数while st < lens:           #设置st和lens是为了爬取热门书籍if startID-1007304 >=1000:print("stop! " + startID)breaktry:startID += 1           #图书id增长score_url = url + str(startID) + "/"       #图书信息的链接地址html = requests.get(url=score_url,headers=header)html.encoding = "utf-8"time.sleep(random.uniform(3, 6))  # 暂停几秒，随机数在2-4s之间if html.status_code == 200:comment_url = score_url + "comments/"  # 评论的链接地址comment_html = requests.get(url=comment_url, headers=header).texttime.sleep(random.uniform(3, 6))  # 暂停几秒，随机数在2-4s之间soup = BeautifulSoup(comment_html, "lxml")total_comments = soup.select("#total-comments")[0].stringcomment_num = total_comments.replace("全部共 ","")comment_num = comment_num.replace(" 条","")if int(comment_num) >= 1000:st +=1print(str(startID)+" is success!" + score_url + " comment_num is " + comment_num)text = html.textget_score(startID,text)get_comments(soup,comment_url,startID,1)       #获取评论信息else:print(score_url + " is failed!" + " comment_num is " + comment_num)else:print(str(startID)+" is failed!")except:print(str(startID) + " is failed!",end='')print(html.status_code)

输出如下（这是以前有输出语句时的代码的输出）

中间一堆数据省略了
这是爬取到了一些冷门书籍，评论数少得可怜，所以直接忽略了