【知乎热榜爬虫】python爬取知乎热榜问题及答案

所用库

requests
xpath解析库
multiprocessing多进程
pymysql数据库操作库

实战背景

主要是爬取知乎热榜的问题及点赞数比较高的答案，通过requests请求库进行爬取，xpath进行解析，并将结果存储至mysql数据库中

爬取的url为：https://www.zhihu.com/hot

源码保存在我的github上：知乎热榜问题及答案数据获取

文章首发于个人网站：大圣的专属空间

代码实现

首先获取问题的标题以及其下属的答案的url地址，代码如下：

import requests
from lxml import etree
import time
import multiprocessing
import pymysqlheaders = {'user-agent': 'Mozilla/5.0 (Windows NT 6.1; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/78.0.3904.108 Safari/537.36','cookie': '***********'
}
url = 'https://www.zhihu.com/hot'
def get_question_num(url,headers):response = requests.get(url,headers=headers)text = response.texthtml = etree.HTML(text)reslut = html.xpath("//section[@class='HotItem']")# 获取问题的IDquestion_list = []for question in reslut:number = question.xpath(".//div[@class='HotItem-index']//text()")[0].strip()title = question.xpath(".//h2[@class='HotItem-title']/text()")[0].strip()href = question.xpath(".//div[@class='HotItem-content']/a/@href")[0].strip()question_num = href.split('/')[-1]question_list.append([question_num,title])# print(number,'\n',title,'\n',href)return question_list

其中，在请求头中加入了user-agent以及自己的cookie，如下图：

使用xpath进行页面解析，最终函数返回的结果为问题ID（即https://www.zhihu.com/question/359056618中的359056618，后面会将为什么是需要返回ID而不是整个url）以及问题的标题

获取到问题的标题以及问题的ID后，我们接下来需要获取问题下面的答案

进入问题答案详细页面，我们发现其问题是通过ajax请求进行加载的，如下：

我们发现其请求json数据的url为：

https://www.zhihu.com/api/v4/questions/359056618/answers?include=data%5B%2A%5D.is_normal%2Cadmin_closed_comment%2Creward_info%2Cis_collapsed%2Cannotation_action%2Cannotation_detail%2Ccollapse_reason%2Cis_sticky%2Ccollapsed_by%2Csuggest_edit%2Ccomment_count%2Ccan_comment%2Ccontent%2Ceditable_content%2Cvoteup_count%2Creshipment_settings%2Ccomment_permission%2Ccreated_time%2Cupdated_time%2Creview_info%2Crelevant_info%2Cquestion%2Cexcerpt%2Crelationship.is_authorized%2Cis_author%2Cvoting%2Cis_thanked%2Cis_nothelp%2Cis_labeled%2Cis_recognized%2Cpaid_info%2Cpaid_info_content%3Bdata%5B%2A%5D.mark_infos%5B%2A%5D.url%3Bdata%5B%2A%5D.author.follower_count%2Cbadge%5B%2A%5D.topics&limit=5&offset=5&platform=desktop&sort_by=default
# 变化量如：question_id：359056618, offset=5,10,15......

其变化的量主要有问题ID以及offset参数，其中offset控制每次加载的数量

因此我们开始实现获取答案的代码：

def data_json_request(question_id,question_title,headers):num = 0i = 1while True:json_url = 'https://www.zhihu.com/api/v4/questions/' + question_id + '/answers?include=data%5B%2A%5D.is_normal%2Cadmin_closed_comment%2Creward_info%2Cis_collapsed%2Cannotation_action%2Cannotation_detail%2Ccollapse_reason%2Cis_sticky%2Ccollapsed_by%2Csuggest_edit%2Ccomment_count%2Ccan_comment%2Ccontent%2Ceditable_content%2Cvoteup_count%2Creshipment_settings%2Ccomment_permission%2Ccreated_time%2Cupdated_time%2Creview_info%2Crelevant_info%2Cquestion%2Cexcerpt%2Crelationship.is_authorized%2Cis_author%2Cvoting%2Cis_thanked%2Cis_nothelp%2Cis_labeled%2Cis_recognized%2Cpaid_info%2Cpaid_info_content%3Bdata%5B%2A%5D.mark_infos%5B%2A%5D.url%3Bdata%5B%2A%5D.author.follower_count%2Cbadge%5B%2A%5D.topics&limit=5&offset={}&platform=desktop&sort_by=default'.format(num)data_json = requests.get(json_url,headers=headers)all_detail_data = data_json.json()['data']length_detail_data = len(all_detail_data)for one_detail_data in all_detail_data:question_title = question_titleanswer_author = one_detail_data['author']['name']author_introduce = one_detail_data['author']['headline']author_followers = one_detail_data['author']['follower_count']answer_vote_num = one_detail_data['voteup_count']answer_comment_num = one_detail_data['comment_count']updated_time = one_detail_data['updated_time']content = one_detail_data['content']# 保存数据至数据库db = pymysql.connect(host='localhost',user='root',password='123456',port=3306,db='spider_data')cursor = db.cursor()sql = 'INSERT INTO zhihu_hot_question(question_title,author_name,author_introduce,author_followers,answer_vote_num,answer_comment_num,updated_time,content) VALUES(%s,%s,%s,%s,%s,%s,%s,%s)'try:if int(answer_vote_num) >= 90:cursor.execute(sql,(question_title,answer_author,author_introduce,author_followers,answer_vote_num,answer_comment_num,updated_time,content))db.commit()print('数据写入成功！！！')else:print('点赞数太少，不保存至数据库！！！')except:print('数据写入失败！')db.rollback()# print(question_title,'\n',answer_author,'\n',author_introduce,'\n',author_followers,'\n',answer_vote_num,'\n',answer_comment_num# ,'\n',updated_time,'\n',content)num = i*5i = i+1if length_detail_data == 0:print('answaer_stop!!!!!')break

其中，我们设定当其答案全部加载完成后，跳出循环，即：

 if lenth_detail_data == 0:print('answaer_stop!!!!!')break

同时也将数据写入数据库的程序直接写入到获取答案的函数中，并且对于点赞数过少的不保存在数据库中（这里设置点赞数大于90进行保存）

最后编写main主函数，如下：

def main():question_id = get_question_num(url,headers)print(question_id)print('当前环境CPU核数是：{}核'.format(multiprocessing.cpu_count()))p = multiprocessing.Pool(4)for q_id in question_id:p.apply_async(data_json_request,args=(q_id[0],q_id[1],headers))p.close()p.join()

这里使用了多进程的方式进行数据的快速获取，最后运行整个程序，并统计最终耗时，如下：

if __name__ == "__main__":start = time.time()main()print('总耗时：%.5f秒'% float(time.time()-start))

最后发现爬取知乎热榜总共用了552s的时间，最终数据库中的保存数据如下：

这样我们就实现了知乎热榜问题及答案的数据获取

总结

使用了最简单的requests请求库
添加请求头的相关数据
分析网页是否使用ajax请求
爬取结果的数据保存（python操作mysql数据库）
多进程的数据快速爬取