requests库和beautifulsoup爬取豆瓣书本数据并存入excel

以下是所有的代码，如果想要改变爬取的书本类型可以在测试函数当中修改

# -*-coding:utf-8 -*-# 豆瓣网爬书参考代码 日期:2018-08-19 参考代码
import sys
import time
from urllib.parse import quote
import requests
import numpy as np
from bs4 import BeautifulSoup
from openpyxl import Workbook#User Agents
hds = [{'User-Agent': 'Mozilla/5.0 (Windows; U; Windows NT 6.1; enUS; rv:1.9.1.6) Gecko/20091201 Firefox/3.5.6'},
{'User-Agent': 'Mozilla/5.0 (Windows NT 6.2) AppleWebKit/ 535.11 (KHTML, like Gecko) Chrome/17.0.963.12 Safari/535.11'},
{'User-Agent': 'Mozilla/5.0 (compatible; MSIE 10.0; Windows NT 6.2; Trident/6.0)'}]#*********爬书虫函数**********
def book_spider(book_tag):page_num = 0 # 页数book_list = []  # 爬到的书籍列表try_times = 0 # 尝试次数while (page_num < 1):  #只爬一页# url='http://www.douban.com/tag/%E5%B0%8F%E8%AF%B4/book?start=0' # 测试用#每类爬15本书url = 'http://www.douban.com/tag/' + quote(book_tag) + '/book?start=' + str(page_num*15)time.sleep(np.random.rand() * 1)#开始爬取try:req = requests.request('get',url, headers=hds[page_num % len(hds)])source_code =req.textplain_text = str(source_code)except Exception as e:print(e)continue##测试 防止IP被禁用#source_code = requests.get(url)#plain_text = source_code.text#使用BS4 解析网页内容soup = BeautifulSoup(plain_text,"lxml")list_soup = soup.find('div', {'class': 'mod book-list'})try_times += 1if list_soup == None and try_times < 200:continueelif list_soup == None or len(list_soup) <= 1:break  # 200 次请求后 任然没有数据就直接退出#分析书籍列表for book_info in list_soup.ﬁndAll('dd'):#标题title = book_info.ﬁnd('a', {'class': 'title'}).string.strip()#描述desc = book_info.ﬁnd('div', {'class': 'desc'}).string.strip()desc_list = desc.split('/')#url 地址book_url = book_info.ﬁnd('a', {'class': 'title'}).get('href')try:author_info = '作者/译者： ' + '/'.join(desc_list[0:-3])except:author_info = '作者/译者： 暂无'try:pub_info = desc_list[-3]except:pub_info = '未知'try:date_info = desc_list[-2]except:date_info = '0000-00-00'try:price_info =desc_list[-1].strip('元')  #去除"元"这个字except:price_info = '0.0'try:rating = book_info.ﬁnd('span',{'class': 'rating_nums'}).string.strip()except:rating = '0.0'try:people_num = get_people_num(book_url)people_num = people_num.strip('人评价')  #去除"人评价"这几个字except:people_num = '0'# 跟踪输出print([title, rating, people_num, author_info, pub_info, date_info,price_info])book_list.append([title, rating, people_num, author_info, pub_info, date_info,price_info])if (len(book_list) > 20):breaktry_times = 0   # 获取无效信息设置成 0page_num += 1# print(page_num)print('从第{}页获取网页信息'.format(str(page_num)))return book_list#*************获取评价人数************
def get_people_num(url):# url='http://book.douban.com/subject/6082808/?from=tag_all' # 地址栏测试用try:req = requests.request('get',url, headers=hds[np.random.randint(0, len(hds))])source_code = req.textplain_text = str(source_code)except Exception as e:print(e)# 获取评价人数soup = BeautifulSoup(plain_text, 'lxml')people_num = soup.ﬁnd('div', {'class': 'rating_sum'}).ﬁndAll('span')[1].string.strip()return people_num#**************分类爬书籍***********
def do_spider(book_tag_lists):book_lists = []for book_tag in book_tag_lists:book_list = book_spider(book_tag)book_list = sorted(book_list, key=lambda x: x[1], reverse=True) #利用第二个参数进行升序排序book_lists.append(book_list)return book_lists#****************数据保存到excel中*********
def print_book_lists_excel(book_lists, book_tag_lists):wb = Workbook()ws = []for i in range(len(book_tag_lists)):ws.append(wb.create_sheet(title=book_tag_lists[i]))for i in range(len(book_tag_lists)):ws[i].append(['序号', '书名', '评分', '评价人数', '作者', '出版社', '出版日期', '价格'])count = 1for bl in book_lists[i]:ws[i].append([count, bl[0], ﬂoat(bl[1]), int(bl[2]), bl[3], bl[4], bl[5], bl[6]])count += 1# save_path = 'book_list'  # 保存# for i in range(len(book_tag_lists)):#     save_path += ('-' + book_tag_lists[i])#     save_path += '.xlsx'#     wb.save(save_path)save_path='book_list2.xlsx'wb.save(save_path)""" 主函数 """
if __name__ == '__main__':# 定义书要爬籍类型别book_tag_lists = ['心理','判断与决策','算法','数据结构','经济','历史']book_lists = do_spider(book_tag_lists)print_book_lists_excel(book_lists, book_tag_lists)#  book_tag_lists = ['传记','哲学','编程','创业','理理财','社会学','佛教']#  book_tag_lists = ['思想','科技','科学','web','股票','爱情','两性']#  book_tag_lists = ['计算机','机器器学习','linux','android','数据库','互联 ⽹网']#  book_tag_lists = ['数学']#  book_tag_lists = ['摄影','设计','音乐','旅⾏','教育','成长','情感','育儿','健康','养生']#  book_tag_lists = ['商业','理财','管理']#  book_tag_lists = ['名著']#  book_tag_lists = ['科普','经典','生活','心灵','文学']#  book_tag_lists = ['科幻','思维','⾦融']# book_tag_lists = ['个人管理', '时间管理', '投资', '文化', '宗教']

excel效果图：

requests库和beautifulsoup爬取豆瓣书本数据并存入excel相关推荐

爬虫beautifulsoup爬取豆瓣读书数据
爬虫beautifulsoup爬取豆瓣读书数据:主要是爬取收集书的名字.类别.简介,用于接下来的聚类学习. 豆瓣链接:https://book.douban.com/tag/?view=type&am ...
BeautifulSoup爬取豆瓣电影数据
BeautifulSoup爬取豆瓣TOP250 豆瓣爬取地址 https://movie.douban.com/top250?format=text BeautifulSoup官网地址 https:/ ...
利用Requests库和正则表达式爬取豆瓣影评Top250
说明最近看了下爬虫基础,想写个博客来记录一下,一来是可以方便和我一样刚入门的小白来参考学习,二来也当做自己的笔记供自己以后查阅. 本文章是利用python3.6和Requests库(需自行安装,cm ...
Requests库实战(三)---爬取豆瓣电影详细信息
完整代码爬取豆瓣电影的详细信息地址:豆瓣电影动画向下滑动时新增的数据也是Ajax请求,原理和上一个项目是一样的.唯一的不同是此处请求url携带了多个参数 import requests impo ...
Python爬取豆瓣高分图书TOP100存入Excel
作为一名转行过来的新手小白,花了两周业余时间,在B站上把北京理工大学嵩天老师的"Python语言程序设计"和"Python网络爬虫与信息提取"看了一遍,心里跃跃 ...
python爬豆瓣电视剧_python requests库爬取豆瓣电视剧数据并保存到本地详解
首先要做的就是去豆瓣网找对应的接口,这里就不赘述了,谷歌浏览器抓包即可,然后要做的就是分析返回的json数据的结构: https://movie.douban.com/j/search_subject ...
python BeautifulSoup爬取豆瓣电影top250信息并写入Excel表格
豆瓣是一个社区网站,创立于2005年3月6日.该网站以书影音起家,提供关于书籍,电影,音乐等作品信息,其描述和评论都是由用户提供的,是Web2.0网站中具有特色的一个网站. 豆瓣电影top250网址: ...
python实现爬虫探探_全栈 - 9 实战爬取豆瓣电影数据
这是全栈数据工程师养成攻略系列教程的第九期:9 实战爬取豆瓣电影数据. 掌握了爬虫的基本原理和代码实现,现在让我们通过实战项目巩固一下. 确定目标在写爬虫之前应当想清楚:我需要哪方面的数据?需要包 ...
python爬取豆瓣影评_【python爬虫实战】爬取豆瓣影评数据
概述: 爬取豆瓣影评数据步骤: 1.获取网页请求 2.解析获取的网页 3.提速数据 4.保存文件源代码: # 1.导入需要的库 import urllib.request from bs4 impo ...

requests库和beautifulsoup爬取豆瓣书本数据并存入excel

requests库和beautifulsoup爬取豆瓣书本数据并存入excel相关推荐

最新文章

热门文章