python爬虫爬取豆瓣读书Top250

话不多说，直接上代码！

from urllib.request import urlopen, Request
from bs4 import BeautifulSoup
import randomclass DBTop250_Spider():url = r"http://book.douban.com/top250?start="page_num = 0top_num = 1headers = [{'User-Agent': 'Mozilla/5.0 (X11; Ubuntu; Linux x86_64; rv:34.0) Gecko/20100101 Firefox/34.0'},{'User-Agent': 'Mozilla/5.0 (Windows; U; Windows NT 6.1; en-US; rv:1.9.1.6) Gecko/20091201 Firefox/3.5.6'},{'User-Agent': 'Mozilla/5.0 (Windows NT 6.2) AppleWebKit/535.11 (KHTML, like Gecko) Chrome/17.0.963.12 ''Safari/535.11'},{'User-Agent': 'Mozilla/5.0 (compatible; MSIE 10.0; Windows NT 6.2; Trident/6.0)'},{'User-Agent': 'Mozilla/5.0 (X11; Ubuntu; Linux x86_64; rv:40.0) Gecko/20100101 Firefox/40.0'},{'User-Agent': 'Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/537.36 (KHTML, like Gecko) Ubuntu ''Chromium/44.0.2403.89 Chrome/44.0.2403.89 Safari/537.36'}]file_path = "..\Files\Books"def Parse_Page(self):req = Request(self.url+str(self.page_num * 25), headers=random.choice(self.headers))page = urlopen(req).read()bsObj = BeautifulSoup(page, "html.parser")books_items = bsObj.find_all("a")for item in books_items:if 'title' in item.attrs:self.Parse_Book(item.attrs['href'])self.top_num += 1self.page_num += 1def Parse_Book(self, book_url):print("book_url is: " + book_url)try:book_request = Request(book_url, headers=random.choice(self.headers))book_html = urlopen(book_request).read()book_bsObj = BeautifulSoup(book_html, "html.parser")book_title = book_bsObj.h1book_info = book_bsObj.find("div", {"id": "info"})intro = book_bsObj.find_all("div", {"class": "intro"})if intro[0].get_text().find("展开全部") != -1:book_intro = intro[1]if intro[2].get_text().find('展开全部') != -1:author_intro = intro[3]else:author_intro = intro[2]else:book_intro = intro[0]if intro[1].get_text().find('展开全部') != -1:author_intro = intro[2]else:author_intro = intro[1]with open(self.file_path + '\Top' + str(self.top_num) + '.txt', 'w', errors='ignore') as f:f.write(book_title.get_text().strip() + '\n')info_text = book_info.get_text().split(' ')for info in info_text:if info != '\n' and info != '':f.write(info.strip())f.write("\n\n内容简介: ")f.write(book_intro.get_text())f.write("\n\n作者简介: ")f.write(author_intro.get_text())except Exception as e:if hasattr(e, "reason"):print("Reason: " + e.reason)sp = DBTop250_Spider()
for i in range(1, 11):sp.Parse_Page()

运行结果：

python爬虫爬取豆瓣读书Top250相关推荐

Python爬虫 - 爬取豆瓣读书TOP250电子书
1.爬取思路访问豆瓣读书top250,并提取书籍名称. 把提取的书籍名称,放到搜书网站(本文使用的是熊猫搜书). 把搜索的结果保存到本地.(底部有保存的文本文件,需要自提) 2.重点解析 1.豆瓣翻 ...
Python爬虫爬取豆瓣电影TOP250
Python爬虫爬取豆瓣电影TOP250 最近在b站上学习了一下python的爬虫,实践爬取豆瓣的电影top250,现在对这两天的学习进行一下总结主要分为三步: 爬取豆瓣top250的网页,并通过 ...
Python爬虫爬取豆瓣电影Top250
爬虫爬取豆瓣Top250 文章目录爬虫爬取豆瓣Top250 完整代码第一步获取整个网页并以html来解析正则表达式来匹配关键词所有信息写入列表中存入Excel中效果如图本文学习自B站UP ...
Python爬虫爬取豆瓣读书
最近用Python写了个豆瓣读书的爬虫玩,方便搜罗各种美美书,分享给大家实现功能: 1 可以爬下豆瓣读书标签下的所有图书 2 按评分排名依次存储 3 存储到Excel中,可方便大家筛选搜罗,比如筛选 ...
python爬虫爬取豆瓣读书
python的课程设计,我爬取的是豆瓣的图书设计题目: 豆瓣读书的Python爬虫一.需求分析由于豆瓣上的图书良莠不齐,很难一下选择适合我们的图书,所以我想通过Python的爬虫程序,实现对豆 ...
Python爬虫-爬取豆瓣读书
爬点什么好呢?最近老是浏览豆瓣上的帖子,那就爬取下豆瓣读书吧! 网络请求,返回值是html页面. 需要对返回回来的结果进行解析.使用Beautiful Soup来解析参见(http://beauti ...
Python3之爬虫爬取豆瓣读书Top250
import requests from bs4 import BeautifulSoup from openpyxl import Workbookexcel_name = "豆瓣读书.x ...
Python 爬虫抓取豆瓣读书TOP250
# -*- coding:utf-8 -*- # author: yukun import requests from bs4 import BeautifulSoup# 发出请求获得HTML源码的函 ...
利用python爬虫爬取豆瓣读书-文学-名著的封面
获取至少两个页面的豆瓣读书-文学-名著的图书封面图片,将图片存到文件夹. 具体代码如下: #dubanimage.py import requests from bs4 import Beautifu ...

python爬虫爬取豆瓣读书Top250

python爬虫爬取豆瓣读书Top250

python爬虫爬取豆瓣读书Top250相关推荐

最新文章

热门文章