python11(BS4模块和request模块)

1.BS4模块

1.BS4简介
Beautiful Soup提供一些简单的、python式的函数用来处理导航、搜索、修改分析树等功能。它是一个工具箱，通过解析文档为tiful Soup自动将输入文档转换为Unicode编码，输出文档转换为utf-8编码。
你不需要考虑编码方式，除非文档没有指定一个编一下原始编码方式就可以了。

2.BS4的4种对象

Tag对象：是html中的一个标签，用BeautifulSoup就能解析出来Tag的具体内容，具体的-格式为‘soup.name‘,其中name是html下的标签。
BeautifulSoup对象：整个html文本对象，可当作Tag对象
NavigableString对象：标签内的文本对象
Comment对象：是一个特殊的NavigableString对象，如果html标签内存在注释，那么它可以过滤掉注释符号保留注释文本
最常用的还是BeautifulSoup对象和Tag对象

3.BeautifulSoup对象

1)实例化soup对象

import refrom bs4 import BeautifulSoup
html = """
<html>
<head><title>story12345</title></head>
<body>
<p class="title" name="dromouse"><b>The Dormouse's story</b></p>
<p class="story">Once upon a time there were three little sisters; and their names were
<a href="http://example.com/elsie" class="sister" id="link1"><span>westos</span><!-- Elsie --></a>,
<a href="http://example.com/lacie" class="sister1" id="link2">Lacie</a> and
<a href="http://example.com/tillie" class="sister" id="link3">Tillie</a>;
and they lived at the bottom of a well.</p>
<p class="story">...</p>
"""
soup = BeautifulSoup(html, 'html.parser')
print(soup.prettify())

lass="story">Once upon a time there were three little sisters; and their names were<a class="sister" href="http://example.com/elsie" id="link1"><span>westos</span><!-- Elsie --></a>,<a class="sister1" href="http://example.com/lacie" id="link2">Lacie</a>and<a class="sister" href="http://example.com/tillie" id="link3">Tillie</a>;
and they lived at the bottom of a well.</p><p class="story">...</p></body>
</html>
{'href': 'http://example.com/elsie', 'class': ['sister'], 'id': 'link1'}
http://example.com/elsie

2)根据标签获取内容
1.标签的常用属性

# ******************标签的常用属性************************
# # 根据格式化， 如果title只有一个， 根据标签可以获取
print(soup.title)
print(type(soup.title))
print(soup.title.name)  # 标签的名称

<title>story12345</title>
<class 'bs4.element.Tag'>
title

# 获取标签里面的属性信息
print(soup.a.attrs)
print(soup.a.attrs['href'])

{'href': 'http://example.com/elsie', 'class': ['sister'], 'id': 'link1'}
http://example.com/elsie

2.标签常用的方法

# *******************标签常用的方法*************************
#get方法用于得到标签下的属性值，注意这是一个重要的方法，在许多场合都能用到，比如你要得到<img src=”#”>标签下的图像url,那么就可以用soup.img.get(‘src’)
print(soup.a.get('href'))
print(soup.a.get('class'))# string得到标签下的文本内容，只有在此标签下没有子标签，或者只有一个子标签的情况下才能返回其中的内容，否则返回的是None;
# get_text()可以获得一个标签中的所有文本内容，包括子孙节点的内容，这是最常用的方法
print(soup.a.string)    # 标签里面的内容
print(soup.a.get_text())

http://example.com/elsie
['sister']
None
westos

3.对获取的属性信息进行修改

# *******************对获取的属性信息进行修改***********************
print(soup.a.get('href'))
soup.a['href'] = 'http://www.baidu.com'
print(soup.a.get('href'))
print(soup.a)

http://example.com/elsie
http://www.baidu.com
<a class="sister" href="http://www.baidu.com" id="link1"><span>westos</span><!-- Elsie --></a>

3)面向对象的匹配

# 查找符合条件的所有标签;
aTagObj =  soup.find_all('a')
print(aTagObj)
for item in aTagObj:print(item)

[<a class="sister" href="http://example.com/elsie" id="link1"><span>westos</span><!-- Elsie --></a>, <a class="sister1" href="http://example.com/lacie" id="link2">Lacie</a>, <a class="sister" href="http://example.com/tillie" id="link3">Tillie</a>]
<a class="sister" href="http://example.com/elsie" id="link1"><span>westos</span><!-- Elsie --></a>
<a class="sister1" href="http://example.com/lacie" id="link2">Lacie</a>
<a class="sister" href="http://example.com/tillie" id="link3">Tillie</a>

 # 需求: 获取所有的a标签， 并且类名为"sister"
aTagObj = soup.find_all('a', class_="sister")
print(aTagObj)

[<a class="sister" href="http://example.com/elsie" id="link1"><span>westos</span><!-- Elsie --></a>, <a class="sister" href="http://example.com/tillie" id="link3">Tillie</a>]

4根据内容进行匹配

print(soup.find_all(text="story"))
print(soup.find_all(text=re.compile('story\d+')))

[]
['story12345']

5.CSS匹配

# 标签选择器
print(soup.select("title"))
# 类选择器(.类名)
print(soup.select(".sister"))
# id选择器(#id名称)
print(soup.select("#link1"))
# 此处不支持正则表达式;
# print(soup.select(re.compile("#link\d+")))
# 属性选择器()
print(soup.select("input[type='password']"))

[<title class="title">story12345</title>]
[<a class="sister" href="http://example.com/elsie" id="link1"><span>westos</span><!-- Elsie --></a>, <a class="sister" href="http://example.com/tillie" id="link3">Tillie</a>]
[<a class="sister" href="http://example.com/elsie" id="link1"><span>westos</span><!-- Elsie --></a>]
[<input type="password"/>]

4.BS4模块的解析器
官方中文文档: https://beautifulsoup.readthedocs.io/zh_CN/v4.4.0/

下表列出了主要的解析器,以及它们的优缺点:

解析器:使用方法:优势:劣势
Python标准库
BeautifulSoup(markup, “html.parser”)
Python的内置标准库
执行速度适中
文档容错能力强
Python 2.7.3 or 3.2.2)前的版本中文档容错能力差

lxml
HTML 解析器 BeautifulSoup(markup, “lxml”)
速度快
文档容错能力强
需要安装C语言库
lxml
XML 解析器
BeautifulSoup(markup, [“lxml-xml”])
BeautifulSoup(markup, “xml”)
速度快
唯一支持XML的解析器
需要安装C语言库

html5lib
BeautifulSoup(markup, “html5lib”)
最好的容错性
以浏览器的方式解析文档
生成HTML5格式的文档
速度慢
不依赖外部扩展

2.request模块

1.爬取个人博客-去掉广告和导航栏

import requests
from bs4 import BeautifulSoupdef get_content(url,):try:user_agent = "Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/59.0.3071.109 Safari/537.36"response = requests.get(url,  headers={'User-Agent': user_agent})response.raise_for_status()   # 如果返回的状态码不是200， 则抛出异常;response.encoding = response.apparent_encoding  # 判断网页的编码格式， 便于respons.text知道如何解码;except Exception as e:print("爬取错误")else:print(response.url)print("爬取成功!")return  response.contentdef parser_content(htmlContent):# 实例化soup对象， 便于处理；soup = BeautifulSoup(htmlContent, 'html.parser')# 提取页面的头部信息， 解决乱码问题headObj = soup.head# 提取需要的内容;divObj = soup.find_all('div', class_="blog-content-box")[0]#scriptObj = soup.scriptwith open('doc/csdn.html', 'w') as f:# 写入头部信息(指定编码格式和标题)f.write(str(headObj))# 写入博客正文;f.write(str(divObj))print("下载成功......")# f.write(str(scriptObj))
if __name__ == '__main__':url = 'https://blog.csdn.net/mangoknight/article/details/87297437'content = get_content(url)parser_content(content)

https://blog.csdn.net/King15229085063/article/details/87380182
爬取成功!
下载成功......

2. 个人博客整理

import requests
from bs4 import BeautifulSoup
import redef get_content(url,):try:user_agent = "Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/59.0.3071.109 Safari/537.36"response = requests.get(url,  headers={'User-Agent': user_agent})response.raise_for_status()   # 如果返回的状态码不是200， 则抛出异常;response.encoding = response.apparent_encoding  # 判断网页的编码格式， 便于respons.text知道如何解码;except Exception as e:print("爬取错误")else:print(response.url)print("爬取成功!")return  response.contentdef parser_content(htmlContent):# 实例化soup对象， 便于处理；soup = BeautifulSoup(htmlContent, 'html.parser')# 1). 获取每个博客的大盒子： 特征: div标签, class名称一致article-item-box csdn-tracking-statistics#  <div class="article-item-box csdn-tracking-statistics" data-articleid="85718923">divObjs = soup.find_all('div', class_="article-item-box")# 2). 依次遍历每一个div标签， 获取博客标题#  博客标题的特征: h4里面的a标签里面的内容# 去掉默认的广告, 留下个人的博客内容;for  divObj in divObjs[1:]:# **2-1. 获取博客标题: 去掉原创或者转载的信息， 只需要博客名称;title = divObj.h4.a.get_text().split()[1]# **2-2. 获取博客链接， 也就是获取a链接中href对应的值；blogUrl = divObj.h4.a.get('href')global  bloginfo# 将爬取的所有内容保存到变量中[(blogtitle, blogurl)]bloginfo.append((title, blogUrl))if __name__ == '__main__':blogPage = 3# 全局变量， 用于保存所有博客信息;bloginfo = []for page in range(1, blogPage+1):url = 'https://blog.csdn.net/mangoknight/article/list/%s' %(page)content = get_content(url)parser_content(content)print("第%d页整理结束" %(page))with open('doc/myblog.md', 'a') as f:for index, info in enumerate(bloginfo[::-1]):f.write('- 第%d篇博客: [%s](%s)\n' %(index+1, info[0], info[1]))print("完成")

https://blog.csdn.net/mangoknight/article/list/1
爬取成功
第1页整理结束
https://blog.csdn.net/mangoknight/article/list/2
爬取成功
第2页整理结束
https://blog.csdn.net/mangoknight/article/list/3
爬取成功
第3页整理结束
完成

3.西刺代理IP爬取

from urllib.error import HTTPError
import requests
from bs4 import BeautifulSoupdef get_content(url):try:user_agent='Mozilla/5.0 (X11; Linux x86_64; rv:45.0) Gecko/20100101 Firefox/45.0'response=requests.get(url,headers={'User-Agent':user_agent})response.raise_for_status()response_encoding=response.apparent_encodingexcept HTTPError as e:print('爬取失败')else:print(response.url)print('爬取成功')return response.contentdef crawl_ip(content):soup=BeautifulSoup(content,'html.parser')for item in soup.find_all('tr')[1:]:ip = item.find_all('td')[1].stringport = item.find_all('td')[2].stringprint((ip, port))if __name__ == '__main__':url='https://www.xicidaili.com/nn/'content=get_content(url)crawl_ip(content)

https://www.xicidaili.com/nn/
爬取成功
('116.209.58.197', '9999')
('121.61.0.106', '9999')
('111.177.190.15', '9999')
('116.209.57.105', '9999')
('121.61.1.33', '9999')
('183.148.153.222', '9999')
('115.151.5.105', '9999')

4.爬取豆瓣TOP250个电影信息

import reimport requests
from bs4 import BeautifulSoupdef get_content(url,):try:user_agent = "Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/59.0.3071.109 Safari/537.36"response = requests.get(url,  headers={'User-Agent': user_agent})response.raise_for_status()   # 如果返回的状态码不是200， 则抛出异常;response.encoding = response.apparent_encoding  # 判断网页的编码格式， 便于respons.text知道如何解码;except Exception as e:print("爬取错误")else:print(response.url)print("爬取成功!")return  response.contentdef parser_content(htmlContent):# 实例化soup对象， 便于处理；soup = BeautifulSoup(htmlContent, 'html.parser')#  1). 电影信息存储在ol标签里面的li标签:#  <ol class="grid_view">olObj = soup.find_all('ol', class_='grid_view')[0]#  2). 获取每个电影的详细信息, 存储在li标签;details = olObj.find_all('li')for detail in details:# #  3). 获取电影名称;movieName = detail.find('span', class_='title').get_text()# 4). 电影评分：movieScore = detail.find('span', class_='rating_num').get_text()# 5). 评价人数***************# 必须要转换类型为字符串movieCommentNum = str(detail.find(text=re.compile('\d+人评价')))# 6). 电影短评movieCommentObj = detail.find('span', class_='inq')if movieCommentObj:movieComment = movieCommentObj.get_text()else:movieComment = "无短评"movieInfo.append((movieName, movieScore, movieCommentNum, movieComment))import openpyxldef create_to_excel(wbname, data, sheetname='Sheet1', ):"""将制定的信息保存到新建的excel表格中;:param wbname::param data: 往excel中存储的数据;:param sheetname::return:"""print("正在创建excel表格%s......" % (wbname))# wb = openpyxl.load_workbook(wbname)#  如果文件不存在， 自己实例化一个WorkBook的对象;wb = openpyxl.Workbook()# 获取当前活动工作表的对象sheet = wb.active# 修改工作表的名称sheet.title = sheetname# 将数据data写入excel表格中;print("正在写入数据........")for row, item in enumerate(data):  # data发现有4行数据， item里面有三列数据;print(item)for column, cellValue in enumerate(item):# cell = sheet.cell(row=row + 1, column=column + 1, value=cellValue)cell = sheet.cell(row=row+1, column=column + 1)cell.value = cellValuewb.save(wbname)print("保存工作薄%s成功......." % (wbname))if __name__ == '__main__':doubanTopPage = 2perPage = 25# [(), (), ()]movieInfo = []# 1, 2, 3 ,4, 5for page in range(1, doubanTopPage+1):# start的值= (当前页-1)*每页显示的数量(25)url = "https://movie.douban.com/top250?start=%s" %((page-1)*perPage)content = get_content(url)parser_content(content)create_to_excel('/tmp/hello.xlsx', movieInfo, sheetname="豆瓣电影信息")

https://movie.douban.com/top250?start=0
爬取成功!
https://movie.douban.com/top250?start=25
爬取成功!
正在创建excel表格/tmp/hello.xlsx......
正在写入数据........
('肖申克的救赎', '9.6', '1325620人评价', '希望让人自由。')
('霸王别姬', '9.6', '979012人评价', '风华绝代。')
('这个杀手不太冷', '9.4', '1213845人评价', '怪蜀黍和小萝莉不得不说的故事。')
('教父2', '9.1', '257644人评价', '优雅的孤独。')
保存工作薄/tmp/hello.xlsx成功.......

5.中国百名大学定向爬虫

import requests
from bs4 import BeautifulSoup
import bs4def get_content(url,):try:user_agent = "Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/59.0.3071.109 Safari/537.36"response = requests.get(url,  headers={'User-Agent': user_agent})response.raise_for_status()   # 如果返回的状态码不是200， 则抛出异常;response.encoding = response.apparent_encoding  # 判断网页的编码格式， 便于respons.text知道如何解码;except Exception as e:print("爬取错误")else:print(response.url)print("爬取成功!")return  response.contentdef getUnivList(html):"""解析页面内容， 需要获取: 学校排名， 学校名称， 省份， 总分"""soup = BeautifulSoup(html, 'lxml')# 该页面只有一个表格， 也只有一个tbody标签;# 获取tbosy里面的所有子标签, 返回的是生成器： soup.find('tbody').children# 获取tbosy里面的所有子标签, 返回的是列表：   soup.find('tbody').contentsuList = []for tr in soup.find('tbody').children:# 有可能没有内容， 获取的tr标签不存在, 判断是否为标签对象?if isinstance(tr, bs4.element.Tag):# print(tr.td)# 返回tr里面的所有td标签;tds = tr('td')# print(tds)# 将每个学校信息以元组的方式存储到列表变量uList中;uList.append((tds[0].string, tds[1].string, tds[2].string, tds[3].string))return  uListdef printUnivList(uList):"""打印学校信息:param uList::return:"""# format的使用: {0} 变量的位置, 冒号后面执行属性信息: ^10占10个字节位置， 并且居中print("{0:^10} {1:^10} {2:^10} {3:^10}".format("排名", '学校名称', "省份/城市", "总分"))for item in uList:# print(item)print("{0:^10} {1:^10} {2:^10} {3:^10}".format(item[0], item[1], item[2], item[3]))def saveUnivData(uList, year):import csvwith open('doc/ranking-%s.csv' %(year), 'w') as f:writer = csv.writer(f)# 将列表的每条数据依次写入csv文件， 并以逗号分隔writer.writerows(uList)print("写入完成....")if __name__ == '__main__':start_year = int(input("开始爬取的年份:"))end_year =  int(input("结束爬取的年份:"))for year in range(start_year, end_year+1):# year = 2017url = "http://www.zuihaodaxue.com/zuihaodaxuepaiming%s.html" %(year)content = get_content(url)uList = getUnivList(content)# printUnivList(uList)saveUnivData(uList, year)print("%s年信息爬取成功......" %(year))

开始爬取的年份:2016
结束爬取的年份:2019
http://www.zuihaodaxue.com/zuihaodaxuepaiming2016.html
爬取成功!
写入完成....
2016年信息爬取成功......
http://www.zuihaodaxue.com/zuihaodaxuepaiming2017.html
爬取成功!
写入完成....
2017年信息爬取成功......
http://www.zuihaodaxue.com/zuihaodaxuepaiming2018.html
爬取成功!
写入完成....
2018年信息爬取成功......
http://www.zuihaodaxue.com/zuihaodaxuepaiming2019.html
爬取成功!
写入完成....
2019年信息爬取成功......

6.豆瓣影评定向爬虫及数据分析处理_绘制词云

import reimport requests
from urllib.error import HTTPError
import csvdef get_content(url):try:user_agent='Mozilla/5.0 (X11; Linux x86_64; rv:45.0) Gecko/20100101 Firefox/45.0'response=requests.get(url,headers={'User-Agent':user_agent})response.raise_for_status()response_encoding=response.apparent_encodingexcept requests.exceptions.HTTPError as e:print('爬取失败',e)return ''except HTTPError as e:print('爬取失败',e)return ''else:print(response.url)print('爬取成功')return response.contentdef parser_content(html):import lxml.etree as etreeselector=etree.HTML(html)typeObj=selector.xpath('//div[@id="nowplaying"]/div[@class="mod-bd"]')[0]moviesObj=typeObj.xpath('.//ul[@class="lists"]/li')new_urls=[]for item in moviesObj:commentId=item.xpath('.//@id')[0]name=item.xpath('.//li[@class="stitle"]/a[@class="ticket-btn"]/text()')[0]new_url='https://movie.douban.com/subject/%s/comments' %(commentId)new_urls.append((name,new_url))return new_urlsdef parser_comment(html,comments):import lxml.etree as etreeimport reselector=etree.HTML(html)comment=str(selector.xpath('//span[@class="short"]/text()'))nextPage=selector.xpath('//a[contains(text(),"后页")]/@href')comments.append(comment)return nextPageimport jieba
import numpy as npfrom PIL import Imagefrom wordcloud import wordclouddef gen_wordCloud(text,filename):result=jieba.lcut(text)imgObj=Image.open('doc/tree.jpg')img_mask=np.array(imgObj)wcObj=wordcloud.WordCloud(font_path='/usr/share/fonts/wqy-zenhei/wqy-zenhei.ttc',mask=img_mask,background_color='snow',min_font_size=5,max_font_size=50,)wcObj.generate(','.join(result))wcObj.to_file(filename)def movieSpider():url='https://movie.douban.com/nowplaying'content=get_content(url)urls=parser_content(content)for item in urls:comments = []movies.append(item[0])url=item[1]html=get_content(url)while True and html:nextPage=parser_comment(html,comments)if nextPage:nextPageUrl=url+nextPage[0]html=get_content(nextPageUrl)else:print('爬取完毕')breakwith open('doc/%s.csv' %(item[0].strip()),'w') as f:writer=csv.writer(f)writer.writerows(comments)if __name__ == '__main__':import csvmovies=[]# 爬取影评，并保存到文件中movieSpider()pattern = re.compile(r'[[\u4e00-\u9fa5]+|[a-zA-Z0-9]+]')for movie in movies:text=''with open('doc/%s.csv' %(movie.strip())) as f:reader=csv.reader(f)for row in reader:text+=''.join(re.findall(pattern,str(row)))gen_wordCloud(text,'doc/img/%s.png' %(movie))