Python爬虫辅助库BeautifulSoup4用法精要

BeautifulSoup是一个非常优秀的Python扩展库，可以用来从HTML或XML文件中提取我们感兴趣的数据，并且允许指定使用不同的解析器。由于beautifulsoup3已经不再继续维护，因此新的项目中应使用beautifulsoup4，目前最新版本是4.5.0，可以使用pip install beautifulsoup4直接进行安装，安装之后应使用from bs4 import BeautifulSoup导入并使用。下面我们就一起来简单看一下BeautifulSoup4的强大功能，更加详细完整的学习资料请参考https://www.crummy.com/software/BeautifulSoup/bs4/doc/。

>>> from bs4 import BeautifulSoup

>>> BeautifulSoup('hello world!', 'lxml') #自动添加和补全标签

<html><body>hello world!</body></html>

>>> html_doc = """

<html><head><title>The Dormouse's story</title></head>

<body>

The Dormouse's story

Once upon a time there were three little sisters; and their names were

<a href="http://example.com/elsie" class="sister" id="link1">Elsie</a>,

<a href="http://example.com/lacie" class="sister" id="link2">Lacie</a> and

<a href="http://example.com/tillie" class="sister" id="link3">Tillie</a>;

and they lived at the bottom of a well.

...

"""

>>> soup = BeautifulSoup(html_doc, 'html.parser') #也可以使用lxml或其他解析器

>>> print(soup.prettify()) #以优雅的方式显示出来

<html>

<head>

<title>

The Dormouse's story

</title>

</head>

<body>

The Dormouse's story

Once upon a time there were three little sisters; and their names were

Elsie

</a>

Lacie

</a>

and

Tillie

</a>

;

and they lived at the bottom of a well.

...

</body>

</html>

>>> soup.title #访问特定的标签

<title>The Dormouse's story</title>

>>> soup.title.name #标签名字

'title'

>>> soup.title.text #标签文本

"The Dormouse's story"

>>> soup.title.string

"The Dormouse's story"

>>> soup.title.parent #上一级标签

<head><title>The Dormouse's story</title></head>

>>> soup.head

<head><title>The Dormouse's story</title></head>

>>> soup.b

The Dormouse's story

>>> soup.body.b

The Dormouse's story

>>> soup.name #把整个BeautifulSoup对象看做标签对象

'[document]'

>>> soup.body

<body>

The Dormouse's story

Once upon a time there were three little sisters; and their names were

<a class="sister" href="http://example.com/elsie" id="link1">Elsie</a>,

<a class="sister" href="http://example.com/lacie" id="link2">Lacie</a> and

<a class="sister" href="http://example.com/tillie" id="link3">Tillie</a>;

and they lived at the bottom of a well.

...

</body>

>>> soup.p

The Dormouse's story

>>> soup.p['class'] #标签属性

['title']

>>> soup.p.get('class') #也可以这样查看标签属性

['title']

>>> soup.p.text

"The Dormouse's story"

>>> soup.p.contents

[The Dormouse's story]

>>> soup.a

<a class="sister" href="http://example.com/elsie" id="link1">Elsie</a>

>>> soup.a.attrs #查看标签所有属性

{'class': ['sister'], 'href': 'http://example.com/elsie', 'id': 'link1'}

>>> soup.find_all('a') #查找所有<a>标签

[<a class="sister" href="http://example.com/elsie" id="link1">Elsie</a>, <a class="sister" href="http://example.com/lacie" id="link2">Lacie</a>, <a class="sister" href="http://example.com/tillie" id="link3">Tillie</a>]

>>> soup.find_all(['a', 'b']) #同时查找<a>和标签

[The Dormouse's story, <a class="sister" href="http://example.com/elsie" id="link1">Elsie</a>, <a class="sister" href="http://example.com/lacie" id="link2">Lacie</a>, <a class="sister" href="http://example.com/tillie" id="link3">Tillie</a>]

>>> import re

>>> soup.find_all(href=re.compile("elsie")) #查找href包含特定关键字的标签

[<a class="sister" href="http://example.com/elsie" id="link1">Elsie</a>]

>>> soup.find(id='link3')

<a class="sister" href="http://example.com/tillie" id="link3">Tillie</a>

>>> soup.find_all('a', id='link3')

[<a class="sister" href="http://example.com/tillie" id="link3">Tillie</a>]

>>> for link in soup.find_all('a'):

print(link.text,':',link.get('href'))

Elsie : http://example.com/elsie

Lacie : http://example.com/lacie

Tillie : http://example.com/tillie

>>> print(soup.get_text()) #返回所有文本

The Dormouse's story

Once upon a time there were three little sisters; and their names were

Elsie,

Lacie and

Tillie;

and they lived at the bottom of a well.

...

>>> soup.a['id'] = 'test_link1' #修改标签属性的值

>>> soup.a

<a class="sister" href="http://example.com/elsie" id="test_link1">Elsie</a>

>>> soup.a.string.replace_with('test_Elsie') #修改标签文本

'Elsie'

>>> soup.a.string

'test_Elsie'

>>> print(soup.prettify())

<html>

<head>

<title>

The Dormouse's story

</title>

</head>

<body>

The Dormouse's story

Once upon a time there were three little sisters; and their names were

test_Elsie

</a>

Lacie

</a>

and

Tillie

</a>

;

and they lived at the bottom of a well.

...

</body>

</html>

>>> for child in soup.body.children: #遍历直接子标签

print(child)

The Dormouse's story

Once upon a time there were three little sisters; and their names were

<a class="sister" href="http://example.com/elsie" id="test_link1">test_Elsie</a>,

<a class="sister" href="http://example.com/lacie" id="link2">Lacie</a> and

<a class="sister" href="http://example.com/tillie" id="link3">Tillie</a>;

and they lived at the bottom of a well.

...

>>> for string in soup.strings: #遍历所有文本，结果略

print(string)

>>> test_doc = '<html><head></head><body></body></heml>'

>>> s = BeautifulSoup(test_doc, 'lxml')

>>> for child in s.html.children: #遍历直接子标签

print(child)

>>> for child in s.html.descendants: #遍历子孙标签

print(child)

Python爬虫辅助库BeautifulSoup4用法精要相关推荐

Python爬虫扩展库BeautifulSoup4用法精要
BeautifulSoup是一个非常优秀的Python扩展库,可以用来从HTML或XML文件中提取我们感兴趣的数据,并且允许指定使用不同的解析器.由于beautifulsoup3已经不再继续维护,因此 ...
python爬虫---requests库的用法
requests是python实现的简单易用的HTTP库,使用起来比urllib简洁很多因为是第三方库,所以使用前需要cmd安装 pip install requests 安装完成后import一下 ...
Python爬虫扩展库scrapy选择器用法入门（一）
关于BeutifulSoup4的用法入门请参考Python爬虫扩展库BeautifulSoup4用法精要,scrapy爬虫案例请参考Python使用Scrapy爬虫框架爬取天涯社区小说"大宗 ...
Python爬虫辅助利器PyQuery模块的安装使用攻略
这篇文章主要介绍了Python爬虫辅助利器PyQuery模块的安装使用攻略,PyQuery可以方便地用来解析HTML内容,使其成为众多爬虫程序开发者的大爱,需要的朋友可以参考下 Windows下的安装 ...
Python Pillow(PIL)库的用法介绍(二)
Python Pillow(PIL)库的用法介绍(二) 在上一篇文章中介绍了Pillow库的一些基本用法,参考:https://blog.csdn.net/weixin_43790276/articl ...
【python】python爬虫requests库详解
1.安装:pip install requests 简介:Requests是一个优雅而简单的Python HTTP库,与之前的urllibPython的标准库相比,Requests的使用方式非常的简单 ...
已解决（Python爬虫requests库报错请求异常SSL错误，证书认证失败问题）requests.exceptions.SSLError: HTTPSConnectionPool
成功解决(Python爬虫requests库报错请求异常,SSL错误,证书认证失败问题)requests.exceptions.SSLError: HTTPSConnectionPool(host= ...
Python爬虫——Requests 库基本使用
文章目录 Python爬虫--Requests 库基本使用 1.Requests简介和下载 2.Requests 库基本使用 Python爬虫--Requests 库基本使用 1.Requests简介 ...
Python爬虫基础库(RBX)的实践_田超凡
转载请注明原作者:田超凡 20190410 CSDN博客:https://blog.csdn.net/qq_30056341# Python爬虫基础库RBX指的是:Requests Beautiful ...

Python爬虫辅助库BeautifulSoup4用法精要

Python爬虫辅助库BeautifulSoup4用法精要相关推荐

最新文章

热门文章