一、xpath提取数据

<bookstore>
<book category="Python 基础"><title lang="cn">cook book</title><author>David Beazley</author><year>2022</year><price>53.20</price>
</book>
<book category="story book"><title lang="en">The Lord of the Rings</title><author>J.R.R.托尔金</author><year>2005</year><price>29.99</price>
</book>
<book category="WEB"><title lang="en">Learning XML</title><author>Erik T. Ray</author><year>2013</year><price>40.05</price>
</book>
</bookstore>

xpath（XML Path Language）是在HTML\XML中查找信息的语句，可在HTML\XML文档中对元素和属性进行遍历

在根结点下面的节点是并列的，如一个树结构，我们也可以像访问文件一样来获得这个元素

xpath插件的安装：

自备梯子（能直接找到国内的.crx插件也可以不挂梯子）
google浏览器搜索xpath_helper，进入谷歌应用商店下载得到一个.crx后缀的文件
打开浏览器的设置，找到拓展（拓展程序），进入开发者模式
直接下载得到的.crx文件用鼠标拖入浏览器，选择安装拓展
点击浏览器右上角拓展程序图标，可将xpath插件锁定

点击xpath图标，出现这个黑框代表xpath插件安装成功

xpath节点选取：

nodename ：选中该元素
/ ：元素间的层级过度
// ：匹配选择，可省略中间节点而不考虑位置
@ ：选取属性
text() ：选取文本

lxml模块：

lxml模块是Python的第三方库，配合path，利用etree.HTML，将获取的网页字符串转化成Element对象，Element对象通过xpath的方法调用，以列表形式返回数据，再进行数据提取

import requests
from lxml import etreetext = """
<bookstore>
<book category="Python 基础"><title lang="cn">cook book</title><author>David Beazley</author><year>2022</year><price>53.20</price>
</book>
<book category="story book"><title lang="en">The Lord of the Rings</title><author>J.R.R.托尔金</author><year>2005</year><price>29.99</price>
</book>
<book category="WEB"><title lang="en">Learning XML</title><author>Erik T. Ray</author><year>2013</year><price>40.05</price>
</book>
</bookstore>
"""html = etree.HTML(text)
print(type(html))   # <class 'lxml.etree._Element'>
html_str = etree.tostring(html).decode()
print(type(html_str))   # <class 'str'># 获取书名
book_name = html.xpath("/html/body/bookstore/book/title/text()")    # 绝对路径
print(book_name)    # ['cook book', 'The Lord of the Rings', 'Learning XML']
book_name = html.xpath("//title/text()")
print(book_name)    # ['cook book', 'The Lord of the Rings', 'Learning XML']# 数据提取列表元素
for i in book_name:print(i, end="\t")
# cook book The Lord of the Rings   Learning XML# 获取作者
book_author = html.xpath("//author/text()")
print("\n", book_author)    # ['David Beazley', 'J.R.R.托尔金', 'Erik T. Ray']# 获取参数
category = html.xpath("//book/@category")
print(category)     # ['Python 基础', 'story book', 'WEB']
lang = html.xpath("//book/title/@lang")
print(lang)         # ['cn', 'en', 'en']book = html.xpath("//book")
for i in book:category = i.xpath("@category")[0]book_info = dict()book_info[category] = dict()book_info[category]['name'] = i.xpath("title/text()")[0]book_info[category]['author'] = i.xpath("author/text()")[0]book_info[category]['year'] = i.xpath("year/text()")[0]book_info[category]['price'] = i.xpath("price/text()")[0]print(book_info)"""
{'Python 基础': {'name': 'cook book', 'author': 'David Beazley', 'year': '2022', 'price': '53.20'}}
{'story book': {'name': 'The Lord of the Rings', 'author': 'J.R.R.托尔金', 'year': '2005', 'price': '29.99'}}
{'WEB': {'name': 'Learning XML', 'author': 'Erik T. Ray', 'year': '2013', 'price': '40.05'}}
"""

二、爬虫爬取图片资源

注：本代码爬取的图片皆无任何商业目的，仅供爬虫技术学习使用

示例：王者荣耀全英雄皮肤图片

import requests
from lxml import etree
import reurl = "https://pvp.qq.com/web201605/herolist.shtml"
response = requests.get(url)response.encoding = "gbk"
html = response.text
# print(html)html = etree.HTML(html)
# li_list = html.xpath("/html/body/div[3]/div/div/div[2]/div[2]/ul")
li_list = html.xpath('//ul[@class="herolist clearfix"]/li/a')
# xpath_helper拿到的xpath是已经被前端渲染过了的，不一定可用
# 如果xpath拿到的xpath不能直接用，就通过标签和属性手动选择数据print(len(li_list))     # 93，英雄数量
for i in li_list:href = i.xpath('./@href')[0]name = i.xpath('./img/@alt')[0]# print(href, name)pattern = r'herodetail/(\d*)\.shtml'id = re.search(pattern, href).group(1)# print(name, id)cnt = 1while True:try:url = f"https://game.gtimg.cn/images/yxzj/img201606/skin/hero-info/{id}/{id}-bigskin-{cnt}.jpg"resp = requests.get(url)if resp.status_code != 200:breakwith open(f"./skins/{name}{cnt}.jpg", "wb") as f:f.write(resp.content)cnt += 1except:print(Exception)break

三、爬虫爬取视频资源

注：本代码爬取的视频无任何商业目的，仅供爬虫技术学习使用

示例：B站李知恩视频

1. 对网页发送网络请求

import requests
from lxml import etreeurl = "https://search.bilibili.com/all?" \"vt=16780856&keyword=李知恩&from_source=webtop_search&spm_id_from=333.1007&search_source=5"response = requests.get(url)
print(response.text)

观察响应结果，请求发送成功，但是并未拿到想要的前端代码，提示需要验证码（登录）

登录信息在请求头里面，我们要获取请求头信息

2. 获取视频链接

import requests
from lxml import etreeurl = "https://search.bilibili.com/all?" \"vt=16780856&keyword=李知恩&from_source=webtop_search&spm_id_from=333.1007&search_source=5"header = {'user-agent': 'Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/110.0.0.0 Safari/537.36','referer': 'https://search.bilibili.com/all?keyword=%E6%9D%8E%E7%9F%A5%E6%81%A9&from_source=webtop_search&spm_id_from=333.1007&search_source=5','cookie': "buvid3=EE26C87B-4D71-DDB1-E3EF-7C231D141FDE66254infoc; b_nut=1676099166; i-wanna-go-back=-1; _uuid=EEA6D79B-8E44-283C-4B1A-F63391023FC41071505infoc; buvid4=8BD6A57D-1756-0D2F-2D58-824CE069B0CF82025-023021115-3r0csnyFmYTJnqj7nA8pAw%3D%3D; DedeUserID=703170552; DedeUserID__ckMd5=921efa783160cc40; rpdid=|(YumR|Yk)|0J'uY~Y|mmklY; b_ut=5; nostalgia_conf=-1; header_theme_version=CLOSE; buvid_fp_plain=undefined; hit-dyn-v2=1; CURRENT_BLACKGAP=0; CURRENT_FNVAL=4048; CURRENT_QUALITY=116; LIVE_BUVID=AUTO5716781015889927; hit-new-style-dyn=1; CURRENT_PID=114c35b0-cd23-11ed-a99d-39a1565dc8a4; fingerprint=07652cefeee9b3af116a4e8892842b91; home_feed_column=5; FEED_LIVE_VERSION=V8; SESSDATA=4afb5a37%2C1696835970%2Cf0b52%2A42; bili_jct=00e8099da712cb9a9738ad46d90f6d21; sid=8tpyjca9; bp_video_offset_703170552=783637244587016200; is-2022-channel=1; buvid_fp=07652cefeee9b3af116a4e8892842b91; PVID=2; innersign=1; b_lsid=326DE875_18776328C0A"
}response = requests.get(url, headers=header)
# print(response.text)html = etree.HTML(response.text)
div_list = html.xpath('//*[@id="i_cecream"]/div/div[2]/div[2]/div/div/div/div[1]/div/div')
print(len(div_list))for i in div_list:i_url = i.xpath('./div/div[2]/a/@href')[0]i_url = "https:" + i_urlprint(i_url)

3. 下载第三方包you-get

pip install you-get

4. 用操作系统调用you-get包下载视频资源

import requests
from lxml import etree
import sys
import os
from you_get import common as you_get
import time
import ffmpyurl = "https://search.bilibili.com/all?keyword=%E6%9D%8E%E7%9F%A5%E6%81%A9&from_source=webtop_search&spm_id_from=333.1007&search_source=5"
header = {'user-agent': 'Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/110.0.0.0 Safari/537.36','referer': 'https://search.bilibili.com/all?keyword=%E6%9D%8E%E7%9F%A5%E6%81%A9&from_source=webtop_search&spm_id_from=333.1007&search_source=5','cookie': "buvid3=EE26C87B-4D71-DDB1-E3EF-7C231D141FDE66254infoc; b_nut=1676099166; i-wanna-go-back=-1; _uuid=EEA6D79B-8E44-283C-4B1A-F63391023FC41071505infoc; buvid4=8BD6A57D-1756-0D2F-2D58-824CE069B0CF82025-023021115-3r0csnyFmYTJnqj7nA8pAw%3D%3D; DedeUserID=703170552; DedeUserID__ckMd5=921efa783160cc40; rpdid=|(YumR|Yk)|0J'uY~Y|mmklY; b_ut=5; nostalgia_conf=-1; header_theme_version=CLOSE; buvid_fp_plain=undefined; hit-dyn-v2=1; CURRENT_BLACKGAP=0; CURRENT_FNVAL=4048; CURRENT_QUALITY=116; LIVE_BUVID=AUTO5716781015889927; hit-new-style-dyn=1; CURRENT_PID=114c35b0-cd23-11ed-a99d-39a1565dc8a4; fingerprint=07652cefeee9b3af116a4e8892842b91; home_feed_column=5; FEED_LIVE_VERSION=V8; SESSDATA=4afb5a37%2C1696835970%2Cf0b52%2A42; bili_jct=00e8099da712cb9a9738ad46d90f6d21; sid=8tpyjca9; bp_video_offset_703170552=783637244587016200; is-2022-channel=1; buvid_fp=07652cefeee9b3af116a4e8892842b91; PVID=2; b_lsid=D3D108210E_1877853C1F0; innersign=0"
}response = requests.get(url, headers=header)
# print(response.text)html = etree.HTML(response.text)
div_list = html.xpath('//*[@id="i_cecream"]/div/div[2]/div[2]/div/div/div/div[1]/div/div')
print(len(div_list))path = "iu_video/"for i in div_list[:5]:v_url = i.xpath('.//a/@href')[0]v_url = "https:" + v_url# print(v_url)# v_title = i.xpath('./div/div[2]/a/div/div[1]/picture/img/@alt')v_title = (i.xpath('.//a/div/div[1]/picture//@alt')[0])print(v_url, v_title)try:# sys.argv = ['you-get', '-o', path, v_url]# you_get.main()os.system(f'you-get -o {path} {v_url}')print(v_url, "下载成功")except:print(Exception)print(v_url, v_title, "下载失败")else:time.sleep(2)

爬取到的资源包括XML类型文件和FLV类型文件，XML类型文件是弹幕文件，FLY类型文件是视频文件，特定播放器可以直接播放，我们也可以将其转换为mp4文件

四、FLV文件转码为MP4文件

安装ffmpy资源包，找到ffmgep.exe将视频文件转码为mp4文件

pip install ffmpy

import ffmpy
import osfolder_path = "./iu_video/"
for filename in os.listdir(folder_path):try:if ".flv" in filename:filename = folder_path + filenamesink_file = filename[:-3] + "mp4"ff = ffmpy.FFmpeg(executable="C:\\Program Files (x86)\\Common Files\DVDVideoSoft\\lib\\ffmpeg.exe",inputs={filename: None},outputs={sink_file: None})ff.run()print(filename, "转码成功")except:print(Exception)print(filename, "转码失败")

【Python】爬虫数据提取相关推荐

python爬虫数据提取_入门Python爬虫——提取数据篇
原标题:入门Python爬虫--提取数据篇作者: 李菲来源:人工智能学习圈前言在提取数据这一环节,爬虫程序会将我们所需要的数据提取出来.在上一篇文章<入门Python爬虫 -- 解析数据 ...
Python 爬虫数据提取
一下子运行七八十个 select 将会是什么样的体验呢? 业务部门提供了一个需要,要求从爬虫数据中提取出88家的数据, 并且也提供了一个excel表格,如下图: 这个时候我们可以通过拍卖行,拍卖时 ...
python爬虫数据提取,Python 信息提取-爬虫，爬虫提取数据, import re
Python 信息提取-爬虫,爬虫提取数据, import re import requestsimport refrom bs4 import BeautifulSoupurl = "ht ...
python 爬虫数据抓取的三种方式
python 爬虫数据抓取的三种方式常用抽取网页数据的方式有三种:正则表达式.Beautiful Soup.lxml 1.正则表达式正则表达式有个很大的缺点是难以构造.可读性差.不易适用未来 ...
Python爬虫数据可视化
Python爬虫--数据可视化导入需要的第三方库 import matplotlib.pyplot as plt import seaborn as sns import pandas as pd ...
python爬取网页表格数据匹配,python爬虫——数据爬取和具体解析
标签:pattern div mat txt 保存关于 json result with open 关于正则表达式的更多用法,可参考链接:https://blog.c ...
Python 爬虫数据写入csv文件中文乱码解决以及天眼查爬虫数据写入csv
python爬虫数据写入csv文件中文乱码,用'utf-8'在pycharm中打开文件没有问题,但是用excel打开却出现了问题,以下为解决方法. (最近在练习爬虫,这个博文是对自己学习的记录和分享, ...
建站四部曲之Python爬虫+数据准备篇(selenium)
本系列分为四篇: 建站四部曲之后端接口篇(SpringBoot+上线) 建站四部曲之Python数据爬虫篇(selenium) 建站四部曲之前端显示篇(React+上线) 建站四部曲之移动端篇(And ...
python爬虫数据解析总结
python爬虫数据解析总结目录 python爬虫数据解析总结 1.概述 2.Xpath解析html数据 2.1.基本语法 1.查询语法 2.2.Xpath解析html数据 1.安装lxml库 2. ...
1.爬虫数据提取——方法总结
爬虫数据提取--方法总结在获取了我们想要的html页面之后,接下来的问题就是如何将我们需要的数据给提取下来,一般来说有三种方式,分别是Xpath语法,正则表达式和bs4库,只要掌握了这三种方法,可以 ...

【Python】爬虫数据提取

一、xpath提取数据

二、爬虫爬取图片资源

三、爬虫爬取视频资源

四、FLV文件转码为MP4文件

【Python】爬虫数据提取相关推荐

最新文章

热门文章