利用脚本动态爬取网页信息

编译环境：Jupyter Notebook (Anaconda3)

调用的包：selenium，webdrive，webdriver_manager

文章目录

利用脚本动态爬取网页信息
- 一、在Anaconda的虚拟环境下安装必要库
- - 1. selenium
  - 2. webdrive
  - 3. webdriver_manager
- 二、实现动态爬取
- - 1. 使用搜索引擎作最基本的脚本测试
  - 2. 爬取指定网页http://quotes.toscrape.com/js/的名言
  - 3. 爬取京东指定商品信息
- 三、总结
- 四、参考链接
- 五、代码下载

一、在Anaconda的虚拟环境下安装必要库

1. selenium

pip install selenium

2. webdrive

要使用selenium去调用浏览器，还需要一个驱动，不同浏览器的webdriver需要独立安装

可以从这里下载：https://npm.taobao.org/mirrors/chromedriver/

并将将该文件添加到PATH下具体方法请参照：win10系统如何配置环境变量

3. webdriver_manager

通过安装webdriver_manager来解决webdriver的管理问题

pip install webdriver_manager

二、实现动态爬取

1. 使用搜索引擎作最基本的脚本测试

使用脚本通过游览器打开百度

from selenium import webdriver
from webdriver_manager.chrome import ChromeDriverManager
driver = webdriver.Chrome(ChromeDriverManager().install())
#进入网页
driver.get("https://www.baidu.com/")

运行结果：

====== WebDriver manager ======
Current google-chrome version is 96.0.4664
Get LATEST chromedriver version for 96.0.4664 google-chrome
There is no [win32] chromedriver for browser  in cache
Trying to download new driver from https://chromedriver.storage.googleapis.com/96.0.4664.45/chromedriver_win32.zip
Driver has been saved in cache [C:\Users\12711\.wdm\drivers\chromedriver\win32\96.0.4664.45]
<ipython-input-2-bed3ab5004c4>:3: DeprecationWarning: executable_path has been deprecated, please pass in a Service objectdriver = webdriver.Chrome(ChromeDriverManager().install())

这里脚本自动下载了另一个所需版本的游览器驱动，通过Win10的任务管理器我们发现新启动的游览器与原本的分开了，显而易见两个版本是不同的。

右键搜索框检查元素

p_input = driver.find_element_by_id("kw")
p_input.send_keys('知乎')

<ipython-input-3-fe5ddeffd9d7>:1: DeprecationWarning: find_element_by_* commands are deprecated. Please use find_element() insteadp_input = driver.find_element_by_id("kw")

右键按钮检查元素

#点击搜索按钮
p_btn=driver.find_element_by_id('su')
p_btn.click()

<ipython-input-4-d6c5181ed729>:2: DeprecationWarning: find_element_by_* commands are deprecated. Please use find_element() insteadp_btn=driver.find_element_by_id('su')

2. 爬取指定网页http://quotes.toscrape.com/js/的名言

同样先打开该网页分析网页元素


from bs4 import BeautifulSoup as bs
from selenium import webdriver
import csv
from selenium.webdriver.chrome.options import Options
from tqdm import tqdm#在电脑终端上显示进度，使代码可视化进度加快
from webdriver_manager.chrome import ChromeDriverManager
driver = webdriver.Chrome(ChromeDriverManager().install())
driver.get('http://quotes.toscrape.com/js/')
#定义csv表头
quote_head=['名言','作者']
#csv文件的路径和名字
quote_path='D:\\1_WorkSpace\\quote.csv'
#存放内容的列表
quote_content=[]'''
function_name:write_csv
parameters:   csv_head,csv_content,csv_path
csv_head:     the csv file head
csv_content:  the csv file content,the number of columns equal to length of csv_head
csv_path:     the csv file route
'''
def write_csv(csv_head,csv_content,csv_path):with open(csv_path, 'w', newline='') as file:fileWriter =csv.writer(file)fileWriter.writerow(csv_head)fileWriter.writerows(csv_content)print('爬取信息成功')###
#可以用find_elements_by_class_name获取所有含这个元素的集合（列表也有可能）
#然后把这个提取出来之后再用继续提取
quote=driver.find_elements_by_class_name("quote")
#将要收集的信息放在quote_content里
for i in tqdm(range(len(quote))):    quote_text=quote[i].find_element_by_class_name("text")quote_author=quote[i].find_element_by_class_name("author")temp=[]temp.append(quote_text.text)temp.append(quote_author.text)quote_content.append(temp)
write_csv(quote_head,quote_content,quote_path)

运行结果：

====== WebDriver manager ======
Current google-chrome version is 96.0.4664
Get LATEST chromedriver version for 96.0.4664 google-chrome
Driver [C:\Users\12711\.wdm\drivers\chromedriver\win32\96.0.4664.45\chromedriver.exe] found in cache
<ipython-input-5-28b14e96b18c>:7: DeprecationWarning: executable_path has been deprecated, please pass in a Service objectdriver = webdriver.Chrome(ChromeDriverManager().install())
<ipython-input-5-28b14e96b18c>:33: DeprecationWarning: find_elements_by_* commands are deprecated. Please use find_elements() insteadquote=driver.find_elements_by_class_name("quote")0%|                                                                                           | 0/10 [00:00<?, ?it/s]C:\Users\12711\anaconda3\lib\site-packages\selenium\webdriver\remote\webelement.py:446: UserWarning: find_element_by_* commands are deprecated. Please use find_element() insteadwarnings.warn("find_element_by_* commands are deprecated. Please use find_element() instead")
100%|██████████████████████████████████████████████████████████████████████████████████| 10/10 [00:00<00:00, 27.87it/s]爬取信息成功

3. 爬取京东指定商品信息

因为最近想买耳机，所以就爬了个Redmi AirDots 3，很方便，可以非常简单地货比三家。

同样先通过检查元素查看指定元素地标签，这里不再展示。

from selenium import webdriver
import time
import csv
from tqdm import tqdm#在电脑终端上显示进度，使代码可视化进度加快
from webdriver_manager.chrome import ChromeDriverManager
driver = webdriver.Chrome(ChromeDriverManager().install())
#加载页面
driver.get("https://www.jd.com/")
time.sleep(3)#定义存放图书信息的列表
goods_info_list=[]
#爬取200本
goods_num=200
#定义表头
goods_head=['价格','名字','链接']
#csv文件的路径和名字
goods_path='D:\\1_Workspace\\erji.csv'#向输入框里输入Java
p_input = driver.find_element_by_id("key")
p_input.send_keys('Redmi AirDots3')#button好像不能根据类名直接获取，先获取大的div，再获取按钮
from_filed=driver.find_element_by_class_name('form')
s_btn=from_filed.find_element_by_tag_name('button')
s_btn.click()#实现点击#获取商品价格、名称、链接
def get_prince_and_name(goods):#直接用css定位元素#获取价格goods_price=goods.find_element_by_css_selector('div.p-price')#获取元素goods_name=goods.find_element_by_css_selector('div.p-name')#获取链接goods_herf=goods.find_element_by_css_selector('div.p-img>a').get_property('href')return goods_price,goods_name,goods_herfdef  drop_down(web_driver):#将滚动条调整至页面底部web_driver.execute_script('window.scrollTo(0, document.body.scrollHeight)')time.sleep(3)#获取爬取一页
def crawl_a_page(web_driver,goods_num):#获取图书列表drop_down(web_driver)goods_list=web_driver.find_elements_by_css_selector('div#J_goodsList>ul>li')#获取一个图书的价格、名字、链接for i in tqdm(range(len(goods_list))):goods_num-=1goods_price,goods_name,goods_herf=get_prince_and_name(goods_list[i])goods=[]goods.append(goods_price.text)goods.append(goods_name.text)goods.append(goods_herf)goods_info_list.append(goods)if goods_num==0:breakreturn goods_numwhile goods_num!=0:goods_num=crawl_a_page(driver,goods_num)btn=driver.find_element_by_class_name('pn-next').click()time.sleep(1)
write_csv(goods_head,goods_info_list,goods_path)

运行结果：

    ====== WebDriver manager ======Current google-chrome version is 96.0.4664Get LATEST chromedriver version for 96.0.4664 google-chromeDriver [C:\Users\12711\.wdm\drivers\chromedriver\win32\96.0.4664.45\chromedriver.exe] found in cache<ipython-input-7-e26dcb5907e1>:6: DeprecationWarning: executable_path has been deprecated, please pass in a Service objectdriver = webdriver.Chrome(ChromeDriverManager().install())<ipython-input-7-e26dcb5907e1>:21: DeprecationWarning: find_element_by_* commands are deprecated. Please use find_element() insteadp_input = driver.find_element_by_id("key")<ipython-input-7-e26dcb5907e1>:25: DeprecationWarning: find_element_by_* commands are deprecated. Please use find_element() insteadfrom_filed=driver.find_element_by_class_name('form')<ipython-input-7-e26dcb5907e1>:49: DeprecationWarning: find_elements_by_* commands are deprecated. Please use find_elements() insteadgoods_list=web_driver.find_elements_by_css_selector('div#J_goodsList>ul>li')100%|██████████████████████████████████████████████████████████████████████████████████| 30/30 [00:01<00:00, 19.09it/s]<ipython-input-7-e26dcb5907e1>:65: DeprecationWarning: find_element_by_* commands are deprecated. Please use find_element() insteadbtn=driver.find_element_by_class_name('pn-next').click()100%|██████████████████████████████████████████████████████████████████████████████████| 60/60 [00:03<00:00, 19.38it/s]100%|██████████████████████████████████████████████████████████████████████████████████| 42/42 [00:02<00:00, 19.66it/s]100%|██████████████████████████████████████████████████████████████████████████████████| 42/42 [00:02<00:00, 19.97it/s]60%|████████████████████████████████████████████████▊                                 | 25/42 [00:01<00:00, 18.78it/s]

爬取信息成功

三、总结

通过本次动态网页信息爬取，我亲身体验到了爬虫为什么被应用得如此广泛，利用脚本和爬虫可以轻易地爬取商城信息，如果在本次实验地基础上再加几个元素，就可以直接通过Excel表上来个十分清晰地货比三家。

四、参考链接

机智的橙子: 动态网页爬虫

weixin_43563705: Selenium:requests+Selenum爬取京东所有图书类的数据

五、代码下载

https://github.com/Wattson1128/HomeWork13-1