python爬虫实例——某二手车数据爬取

某二手车网站数据爬取

要求：

找到所要爬取的网站网址（url）；
今天案例的网址（url）：https://www.guazi.com/gy/dazhong/o1/#bread。
观察网站，点开检查，找到我们所要爬取的数据。
通过对网站数据的分析，明确目标数据的爬取方式后，通过请求获得数据。
最后将得到的数据保存在文件中。

爬取前的准备工作
下载相关库：
今天我们要用到的库有：

第三方：requests库
xlml库
bs4—BeautifulSoup库

开始代码演示及分析

import requests  #导入requests库，对url发送请求
import xlml
from bs4 import BeautifulSoup   #用bs4来解析网页，帮助我们爬取到我们想要爬取的内容
import re#首先确定我们的url
url = 'https://www.guazi.com/gy/dazhong/o2/#bread'   #这里我们爬取数据的对象为某二手车网站上大众品牌汽车的数据
#观察网页我们发现我们不止要在请求的时候加上user_agent，还需要加入cookie
headers = {'User-Agent':'Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/78.0.3904.70 Safari/537.36','Cookie':'antipas=0601819329K313586692192455166; uuid=3bfbeb56-9f3e-4d0d-cbe3-260b14154476; cityDomain=gy; clueSourceCode=%2A%2300; user_city_id=36; Hm_lvt_936a6d5df3f3d309bda39e92da3dd52f=1595389723; ganji_uuid=2943330065231720816366; sessionid=641c79ee-5210-4be7-892b-466faf206b39; lg=1; close_finance_popup=2020-07-22; cainfo=%7B%22ca_a%22%3A%22-%22%2C%22ca_b%22%3A%22-%22%2C%22ca_s%22%3A%22seo_baidu%22%2C%22ca_n%22%3A%22default%22%2C%22ca_medium%22%3A%22-%22%2C%22ca_term%22%3A%22-%22%2C%22ca_content%22%3A%22-%22%2C%22ca_campaign%22%3A%22-%22%2C%22ca_kw%22%3A%22-%22%2C%22ca_i%22%3A%22-%22%2C%22scode%22%3A%22-%22%2C%22keyword%22%3A%22-%22%2C%22ca_keywordid%22%3A%22-%22%2C%22display_finance_flag%22%3A%22-%22%2C%22platform%22%3A%221%22%2C%22version%22%3A1%2C%22client_ab%22%3A%22-%22%2C%22guid%22%3A%223bfbeb56-9f3e-4d0d-cbe3-260b14154476%22%2C%22ca_city%22%3A%22gy%22%2C%22sessionid%22%3A%22641c79ee-5210-4be7-892b-466faf206b39%22%7D; preTime=%7B%22last%22%3A1595391260%2C%22this%22%3A1595389721%2C%22pre%22%3A1595389721%7D; Hm_lpvt_936a6d5df3f3d309bda39e92da3dd52f=1595391261'}

在这里，如果我们不加cookie的话会出现以下情况

url = 'https://www.guazi.com/gy/dazhong/o2/#bread'
headers = {'User-Agent':'Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/78.0.3904.70 Safari/537.36'}
response = requests.get(url,headers=headers).content.decode()
print(response)

从而得不到我们想要的数据，但是当我们加上cookie时：

url = 'https://www.guazi.com/gy/dazhong/o1/#bread'
headers = {'User-Agent':'Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/78.0.3904.70 Safari/537.36','Cookie':'antipas=0601819329K313586692192455166; uuid=3bfbeb56-9f3e-4d0d-cbe3-260b14154476; cityDomain=gy; clueSourceCode=%2A%2300; user_city_id=36; Hm_lvt_936a6d5df3f3d309bda39e92da3dd52f=1595389723; ganji_uuid=2943330065231720816366; sessionid=641c79ee-5210-4be7-892b-466faf206b39; lg=1; close_finance_popup=2020-07-22; cainfo=%7B%22ca_a%22%3A%22-%22%2C%22ca_b%22%3A%22-%22%2C%22ca_s%22%3A%22seo_baidu%22%2C%22ca_n%22%3A%22default%22%2C%22ca_medium%22%3A%22-%22%2C%22ca_term%22%3A%22-%22%2C%22ca_content%22%3A%22-%22%2C%22ca_campaign%22%3A%22-%22%2C%22ca_kw%22%3A%22-%22%2C%22ca_i%22%3A%22-%22%2C%22scode%22%3A%22-%22%2C%22keyword%22%3A%22-%22%2C%22ca_keywordid%22%3A%22-%22%2C%22display_finance_flag%22%3A%22-%22%2C%22platform%22%3A%221%22%2C%22version%22%3A1%2C%22client_ab%22%3A%22-%22%2C%22guid%22%3A%223bfbeb56-9f3e-4d0d-cbe3-260b14154476%22%2C%22ca_city%22%3A%22gy%22%2C%22sessionid%22%3A%22641c79ee-5210-4be7-892b-466faf206b39%22%7D; preTime=%7B%22last%22%3A1595391260%2C%22this%22%3A1595389721%2C%22pre%22%3A1595389721%7D; Hm_lpvt_936a6d5df3f3d309bda39e92da3dd52f=1595391261'}
response =requests.get(url,headers=headers).content.decode()
print(response)

这样我们就可以进行下一步了
用bs4进行数据的解析

从这里我们可以看到，我们想要的数据包括在一个 ul 中 class = ‘carlist clearfix js-top’中
而每一个li data-scroll-track标签中的数据就是我们需要解析得到的数据。

soup = BeautifulSoup(response,'lxml')
li_list = soup.find('ul',{'class':'carlist clearfix js-top'}).find_all('li')
print(li_list)

通过用bs4对数据进行解析，我们得到如下数据：

从上面解析出来的数据中我们就可以得到

有下面这个从上面提取出来的代码可以看出，要拿到车的名称就可以：#<h2 class="t">大众 途观L 2018款 330TSI 自动两驱风尚版</h2>
for i in li_list:title = i.find('h2',{'class':"t"}).get_text()film_title = re.sub(r' ', '', title).split(' ')print(film_title)

根据观察我们的li_list结果我们可以按照上面爬取汽车名一样爬取到他们的：年份、公里数、售价以及原价。

数据：

data = i.find('div',{'class':'t-i'}).get_text()film_data = re.sub(r'|', '', data).split('|')print(film_data)

这个数据里面包括了年份与公里数，所以我们想要这两个数据，直接从里面取出来就可以了，所以：

年份：

year = film_data[0]
print(year)

公里数：

lc = film_data[1]
print(cl)

最后我们还剩下售价与原价：
从网页页面我们可以看到：

一些数据售价与原价都有，但是有的数据只有售价，售价是所有商品都有的，所以我们先爬取它的售价。
售价包含在这个标签下：

shoujia = i.find('div',{'class':'t-price'}).find('p').get_text()print(shoujia)

这样我们就拿到了售价，但是还有原价，这里因为原价有的有，有的没有，所以我们需要做一个一场处理：

可以看到我们的原价在em标签中

    try:yuanjia = i.find('div',{'class':'t-price'}).find('em').get_text()print(yuanjia)except AttributeError:yuanjia = ''

这样我们就能将原价拿出来了

到现在我们一共爬取到一页的二手车的：型号、年份、公里数、售价、原价5个类型的数据。现在我们需要爬取更多页的数据。
所以我们就需要在最开始的代码前加上：

for page in range(1,50):print("*****************正在下载第{}页数据********************".format(page))url = 'https://www.guazi.com/gy/dazhong/o{}/#bread'.format(page)

来更改我们所要发请求的url
这里为什么这样加，是因为，当我们去网页里点击下一页，在下一页的时候，url改变的都是dazhong/o{}/#bread o后面的数字，当数字为1时就是第一页，5时就是第五页，所以我们range一下，来决定我们要爬取多少页

这样我们就可以得到更多的数据

最后就是将这些数据进行保存，方便下一步的分析：

    with open(r'瓜子二手车数据1.csv', 'a', encoding='utf-8')as f:for i in li_list:title = i.find('h2',{'class':"t"}).get_text()film_title = re.sub(r' ', '', title).split(' ')print(film_title)data = i.find('div',{'class':'t-i'}).get_text()film_data = re.sub(r'|', '', data).split('|')# print(film_data)year = film_data[0]print(year)lc = film_data[1]print(lc)shoujia = i.find('div',{'class':'t-price'}).find('p').get_text()print(shoujia)try:yuanjia = i.find('div',{'class':'t-price'}).find('em').get_text()print(yuanjia)except AttributeError:yuanjia = ''f.write('{},{},{},{},{}\n'.format(film_title,year,lc,shoujia,yuanjia))

添加一个with open将数据写入到csv文件中。

结果展示

完整代码

import requests
import lxml
from bs4 import BeautifulSoup
import parsel
import re
for page in range(1,50):print("************第{}页正在保存**********".format(page))url = 'https://www.guazi.com/gy/dazhong/o{}/#bread'.format(page)headers = {'User-Agent':'Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/78.0.3904.70 Safari/537.36','Cookie':'uuid=3bfbeb56-9f3e-4d0d-cbe3-260b14154476; cityDomain=gy; ganji_uuid=2943330065231720816366; lg=1; antipas=z6683504k93003WPO325643r3; clueSourceCode=%2A%2300; user_city_id=36; sessionid=c1cf6d12-f864-40c1-be6a-689466580011; close_finance_popup=2020-07-27; cainfo=%7B%22ca_a%22%3A%22-%22%2C%22ca_b%22%3A%22-%22%2C%22ca_s%22%3A%22seo_baidu%22%2C%22ca_n%22%3A%22default%22%2C%22ca_medium%22%3A%22-%22%2C%22ca_term%22%3A%22-%22%2C%22ca_content%22%3A%22-%22%2C%22ca_campaign%22%3A%22-%22%2C%22ca_kw%22%3A%22-%22%2C%22ca_i%22%3A%22-%22%2C%22scode%22%3A%22-%22%2C%22keyword%22%3A%22-%22%2C%22ca_keywordid%22%3A%22-%22%2C%22display_finance_flag%22%3A%22-%22%2C%22platform%22%3A%221%22%2C%22version%22%3A1%2C%22client_ab%22%3A%22-%22%2C%22guid%22%3A%223bfbeb56-9f3e-4d0d-cbe3-260b14154476%22%2C%22ca_city%22%3A%22gy%22%2C%22sessionid%22%3A%22c1cf6d12-f864-40c1-be6a-689466580011%22%7D; Hm_lvt_936a6d5df3f3d309bda39e92da3dd52f=1595389723,1595834426,1595834461; guazitrackersessioncadata=%7B%22ca_kw%22%3A%22-%22%7D; preTime=%7B%22last%22%3A1595835397%2C%22this%22%3A1595389721%2C%22pre%22%3A1595389721%7D; Hm_lpvt_936a6d5df3f3d309bda39e92da3dd52f=1595835398'}response =requests.get(url,headers=headers).content.decode()# print(response)soup = BeautifulSoup(response,'lxml')li_list = soup.find('ul',{'class':'carlist clearfix js-top'}).find_all('li')# print(li_list)with open(r'python爬虫实例——某二手车数据.csv', 'a', encoding='utf-8')as f:for i in li_list:title = i.find('h2',{'class':"t"}).get_text()film_title = re.sub(r' ', '', title).split(' ')print(film_title)data = i.find('div',{'class':'t-i'}).get_text()film_data = re.sub(r'|', '', data).split('|')# print(film_data)year = film_data[0]print(year)lc = film_data[1]print(lc)shoujia = i.find('div',{'class':'t-price'}).find('p').get_text()print(shoujia)try:yuanjia = i.find('div',{'class':'t-price'}).find('em').get_text()print(yuanjia)except AttributeError:yuanjia = ''f.write('{},{},{},{},{}\n'.format(film_title,year,lc,shoujia,yuanjia))

下篇文章我们来分析我们说爬取的数据

特别注意！！！！！！！！！！！！

             ## 每一次运行程序都要用登录网站成功的cookie！！！！！！！！不是会与不加cookie一样拿不到网页的数据