基于requests和lxml库爬取优信二手车

工具：
requests库和lxml库
具体代码：

# _*_ coding:utf-8 _*_
import requests
import re
import time
import MySQLdb
import MySQLdb.cursors
from lxml import etree
from urllib import parse
from random import randintfiter_url = []       #用于过滤已经爬取的url
url_list = ['https://www.xin.com/guangzhou/i1/']
start_url = 'https://www.xin.com/guangzhou/i1/'
header = {'User-Agent':'Mozilla/5.0 (Windows NT 10.0; Win64; x64; rv:63.0) Gecko/20100101 Firefox/63.0',
}def start_request(url,header):response = requests.get(url=url,headers=header)response.enconding = 'utf-8'text = response.textreturn textdef get_next_url(url):for i in url:if i in fiter_url:passelse:fiter_url.append(i)time.sleep(randint(1,3))content = start_request(i,header)return contentdef get_car_info_url(text):html = etree.HTML(text,etree.HTMLParser())next_url = html.xpath('//div[contains(@class,"con-page")]//a/@href')for t in next_url:next_url = parse.urljoin(start_url,t)url_list.append(next_url)next_nodes = html.xpath('//div[@class="across"]//a[@class="aimg"]/@href')for next_node in next_nodes:detail_url = parse.urljoin(start_url,next_node)yield detail_urldef get_car_info(detail_url):for i in detail_url:time.sleep(randint(1,3))text = start_request(i,header=header)html = etree.HTML(text,etree.HTMLParser())data = {}data['title'] = html.xpath('//span[@class="cd_m_h_tit"]/text()')[0]data['register_time'] = html.xpath('//span[@class="cd_m_desc_key"][1]/text()')[0]data['miles'] = html.xpath('//a[@class="cd_m_desc_val"]/text()')[0]data['city'] = '广州'data['oil_mount'] = html.xpath('//li[contains(@class,"cd_m_desc_line")][4]/span[@class="cd_m_desc_val"]/text()')[0]data['price'] = html.xpath('//span[@class="cd_m_info_jg"]/b/text()')[0]yield datadef data_clean(datas):for data in datas:data['miles'] = data['miles'].strip()data['title'] = data['title'].strip()speed_pattern = r'.*(手动|自动).*'m = re.match(speed_pattern,data['title'])data['speed_box'] = m.group(1)title_pattern = r'(.*\d+款).*'t = re.match(title_pattern,data['title'])data['title'] = t.group(1)data['register_time'] = data['register_time'].replace('上牌','')yield datadef insert_into_sql(data):conn = MySQLdb.connect('localhost','root','9901914846','guazi',charset='utf8',use_unicode=True)cursor = conn.cursor()insert_sql = """insert into youxin_data(title,register_time,miles,city,oil_mount,speed_box,price)VALUES(%s,%s,%s,%s,%s,%s,%s)"""params = (data['title'],data['register_time'],data['miles'],data['city'],data['oil_mount'],data['speed_box'],data['price'])cursor.execute(insert_sql,params)conn.commit()if __name__ == '__main__':while fiter_url != url_list:text = get_next_url(url_list)detail_url = get_car_info_url(text)datas = get_car_info(detail_url)data = data_clean(datas)for i in data:if i:insert_into_sql(i)print('插入成功')else:print('插入失败')

100行左右代码简单地实现爬取优信二手车车辆的具体信息。当然信息不止那么少，小编随便选了几个。代码写得不好的地方，请各位大神指教。

基于requests和lxml库爬取优信二手车相关推荐

基于requests库和lxml库爬取优信二手车
工具:lxml库和requests库 # _*_ coding:utf-8 _*_ import requests import re import time import MySQLdb impor ...
python 循环定时器 timer显示数据_【Python】多线程、定时循环爬取优信二手车信息...
爬虫爬取优信二手车:循环遍历每页,获取相应的有价值字段信息,这里不详细阐释了. 多线程 Python中,使用concurrent.futures模块下的ThreadPoolExecutor类来实现线 ...
python爬虫---爬取优信二手车
import requests import re import redis from lxml import etree import pymysql #获取城市数据 class City():de ...
scrapy爬取优信二手车
由于首页车辆分类是动态数据(解析动态页面),因此在页面源码找不到需要的数据,可以通过以下步骤找 car.py(爬虫文件) import scrapy import json from ..items ...
urllib库爬取拍信创意图片(post请求)json传参
urllib库爬取拍信创意图片解决urllib库遇到Request payload传参问题分析网页: 找到接口: 发现图片数据都是以json格式存储在这个接口里我们在来看接口所需要的data,这 ...
requests库和lxml库爬取彼岸图网的图片
```python import requests,os from lxml import etree# 创建对应的存储的文件夹 if(not os.path.exists('./彼岸图网图片')): ...
python爬取小说网站资源_利用python的requests和BeautifulSoup库爬取小说网站内容
1. 什么是Requests?html Requests是用Python语言编写的,基于urllib3来改写的,采用Apache2 Licensed 来源协议的HTTP库.python 它比urlli ...
使用Python requests和BeautifulSoup库爬取去哪儿网
功能说明:爬取去哪儿网城市下面若干条景点详细信息并将数据导入Excel表(使用xlwt库) 爬取去哪儿网的教程参考自 https://blog.csdn.net/gscsd_t/article/det ...
基于requests百度图库图片爬取
这几天尝试图片的爬取,根据一篇博客,有了一些收获,来总结一下,此次爬取的对象是百度图库.百度爬虫分为就3部分: (1)获得批量图片的url: (2)通过每张图片的url获取图像内容: (3)保存图片. ...

基于requests和lxml库爬取优信二手车

基于requests和lxml库爬取优信二手车相关推荐

最新文章

热门文章