python之requests爬虫Boss数据

需要用到的库：reqeusts、lxml

没有的可以用直接下载

pip install requests

pip install lxm

这里以python岗位，地点北京为例

爬取的数据就是岗位名称、薪资、地点

首先导入需要用到的模块

import requests

from lxml import etree

岗位可以通过input提前输入好，传参给url

job = input('输入职位')

将需要访问的url赋给一个变量

url = 'https://www.zhipin.com/job_detail/?query=%s&city=101010100&industry=&position='%job

query=%s(%s是一个占位)在引号后面的%job就是占的值

访问这个页面需要加一个头部(headers)降低被识别爬虫的概率

在当前页面按f12点击Network，如果没有东西的话可以刷新一下页面

这里只用到两个参数，一个user-agent，一个cookie

headers = {

'user-agent': 'Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/75.0.3770.100 Safari/537.36',

'cookie': '_uab_collina=157853739340991408682799; Hm_lvt_194df3105ad7148dcf2b98a91b5e727a=1578537393,1578554153; __c=1578554153; __g=-; Hm_lpvt_194df3105ad7148dcf2b98a91b5e727a=1578554168; __zp_stoken__=d0e7eq77rh2ql3R%2F5VwP4mPjHKu%2BjYVQMbIFSPnpEWipSXfKaWf%2FM%2FxBRat22vE%2FR4PdiD%2BDhDiSNaW%2FTjVMpYOEMTTUmxg7WSFqYfpdWi5SSIMEcHuwoKbmd%2B6tlv5ONmSF; __l=l=%2Fwww.zhipin.com%2Fjob_detail%2F%3Fquery%3D%25E4%25BA%25BA%25E5%25B7%25A5%25E6%2599%25BA%25E8%2583%25BD%26city%3D101010100%26industry%3D%26position%3D&r=&friend_source=0&friend_source=0; __a=32343010.1578537387.1578537387.1578554153.9.2.2.9'

}

请求数据并且返回值

res = requests.get(url,headers=headers).text

利用etree中的HTML解析数据

html = etree.HTML(res)

#岗位名称

job_name = html.xpath('//*[@id="main"]/div/div[2]/ul/li/div/div[1]/h3/a/div[1]/text()')

#薪资

salary = html.xpath('//*[@id="main"]//ul/li//h3/a/span/text()')

#地点、工作经验、学历

site = html.xpath('//*[@id="main"]/div/div[2]/ul/li/div/div[1]/p')

print('工作岗位:',job)

print('薪资:',salary)

print('地点：',site)

输出的数据如下

整体代码

import requests

from lxml import etree

import json

job = input('输入职位')

url = 'https://www.zhipin.com/job_detail/?query=%s&city=101010100&industry=&position='%job

headers = {

'user-agent': 'Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/75.0.3770.100 Safari/537.36',

'cookie': '_uab_collina=157853739340991408682799; Hm_lvt_194df3105ad7148dcf2b98a91b5e727a=1578537393,1578554153; __c=1578554153; __g=-; __l=l=%2Fwww.zhipin.com%2Fjob_detail%2F%3Fquery%3D%25E4%25BA%25BA%25E5%25B7%25A5%25E6%2599%25BA%25E8%2583%25BD%26city%3D101010100%26industry%3D%26position%3D&r=&friend_source=0&friend_source=0; lastCity=101010100; Hm_lpvt_194df3105ad7148dcf2b98a91b5e727a=1578556532; __zp_stoken__=d0e7eq77rh2ql3R%2F5VwP4mPjHOT%2BY0u%2F2GMG6hriOPZlx6iA6NPb%2FycP1M1RRJxkLq%2FdiD%2BDhDiSNaW%2FTjVMpYOEMScFTSjVVO31G%2B8%2Bwf%2Bxs7gEcHuwoKbmd%2B6tlv5ONmSF; __a=32343010.1578537387.1578537387.1578554153.29.2.22.29'

}

res = requests.get(url,headers=headers).text

html = etree.HTML(res)

job_name = html.xpath('//*[@id="main"]/div/div[2]/ul/li/div/div[1]/h3/a/div[1]/text()')

salary = html.xpath('//*[@id="main"]//ul/li//h3/a/span/text()')

site = html.xpath('//*[@id="main"]/div/div[2]/ul/li/div/div[1]/p/text()')

print('工作岗位:',job_name)

print('薪资:',salary)

print('地点：',site)

最后需要注意网站中的cookie是实时更新，如果数据没有出来再去网页中查看cookie值