python爬取拉钩网招聘信息

拉钩网网址为：https://www.lagou.com/

点击F12进入控制台观察结构，发现所有的招聘内容都在此json文件中：

注意headers中的请求url以及请求方法：

还有表单数据：

获取以上信息后，基本就可以开始爬取工作，注意，拉钩网有反爬机制，所以需要使用cookie，referer，user-agent模拟浏览器登录。

代码如下：

#导入模块
import requests
from bs4 import BeautifulSoup
from urllib.parse import quote#键入信息
keyword = input("关键词：")
city = input("所在城市：")
#将所在城市字符串转换成url编码
city = quote(city)headers = {'Cookie':'_ga=GA1.2.1209754414.1514967030; user_trace_token=20180103161031-90df0df0-f05d-11e7-9fc4-5254005c3644; LGUID=20180103161031-90df13b0-f05d-11e7-9fc4-5254005c3644; _gid=GA1.2.1398638690.1528077740; index_location_city=%E6%88%90%E9%83%BD; WEBTJ-ID=20180604211403-163caeee34932f-0d26f742560af7-3c60460e-1049088-163caeee34a74b; _gat=1; PRE_HOST=www.baidu.com; LGSID=20180604211405-287e8893-67f9-11e8-9199-525400f775ce; PRE_UTM=m_cf_cpc_baidu_pc; PRE_SITE=https%3A%2F%2Fwww.baidu.com%2Fbaidu.php%3Fsc.K000000fJeHuq9k182ORUWSBOwQf0uubLYJOnccqK-6lsOf9B--xNbB1V0Oak5wAYokuFvNP9W5EWMgVVbG7h4DURdIbdtIKzQccpCTHJe_BvkYwDT-P7rrahydjnpGo9b-DSOk6Sf9CVzYSzYH_KJs7FQ2sTKX7lyFxD_yEKva762AyN6.DD_NR2Ar5Od663rj6tJQrGvKD7ZZKNfYYmcgpIQC8xxKfYt_U_DY2yP5Qjo4mTT5QX1BsT8rZoG4XL6mEukmryZZjzsLTJplePXO-zIr4PXE-sSxH9vX8ZuEsSXOjEzmxUEsSxW9qx-9LdoDkbLyNSPhHWzdvT85R_nYQAHWEotN.U1Yk0ZDqs2v4_tL30A7bTgbqs2v4_tL30A7bTgfqn6KspynqnfKY5TaV8U5PS0KGUHYznjf0u1dsTLwz0ZNG5yF9pywdUAY0TA-b5Hc30APGujYznWm0UgfqnH0krNtknjDLg1DsnWPxn10kPNt1PW0k0AVG5H00TMfqnWDL0ANGujY0mhbqnW0Y0AdW5HDsnj7xP1nsnHRYrjcYg17xnH0zg100TgKGujYs0Z7Wpyfqn0KzuLw9u1Ys0A7B5HKxn0K-ThTqn0KsTjYknjf1njRvrHbv0A4vTjYsQW0snj0snj0s0AdYTjYs0AwbUL0qn0KzpWYs0Aw-IWdsmsKhIjYs0ZKC5H00ULnqn0KBI1Ykn0K8IjYs0ZPl5fKYIgnqn1mvPWb1nHb3PW0YnjTvP1msP0Kzug7Y5HDdnW6knH6sn1TvrjR0Tv-b5H-buWb3Pjubnj0snAm3Pj00mLPV5HKKP1uDrDRYwWfdwDDYfWf0mynqnfKsUWYs0Z7VIjYs0Z7VT1Ys0ZGY5H00UyPxuMFEUHYsg1Kxn7tsg100uA78IyF-gLK_my4GuZnqn7tsg1Kxn1D3PWbkg100TA7Ygvu_myTqn0Kbmv-b5Hcvrjf1PHfdP6K-IA-b5iYk0A71TAPW5H00IgKGUhPW5H00Tydh5HDv0AuWIgfqn0KhXh6qn0Khmgfqn0KlTAkdT1Ys0A7buhk9u1Yk0Akhm1Ys0APzm1Ydnj01n0%26ck%3D3433.2.110.206.561.239.621.215%26shh%3Dwww.baidu.com%26sht%3D25017023_10_pg%26us%3D1.0.2.0.0.0.0%26ie%3Dutf-8%26f%3D8%26tn%3D25017023_10_pg%26wd%3D%25E6%258B%2589%25E9%2592%25A9%25E7%25BD%2591%26oq%3D%25E6%258B%2589%25E9%2592%25A9%25E7%25BD%2591%26rqlang%3Dcn%26lm%3D-1%26ssl_s%3D1%26ssl_c%3Dssl1_163caeed355%26bc%3D110101; PRE_LAND=https%3A%2F%2Fwww.lagou.com%2Flp%2Fhtml%2Fcommon.html%3Futm_source%3Dm_cf_cpc_baidu_pc%26m_kw%3Dbaidu_cpc_cd_e110f9_265e1f_%25E6%258B%2589%25E9%2592%25A9%25E7%25BD%2591; JSESSIONID=ABAAABAAAGFABEF3C6E46C38A26E7FFF00985171CC476C0; Hm_lvt_4233e74dff0ae5bd0a3d81c6ccf756e6=1528077739,1528118044,1528118048,1528118062; TG-TRACK-CODE=index_search; Hm_lpvt_4233e74dff0ae5bd0a3d81c6ccf756e6=1528118071; LGRID=20180604211429-368ac179-67f9-11e8-9199-525400f775ce; SEARCH_ID=1976fcf584114b59811d845ae44421b1','Referer':'https://www.lagou.com/jobs/list_python?city=%s&cl=false&fromSearch=true&labelWords=&suginput=' % (city),'User-Agent':'Mozilla/5.0 (Windows NT 10.0; WOW64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/63.0.3239.108 Safari/537.36'
}demo_URL = 'https://www.lagou.com/jobs/list_{0}?city={1}&cl=false&fromSearch=true&labelWords=&suginput='
URL = demo_URL.format(keyword,city)
HTML = requests.get(URL,headers=headers)
soup = BeautifulSoup(HTML.content,'html.parser')
page_total = int(soup.select('.page-number')[0].text.strip().replace(' ','').replace('\n','').split('/')[1])
#输出该职业信息总共有多少页
print(page_total)#遍历所有页
for page_number in range(1,(page_total+1)):data = {'first': 'true','pn': page_number,'kd': keyword}demo_url = 'https://www.lagou.com/jobs/positionAjax.json?city={}&needAddtionalResult=false'url = demo_url.format(city)html = requests.post(url,headers=headers,data=data)result = html.json()info = result['content']['positionResult']['result']for i in info:#输出职业名，工作时间，文凭，薪资，公司简称，公司全称，所在城市print(i['positionName'],i['workYear'],i['education'],i['salary'],i['companyShortName'],i['companyFullName'],i['city'])

效果展示：

首先输入关键词：

在输入所在城市：

点击enter，爬取信息展示（信息量过多，只展示了一部分信息）：

python爬取拉钩网招聘信息相关推荐

python 爬取拉钩网数据
python 爬取拉钩网数据完整代码下载:https://github.com/tanjunchen/SpiderProject/blob/master/lagou/LaGouSpider.py # ...
Python 爬取拉勾招聘信息
Python 爬取拉勾招聘信息故事背景最近有个好哥们啊浪迫于家里工资太低,准备从北方老家那边来深圳这边找工作,啊浪是学平面设计的知道我在深圳这边于是向我打听深圳这边平面设计薪资水平,当时我有点懵逼 ...
python 爬取拉钩招聘数据
上一篇介绍在linux 搭建jupter lab,本文将介绍python数据接口的爬取以及提取建模分析的数据导入依赖的包 import requests import time from urlli ...
python爬取实习僧招聘信息字体反爬
参考博客:http://www.cnblogs.com/eastonliu/p/9925652.html 实习僧招聘的网站采用了字体反爬,在页面上显示正常,查看源码关键信息乱码,如下图所示: 查看网页 ...
scrapy爬虫实践之抓取拉钩网招聘信息（4）
拉勾的302搞的我不心力憔悴,几乎失去了动力继续再研究拉勾爬虫-实际上,这种无力感很大程度上来源于知识结构的匮乏(尤其是基础方面)和毫无进展带来的挫败感. 于是乎去读基础教程<learning ...
scrapy爬虫实践之抓取拉钩网招聘信息（2）
今天遇到了一个百思不得其解的问题.我用xpath获取目标网页的divs,理论上来说,应该是把这个div下的所有div存进了列表里,但是语句却是这样写的 divs = response.xpath('/ ...
ruby 爬虫爬取拉钩网职位信息，产生词云报告
思路:1.获取拉勾网搜索到职位的页数 2.调用接口获取职位id 3.根据职位id访问页面,匹配出关键字 url访问采用unirest,由于拉钩反爬虫,短时间内频繁访问会被限制访问,所以没有采用多线程, ...
python爬取智联招聘信息
import random import re from time import sleep import requests from tqdm import tqdm import user_age ...
用request模块爬取拉钩招聘信息
from urllib import request from urllib import parse# url = "https://www.lagou.com/jobs/list_pyt ...
python爬取拉勾网_python爬虫—爬取拉钩网
本人自学python,小试牛刀,爬取广州片区拉钩网招聘信息.仅用于学习参考文章:https://blog.csdn.net/SvJr6gGCzUJ96OyUo/article/details/805 ...

python爬取拉钩网招聘信息

python爬取拉钩网招聘信息相关推荐

最新文章

热门文章