爬虫之websocket数据爬取

收集了大家的问题。我又重新写了一篇websocket的代码，并添加了注释，在文章最后。希望可以解决大家遇到的问题～

websocket是最近开发很常用的技术之一，他可以一直保持着连接不断，但是你的页面还可以继续展示其它任务，很适用于直播时候的弹幕等。这个是我自己的基础理解，详细理解大家可以参考：https://segmentfault.com/a/1190000013149749

我觉得针对websocket的数据爬取，一定要有逆向思维，首先找到数据，然后去倒推这个流程，进而掌握数据传送的全过程。而且针对websocket数据爬取，有一个很好的框架大家可以使用


import time
import json
import pandas as pd
import requests
import websocket
from requests.adapters import HTTPAdapter
import timetry:import thread
except ImportError:import _thread as threadclass websocket_class :def __init__(self):pass#这里就是websocket爬虫的核心，发送请求，接收数据并做最后处理，def on_message(self,ws, message):passdef on_error(self,ws, error):print(error)#关闭websocket长连接def on_close(self,ws):print("关闭连接")#程序运行第一步def on_open(self,ws):def run(*args):#这里面就是写大家倒退出来页面请求第一步的代码passthread.start_new_thread(run, ())if __name__ == "__main__":header = {'Accept-Encoding': 'gzip, deflate, br','Accept-Language': 'zh-CN,zh;q=0.9,','Cache-Control': 'no-cache','Connection': 'Upgrade','Cookie': cookie,'Host': 'ws-nextbi.yushanfang.com','Origin': 'https://nextbi.yushanfang.com','Pragma': 'no-cache','Sec-WebSocket-Extensions': 'permessage-deflate; client_max_window_bits',#这个参数要进行实时修改'Sec-WebSocket-Key': 'QBn6rnK29DZL6BC6+O2TRA==','Sec-WebSocket-Version': '13','Upgrade': 'websocket','User-Agent': 'Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/68.0.3440.106 Safari/537.36'}websocket.enableTrace(True)websocket_obj = websocket_class(cookie, appId, cateId,filename)ws = websocket.WebSocketApp("wss://ws-nextbi.yushanfang.com/",on_message = websocket_obj.on_message,on_error = websocket_obj.on_error,on_close = websocket_obj.on_close,header=header)ws.on_open = websocket_obj.on_openws.run_forever()

首先大家可以看到下图，就证明此数据是websocket传输
w
大家可以看到我标红的地方有两个箭头，绿色的就是我们需要模拟的请求，红色朝下的就是请求对应的数据，那么问题来了，页面上那么多请求和接收，我们如何知道哪个请求对应哪个数据呢？

这个rid就是我们找一对请求和接受的依据。大家肯定注意到，这个rid和时间戳很类似，没错这就是一个13位的时间戳，和随机数组合而成的

randomID = str(int(time.time()*1000))+str(self.count).zfill(3)

大家可以依据我上面说的这些，结合自己的网站写爬虫代码。

#这是我的一部分请求代码randomID = str(int(time.time()*1000))+str(self.count).zfill(3)self.targetID['tagTitle'][randomID]= self.targetID['targetSecondID'][subrid]sendMessage = {"method":"/iWidget/list","headers":{"rid":randomID,"type":"PULL"},"body":{"args":{"sheetId":message['body'],"appId":self.appId}}}ws.send(json.dumps(sendMessage))

我遇到的问题：
1.爬取的时候总是有一大串的报红

我的理解是发送的请求并没有接受的数据，因为rid是根据当前的时间在不停的生成时间戳，所以会出现一些时间戳并没有匹配的数据，当人大家可以看到白色的字体就是我接受的数据。

2.无法批量拉取数据，因为websocket是长连接，批量拉取的话，如果爬取第二个数据，第一个数据的长连接总是无法关掉，导致数据重复爬取，我有进行强制关闭，依然无法批量。

#下面我放一份我自己写的一个关于websocket 抓取数据的代码。

import websocket
from requests.adapters import HTTPAdapterfrom mybaseCode import *try:import thread
except ImportError:import _thread as threadclass websocket_class :count =0sheetIds = [] #获取所有的标签IDtargetID = {}targetID['getDataOnWidgetV2'] = {}targetID['targetSecondID'] = {}targetID['tagTitle'] = {} #标签名称targetID['tagTitleTT'] = {}  # 标签名称#需要定期去修改newDF = pd.DataFrame()def __init__(self,cookie, appId, brandId,date):# 配置requestsself.s = requests.Session()self.NETWORK_STATUS = Falseself.REQUEST_TIMEOUT = False# 配置requests超时重试self.s.mount('http://', HTTPAdapter(max_retries=3))self.s.mount('https://', HTTPAdapter(max_retries=3))self.date=dateself.appId = appIdself.brandId = brandIdself.insertCookie(cookie)self.getsheetIds()self.n=0self.final=pd.DataFrame(columns=['类目','渠道','时间','销售金额-本品牌', '销售金额同比-本品牌','购买人数-本品牌','购买人数同比-本品牌','客单价-本品牌','客单价同比-本品牌','人均购买叶子类目数-本品牌', '人均购买叶子类目数同比-本品牌','新进叶子类目数-本品牌','新进叶子类目数同比-本品牌'])#这个是我封装的cooke请求def insertCookie(self,cookie):# 读取cookiecookies = []try:for line in cookie.split(';'):name, value = line.strip().split('=', 1)cookies.append({'name': name, 'value': value})except ValueError:print('cookie格式错误!')return {'errCode': -1, 'errMsg': 'cookie格式错误!'}# cookie注入到requests.Sessionfor cookie in cookies:if cookie['name'] == '_tb_token_':self.x_csrf_token = cookie['value']self.s.cookies.set(cookie['name'], cookie['value'])#这里是为了爬虫统一封装的参数def requests_method(self, url):try:result = self.s.get(url,headers={'accept-encoding': 'gzip, deflate, br','accept-language': 'zh-CN,zh;q=0.9,en-US;q=0.8,en;q=0.7,la;q=0.6','cache-control': 'no-cache','pragma':'no-cache','referer':'https://databank.tmall.com/','user-agent': 'Mozilla/5.0 (Macintosh; Intel Mac OS X 10_13_6) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/67.0.3396.99 Safari/537.36'}).textreturn resultexcept Exception as e:print(e)return False#这个自定义函数是我发现websocket请求之前有一些参数我需要通过这个请求拿到def getsheetIds(self):url='https://strategy.tmall.com/api/scapi?path=/api/v1/category/strategy/listRelation&brandId='+self.brandIdsendPackage = self.requests_method(url)res = json.loads(sendPackage)self.tagSheetIds = {}for item in res['data']:if item['children'] == None:self.tagSheetIds[item['cateId']] = item['cateFullName']else:self.tagSheetIds[item['cateId']] = item['cateFullName']for childrenId in item['children']:self.tagSheetIds[childrenId['cateId']] = childrenId['cateFullName']#这里其实有点自调函数的意思，因为所有通过ws.send()方法接收的message都会到这里，# 所以你就需要通过下面的rid取判断，此时接收的message到哪一层（或者说是哪个请求反馈的结果。def on_message(self,ws, message):print('---------------进入方法-----------------------')print(message)message = json.loads(message)subrid = message['headers']['rid']#这里我写了3层判断，大家注意我都是通过rid来判断。回顾一下我上面说的 rid是判断 请求对应的结果 唯一标符号if (subrid in self.targetID['tagTitleTT'].keys()):randomID = str(int(time.time() * 1000)) + str(self.count).zfill(2)messageTwo={"method":"/iWidget/list","headers":{"rid":randomID,"type":"PULL"},"body":{"args":{"sheetId":"11531","appId":self.appId}}}ws.send(json.dumps(messageTwo))self.targetID['tagTitle'][randomID]= ''# 这里发送第二个请求，到下面的第三层elif(subrid in self.targetID['tagTitle'].keys()):#measureInfo = {item['id']: item['measure'][0]['showName'] for item in message['body']['list'] if'measure' in item.keys() and '本品牌' in item['measure'][0]['showName']}# measureInfo={331756: '销售金额-本品牌', 331758: '销售金额同比-本品牌'}# print(measureInfo)# exit()for tag_k,tag_v in self.tagSheetIds.items():print('现在跑的是---------',tag_v)for k,v in measureInfo.items():randomID = str(int(time.time() * 1000)) + str(self.count).zfill(2)print('v----------',v)#这里就是我在组装第二次请求，大家可根据自己的情况进行修改for channel_k,channel_v in {'4':'天猫'}.items():print('现在跑的渠道是---------', channel_v)if tag_v == '全部':messageThree = {"method": "/queryDataService/queryDataOnWidget","headers": {"rid": randomID,"type": "PULL"},"body": {"args": {"referer": "strategy-brandGrowOverview","id": k,"isMock": 0,"whatIfParam": {"widgetParamList": [],"customParamList": []},"selections": [{"dimensionName": "cateId","restrictList": [{"hide": 1,"oper": "eq","value": "-999"}],"eq": [{"hide": 1,"oper": "eq","value": "-999"}],"lt": None,"gt": None,"ge": None,"le": None,"ne": None,"showText": "-999"}, {"dimensionName": "cateName","restrictList": [{"hide": 1,"oper": "eq","value": "cate-999"}],"eq": [{"hide": 1,"oper": "eq","value": "cate-999"}],"lt": None,"gt": None,"ge": None,"le": None,"ne": None,"showText": "cate-999"}, {"dimensionName": "cateLevel","restrictList": [{"hide": 1,"oper": "eq","value": "-999"}],"eq": [{"hide": 1,"oper": "eq","value": "-999"}],"lt": None,"gt": None,"ge": None,"le": None,"ne": None,"showText": "-999"}, {"dimensionName": "channelId","restrictList": [{"hide": 1,"oper": "eq","value": channel_k}],"eq": [{"hide": 1,"oper": "eq","value": channel_k}],"lt": None,"gt": None,"ge": None,"le": None,"ne": None,"showText": channel_k}, {"dimensionName": "userType","restrictList": [{"hide": 1,"oper": "eq","value": "-999"}],"eq": [{"hide": 1,"oper": "eq","value": "-999"}],"lt": None,"gt": None,"ge": None,"le": None,"ne": None,"showText": "-999"}, {"dimensionName": "bizDate","restrictList": [{"hide": 1,"oper": "eq","value": "202003"}],"eq": [{"hide": 1,"oper": "eq","value": "202003"}],"lt": None,"gt": None,"ge": None,"le": None,"ne": None,"showText": "202003"}, {"dimensionName": "env","restrictList": [{"hide": 1,"oper": "eq","value": "//strategy.tmall.com"}],"eq": [{"hide": 1,"oper": "eq","value": "//strategy.tmall.com"}],"lt": None,"gt": None,"ge": None,"le": None,"ne": None,"showText": "//strategy.tmall.com"}, {"dimensionName": "meaTypeBrandGrow","restrictList": [{"hide": 1,"oper": "eq","value": "brand_cate_new_byr_cnt","showName": "品牌新客"}],"eq": [{"hide": 1,"oper": "eq","value": "brand_cate_new_byr_cnt","showName": "品牌新客"}],"lt": None,"gt": None,"ge": None,"le": None,"ne": None,"showText": "品牌新客"}, {"dimensionName": "meaTypeBrandCate","restrictList": [{"hide": 1,"oper": "eq","value": "pay_ord_amt","showName": "销售金额"}],"eq": [{"hide": 1,"oper": "eq","value": "pay_ord_amt","showName": "销售金额"}],"lt": None,"gt": None,"ge": None,"le": None,"ne": None,"showText": "销售金额"}],"rdPathInfoList": [],"appId": "7"}}}else:messageThree ={"method": "/queryDataService/queryDataOnWidget","headers": {"rid": randomID,"type": "PULL"},"body": {"args": {"referer": "strategy-brandGrowOverview","id":k,"isMock": 0,"whatIfParam": {"widgetParamList": [],"customParamList": []},"selections": [{"dimensionName": "cateId","restrictList": [{"hide": 1,"oper": "eq","value": tag_k}],"eq": [{"hide": 1,"oper": "eq","value": tag_k}],"lt": None,"gt": None,"ge": None,"le": None,"ne": None,"showText": tag_k}, {"dimensionName": "cateName","restrictList": [{"hide": 1,"oper": "eq","value": "cate_leaf"}],"eq": [{"hide": 1,"oper": "eq","value": "cate_leaf"}],"lt": None,"gt": None,"ge": None,"le": None,"ne": None,"showText": "cate_leaf"}, {"dimensionName": "cateLevel","restrictList": [{"hide": 1,"oper": "eq","value": "2"}],"eq": [{"hide": 1,"oper": "eq","value": "2"}],"lt": None,"gt": None,"ge": None,"le": None,"ne": None,"showText": "2"}, {"dimensionName": "channelId","restrictList": [{"hide": 1,"oper": "eq","value": channel_k}],"eq": [{"hide": 1,"oper": "eq","value": channel_k}],"lt": None,"gt": None,"ge": None,"le": None,"ne": None,"showText": channel_k}, {"dimensionName": "userType","restrictList": [{"hide": 1,"oper": "eq","value": "-999"}],"eq": [{"hide": 1,"oper": "eq","value": "-999"}],"lt": None,"gt": None,"ge": None,"le": None,"ne": None,"showText": "-999"}, {"dimensionName": "bizDate","restrictList": [{"hide": 1,"oper": "eq","value": "202003"}],"eq": [{"hide": 1,"oper": "eq","value": "202003"}],"lt": None,"gt": None,"ge": None,"le": None,"ne": None,"showText": "202003"}, {"dimensionName": "env","restrictList": [{"hide": 1,"oper": "eq","value": "//strategy.tmall.com"}],"eq": [{"hide": 1,"oper": "eq","value": "//strategy.tmall.com"}],"lt": None,"gt": None,"ge": None,"le": None,"ne": None,"showText": "//strategy.tmall.com"}, {"dimensionName": "meaTypeBrandGrow","restrictList": [{"hide": 1,"oper": "eq","value": "brand_cate_new_byr_cnt","showName": "品牌新客"}],"eq": [{"hide": 1,"oper": "eq","value": "brand_cate_new_byr_cnt","showName": "品牌新客"}],"lt": None,"gt": None,"ge": None,"le": None,"ne": None,"showText": "品牌新客"}, {"dimensionName": "meaTypeBrandCate","restrictList": [{"hide": 1,"oper": "eq","value": "pay_ord_amt","showName": "销售金额"}],"eq": [{"hide": 1,"oper": "eq","value": "pay_ord_amt","showName": "销售金额"}],"lt": None,"gt": None,"ge": None,"le": None,"ne": None,"showText": "销售金额"}],"rdPathInfoList": [],"appId": self.appId}}}#这里发送第三个请求，到下面的第四self.targetID['getDataOnWidgetV2'][randomID]= v+','+tag_v+','+channel_vws.send(json.dumps(messageThree))self.count+=1time.sleep(random.randint(3,6))elif(subrid in self.targetID['getDataOnWidgetV2'].keys()):#经过层层递进，到这一层，我就可以取数填表，下面的代码是我填表的代码，大家可根据自己的情况进行更改print('--------------------进入第三个方法-----------------------------------')# print(message)print(self.targetID['getDataOnWidgetV2'][subrid])showName = [item['showName'] for item in message['body']['axises'][0]['values'] if'showName' in item.keys()]pay_ord_amt = message['body']['datas'][0]['values']pay_ord_amt_industry = message['body']['datas'][1]['values']valuesT=self.targetID['getDataOnWidgetV2'][subrid].split(',')[0]channels=self.targetID['getDataOnWidgetV2'][subrid].split(',')[2]mylisy=['','','','','','','','','','']if len(self.final.loc[(self.final['类目']==self.targetID['getDataOnWidgetV2'][subrid].split(',')[1])&(self.final['渠道'] == channels)])==0 and pay_ord_amt != [None, None, None, None, None, None, None, None, None, None, None, None]:for index,item in enumerate(showName):mylisy[0] = pay_ord_amt[index] if valuesT == '销售金额-本品牌' else ''mylisy[1] = pay_ord_amt[index] if valuesT == '销售金额同比-本品牌' else ''mylisy[2] = pay_ord_amt[index] if valuesT == '购买人数-本品牌' else ''mylisy[3] = pay_ord_amt[index] if valuesT == '购买人数同比-本品牌' else ''mylisy[4] = pay_ord_amt[index] if valuesT == '客单价-本品牌' else ''mylisy[5] = pay_ord_amt[index] if valuesT == '客单价同比-本品牌' else ''mylisy[6] = pay_ord_amt[index] if valuesT == '人均购买叶子类目数-本品牌' else ''mylisy[7] = pay_ord_amt[index] if valuesT == '人均购买叶子类目数同比-本品牌' else ''mylisy[8] = pay_ord_amt[index] if valuesT == '新进叶子类目数-本品牌' else ''mylisy[9] = pay_ord_amt[index] if valuesT == '新进叶子类目数同比-本品牌' else ''self.final.loc[self.n]=[self.targetID['getDataOnWidgetV2'][subrid].split(',')[1],channels,item,mylisy[0],mylisy[1],mylisy[2],mylisy[3],mylisy[4],mylisy[5],mylisy[6],mylisy[7],mylisy[8],mylisy[9]]self.n+=1elif len(self.final) > 0 and pay_ord_amt != [None, None, None, None, None, None, None, None, None, None, None, None]:for index, item in enumerate(showName):self.final.loc[(self.final['时间'] == item) &(self.final['渠道'] == channels)& (self.final['类目']==self.targetID['getDataOnWidgetV2'][subrid].split(',')[1]),[valuesT]]=pay_ord_amt[index]else:print('')def on_error(self,ws, error):print('error------------------:%s'%(error))def on_close(self,ws):print("关闭连接")self.final.to_excel('./_品牌老客_'+''+'_2020.xlsx')def on_open(self,ws):def run(*args):#websocket 从这里开始发出第一个请求，之后就进入到on_message 的方法里面randomID = str(int(time.time() * 1000)) + str(self.count).zfill(2)sendMessage ={"method":"/iSheet/get","headers":{"rid":randomID,"type":"PULL"},"body":{"args":{"id":"11533","appId":self.appId}}}ws.send(json.dumps(sendMessage))self.targetID['tagTitleTT'][randomID]= ''self.count +=1time.sleep(1)# run()thread.start_new_thread(run, ())if __name__ == '__main__':cookie ='cna=dPQBF57PeBICAdINZPY4DT5o; t=4bf5ddb79f22d9d36a2850d260a96343; _tb_token_=5e680719356e7; cookie2=1a5abd5978269b5c1741aa3b05ed877b; dnk=%5Cu4E0A%5Cu6D77%5Cu70B9%5Cu6B63%5Cu4E92%5Cu8054%5Cu7F51%5Cu79D1%5Cu6280; lid=%E4%B8%8A%E6%B5%B7%E7%82%B9%E6%AD%A3%E4%BA%92%E8%81%94%E7%BD%91%E7%A7%91%E6%8A%80; tracknick=%5Cu4E0A%5Cu6D77%5Cu70B9%5Cu6B63%5Cu4E92%5Cu8054%5Cu7F51%5Cu79D1%5Cu6280; lgc=%5Cu4E0A%5Cu6D77%5Cu70B9%5Cu6B63%5Cu4E92%5Cu8054%5Cu7F51%5Cu79D1%5Cu6280; login=true; __YSF_SESSION__={"baseId":"fed21222d74ba9fc","brandId":"97dafe8e6eb4147b","departmentId":"388dd60f6d0f422b","smartId":"570f63fcca6a8aa0","databankProjectId":"388dd60f6d0f422b","dataFactoryProjectId":"b02b1accf92683ba"}; _mw_us_time_=1586237833920; welcomeShownTime=1586237834069; uc1=tag=8&cookie21=UtASsssmfuQi&pas=0&lng=zh_CN&cookie14=UoTUPOaFWzXASA%3D%3D&cookie16=W5iHLLyFPlMGbLDwA%2BdvAGZqLg%3D%3D&existShop=false&cookie15=UIHiLt3xD8xYTw%3D%3D; uc3=lg2=UIHiLt3xD8xYTw%3D%3D&nk2=qiAr7C1v6U%2BpUQ0vr5j5%2BjWB&id2=UNJV26FcR9wX2A%3D%3D&vt3=F8dBxdAaVyonn%2BzVax0%3D; _l_g_=Ug%3D%3D; uc4=nk4=0%40qBqkhbtg7i8tPi8TvK%2B0wLlFQPeF%2FtOw1FmY5JQ%3D&id4=0%40UgXSr0f7sKvYnHwgJa4Y47HKEFVJ; unb=3246332809; cookie1=BYXMVOB%2F0gpSvIKO64slLGtNsh3h2mrU2YNsQG%2BLLag%3D; cookie17=UNJV26FcR9wX2A%3D%3D; _nk_=%5Cu4E0A%5Cu6D77%5Cu70B9%5Cu6B63%5Cu4E92%5Cu8054%5Cu7F51%5Cu79D1%5Cu6280; sgcookie=EP6s1pySywojoogRO5cT4; sg=%E6%8A%8090; csg=f4fdfb37; l=dBOoYs0nQconVQr-BOCZSUv_mU7OSIRAguJl5Ei9i_5Cb6T6j2_Oo6B_VF96VjWfGFTB4fyJ7oy9-etXwQHmndHJyFr3xxDc.; isg=BBoasSjbEJlV15zDlFAAP3Qxa8Y8S54lZ5HLmCSTxq1-l7rRDNvuNeDmZmMLeha9'appId = '7'       #品牌IDdate='2020'#就是人群标签的id#二级类目IDbrandId=''##############################上方的三个参数是需要修改的，AIPL对应4个不同的cookie来运行header = {'Accept-Encoding': 'gzip, deflate, br','Accept-Language': 'zh-CN,zh;q=0.9,','Cache-Control': 'no-cache','Connection': 'Upgrade','Cookie': cookie,'Host': 'ws-insight-engine.tmall.com','Origin': 'https://insight-engine.tmall.com','Pragma': 'no-cache','Sec-WebSocket-Extensions': 'permessage-deflate; client_max_window_bits','Sec-WebSocket-Key': 'chECld5skiow6Q5/44bBbw==','Sec-WebSocket-Version': '13','Upgrade': 'websocket','User-Agent': 'Mozilla/5.0 (Macintosh; Intel Mac OS X 10_14_0) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/78.0.3904.108 Safari/537.36'}#这里是python websocket 固定模版，大家可以根据我贴的链接 试一试websocket.enableTrace(True)websocket_obj = websocket_class(cookie, appId, brandId,date)ws = websocket.WebSocketApp("wss://ws-insight-engine.tmall.com/",on_message = websocket_obj.on_message,on_error = websocket_obj.on_error,on_close = websocket_obj.on_close,header=header)ws.on_open = websocket_obj.on_openws.run_forever()ws.close()print('跑完一个')time.sleep(5)

爬虫之websocket数据爬取相关推荐

python财务报表预测股票价格_机器学习股票价格预测从爬虫到预测-数据爬取部分...
声明:本文已授权公众号「AI极客研修站」独家发布前言各位朋友大家好,小之今天又来给大家带来一些干货了.上篇文章机器学习股票价格预测初级实战是我在刚接触量化交易那会,因为苦于找不到数据源,所以找的一 ...
python爬虫从入门到实战笔记——第一章爬虫原理和数据爬取
爬虫原理和数据抓取 1.1 通用爬虫和聚焦爬虫通用爬虫聚焦爬虫 1.2 HTTP和HTTPS HTTP的请求与响应浏览器发送HTTP请求的过程: 客户端HTTP请求请求方法常用的请求报头服 ...
python爬虫的一次尝试——华北电力大学图书馆读者荐购系统：基于python爬虫的web数据爬取
华北电力大学图书馆读者荐购系统数据爬取前言本章工具网页分析 1.荐购数据 2.书目具体信息代码部分 1. 荐购数据爬取 2. 完整书目信息爬取前言本学期数据仓库与数据挖掘课程大作业是编程实 ...
mysql 轨迹数据存储_python爬虫26 | 把数据爬取下来之后就存储到你的MySQL数据库...
小帅b说过在这几篇中会着重说说将爬取下来的数据进行存储上次我们说了一种 csv 的存储方式这次主要来说说怎么将爬取下来的数据保存到 MySQL 数据库接下来就是学习python的正确姿势真 ...
python爬虫基础Ⅱ——Ajax数据爬取、带参请求：QQ音乐歌单、QQ音乐评论
文章目录基础爬虫部分Ⅱ Ajax技术 json 1. Network 2. XHR怎么请求? 3. 什么是json? 4. json数据如何解析? 带参数请求 1. 复习 2. params 3. ...
爬虫：Ajax数据爬取
目录 1.什么是Ajax 1.1 实例的引入 1.2 基本原理 2.Ajax分析方法 1.查看请求 2.过滤请求 3.Ajax结果提取 1.分析请求 2.分析响应 3.例子我们在用 requests ...
爬虫案例—京东数据爬取、数据处理及数据可视化（效果+代码）
一.数据获取使用PyCharm(引用requests库.lxml库.json库.time库.openpyxl库和pymysql库)爬取京东网页相关数据(品牌.标题.价格.店铺等) 数据展示(片段): ...
【Python爬虫】东方财富数据爬取
0.背景由于工作需要,最近在研究如何从东方财富网页爬取股东增持数据. 网页:http://data.eastmoney.com/executive/gdzjc-jzc.html 1.分析网页请求打 ...
Python爬虫之微信数据爬取（十三）
原创不易,转载前请注明博主的链接地址:Blessy_Zhu https://blog.csdn.net/weixin_42555080 本次代码的环境: 运行平台: Windows Python版本: ...

爬虫之websocket数据爬取

爬虫之websocket数据爬取相关推荐

最新文章

热门文章