fiddler移动端抓取

夜神模拟器配置

fiddler设置

选择这个选项

打开这个浏览器

直接点击继续

提示有证书问题，我们来安装证书，点击继续

输入我们的ip地址

总体过程

代码爬取豆果美食

所要携带的信息


header = {"client": "4","version": "6922.2","device": "MI 6","sdk": "19,4.4.2","imei": "863254010448503","channel": "qqkp","mac":"44:85:00:5E:5B:28","resolution": "720*1280","dpi": "1.5","android-id":"4485005e5b281516","pseudo-id":"05e5b28151644850","brand": "Xiaomi","scale": "1.5","timezone": "28800","language": "zh","cns": "3","carrier": "CMCC","imsi":"460074485009491","user-agent": "Mozilla/5.0 (Linux; Android 4.4.2; MI 6  Build/NMF26X) AppleWebKit/537.36 (KHTML, like Gecko) Version/4.0 Chrome/30.0.0.0 Mobile Safari/537.36","reach": "1","newbie": "1","lon":"116.568176","lat":"26.997867","cid":"361000","Content-Type": "application/x-www-form-urlencoded; charset=utf-8","Accept-Encoding": "gzip, deflate","Connection": "Keep-Alive","Cookie":"duid=57158696","Host": "api.douguo.net","Content-Length":"68",
}data = {"client": "4","_session": "1537295931652863254010448503","v": "1503650468","_vs": "2305",
}data_2 = {"client": "4","_session": "1537295931652863254010448503","keyword": item['name'],"order": "3","_vs": "400",
}detail_data = {"client": "4",# "_session": "1537295931652863254010448503","author_id": "0","_vs": "2803","_ext": '{"query":{"id":' + str(caipu_info['shicai_id']) + ',"kw":' + caipu_info['shicai'] + ',"idx":"4","src":"2803","type":"13"}}',
}

爬取代码

# @Time    : 2021/3/24 20:46
# @Author  : Jerry
# @File    : 豆果美食.py
# @Software: PyCharmimport requests
import json
from multiprocessing import Queue
from handle_mongo import mongo_infoqueue_list = Queue()def handel_request(url,data):#移动端请求头的信息header = {"client": "4","version": "6922.2","device": "MI 6","sdk": "19,4.4.2","imei": "863254010448503","channel": "qqkp","mac": "44:85:00:5E:5B:28","resolution": "720*1280","dpi": "1.5",# "android-id": "4485005e5b281516",# "pseudo-id": "05e5b28151644850","brand": "Xiaomi","scale": "1.5","timezone": "28800","language": "zh","cns": "3","carrier": "CMCC",# "imsi": "460074485009491","user-agent": "Mozilla/5.0 (Linux; Android 4.4.2; MI 6  Build/NMF26X) AppleWebKit/537.36 (KHTML, like Gecko) Version/4.0 Chrome/30.0.0.0 Mobile Safari/537.36","reach": "1","newbie": "1","lon": "116.568176","lat": "26.997867","cid": "361000","Content-Type": "application/x-www-form-urlencoded; charset=utf-8","Accept-Encoding": "gzip, deflate","Connection": "Keep-Alive",# "Cookie": "duid=57158696","Host": "api.douguo.net",# "Content-Length": "68",}response = requests.post(url=url,headers=header,data=data)return responsedef handle_index():url = 'http://api.douguo.net/recipe/flatcatalogs'data = {"client": "4",# "_session": "1537295931652863254010448503",# "v": "1503650468","_vs": "2305",}response = handel_request(url=url,data=data)index_response_dict = json.loads(response.text) # 数据转换for index_item in index_response_dict['result']['cs']:for index_item_1 in index_item['cs']:for item in index_item_1['cs']:data_2 = {"client": "4",# "_session": "1537295931652863254010448503","keyword": item['name'],"order": "3","_vs": "400",}queue_list.put(data_2)def handle_caipu_list(data):print('当前处理的食材:',data['keyword'])caipu_list_url = 'http://api.douguo.net/recipe/v2/search/0/20'caipu_list_response = handel_request(url=caipu_list_url, data=data)caipu_list_response_dict = json.loads(caipu_list_response.text)for item in caipu_list_response_dict['result']['list']:caipu_info = {}caipu_info['shicai'] = data['keyword']if item['type'] == 13:caipu_info['user_name'] = item['r']['an']caipu_info['shicai_id'] = item['r']['id']caipu_info['describe'] = item['r']['cookstory']caipu_info['shicai_name'] = item['r']['n']caipu_info['zuoliao_list'] = item['r']['major']detail_url = 'http://api.douguo.net/recipe/detail/'+str(caipu_info['shicai_id'])detail_data = {"client": "4",# "_session": "1537295931652863254010448503","author_id": "0","_vs": "2803","_ext": '{"query":{"id":' + str(caipu_info['shicai_id']) + ',"kw":' + caipu_info['shicai'] + ',"idx":"4","src":"2803","type":"13"}}',}detail_response = handel_request(url=detail_url,data=detail_data)detail_response_dict = json.loads(detail_response.text)caipu_info['tips'] = detail_response_dict['result']['recipe']['tips']caipu_info['cook_step'] = detail_response_dict['result']['recipe']['cookstep']print('当前入库的菜谱是:',caipu_info['shicai_name'])mongo_info.insert_item(caipu_info)else:continueprint(item)handle_index()
handle_caipu_list(queue_list.get())

存入mongo数据库

# @Time    : 2021/3/24 21:47
# @Author  : Jerry
# @File    : handle_mongo.py
# @Software: PyCharmimport pymongo
from pymongo.collection import Collectionclass Connect_mongo():def __init__(self):self.client = pymongo.MongoClient(host='127.0.0.1',port=27017)# 定义数据库的名字self.db_data = self.client['douguo']def insert_item(self,item):# 定义表名db_collection = Collection(self.db_data,'douguo_item')db_collection.insert(item)mongo_info = Connect_mongo()

fiddler移动端抓取相关推荐

fiddler移动端抓包使用教程（详解）
目录:导读前言抓包什么是抓包哪些场景下需要抓包 Fiddler Fiddler抓包原理安装 Fiddler移动端抓包第一步:允许远程计算机连接第二步,设置手机网络代理第三步,允许捕获H ...
【Fiddler移动端抓包】~抓包不是偷窥，Fiddler教你看透移动应用背后的秘密~
目录引言抓包什么是抓包哪些场景下需要抓包 Fiddler Fiddler抓包原理安装 Fiddler移动端抓包第一步:允许远程计算机连接第二步,设置手机网络代理第三步,允许捕获HTTP ...
FIddler之Fiddler移动端抓包
前言笔者今天的这篇文章呢,想使用通俗易懂的话语,让大家明白以下内容: 什么是抓包哪些场景需要用到抓包Fiddler抓包的原理怎样使用Fiddler进行移动端抓包一.抓包包 (Packet) 是T ...
Bus Hound 工具抓取串口数据（PC端抓取USB转串口数据）
测试环境: PC端 USB转串口链接终端板卡串口目标:抓取通信过程中的通信数据工具介绍: Bus Hound是是由美国perisoft公司研制的一款超级软件总线协议分析器,它是一种专用于PC机各 ...
fiddler 升级后抓取https流量各种失败后一定不要忘记这一招
fiddler抓取https请求,官网有配置教程,http://docs.telerik.com/fiddler/Configure-Fiddler/Tasks/DecryptHTTPS 一般情况下, ...
fiddler移动端抓包wifi不可上网（配置都正确，防火墙已关闭）
记录一下这个困扰我一天的问题() 公司有两个wifi可以连接,一个是XX,一个是XX_5G 我的问题:按照公司给的使用教程里描述的,移动端抓包必须电脑和手机连接同一个网络,所以我严格按照教程走,电脑连 ...
Fiddler显示只抓取特定网站或APP
可参考: http://t.istester.com:3000/istester/21Day/src/master/Fiddler/V1.0/21Day_Fiddler_07.md
fiddler不能抓取浏览器数据_抓包软件 Fiddler 了解一下？
学会如何抓包,是爬虫的必备技能,甚至可以说,不会抓包就等同于不会爬虫. 那我们怎样抓包呢?如果直接抓取浏览器上的内容,可以直接使用开发者工具进行抓包,但有个局限,只能抓浏览器的,功能也没有多少.还可以 ...
fiddler不能抓取浏览器数据_手机连接fiddler后，浏览器无法打开网页或者fiddler抓取不到手机应用相关数据的情况...
关于手机如何连接fiddler,网上有很多教程,我暂时就不写了今天在使用fiddler的过程中,发现fiddler突然无法抓取移动端应用的数据包,再三确认连接无误.因此就开始了解决之旅起因是安卓手 ...

fiddler移动端抓取

夜神模拟器配置

fiddler设置

总体过程

代码爬取豆果美食

fiddler移动端抓取相关推荐

最新文章

热门文章