python_爬虫_七麦网

本文用于学习交流使用，如有侵权，联系删除

1 爬取需求

1.1 七麦网简介

七麦网（https://www.qimai.cn/），该平台支持提供iOS、Android应用市场、微信、小程序等数据查询，是同时打通App数据、微信公众号数据、小程序数据的数据分析平台。

1.2 爬取需求

根据关键词文件，在检索栏内输入检索关键词。以滴滴出行为例，获得如下界面，在这里点击第一个获得相应的appID（554499054），在此基础上获得更详细的信息。

2 url构造

2.1 analysis加密

相信来看的各位都是因为七麦网的analysis的加密，参考了一系列的文章解决了这个问题，由于侵权问题，就不放破解的js代码了。下面将参考用的博客放给大家

https://mp.weixin.qq.com/s/1E_ONUnMwTFozd9-dB70Yw
https://blog.csdn.net/wg5foc08/article/details/102831489
https://blog.csdn.net/Herishwater/article/details/90733267

由于此文做的是研究某一个APP的ID，之前人做的JS是研究IOS的排行信息，所以在JS传参方向有细节不同。可以通过申请网址来看（https://www.qimai.cn/search/index/country/cn/search/%E6%BB%B4%E6%BB%B4%E5%87%BA%E8%A1%8C），这里传参的param不一样，除此以外，加密方式一样。

2.2 构造链接代码

我的代码实在jupyter notebook里面写的，下面是图片，把代码集中贴在后面。

import requests
import execjs
import urllib
import os
import json
import random
import re
import csv# 读入整理的app名称
search_list = []
with open("/Users/didi/Downloads/七麦网爬虫/key_word.txt", "r") as f:for line in f.readlines():line = line.strip('\n')search_list.append(line)# 读入解密js
with open('/Users/didi/Downloads/七麦网爬虫/qimai_app.js', 'r', encoding='utf-8') as f:get_analy_js = f.read()
jsData = execjs.compile(get_analy_js)# 准备UA
user_agent = ["Mozilla/5.0 (Macintosh; U; Intel Mac OS X 10_6_8; en-us) AppleWebKit/534.50 (KHTML, like Gecko) Version/5.1 Safari/534.50","Mozilla/5.0 (Windows; U; Windows NT 6.1; en-us) AppleWebKit/534.50 (KHTML, like Gecko) Version/5.1 Safari/534.50","Mozilla/5.0 (Windows NT 10.0; WOW64; rv:38.0) Gecko/20100101 Firefox/38.0","Mozilla/5.0 (Windows NT 10.0; WOW64; Trident/7.0; .NET4.0C; .NET4.0E; .NET CLR 2.0.50727; .NET CLR 3.0.30729; .NET CLR 3.5.30729; InfoPath.3; rv:11.0) like Gecko","Mozilla/5.0 (compatible; MSIE 9.0; Windows NT 6.1; Trident/5.0)","Mozilla/4.0 (compatible; MSIE 8.0; Windows NT 6.0; Trident/4.0)","Mozilla/5.0 (Macintosh; Intel Mac OS X 10.6; rv:2.0.1) Gecko/20100101 Firefox/4.0.1","Mozilla/5.0 (Windows NT 6.1; rv:2.0.1) Gecko/20100101 Firefox/4.0.1","Opera/9.80 (Macintosh; Intel Mac OS X 10.6.8; U; en) Presto/2.8.131 Version/11.11","Opera/9.80 (Windows NT 6.1; U; en) Presto/2.8.131 Version/11.11","Mozilla/5.0 (Macintosh; Intel Mac OS X 10_7_0) AppleWebKit/535.11 (KHTML, like Gecko) Chrome/17.0.963.56 Safari/535.11","Mozilla/4.0 (compatible; MSIE 7.0; Windows NT 5.1; Maxthon 2.0)","Mozilla/4.0 (compatible; MSIE 6.0; ) Opera/UCWEB7.0.2.37/28/999","Mozilla/6.0 (iPhone; CPU iPhone OS 8_0 like Mac OS X) AppleWebKit/536.26 (KHTML, like Gecko) Version/8.0 Mobile/10A5376e Safari/8536.25",
]# 通过访问七麦得到时间戳
def get_synct():headers = {"User-Agent": random.choice(user_agent),'Origin': 'https://www.qimai.cn','Referer': 'https://www.qimai.cn/rank',"Accept": "application/json,text/plain,*/*",}resp = requests.get('https://www.qimai.cn/rank', headers=headers, verify=False)t = resp.cookies.get_dict()synct = t.get('synct')return synct# 获得加密后的url
def get_analysis_url(synct,app_name):params = {'country':'cn','search':app_name}analysis = jsData.call('getAnalysis',synct,params)base_url = "https://api.qimai.cn/search/index?analysis="params_str = "&country=cn&search=" + str(app_name)url = str(base_url) + str(analysis) + str(params_str)return urldef get_appID(url):headers = {"User-Agent": random.choice(user_agent),'Origin': 'https://www.qimai.cn'}html = requests.get(url, headers=headers)response = html.texttry:pattern = r'"appId":"(.*?)","appName"'  appID = re.search(pattern, response, re.I)[1]except:appID = Nonereturn appIDdef write_csv(file,appInfo):with open(file, 'a', encoding='utf-8',newline='') as csv_file:writer = csv.weiter(csv_file)writer.writerow(appInfo)def main(app_name):synct = get_synct()url = get_analysis_url(synct,app_name)appID = get_appID(url)print(appID)if __name__ == '__main__':for i in range(len(search_list))app_name = "滴滴出行"main(app_name)

3 其他反爬

破解analysis后还有其他的反爬，下面是一张测试结果，这是对于访问频率的封禁，可以看出七麦网的反爬是非常厉害的。