跳过selenium检测爬取淘宝直通车

最近，有对阿里商家端进行一些数据爬取，这次爬取的是直通车人群溢价数据，发现对selenium的检测相当厉害，然而我的回答是，你强任你强，清风拂山岗。咱人工登录怕过谁，什么cokies,user-agent,selenium检测,token，统统与我无瓜,我们的宗旨就是,能用就行

核心就是,打开真实浏览器，用selenium接管，登录之后，再关闭提示框，立刻开始爬虫
os.system('cd "C:\\Program Files (x86)\\Google\\Chrome\\Application"&start chrome.exe --remote-debugging-port=9999 --user-data-dir="C:\selenum\AutomationProfile" https://subway.simba.taobao.com')

顺便用tkinter简单搞了个账号密码提示框，省的老去找

直接上代码

# -*- coding: utf-8 -*-
import os
import re
import time
from datetime import date, timedelta
from lxml import etree
from selenium import webdriver
from selenium.webdriver.chrome.options import Options
from selenium.webdriver.support.ui import WebDriverWait
from selenium.webdriver.common.by import By
from selenium.webdriver.support import expected_conditions as EC
import pandas as pd
from user_pass import time_countdef main():"""人群溢价爬取"""#打开本地chrome，同时打开直通车登录页面，需要提前配置环境变量pathos.system('cd "C:\\Program Files (x86)\\Google\\Chrome\\Application"&start chrome.exe --remote-debugging-port=9999 --user-data-dir="C:\selenum\AutomationProfile" https://subway.simba.taobao.com')chrome_debug_port = 9999chrome_options = Options()# chrome_options.add_argument('--headless')chrome_options.add_experimental_option("debuggerAddress", f"127.0.0.1:{chrome_debug_port}")#selenium接管当前网页browser = webdriver.Chrome(options=chrome_options)wait = WebDriverWait(browser, 15)print(browser.title)# 当前句柄,以防有iframe,事实是我想多了current_handle = browser.current_window_handleprint("准备好，爬虫开始:")# 账号密码提示：time_count()# 进入推广页面url = 'https://subway.simba.taobao.com/#!/manage/campaign/index'browser.get(url)time.sleep(3)# 获取cookies# cookies = browser.get_cookies()# jsonCookies = json.dumps(cookies)# # 把cookies保存在本地# with open('ffCookies.json', 'w') as f:#    f.write(jsonCookies)# time.sleep(2)# 获取人群数据get_CrowdDetail(browser,wait)browser.close()# 获取人群溢价数据
def get_CrowdDetail(browser,wait):# 计数器count = 0#设定开始与结束时间startDate = str((date.today() + timedelta(days=-7)).strftime("%Y-%m-%d"))endDate = str((date.today() + timedelta(days=-1)).strftime("%Y-%m-%d"))print('爬取日期:'+startDate+'至'+endDate)#保存好要爬取的人群溢价网址,如果能得到campaignId,adGroupId两个关键字段大可不必这样麻烦，可直接F12从xhr中获取CrowDetail = {'https://subway.simba.taobao.com/#!/manage/adgroup/detail?productId=101001005&tab=crowd&campaignId=66281901&adGroupId=1745413653&start='+startDate+'&end='+endDate+'&rptType=realTime&adpage=1','https://subway.simba.taobao.com/#!/manage/adgroup/detail?productId=101001005&tab=crowd&campaignId=66281901&adGroupId=1745413655&start='+startDate+'&end='+endDate+'&rptType=realTime&adpage=1','https://subway.simba.taobao.com/#!/manage/adgroup/detail?productId=101001005&tab=crowd&campaignId=66281901&adGroupId=1745413656&start='+startDate+'&end='+endDate+'&rptType=realTime&adpage=1','https://subway.simba.taobao.com/#!/manage/adgroup/detail?productId=101001005&tab=crowd&campaignId=66281901&adGroupId=1745413657&start='+startDate+'&end='+endDate+'&rptType=realTime&adpage=1','https://subway.simba.taobao.com/#!/manage/adgroup/detail?productId=101001005&tab=crowd&campaignId=66281901&adGroupId=1850331434&start='+startDate+'&end='+endDate+'&rptType=realTime&adpage=1','https://subway.simba.taobao.com/#!/manage/adgroup/detail?productId=101001005&tab=crowd&campaignId=66281901&adGroupId=1877620378&start='+startDate+'&end='+endDate+'&rptType=realTime&adpage=1','https://subway.simba.taobao.com/#!/manage/adgroup/detail?productId=101001005&tab=crowd&campaignId=66173371&adGroupId=1733518315&start='+startDate+'&end='+endDate+'&rptType=realTime&adpage=1','https://subway.simba.taobao.com/#!/manage/adgroup/detail?productId=101001005&tab=crowd&campaignId=66173370&adGroupId=1734092022&start='+startDate+'&end='+endDate+'&rptType=realTime&adpage=1','https://subway.simba.taobao.com/#!/manage/adgroup/detail?productId=101001005&tab=crowd&campaignId=66173370&adGroupId=1734092023&start='+startDate+'&end='+endDate+'&rptType=realTime&adpage=1','https://subway.simba.taobao.com/#!/manage/adgroup/detail?productId=101001005&tab=crowd&campaignId=66157385&adGroupId=1732352485&start='+startDate+'&end='+endDate+'&rptType=realTime&adpage=1','https://subway.simba.taobao.com/#!/manage/adgroup/detail?productId=101001005&tab=crowd&campaignId=66157385&adGroupId=1732352486&start='+startDate+'&end='+endDate+'&rptType=realTime&adpage=1','https://subway.simba.taobao.com/#!/manage/adgroup/detail?productId=101001005&tab=crowd&campaignId=66157385&adGroupId=1732352487&start='+startDate+'&end='+endDate+'&rptType=realTime&adpage=1','https://subway.simba.taobao.com/#!/manage/adgroup/detail?productId=101001005&tab=crowd&campaignId=66157385&adGroupId=1732352489&start='+startDate+'&end='+endDate+'&rptType=realTime&adpage=1','https://subway.simba.taobao.com/#!/manage/adgroup/detail?productId=101001005&tab=crowd&campaignId=66157385&adGroupId=1855117081&start='+startDate+'&end='+endDate+'&rptType=realTime&adpage=1','https://subway.simba.taobao.com/#!/manage/adgroup/detail?productId=101001005&tab=crowd&campaignId=66157385&adGroupId=1881028817&start='+startDate+'&end='+endDate+'&rptType=realTime&adpage=1','https://subway.simba.taobao.com/#!/manage/adgroup/detail?productId=101001005&tab=crowd&campaignId=66281882&adGroupId=1732330386&start='+startDate+'&end='+endDate+'&rptType=realTime&adpage=1','https://subway.simba.taobao.com/#!/manage/adgroup/detail?productId=101001005&tab=crowd&campaignId=66145329&adGroupId=1739717496&start='+startDate+'&end='+endDate+'&rptType=realTime&adpage=1','https://subway.simba.taobao.com/#!/manage/adgroup/detail?productId=101001005&tab=crowd&campaignId=66145329&adGroupId=1739717497&start='+startDate+'&end='+endDate+'&rptType=realTime&adpage=1'}for i in CrowDetail:count=count+1header = []# 访问网址，准备获取数据browser.get(i)wait.until(EC.presence_of_element_located((By.XPATH, '//div[@class="manage-common-view-list"]//div[@class="freeze-table bp-table"]//div[@class="table-content osx custom-scrollbar"]//table[@class="bp-table"]/tbody/tr/td[3]')))print('当前访问网址:'+i)time.sleep(1.5)#解析page_sourcesource = etree.HTML(browser.page_source)title = source.xpath('//*[@id="magix_vf_main"]/article/div[@class="grid-p clearfix"]//ul/li[5]/div/div/span[1]')[0].textproduct= source.xpath('//*[@id="magix_vf_main"]/article//div[@class="adgroup-detail"]/p//a')[0].textpeople_list = source.xpath('//div[@class="manage-common-view-list"]//div[@class="freeze-table bp-table"]//div[@class="freeze-col"]//table[@class="bp-table"]/tbody/tr')info_list = source.xpath('//div[@class="manage-common-view-list"]//div[@class="freeze-table bp-table"]//div[@class="table-content osx custom-scrollbar"]//table[@class="bp-table"]/tbody/tr')time.sleep(1.5)#获取标题等信息for people in people_list:t = {}if people.xpath('./td[2]/strong'):t['标题'] = titlet['产品'] = productt['序号'] = people.xpath('./@index')[0]t['状态'] = people.xpath('./td[2]/strong')[0].textt['推广人群'] = people.xpath('./td[3]/div[1]/span')[0].text#获取详细信息for detail in info_list:if detail.xpath('./@index')[0]==t['序号'] and detail.xpath('./td[1]/div'):t['对应序号'] = detail.xpath('./@index')[0]t['人群分类'] = detail.xpath('./td[1]/div')[0].textt['溢价'] = detail.xpath('./td[2]/div/span/span')[0].textt['展现'] = detail.xpath('./td[3]')[0].textt['点击'] = detail.xpath('./td[4]')[0].textt['CTR'] = detail.xpath('./td[5]')[0].textt['花费'] = detail.xpath('./td[6]')[0].textt['CPC'] = detail.xpath('./td[7]')[0].textt['加购数'] = detail.xpath('./td[8]')[0].textt['转化率'] = detail.xpath('./td[9]')[0].textt['成交金额'] = detail.xpath('./td[10]')[0].textt['ROI'] = detail.xpath('./td[11]')[0].textt['成交订单'] = detail.xpath('./td[12]')[0].textheader.append(t)print('已获取:' + t['标题'] + '-' + t['产品'] + '-' + t['推广人群'])#re.findall("[0-9\,\.]*",)#写入本地CSV文件df = pd.DataFrame(header)if count==1:df.to_csv('./dd.csv', index=None, encoding='utf-8-sig', mode='a')else:df.to_csv('./dd.csv', index=None, encoding='utf-8-sig', mode='a', header=None)time.sleep(1)print('已保存该数据到本地dd.csv文件夹')if __name__ == '__main__':main()print("结束---------------------------------")

小提示框user_pass.py


import tkinterdef close():root.destroy()def time_count():global  root#创建应用程序窗口,设置标题和大小root = tkinter.Tk()root.wm_attributes('-topmost',1)root.title('我只是一个提示框')root['width'] = 200root['height'] = 110#不允许改变窗口大小root.resizable(False, False)#创建Text组件，放置一些文字richText = tkinter.Text(root, width=50)richText.place(x=10, y=10, width=180, height=30)richText.insert('0.0', '你的账号')eeText = tkinter.Text(root, width=50)eeText.place(x=10, y=40, width=180, height=30)eeText.insert('0.0', '你的密码')#显示关闭按钮buff = tkinter.Button(root,text="关闭",command=close)buff.place(x=70, y=75, width=50, height=30)root.mainloop()

最终的结果是这样的（销售数据未展示）

跳过selenium检测爬取淘宝直通车相关推荐

python爬取bilibili数据_python基础教程之selenium+phantomjs爬取bilibili
selenium+phantomjs爬取bilibili 首先我们要下载phantomjs 你可以到 http://phantomjs.org/download.html 这里去下载下载完之后解压到 ...
[python爬虫] BeautifulSoup和Selenium简单爬取知网信息测试
作者最近在研究复杂网络和知识图谱内容,准备爬取知网论文相关信息进行分析,包括标题.摘要.出版社.年份.下载数和被引用数.作者信息等.但是在爬取知网论文时,遇到问题如下: 1.爬取内容总为空,其原因 ...
[转载] python+selenium定时爬取丁香园的新冠病毒每天的数据，并制作出类似的地图（部署到云服务器）
参考链接: Python vars() python+selenium定时爬取丁香园的新冠病毒每天的数据,并制作出类似的地图(部署到云服务器) 声明:仅供技术交流,请勿用于非法用途,如有其它非法用途造 ...
Selenium+ChromeDriver爬取链家网二手房价（包括经纬度）信息
[需求] 爬取青岛市二手房源信息,要求包括房源基本信息.小区名称.总价.单价以及房源的经纬度信息. 根据以上需求,进入青岛市二手房首页,如图1所示,进行页面分析: 图1 [页面分析] 首先 ...
python + selenium多进程爬取淘宝搜索页数据
python + selenium多进程爬取淘宝搜索页数据 1. 功能描述按照给定的关键词,在淘宝搜索对应的产品,然后爬取搜索结果中产品的信息,包括:标题,价格,销量,产地等信息,存入mongodb ...
基于selenium+scrapy爬取复仇者联盟4豆瓣影评数据
基于selenium+scrapy爬取复仇者联盟4豆瓣影评数据参考资料: 黑马程序员爬虫教程静觅爬虫教程 mac下anaconda安装selenium+PhantomJS scrapy下载中间件结 ...
python+selenium定时爬取丁香园的新冠病毒每天的数据，并制作出类似的地图（部署到云服务器）
python+selenium定时爬取丁香园的新冠病毒每天的数据,并制作出类似的地图(部署到云服务器) 声明:仅供技术交流,请勿用于非法用途,如有其它非法用途造成损失,和本博客无关目录 python ...
python+selenium+phantomJS爬取国家地表水水质自动监测实时数据发布系统——动态网页爬虫
一.关于phantomjs 1.介绍 PhantomJS是一个为自动化而生的利器,它本质上是一个基于webkit内核的无界面浏览器,并可使用JavaScript或CoffeeScript进行编程.由于 ...
[python爬虫] Selenium定向爬取PubMed生物医学摘要信息
本文主要是自己的在线代码笔记.在生物医学本体Ontology构建过程中,我使用Selenium定向爬取生物医学PubMed数据库的内容. PubMed是一个免费的搜寻引擎,提供生物医学 ...
Selenium+Python3爬取微博私信
Selenium+Python3爬取微博私信需求缺陷代码需求爬取微博私信信息,包括:文本.对话者信息.图片路径,并将截图.文本留存. 缺陷微博私信网页长时间查看时,经常有刷新失败.页面崩溃 ...

跳过selenium检测爬取淘宝直通车

直接上代码

小提示框user_pass.py

最终的结果是这样的（销售数据未展示）

跳过selenium检测爬取淘宝直通车相关推荐

最新文章

热门文章