Selenium爬虫 -- 用户动态数据爬取

突然在一次熬夜写完代码后，发现自己是真的有点悟元素了......

定位方式需要灵活变通，比如脸书的元素使用纯树形路径的xpath比较稳定，而领英的元素显然使用@classname比较稳定，xpath需要处理异常。

爬取个人用户的动态。

下载链接（部分信息已用******代替）

代码片段

        usersl = ['kaifulee'] for uidN in usersl:self.browser1.get('https://www.linkedin.com/in/' + uidN + '/detail/recent-activity/')  # 个人动态time.sleep(random.randint(7, 15))pre_height = 0now_height = 0while True:self.browser1.execute_script("scrollBy(0,100000)")time.sleep(random.randint(7, 15))now_height = self.browser1.execute_script("return document.documentElement.scrollHeight;")print(now_height)if now_height == pre_height:breakpre_height = now_heighttime.sleep(random.randint(7, 15))pic_video_count = 0articles = self.browser1.find_elements_by_xpath('/html/body/div[7]/div[3]/div/div/div/div/div/div[2]/div/div')for article in articles:try:article.find_element_by_xpath('./div/div[1]/span/div/span/a/span')except Exception:print('原创帖子')name = article.find_element_by_xpath('./div/div[1]/a/div[3]/span[1]/span[1]/span').textprint('用户名：' + name)pubtime = article.find_element_by_xpath('./div/div[1]/a/div[3]/span[3]/span/span[2]').textprint('发布时间：' + pubtime)content = '空'try:content = article.find_element_by_xpath('./div/div[3]/div[1]/div/span/span').textexcept Exception:print(Exception)else:print('内容：' + content)like_num = 0try:like_num = article.find_element_by_xpath('./div/div[4]/div/ul/li[1]/button/span').textexcept Exception:print(Exception)print('点赞数：' + str(like_num))cmt_num = 0try:cmt_num0 = article.find_element_by_xpath('./div/div[4]/div/ul/li[2]/button/span').textexcept Exception:print(Exception)else:cmt_num = cmt_num0[4:len(cmt_num0) - 1]print('评论数：' + str(cmt_num))pic_path = '无'de_path = 'D:/**************/LinkedIn/LiSpider/pics_videos'picsrcs = []try:article.find_element_by_xpath('./div/article/div/div[1]/a/div/div/img')except Exception:print(Exception)else:spics = article.find_elements_by_xpath('./div/article/div/div[1]/a/div/div/img')for apic in spics:picsrc = apic.get_attribute('src')picsrcs.append(picsrc)print('picsrc: ' + picsrc)urllib.request.urlretrieve(picsrc, './pics_videos/%s.jpg' % pic_video_count)pic_path = de_path + str(pic_video_count) + '.jpg'pic_video_count += 1print('保存图片的本地路径：' + pic_path)try:article.find_element_by_xpath('./div/div[4]/div/div/button/div/div/img')except Exception:print(Exception)else:spics = article.find_elements_by_xpath('./div/div[4]/div/div/button/div/div/img').get_attribute('src')for apic in spics:print('picsrc: ' + apic)urllib.request.urlretrieve(apic, './pics_videos/%s.jpg' % pic_video_count)pic_path = de_path + str(pic_video_count) + '.jpg'pic_video_count += 1print('保存图片的本地路径：' + pic_path)video_link = '无'video_path = 'D:/_HhrWorkSpace/_Python/Spiders/LinkedIn/LiSpider/pics_videos'try:vidsrc = article.find_element_by_xpath('./div/div[4]/div[1]/div/div/a').get_attribute('href')except Exception:print(Exception)else:video_link = vidsrcprint('video download start')os.system('you-get  -o %s %s --debug' % (video_path, video_link))time.sleep(random.randint(7, 15))print('video download OK, continue loading...')data_type = ''if pic_path != '无':data_type += '图片 + 'if video_link != '无':data_type += '视频 + 'if content != '空':data_type += '文本'print('信息类型：' + data_type)cmts = list()strcmts = '无'try:article.find_elements_by_xpath('./div/div[4]/div/ul/li[2]/button').click()except Exception:print(Exception)else:while True:try:article.find_element_by_xpath('./div/div[4]/div/div[3]/div[3]/div/button').click()time.sleep(random.randint(7, 15))except Exception:print(Exception)else:breakcmtidtx = article.find_elements_by_xpath('./div/div[4]/div/div[3]/div[3]/article')for act in cmtidtx:cmts.append('\"' + act.find_element_by_xpath('./div[1]/a[2]/h3/span[1]/span[1]').text + '\":\"' + act.find_element_by_xpath('./div[3]/div/div[1]/p/span/span').text + '\"')strcmts = ','.join(cmts)article.find_element_by_xpath('./div/div[4]/div/ul/li[1]/button').click()like1_num = 0like1_us = list()strlike1_us = '无'if like_num != 0:time.sleep(random.randint(7, 15))try:self.browser1.find_element_by_xpath('/html/body/div[4]/div/div/div[1]/div/div/div/button[2]').click()except Exception:print(Exception)else:# pre_height = 0# now_height = 0# while True:#     # self.browser1.execute_script("window.scrollTo(0,document.body.scrollHeight)")#     self.browser1.execute_script("scrollBy(0,100000)")#     time.sleep(random.randint(7, 15))#     now_height = self.browser1.execute_script("return document.documentElement.scrollHeight;")#     print(now_height)#     if now_height == pre_height:#         break#     pre_height = now_height# time.sleep(random.randint(7, 15))like1_num = self.browser1.find_element_by_xpath('/html/body/div[4]/div/div/div[1]/div/div/div/button[2]/div/span[2]').textlikes1 = self.browser1.find_elements_by_xpath('/html/body/div[4]/div/div/div[2]/div/ul/li')for al in likes1:like1_us.append(al.find_element_by_xpath('./a/div/div[2]/div[1]/span').text)strlike1_us = ','.join(like1_us)time.sleep(random.randint(7, 15))self.browser1.find_element_by_xpath('/html/body/div[4]/div/div/button').click()insert_time = datetime.datetime.now().strftime("%Y-%m-%d %H:%M:%S")update_time = datetime.datetime.now().strftime("%Y-%m-%d %H:%M:%S")sql_insert = "insert into linkedin_blog(uname,content," \"pubtime,cmt_num,like_num,insert_time,update_time," \"data_type,pic_path,video_path,like1_num,like1_us," \"like2_num,like2_us,like3_num,like3_us,like4_num,like4_us,like5_num,like5_us," \"like6_num,like6_us,comments) " \"values ('%s','%s','%s','%s','%s','%s','%s','%s','%s','%s','%s','%s','%s'," \"'%s','%s','%s','%s','%s','%s','%s','%s','%s','%s')" \% (name, content, pubtime, cmt_num, like_num, insert_time, update_time,data_type, pic_path, video_path, like1_num, strlike1_us, like2_num, strlike2_us,like3_num, strlike3_us, like4_num, strlike4_us, like5_num, strlike5_us, like6_num,strlike6_us, strcmts)try:cur.execute(sql_insert)db.commit()print("SQL sucess")except Exception as e:print("SQL error")print(e)db.rollback()db.close()
if __name__ == '__main__':spider = LinkedInSpider()spider.main()

Selenium爬虫 -- 用户动态数据爬取相关推荐

结合Selenium 和 Requests完成动态数据爬取
Selenium 简介 Selenium是一个用于Web应用程序测试的工具.Selenium测试直接调用操作浏览器,就像真正的用户在操作一样.支持的浏览器包括IE(7, 8, 9, 10, 11),M ...
网络爬虫（动态网页爬取关于混合动力汽车的论文数据）
一.实验目的 (1) 熟悉网页浏览器开发工具的使用: (2) 掌握动态网页加载过程: (3) 掌握 post 请求参数设置: (4) 掌握基本的反反爬技术: (5) 能够根据问题需求,指定网络爬虫方案 ...
python爬虫从入门到实战笔记——第一章爬虫原理和数据爬取
爬虫原理和数据抓取 1.1 通用爬虫和聚焦爬虫通用爬虫聚焦爬虫 1.2 HTTP和HTTPS HTTP的请求与响应浏览器发送HTTP请求的过程: 客户端HTTP请求请求方法常用的请求报头服 ...
python财务报表预测股票价格_机器学习股票价格预测从爬虫到预测-数据爬取部分...
声明:本文已授权公众号「AI极客研修站」独家发布前言各位朋友大家好,小之今天又来给大家带来一些干货了.上篇文章机器学习股票价格预测初级实战是我在刚接触量化交易那会,因为苦于找不到数据源,所以找的一 ...
爬虫之websocket数据爬取
收集了大家的问题.我又重新写了一篇websocket的代码,并添加了注释,在文章最后.希望可以解决大家遇到的问题- websocket是最近开发很常用的技术之一,他可以一直保持着连接不断,但是你的页面 ...
python爬虫的一次尝试——华北电力大学图书馆读者荐购系统：基于python爬虫的web数据爬取
华北电力大学图书馆读者荐购系统数据爬取前言本章工具网页分析 1.荐购数据 2.书目具体信息代码部分 1. 荐购数据爬取 2. 完整书目信息爬取前言本学期数据仓库与数据挖掘课程大作业是编程实 ...
汽车用户消费投诉数据爬取分析（Python爬虫）
多线程爬虫代码 """ name:汽车用户消费投诉_品牌url爬取,已完成 author:zhangxiaoyu """ import _t ...
爬虫day05 动态网站爬取，json模块 selenium , phantomjs , BeautifulSoup
目录 1.糗事百科-xpath 2.动态网站数据抓取 - Ajax 3.json模块 4.selenium + phantomjs 强大的网络爬虫 1.selenium 2.phantomjs 3.常 ...
爬虫实战（二）猫眼电影动态数据爬取
前言爬虫部分的知识算是告一段落了,又是几天过去了,感觉挺充实的.里面可能还有一些需要改进的地方,慢慢完善吧. 正文接下来我们就说说上次遗留下来的问题,如何爬取动态数据.先告诉你一下爬取的数据中为什 ...
python汽车用户消费投诉数据爬取_爬虫案例：汽车用户消费投诉数据爬取
任务:爬取某公开网站上的汽车用户消费投诉数据('http://tousu.315che.com/tousulist/serial/55467/'),字段包含品牌.投诉内容.品牌车型.单号.投诉问题.投 ...

Selenium爬虫 -- 用户动态数据爬取

代码片段

Selenium爬虫 -- 用户动态数据爬取相关推荐

最新文章

热门文章