一、背景

开发背景：女朋友论文需要
开源背景：既然一直加班，那就索性回家也加吧，学弟们校招要避坑哦~凡是让你自愿996的，都不要去哦

二、整体思路

搭建kexue上网环境，我用的是v2rayN-Core（可百度找到使用方法）
爬虫采集评论（requests）

三、代码讲解

3.1 代码结构

img：工具图标
model：爬虫主程序、扩展分析
present：界面控件方法及信号连接
view：界面ui

3.2 核心代码

3.2.1 爬虫核心代码

篇幅有点长，这里只贴核心的代码
**Tips:**可能要修改一下本地代理配置os.environ["https_proxy"]="http:127.0.0.1:10809"，端口和你的上网工具的设置有关，这个配置当时坑了我一晚上

# -*- coding: utf-8 -*-
# time: 2022/7/3 4:52
# file: downloader.py
# author: Shi Yasongfrom __future__ import print_functionimport json
import re
import time
import dateparser
import requests
import os
os.environ["https_proxy"]="http:127.0.0.1:10809"YOUTUBE_VIDEO_URL = 'https://www.youtube.com/watch?v={youtube_id}'USER_AGENT = 'Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/79.0.3945.130 Safari/537.36'SORT_BY_POPULAR = 0
SORT_BY_RECENT = 1YT_CFG_RE = r'ytcfg\.set\s*\(\s*({.+?})\s*\)\s*;'
YT_INITIAL_DATA_RE = r'(?:window\s*\[\s*["\']ytInitialData["\']\s*\]|ytInitialData)\s*=\s*({.+?})\s*;\s*(?:var\s+meta|</script|\n)'class YoutubeCommentDownloader:def __init__(self):self.session = requests.Session()self.session.headers['User-Agent'] = USER_AGENTdef ajax_request(self, endpoint, ytcfg, retries=5, sleep=20):url = 'https://www.youtube.com' + endpoint['commandMetadata']['webCommandMetadata']['apiUrl']data = {'context': ytcfg['INNERTUBE_CONTEXT'],'continuation': endpoint['continuationCommand']['token']}for _ in range(retries):response = self.session.post(url, params={'key': ytcfg['INNERTUBE_API_KEY']}, json=data)if response.status_code == 200:return response.json()if response.status_code in [403, 413]:return {}else:time.sleep(sleep)def get_comments(self, youtube_id, *args, **kwargs):return self.get_comments_from_url(YOUTUBE_VIDEO_URL.format(youtube_id=youtube_id), *args, **kwargs)def get_comments_from_url(self, youtube_url, sort_by=SORT_BY_RECENT, language=None, sleep=1):response = self.session.get(youtube_url)if 'uxe=' in response.request.url:self.session.cookies.set('CONSENT', 'YES+cb', domain='.youtube.com')response = self.session.get(youtube_url, verify=False)html = response.textytcfg = json.loads(self.regex_search(html, YT_CFG_RE, default=''))if not ytcfg:return  # Unable to extract configurationif language:ytcfg['INNERTUBE_CONTEXT']['client']['hl'] = languagedata = json.loads(self.regex_search(html, YT_INITIAL_DATA_RE, default=''))section = next(self.search_dict(data['contents'], 'itemSectionRenderer'), None)renderer = next(self.search_dict(section, 'continuationItemRenderer'), None) if section else Noneif not renderer:# Comments disabled?returnneeds_sorting = sort_by != SORT_BY_POPULARcontinuations = [renderer['continuationEndpoint']]while continuations:continuation = continuations.pop()response = self.ajax_request(continuation, ytcfg)if not response:breakerror = next(self.search_dict(response, 'externalErrorMessage'), None)if error:raise RuntimeError('Error returned from server: ' + error)if needs_sorting:sort_menu = next(self.search_dict(response, 'sortFilterSubMenuRenderer'), {}).get('subMenuItems', [])if sort_by < len(sort_menu):continuations = [sort_menu[sort_by]['serviceEndpoint']]needs_sorting = Falsecontinueraise RuntimeError('Failed to set sorting')actions = list(self.search_dict(response, 'reloadContinuationItemsCommand')) + \list(self.search_dict(response, 'appendContinuationItemsAction'))for action in actions:for item in action.get('continuationItems', []):if action['targetId'] == 'comments-section':# Process continuations for comments and replies.continuations[:0] = [ep for ep in self.search_dict(item, 'continuationEndpoint')]if action['targetId'].startswith('comment-replies-item') and 'continuationItemRenderer' in item:# Process the 'Show more replies' buttoncontinuations.append(next(self.search_dict(item, 'buttonRenderer'))['command'])for comment in reversed(list(self.search_dict(response, 'commentRenderer'))):result = {'cid': comment['commentId'],'text': ''.join([c['text'] for c in comment['contentText'].get('runs', [])]),'time': comment['publishedTimeText']['runs'][0]['text'],'author': comment.get('authorText', {}).get('simpleText', ''),'channel': comment['authorEndpoint']['browseEndpoint'].get('browseId', ''),'votes': comment.get('voteCount', {}).get('simpleText', '0'),'photo': comment['authorThumbnail']['thumbnails'][-1]['url'],'heart': next(self.search_dict(comment, 'isHearted'), False)}try:result['time_parsed'] = dateparser.parse(result['time'].split('(')[0].strip()).timestamp()except AttributeError:passpaid = (comment.get('paidCommentChipRenderer', {}).get('pdgCommentChipRenderer', {}).get('chipText', {}).get('simpleText'))if paid:result['paid'] = paidyield resulttime.sleep(sleep)@staticmethoddef regex_search(text, pattern, group=1, default=None):match = re.search(pattern, text)return match.group(group) if match else default@staticmethoddef search_dict(partial, search_key):stack = [partial]while stack:current_item = stack.pop()if isinstance(current_item, dict):for key, value in current_item.items():if key == search_key:yield valueelse:stack.append(value)elif isinstance(current_item, list):for value in current_item:stack.append(value)

3.2.2 界面核心代码

界面是用pyqt拖出来的，没啥好说的，拖的不好看，如果可以，你可以继续优化界面
数据大的话工具可能会卡死，所以这里采用了多线程

# -*- coding: utf-8 -*-
# time: 2022/7/3 2:52
# file: p_home.py
# author: Shi Yasongfrom model.youtube_comment_downloader import GetYoutube
from threading import Threadimport view.v_home
import threadingclass HomePresenter(object):def __init__(self):self.CreateView()self.CreateConnect()def CreateView(self):self.m_HomeWidget = view.v_home.HomeWidget()self.m_HomeWidget.show()self.m_HomeWidget.ShowInitMsg()def CreateConnect(self):self.m_HomeWidget.m_ConfirmBtn.clicked.connect(self.SlotConfirmBtnClicked)def SlotConfirmBtnClicked(self):youtube_id = self.m_HomeWidget.GetID()if not youtube_id:self.m_HomeWidget.ShowMessage("提示", "请输入视频id")returnoutput = self.m_HomeWidget.GetName()if not output:self.m_HomeWidget.ShowMessage("提示", "请输入输出名")returnt1 = threading.Thread(target=self.Test, args=(youtube_id, output,))t1.start()t1.join()def Test(self, youtube_id, output):oYoutube = GetYoutube()thread = Thread(target=oYoutube.Test, args=(youtube_id, output,))thread.start()def HomePresentInit():global g_HomePresentInitg_HomePresentInit = HomePresenter()def GetHomePresenter():global g_HomePresentInitreturn g_HomePresentInit

3.2.3 json转excel

# -*- coding: utf-8 -*-
# time: 2022/7/3 5:10
# file: json2xlsx.py
# author: Shi Yasongimport pandas as pd
import osfor file in os.listdir('../'):if file.endswith('json'):print(file)f_head, f_tail = file.split(".")print(f_head, " || ", f_tail)try:df = pd.read_json(file, lines=True)df.to_excel('{}.xlsx'.format(f_head), index=False, engine='xlsxwriter', encoding='UTF-8')except Exception as e:print('Excepted-》{}: {}'.format(file, str(e)))

四、爬取结果

每条数据包含的字段：
视频id，评论文本，评论时间，评论者，频道，点赞，图片等
这里只抓取了评论的数据，你可以抓取更多的数据来完成你的需求，比如情感分析等等

五、工具截图

最后附上工具截图和下载链接
（这里是一条链接，还在审核明天再发）

python爬取YB视频评论相关推荐

Python爬取京东商品评论和图片下载
Python爬取京东商品评论和图片下载代码仅供学习使用,切勿扩散并且频繁爬取网站贴上代码和注释,方便自己下次看 import requests import time import json im ...
python爬取B站评论制作词云
python爬取B站评论制作词云江山代有才人出,B站评论占一半废话不多说,咱们直接上代码` import imageio import jieba import wordcloud import ...
Python爬取京东商品评论
京东商城的页面不是静态页面,其评论信息存放于json文件中,由ajax控制,所以我们在浏览器上看到的网页源代码和用Python爬下来的是不一样的,所以我们真正要爬取的是评论数据所存放的json文件. ...
python爬取bilibili数据_如何使用Python爬取bilibili视频（详细教程）
Python爬取bilibili视频摘要为了解决PC端的bilibili无法下载视频的问题,使用python语言可以实现一个能够爬取bilibili某个视频资源(不包括会员视频)的程序.采用整个视 ...
python如何爬取网页视频_快就完事了！10分钟用python爬取网站视频和图片
原标题:快就完事了!10分钟用python爬取网站视频和图片话不多说,直接开讲!教你如何用Python爬虫爬取各大网站视频和图片. 638855753 网站分析: 我们点视频按钮,可以看到的链接是: ...
python爬淘宝app数据_一篇文章教会你用Python爬取淘宝评论数据（写在记事本）
[一.项目简介] 本文主要目标是采集淘宝的评价,找出客户所需要的功能.统计客户评价上面夸哪个功能多,比如防水,容量大,好看等等. [二.项目准备工作] 准备Pycharm,下载安装等,可以参考这篇文章 ...
Python - 爬取豆瓣短评评论
Python - 爬取豆瓣短评评论 import requests from bs4 import BeautifulSoup import re import time# 保存豆瓣评分 source ...
用python爬取天猫商品评论并分析（2）
用python爬取天猫商品评论并分析(2) 之前介绍过天猫数据的爬取和数据初步处理,今天介绍下将采集的评论进行文本分析!下面是总流程: 0. 主要流程 0. 数据采集这一步参考网址:https:/ ...
python爬取爱情公寓电影评论并制作词云
python爬取爱情公寓电影评论并制作词云前言: 一直想研究研究如何生成词云,今天抽点时间给大家分享一下制作词云的过程,本文重在研究词云如何制作,由于时间仓促,至于爬取的数据量不大,大家可自 ...

python爬取YB视频评论

python爬取评论