文章目录

1. 爬取评论内容代码
2.爬取评论时间代码
3.数据处理部分
- 3.1 评论的时间戳转换为正常时间
- 3.2 评论内容读入csv
- 3.3 统计一天各个时间段内的评论数
- 3.4 统计最近评论数
4.数据分析
- 4.1 1.制作词云图
- 4.2 制作最近评论数条形图与折线图
- 4.3 制作每小时评论条形图与折线图
- 4.4 制作最近评论数饼图
- 4.5 制作每小时评论饼图
- 4.6 制作观看时间区间评论统计饼图
- 4.7 制作主演提及占比饼图
- 4.8 评论内容情感分析

本文爬取腾讯视频网页以《乔家的儿女》这部电视剧后台评论，对数据进行爬虫与数据分析，爬取30页后台评论（完全爬取耗时巨大，所以选取30页作为本文数据爬取范围），对评论进行情绪文本分析处理

腾讯的评论数据在json文件里面，需要找到Json文件中的comment_id进行数据提取

1. 爬取评论内容代码

爬去数据注意要找到comment_id，乔家的儿女评论id是：7359549499，page_num是爬取的页数，本文选择爬取30页

import requests
import re
import randomdef get_html(url, params):uapools = ['Mozilla/5.0 (Macintosh; Intel Mac OS X 10_9_2) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/35.0.1916.153 Safari/537.36','Mozilla/5.0 (Windows NT 6.1; WOW64; rv:30.0) Gecko/20100101 Firefox/30.0','Mozilla/5.0 (Macintosh; Intel Mac OS X 10_9_2) AppleWebKit/537.75.14 (KHTML, like Gecko) Version/7.0.3 Safari/537.75.14']thisua = random.choice(uapools)headers = {"User-Agent": thisua}r = requests.get(url, headers=headers, params=params)r.raise_for_status()r.encoding = r.apparent_encodingr.encoding = 'utf-8'# 不加此句出现乱码return r.textdef parse_page(infolist, data):commentpat = '"content":"(.*?)"'lastpat = '"last":"(.*?)"'commentall = re.compile(commentpat, re.S).findall(data)next_cid = re.compile(lastpat).findall(data)[0]infolist.append(commentall)return next_ciddef print_comment_list(infolist):j = 0for page in infolist:print('第' + str(j + 1) + '页\n')commentall = pagefor i in range(0, len(commentall)):print(commentall[i] + '\n')j += 1def save_to_txt(infolist, path):fw = open(path, 'w+', encoding='utf-8')j = 0for page in infolist:#fw.write('第' + str(j + 1) + '页\n')commentall = pagefor i in range(0, len(commentall)):fw.write(commentall[i] + '\n')j += 1fw.close()def main():infolist = []vid = '7359549499';cid = "0";page_num = 30url = 'https://video.coral.qq.com/varticle/' + vid + '/comment/v2'print(url)for i in range(page_num):params = {'orinum': '10', 'cursor': cid}html = get_html(url, params)cid = parse_page(infolist, html)print_comment_list(infolist)save_to_txt(infolist, 'content.txt')main()

2.爬取评论时间代码

保留评论时间代码为time.txt

import requests
import re
import randomdef get_html(url, params):uapools = ['Mozilla/5.0 (Macintosh; Intel Mac OS X 10_9_2) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/35.0.1916.153 Safari/537.36','Mozilla/5.0 (Windows NT 6.1; WOW64; rv:30.0) Gecko/20100101 Firefox/30.0','Mozilla/5.0 (Macintosh; Intel Mac OS X 10_9_2) AppleWebKit/537.75.14 (KHTML, like Gecko) Version/7.0.3 Safari/537.75.14']thisua = random.choice(uapools)headers = {"User-Agent": thisua}r = requests.get(url, headers=headers, params=params)r.raise_for_status()r.encoding = r.apparent_encodingr.encoding = 'utf-8'# 不加此句出现乱码return r.textdef parse_page(infolist, data):commentpat = '"time":"(.*?)"'lastpat = '"last":"(.*?)"'commentall = re.compile(commentpat, re.S).findall(data)next_cid = re.compile(lastpat).findall(data)[0]infolist.append(commentall)return next_ciddef print_comment_list(infolist):j = 0for page in infolist:print('第' + str(j + 1) + '页\n')commentall = pagefor i in range(0, len(commentall)):print(commentall[i] + '\n')j += 1def save_to_txt(infolist, path):fw = open(path, 'w+', encoding='utf-8')j = 0for page in infolist:#fw.write('第' + str(j + 1) + '页\n')commentall = pagefor i in range(0, len(commentall)):fw.write(commentall[i] + '\n')j += 1fw.close()def main():infolist = []vid = '7359549499';cid = "0";page_num =30url = 'https://video.coral.qq.com/varticle/' + vid + '/comment/v2'#print(url)for i in range(page_num):params = {'orinum': '10', 'cursor': cid}html = get_html(url, params)cid = parse_page(infolist, html)print_comment_list(infolist)save_to_txt(infolist, 'time.txt')main()

3.数据处理部分

3.1 评论的时间戳转换为正常时间

# coding=gbk
import csv
import timecsvFile = open("data.csv",'w',newline='',encoding='utf-8')
writer = csv.writer(csvFile)
csvRow = []
#print(csvRow)
f = open("time.txt",'r',encoding='utf-8')
for line in f:csvRow = int(line)#print(csvRow)timeArray = time.localtime(csvRow)csvRow = time.strftime("%Y-%m-%d %H:%M:%S", timeArray)print(csvRow)csvRow = csvRow.split()writer.writerow(csvRow)f.close()
csvFile.close()

3.2 评论内容读入csv

在文件夹中能找到csv文件

# coding=gbk
import csv
import timecsvFile = open("data.csv",'w',newline='',encoding='utf-8')
writer = csv.writer(csvFile)
csvRow = []
#print(csvRow)
f = open("time.txt",'r',encoding='utf-8')
for line in f:csvRow = int(line)#print(csvRow)timeArray = time.localtime(csvRow)csvRow = time.strftime("%Y-%m-%d %H:%M:%S", timeArray)print(csvRow)csvRow = csvRow.split()writer.writerow(csvRow)f.close()
csvFile.close()

3.3 统计一天各个时间段内的评论数

# coding=gbk
import csvfrom pyecharts import options as opts
from sympy.combinatorics import Subset
from wordcloud import WordCloudwith open('data.csv') as csvfile:reader = csv.reader(csvfile)data1 = [str(row[1])[0:2] for row in reader]print(data1)
print(type(data1))#先变成集合得到seq中的所有元素，避免重复遍历
set_seq = set(data1)
rst = []
for item in set_seq:rst.append((item,data1.count(item)))  #添加元素及出现个数
rst.sort()
print(type(rst))
print(rst)with open("time2.csv", "w+", newline='', encoding='utf-8') as f:writer = csv.writer(f, delimiter=',')for i in rst:                # 对于每一行的，将这一行的每个元素分别写在对应的列中writer.writerow(i)with open('time2.csv') as csvfile:reader = csv.reader(csvfile)x = [str(row[0]) for row in reader]print(x)
with open('time2.csv') as csvfile:reader = csv.reader(csvfile)y1 = [float(row[1]) for row in reader]print(y1)

3.4 统计最近评论数

# coding=gbk
import csvfrom pyecharts import options as opts
from sympy.combinatorics import Subset
from wordcloud import WordCloudwith open('data.csv') as csvfile:reader = csv.reader(csvfile)data1 = [str(row[0]) for row in reader]#print(data1)
print(type(data1))#先变成集合得到seq中的所有元素，避免重复遍历
set_seq = set(data1)
rst = []
for item in set_seq:rst.append((item,data1.count(item)))  #添加元素及出现个数
rst.sort()
print(type(rst))
print(rst)with open("time1.csv", "w+", newline='', encoding='utf-8') as f:writer = csv.writer(f, delimiter=',')for i in rst:                # 对于每一行的，将这一行的每个元素分别写在对应的列中writer.writerow(i)with open('time1.csv') as csvfile:reader = csv.reader(csvfile)x = [str(row[0]) for row in reader]print(x)
with open('time1.csv') as csvfile:reader = csv.reader(csvfile)y1 = [float(row[1]) for row in reader]print(y1)

4.数据分析

数据分析方面：涉及到了词云图，条形，折线，饼图，后三者是对评论时间与主演占比的分析，然而腾讯的评论时间是以时间戳的形式显示，所以要进行转换，再去统计出现次数，最后，新加了对评论内容的情感分析。

4.1 1.制作词云图

import numpy as np
import re
import jieba
from wordcloud import WordCloud
from matplotlib import pyplot as plt
from PIL import Image# 上面的包自己安装，不会的就百度f = open('content.txt', 'r', encoding='utf-8')  # 这是数据源，也就是想生成词云的数据
txt = f.read()  # 读取文件
f.close()  # 关闭文件，其实用with就好，但是懒得改了
# 如果是文章的话，需要用到jieba分词，分完之后也可以自己处理下再生成词云
newtxt = re.sub("[A-Za-z0-9\!\%\[\]\,\。]", "", txt)
print(newtxt)
words = jieba.lcut(newtxt)img = Image.open(r'wc.jpg')  # 想要搞得形状
img_array = np.array(img)# 相关配置，里面这个collocations配置可以避免重复
wordcloud = WordCloud(background_color="white",width=1080,height=960,font_path="../文悦新青年.otf",max_words=150,scale=10,#清晰度max_font_size=100,mask=img_array,collocations=False).generate(newtxt)plt.imshow(wordcloud)
plt.axis('off')
plt.show()
wordcloud.to_file('wc.png')

4.2 制作最近评论数条形图与折线图

# encoding: utf-8
import csv
import pyecharts.options as opts
from pyecharts.charts import Bar
from pyecharts.globals import ThemeTypeclass DrawBar(object):"""绘制柱形图类"""def __init__(self):"""创建柱状图实例，并设置宽高和风格"""self.bar = Bar(init_opts=opts.InitOpts(width='1500px', height='700px', theme=ThemeType.LIGHT))def add_x(self):"""为图形添加X轴数据"""with open('time1.csv') as csvfile:reader = csv.reader(csvfile)x = [str(row[0]) for row in reader]print(x)self.bar.add_xaxis(xaxis_data=x,)def add_y(self):with open('time1.csv') as csvfile:reader = csv.reader(csvfile)y1 = [float(row[1]) for row in reader]print(y1)"""为图形添加Y轴数据，可添加多条"""self.bar.add_yaxis(  # 第一个Y轴数据series_name="评论数",  # Y轴数据名称y_axis=y1,  # Y轴数据label_opts=opts.LabelOpts(is_show=True,color="black"),  # 设置标签bar_max_width='100px',  # 设置柱子最大宽度)def set_global(self):"""设置图形的全局属性"""#self.bar(width=2000,height=1000)self.bar.set_global_opts(title_opts=opts.TitleOpts(  # 设置标题title='乔家的儿女近日评论统计',title_textstyle_opts=opts.TextStyleOpts(font_size=35)),tooltip_opts=opts.TooltipOpts(  # 提示框配置项（鼠标移到图形上时显示的东西）is_show=True,  # 是否显示提示框trigger="axis",  # 触发类型（axis坐标轴触发，鼠标移到时会有一条垂直于X轴的实线跟随鼠标移动，并显示提示信息）axis_pointer_type="cross"# 指示器类型（cross将会生成两条分别垂直于X轴和Y轴的虚线，不启用trigger才会显示完全）),toolbox_opts=opts.ToolboxOpts(),  # 工具箱配置项(什么都不填默认开启所有工具))def draw(self):"""绘制图形"""self.add_x()self.add_y()self.set_global()self.bar.render('DrawBar.html')  # 将图绘制到 test.html 文件内，可在浏览器打开def run(self):"""执行函数"""self.draw()if __name__ == '__main__':app = DrawBar()app.run()
DrawBar

4.3 制作每小时评论条形图与折线图

# encoding: utf-8
# encoding: utf-8
import csv
import pyecharts.options as opts
from pyecharts.charts import Bar
from pyecharts.globals import ThemeTypeclass DrawBar(object):"""绘制柱形图类"""def __init__(self):"""创建柱状图实例，并设置宽高和风格"""self.bar = Bar(init_opts=opts.InitOpts(width='1500px', height='700px', theme=ThemeType.MACARONS))def add_x(self):"""为图形添加X轴数据"""str_name1 = '点'with open('time2.csv') as csvfile:reader = csv.reader(csvfile)x = [str(row[0] + str_name1) for row in reader]print(x)self.bar.add_xaxis(xaxis_data=x)def add_y(self):with open('time2.csv') as csvfile:reader = csv.reader(csvfile)y1 = [int(row[1]) for row in reader]print(y1)"""为图形添加Y轴数据，可添加多条"""self.bar.add_yaxis(  # 第一个Y轴数据series_name="评论数",  # Y轴数据名称y_axis=y1,  # Y轴数据label_opts=opts.LabelOpts(is_show=False),  # 设置标签bar_max_width='50px',  # 设置柱子最大宽度)def set_global(self):"""设置图形的全局属性"""#self.bar(width=2000,height=1000)self.bar.set_global_opts(title_opts=opts.TitleOpts(  # 设置标题title='扫黑风暴各时间段评论统计',title_textstyle_opts=opts.TextStyleOpts(font_size=35)),tooltip_opts=opts.TooltipOpts(  # 提示框配置项（鼠标移到图形上时显示的东西）is_show=True,  # 是否显示提示框trigger="axis",  # 触发类型（axis坐标轴触发，鼠标移到时会有一条垂直于X轴的实线跟随鼠标移动，并显示提示信息）axis_pointer_type="cross"# 指示器类型（cross将会生成两条分别垂直于X轴和Y轴的虚线，不启用trigger才会显示完全）),toolbox_opts=opts.ToolboxOpts(),  # 工具箱配置项(什么都不填默认开启所有工具))def draw(self):"""绘制图形"""self.add_x()self.add_y()self.set_global()self.bar.render('DrawBar2.html')  # 将图绘制到 test.html 文件内，可在浏览器打开def run(self):"""执行函数"""self.draw()if __name__ == '__main__':app = DrawBar()app.run()

4.4 制作最近评论数饼图

import csvfrom pyecharts import options as opts
from pyecharts.charts import Pie
from random import randintfrom pyecharts.globals import ThemeTypewith open('time1.csv') as csvfile:reader = csv.reader(csvfile)x = [str(row[0]) for row in reader]print(x)
with open('time1.csv') as csvfile:reader = csv.reader(csvfile)y1 = [float(row[1]) for row in reader]print(y1)num = y1
lab = x
(Pie(init_opts=opts.InitOpts(width='1700px',height='450px',theme=ThemeType.LIGHT))#默认900，600.set_global_opts(title_opts=opts.TitleOpts(title="乔家的儿女近日评论统计",title_textstyle_opts=opts.TextStyleOpts(font_size=27)),legend_opts=opts.LegendOpts(pos_top="10%", pos_left="1%",# 图例位置调整),).add(series_name='',center=[280, 270], data_pair=[(j, i) for i, j in zip(num, lab)])#饼图.add(series_name='',center=[845, 270],data_pair=[(j,i) for i,j in zip(num,lab)],radius=['40%','75%'])#环图.add(series_name='', center=[1380, 270],data_pair=[(j, i) for i, j in zip(num, lab)], rosetype='radius')#南丁格尔图
).render('pie_pyecharts.html')

4.5 制作每小时评论饼图

import csvfrom pyecharts import options as opts
from pyecharts.charts import Pie
from random import randintfrom pyecharts.globals import ThemeTypestr_name1 = '点'with open('time2.csv') as csvfile:reader = csv.reader(csvfile)x = [str(row[0]+str_name1) for row in reader]print(x)
with open('time2.csv') as csvfile:reader = csv.reader(csvfile)y1 = [int(row[1]) for row in reader]print(y1)num = y1
lab = x
(Pie(init_opts=opts.InitOpts(width='1650px',height='500px',theme=ThemeType.LIGHT,))#默认900，600.set_global_opts(title_opts=opts.TitleOpts(title="乔家的儿女每小时评论统计",title_textstyle_opts=opts.TextStyleOpts(font_size=27)),legend_opts=opts.LegendOpts(pos_top="8%", pos_left="4%",# 图例位置调整),).add(series_name='',center=[250, 300], data_pair=[(j, i) for i, j in zip(num, lab)])#饼图.add(series_name='',center=[810, 300],data_pair=[(j,i) for i,j in zip(num,lab)],radius=['40%','75%'])#环图.add(series_name='', center=[1350, 300],data_pair=[(j, i) for i, j in zip(num, lab)], rosetype='radius')#南丁格尔图
).render('pie_pyecharts2.html')

4.6 制作观看时间区间评论统计饼图

# coding=gbk
import csvfrom pyecharts import options as opts
from pyecharts.globals import ThemeType
from sympy.combinatorics import Subset
from wordcloud import WordCloudwith open('data.csv') as csvfile:reader = csv.reader(csvfile)data2 = [int(row[1].strip('')[0:2]) for row in reader]#print(data2)
print(type(data2))#先变成集合得到seq中的所有元素，避免重复遍历
set_seq = set(data2)
list = []
for item in set_seq:list.append((item,data2.count(item)))  #添加元素及出现个数
list.sort()
print(type(list))
#print(list)with open("time2.csv", "w+", newline='', encoding='utf-8') as f:writer = csv.writer(f, delimiter=',')for i in list:                # 对于每一行的，将这一行的每个元素分别写在对应的列中writer.writerow(i)n = 4#分成n组
m = int(len(list)/n)
list2 = []
for i in range(0, len(list), m):list2.append(list[i:i+m])print("凌晨 : ",list2[0])
print("上午 : ",list2[1])
print("下午 : ",list2[2])
print("晚上 : ",list2[3])with open('time2.csv') as csvfile:reader = csv.reader(csvfile)y1 = [int(row[1]) for row in reader]print(y1)n =6
groups = [y1[i:i + n] for i in range(0, len(y1), n)]print(groups)x=['凌晨','上午','下午','晚上']
y1=[]
for y1 in groups:num_sum = 0for groups in y1:num_sum += groupsprint(x)
print(y1)import csvfrom pyecharts import options as opts
from pyecharts.charts import Pie
from random import randintstr_name1 = '点'num = y1
lab = x
(Pie(init_opts=opts.InitOpts(width='1500px',height='450px',theme=ThemeType.LIGHT))#默认900，600.set_global_opts(title_opts=opts.TitleOpts(title="乔家的儿女观看时间区间评论统计", title_textstyle_opts=opts.TextStyleOpts(font_size=30)),legend_opts=opts.LegendOpts(pos_top="8%",  # 图例位置调整),).add(series_name='',center=[260, 270], data_pair=[(j, i) for i, j in zip(num, lab)])#饼图.add(series_name='',center=[1230, 270],data_pair=[(j,i) for i,j in zip(num,lab)],radius=['40%','75%'])#环图.add(series_name='', center=[750, 270],data_pair=[(j, i) for i, j in zip(num, lab)], rosetype='radius')#南丁格尔图
).render('pie_pyecharts3.html')

4.7 制作主演提及占比饼图

import csvimport numpy as np
import re
import jieba
from matplotlib.pyplot import scatter
from wordcloud import WordCloud
from matplotlib import pyplot as plt
from PIL import Image# 上面的包自己安装，不会的就百度f = open('content.txt', 'r', encoding='utf-8')  # 这是数据源，也就是想生成词云的数据
words = f.read()  # 读取文件
f.close()  # 关闭文件，其实用with就好，但是懒得改了name=["白宇","宋祖儿","毛晓彤","张晚意","周翊然","刘钧","李佳航",'朱珠']print(name)
count=[float(words.count("白宇")),float(words.count("宋祖儿")),float(words.count("毛晓彤")),float(words.count("张晚意")),float(words.count("周翊然")),float(words.count("刘钧")),float(words.count("李佳航")),float(words.count("朱珠"))]
print(count)import csvfrom pyecharts import options as opts
from pyecharts.charts import Pie
from random import randintfrom pyecharts.globals import ThemeTypenum = count
lab = name
(Pie(init_opts=opts.InitOpts(width='1650px',height='450px',theme=ThemeType.LIGHT))#默认900，600.set_global_opts(title_opts=opts.TitleOpts(title="乔家的儿女主演提及占比",title_textstyle_opts=opts.TextStyleOpts(font_size=27)),legend_opts=opts.LegendOpts(pos_top="3%", pos_left="33%",# 图例位置调整),).add(series_name='',center=[280, 270], data_pair=[(j, i) for i, j in zip(num, lab)])#饼图.add(series_name='',center=[800, 270],data_pair=[(j,i) for i,j in zip(num,lab)],radius=['40%','75%'])#环图.add(series_name='', center=[1300, 270],data_pair=[(j, i) for i, j in zip(num, lab)], rosetype='radius')#南丁格尔图
).render('pie_pyecharts4.html')

4.8 评论内容情感分析

import numpy as np
from snownlp import SnowNLP
import matplotlib.pyplot as pltf = open('content.txt', 'r', encoding='UTF-8')
list = f.readlines()
sentimentslist = []
for i in list:s = SnowNLP(i)print(s.sentiments)sentimentslist.append(s.sentiments)
plt.hist(sentimentslist, bins=np.arange(0, 1, 0.01), facecolor='g')
plt.xlabel('Sentiments Probability')
plt.ylabel('Quantity')
plt.title('Analysis of Sentiments')
plt.show()

python爬虫取腾讯视频评论相关推荐

python实现qq登录腾讯视频_Python爬取腾讯视频评论的思路详解
一.前提条件安装了Fiddler了(用于抓包分析) 谷歌或火狐浏览器如果是谷歌浏览器,还需要给谷歌浏览器安装一个SwitchyOmega插件,用于代理服务器有Python的编译环境,一般选择Py ...
Python爬虫爬取腾讯视频评论内容
Python爬虫爬取腾讯视频评论内容腾讯视频评论内容在源码里是不可见,所以要使用抓包分析. 可安装fiddler代理服务器实现. 下面爬了腾讯视频评论内容(不包括回复),及发表评论者的id.昵称和 ...
python爬取腾讯视频弹幕_用Python爬取腾讯视频弹幕
原标题:用Python爬取腾讯视频弹幕 via:菜J学Python 1.网页分析本文以爬取<脱口秀大会第3季>最后一期视频弹幕为例,首先通过以下步骤找到存放弹幕的真实url. 通过删减 ...
python爬取b站评论_学习笔记(1):写了个python爬取B站视频评论的程序
学习笔记(1):写了个python爬取B站视频评论的程序 import requests import json import os table='fZodR9XQDSUm21yCkr6zBqiveY ...
python爬b站评论_学习笔记(1):写了个python爬取B站视频评论的程序
学习笔记(1):写了个python爬取B站视频评论的程序 import requests import json import os table='fZodR9XQDSUm21yCkr6zBqiveY ...
python爬取腾讯视频会员V力值
python爬取腾讯视频会员V力值练练手,只需要将cookies改成自己的便即可运行. from bs4 import BeautifulSoup import requests import r ...
谷歌爬虫ua_Python爬虫，实战爬取腾讯视频评论！
文章目录: 一.前提条件二.分析思路三.代码编写四.结果展示一.前提条件安装了Fiddler了(用于抓包分析) 谷歌或火狐浏览器如果是谷歌浏览器,还需要给谷歌浏览器安装一个SwitchyO ...
python抓取腾讯视频弹幕_Python实战 | 如何抓取腾讯视频弹幕
原标题:Python实战 | 如何抓取腾讯视频弹幕当代年轻人的快乐是网络给的. 如果有人吐槽周末太无聊,他们一定会反驳: 是追剧不香吗? 是吃鸡不好玩吗? 周末辣么短,怎么会无聊呢? 诚然,追剧和游 ...
Python 爬取腾讯电视剧评论
定向爬取腾讯电视剧评论本例思路:打开评论页面,通过fiddler提取加载评论页面的网址,对比分析url,构造内容和用户pattern,然后爬取输出. 1,打开电视剧如果爱页面https://v.qq ...

python爬虫取腾讯视频评论