2021-04-21爬虫爬取b站直播间弹幕并制作词云

直播间弹幕接口

'https://api.live.bilibili.com/xlive/web-room/v1/dM/gethistory?roomid=' + roomid

示例：

json格式化

{"code": 0,"data": {"admin": [{"text": "哈哈哈哈哈","uid": 4760898,"nickname": "clown丶cry","uname_color": "#E17AFF","timeline": "2021-04-20 20:55:55","isadmin": 1,"vip": 0,"svip": 0,"medal": [22, "欧皇菜", "clown丶cry", 22744074, 1725515, "", 0, 6809855, 1725515, 5414290, 3, 1, 4760898],"title": ["", ""],"user_level": [38, 0, 10512625, 40644],"rank": 10000,"teamid": 0,"rnd": "1618923349","user_title": "","guard_level": 2,"bubble": 2,"bubble_color": "#19897EFF,#403F388E,#33897EFF","check_info": {"ts": 1618923355,"ct": "6E3407C2"},"lpl": 0},...],"room": [{"text": "。","uid": 1756655356,"nickname": "名称不能更改","uname_color": "","timeline": "2021-04-21 17:17:06","isadmin": 0,"vip": 0,"svip": 0,"medal": [10, "棉花花", "棉花大哥哥", 34348, 9272486, "", 0, 9272486, 9272486, 9272486, 0, 1, 1871001],"title": ["", ""],"user_level": [0, 0, 9868950, "\u003e50000"],"rank": 10000,"teamid": 0,"rnd": "-732378221","user_title": "","guard_level": 0,"bubble": 0,"bubble_color": "","check_info": {"ts": 1618996626,"ct": "DD23A60A"},"lpl": 0}, ...]},"message": "","msg": ""
}

"admin"对应舰长弹幕

"room"对应全部弹幕，每次请求只能获取最近的十条

获取弹幕列表

import requests
import jsonroomid = '313285'
url = 'https://api.live.bilibili.com/xlive/web-room/v1/dM/gethistory?roomid=' + roomiddef getNewDanmu(url):response = requests.get(url).textdanmuJson = json.loads(response)['data']['room']danmuList = []danmu = {}for dm in danmuJson:# 取出关键信息danmu['text'] = dm['text']danmu['uid'] = dm['uid']danmu['nickname'] = dm['nickname']danmu['timeline'] = dm['timeline']danmuList.append(danmu.copy())danmu.clear()return danmuList

代理池防反爬

import requests
import json
import randomroomid = '313285'
url = 'https://api.live.bilibili.com/xlive/web-room/v1/dM/gethistory?roomid=' + roomidproxies = [{"http":"http://175.42.129.120:9999"},{"http":"http://36.250.156.135:9999"},{"http":"http://163.204.242.238:9999"},{"http":"http://60.174.190.15:9999"},{"http":"http://117.95.198.132:9999"},{"http":"http://114.239.151.229:9999"},{"http":"http://36.249.48.47:9999"},{"http":"http://175.44.109.205:9999"},{"http":"http://58.22.177.224:9999"},{"http":"http://60.169.133.225:9999"},{"http":"http://1.198.42.177:9999"},{"http":"http://49.70.94.154:9999"},{"http":"http://42.238.87.240:9999"},{"http":"http://182.34.26.144:9999"},{"http":"http://112.111.77.70:9999"}
]def getNewDanmu(url):rand = random.randint(0, len(proxies)-1)response = requests.get(url, proxies[rand]).textdanmuJson = json.loads(response)['data']['room']danmuList = []danmu = {}for dm in danmuJson:danmu['text'] = dm['text']danmu['uid'] = dm['uid']danmu['nickname'] = dm['nickname']danmu['timeline'] = dm['timeline']danmuList.append(danmu.copy())danmu.clear()return danmuList

弹幕写入文件

lastDanmu = {}        # 记录最后一条弹幕
def writeDanmu(roomid, danmuList, txtTime):global lastDanmu        # 声明使用全局变量if lastDanmu in danmuList:        # 如果最后一条弹幕在弹幕列表中flushIndex = danmuList.index(lastDanmu) + 1        # 则从该条弹幕以后开始向文件中写入else:flushIndex = 0        # 否则从头开始写入# print(flushIndex)fw = open(roomid+txtTime+'.txt', 'a+', encoding='utf-8')for dm in danmuList[flushIndex: 10]:fw.write(dm['text'] + '\t' + str(dm['uid']) + '\t' + dm['nickname'] + '\t' + dm['timeline'] + '\n')fw.close()lastDanmu = danmuList[-1]

主体函数

def run(roomid):txtTime = time.strftime("%Y-%m-%d_%H_%M_%S", time.localtime())url = 'https://api.live.bilibili.com/xlive/web-room/v1/dM/gethistory?roomid=' + roomid    while True:newDanmuList = getNewDanmu(url)writeDanmu(roomid, newDanmuList,txtTime)

WordCloud模块生成词云

from wordcloud import WordCloud
import jieba
import matplotlib.pyplot as plt# 清洗数据集
filename = '219876152021-04-16_20_00_51.txt'
fr = open(filename, 'r', encoding='UTF-8')
text = ''
items = fr.readlines()
for item in items:text += item.split('\t')[0] + ' '# 屏蔽词
stopwords = {'我', '的', '了', '吧', '啊', '吗', '是', '来', '你', '好耶', 'EBPWQU6SNZ22', '2T7E9CPA7YJE', 'CBNXRD6S7H3N'}cutTxt = jieba.lcut(text)        # 可省
result = ' '.join(cutTxt)        # 可省
wc = WordCloud(background_color='white', width=800, height=800, font_path='simhei.ttf', stopwords=stopwords)
wc.generate(result)                # wc.generate(text)
plt.imshow(wc)
wc.to_file("test.png")

2021-04-21爬虫爬取b站直播间弹幕并制作词云相关推荐

【Python】大数据挖掘课程作业1——使用爬虫爬取B站评论、弹幕与UP主的投稿视频列表
[Python]大数据挖掘课程作业1--使用爬虫爬取B站评论.弹幕与UP主的投稿视频列表数据挖掘部分的基本目标是:对于指定的UP主,能够获取其投稿视频列表:对于指定的视频,能够获取其视频标签.评论( ...
Python 爬取B站视频信息弹幕信息生成词云统计
本文介绍功能:实现爬取B站视频信息(用户输入关键词).爬取弹幕信息(支持自定义天数).生成词云图完整代码地址:https://github.com/736755244/py_bilibili 一.数 ...
python3网络爬虫--爬取b站用户投稿视频信息（附源码）
文章目录一．准备工作 1．工具二．思路 1．整体思路 2．爬虫思路三．分析网页 1.分析数据加载方式 2．分词接口url 3.分析用户名(mid) 四．撰写爬虫五．得到数据六．总结上次写了 ...
【Python】使用Python做简易爬虫爬取B站评论
目录一.前言二.分析网页三.代码 1.头 2.获取根评论 3.获取子评论四.总代码五.总结一.前言 B站评论没有查找功能,就随手写了一个爬虫爬取B站评论存储到本地txt中首先需要安装py ...
爬取B站直播流 - http+flv的相关研究
参考链接 HTTP-FLV直播初探 HTTP-FLV的两种方式 Json将&符号转成了 \u0026 python requests提示警告InsecureRequestWarning 爬取B ...
python3爬取B站视频历史弹幕
python爬取B站视频历史弹幕演示演示 1.运行程序,输入Bvid和爬取日期. 2.程序运行完成后会在当前文件夹下生成一个csv格式文件. 百度网盘链接: https://pan.baidu.c ...
python爬取b站直播弹幕代码
以下是使用Python爬取B站直播弹幕的代码示例: import requests import json import timedef get_real_url(room_id):# 获取直播间的真 ...
豆酱小白与python（一）：提取B站弹幕并制作词云
文章目录 1.python环境安装 2.python语法和知识学习 4.爬取B站单个视频弹幕,并绘制词云 5.爬取B站多个视频弹幕(自定义关键词),并绘制词云 6 .1.爬取单个视频弹幕绘制词云cod ...
自定义事件详解以及实现有趣B站直播间弹幕轰炸功能
案例:B站直播间弹幕轰炸跳过按键监听办法: var event = document.createEvent('Event'); event.initEvent('input', true, true ...

2021-04-21爬虫爬取b站直播间弹幕并制作词云

直播间弹幕接口

获取弹幕列表

代理池防反爬

弹幕写入文件

主体函数

WordCloud模块生成词云

2021-04-21爬虫爬取b站直播间弹幕并制作词云相关推荐

最新文章

热门文章