基于Python猫眼票房TOP100电影数据抓取

使用requests库抓取数据
BeatifulSoup解析URL，将数据存入csv文件
处理数据，生成图表

本次爬取数据使用Python语言在jupyter notebook上爬取的，并使用数据生成了一些图表。

使用requests库抓取数据

本段代码提取（www.maoyan.com/board/4）网页中所有信息.

import requests as req
import re
from bs4 import BeautifulSoup as bs
import time as tidef link(url):header = {"User-Agent" : "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/94.0.4606.81 Safari/537.36 Edg/94.0.992.50","cookie" : "__mta=49569177.1634546762447.1635751930589.1635755537866.189; __mta=49569177.1634546762447.1635755537866.1636089904755.190; _lxsdk_cuid=17c92944154c8-0976821c9ff2ba-513c1f42-1fa400-17c9294415486; _lx_utm=utm_source=Baidu&utm_medium=organic; Hm_lvt_703e94591e87be68cc8da0da7cbd0be2=1634709537,1634709791,1634709795,1635568974; __mta=246700987.1634548728665.1634549353008.1635568973951.3; uuid_n_v=v1; uuid=2B2B05903C7611EC9D395F5B080ABF0B13E32B82F40F483EA9968669224339F7; _csrf=04f1258f2b355ec8fc0e1f4b729b4545d23fc1610830265d79846b5d5b3fc7cf; _lxsdk=2B2B05903C7611EC9D395F5B080ABF0B13E32B82F40F483EA9968669224339F7; Hm_lpvt_703e94591e87be68cc8da0da7cbd0be2=1636089905; _lxsdk_s=17cee6e2ca2-e12-d9c-bd9||14"}res = req.get(url,headers = header)if res.status_code == 200:return bs(res.text,"lxml")return None

上面的代码中，cookie需要自己去更改。
打开猫眼界面（www.maoyan.com/board/4），cookie方式如图所示

BeatifulSoup解析URL，将数据存入csv文件

Title = []
Director = []
Actors = []
Rating = []
Income = []
Duration = []
Type = []
Region = []
Release = []
main_Type = []
Score = []for i in range(0,100,10):url = "https://maoyan.com/board/4?offset=" + str(i)movies = link(url).find_all("dd")for i in movies:img = i.find("img",class_ = "board-img").get("data-src")num = i.find("i").textname = i.find("a").get("title")actor = re.findall("主演：(.*)",i.find("p",class_ = "star").text)[0]release = re.findall("上映时间：(.*)",i.find("p",class_ = "releasetime").text)[0].split('(')[0].split('-')[0]      url1 = "https://maoyan.com" + i.find("p",class_ = "name").a.get("href")score = i.find("i",class_ = "integer").text + i.find("i",class_ = "fraction").textmovie = link(url1)       ti.sleep(2)director = movie.find("a",class_= "name").text.replace("\n","").replace(" ","")type1 = movie.find("li",class_="ellipsis").text.replace("\n"," ")type2 = movie.find("a",class_="text-link").text.strip()last_income = movie.find_all("div",class_="mbox-name")duration = re.findall(r'\d+',movie.find_all("li",class_="ellipsis")[1].text)region = movie.find_all("li",class_="ellipsis")[1].text.split('/')[0].strip()Region.append(region)Title.append(name)Director.append(director)Actors.append(actor)Rating.append(num)Release.append(release)main_Type.append(type2)Score.append(score)Duration.append(duration[0]+"分钟")         Type.append(type1)if(movie.find("div",class_="mbox-name")):income = Nonenum_temp = 0for income in  last_income:num_temp = num_temp+1                passif income:content = income.getText()income_temp = movie.find_all("div",class_="mbox-name")[num_temp-2].textelse:Income.append("暂无")continueif(income_temp=="暂无"):Income.append(income_temp)else:Income.append(income_temp+"万")print("Grab complete")

将数据存入csv文件

import pandas as pd
df = pd.DataFrame({'Rating':Rating,'Title':Title,'Director':Director,'Actors':Actors,'Income':Income,'Score':Score,'Duration':Duration,'Release':Release,'Type':Type,'Region':Region,'Main Type':main_Type})
df = df.set_index('Rating')
df.to_csv('./test.csv',encoding='utf_8_sig')
print('ok')

csv文件

处理数据，生成图表

读取csv文件

import csv
from datetime import datetime
import matplotlib.pyplot as plt
import pandas as pd
import seaborn.apionly as sns
from pylab import *
filename = 'test.csv'
data = pd.read_csv(filename)

各个区域电影占比的饼状图

readtxt = (",".join( data.Region.values))
readlist = readtxt.split(',')
dict_region = {}for every_region in readlist:if every_region in dict_region:dict_region[every_region] += 1else:dict_region[every_region] = 1region = []
num_region = []
region_value = 0
for key in dict_region:if(dict_region[key]<4):region_key = "其他"region_value += dict_region[key]else:region.append(key)        num_region.append(dict_region[key])region.append(region_key)
num_region.append(region_value)    plt.rcParams['font.sans-serif']=['SimHei']
plt.rcParams['axes.unicode_minus'] = False
plt.figure(figsize = (20,8),dpi = 100)color = ['#b1d1fc','#75bbfd','#2976bb','#acc2d9','#a2cffe','#c5c9c7','#276ab3','#b7c9e2','#49759c']
plt.title('Region')
explode = [0.04,0.05,0.04,0.04,0.04,0.04,0.04,0.04,0.04,0.04,0.04]
plt.pie(x = num_region,labels = region,colors = color,autopct = '%0.1f%%',shadow = False,pctdistance = 0.8,explode = explode,startangle = 286)plt.legend(loc = "upper right",bbox_to_anchor = (0.7, 0, 0.5, 1))
plt.savefig('./region.png')
plt.show()

饼状图展示

代码资源已经上传
https://download.csdn.net/download/weixin_43165086/37368600

基于Python猫眼票房TOP100电影数据抓取相关推荐

基于Python爬虫的股票成交量数据抓取分析系统
目录数据获取 2 1.1. 实验环境搭建 2 1.2. 抓取数据 2 1.2.1. 新浪财经 3 1.2.2. 网易财经 6 1.2.3. 东方财富 12 1.2.4. TuShare (挖地兔) ...
如何用python抓取文献_浅谈Python爬虫技术的网页数据抓取与分析
浅谈 Python 爬虫技术的网页数据抓取与分析吴永聪 [期刊名称] <计算机时代> [年 ( 卷 ), 期] 2019(000)008 [摘要] 近年来 , 随着互联网的发展 , 如何 ...
猫眼api html,猫眼电影数据抓取
猫眼电影数据抓取实现分析 1.基准xpath: 匹配所有电影信息的节点对象列表 '//dl[@class="board-wrapper"]/dd' 2.遍历对象列表,依次获取每个电 ...
python中国大学排名爬虫写明详细步骤-Python爬虫--2019大学排名数据抓取
Python爬虫--2019大学排名数据抓取准备工作输入:大学排名URL连接输出:大学排名信息屏幕输出所需要用到的库:requests,bs4 思路获取网页信息提取网页中的内容并放到数据结 ...
Python爬虫入门实战之猫眼电影数据抓取(理论篇)
前言本文可能篇幅较长,但是绝对干货满满,提供了大量的学习资源和途径.达到让读者独立自主的编写基础网络爬虫的目标,这也是本文的主旨,输出有价值能够真正帮助到读者的知识,即授人以鱼不如授人以渔,让我们直 ...
python编程理论篇_Python爬虫入门实战之猫眼电影数据抓取(理论篇)
前言本文可能篇幅较长,但是绝对干货满满,提供了大量的学习资源和途径.达到让读者独立自主的编写基础网络爬虫的目标,这也是本文的主旨,输出有价值能够真正帮助到读者的知识,即授人以鱼不如授人以渔,让我们直 ...
python爬虫入门实战争胜法_Python爬虫入门实战之猫眼电影数据抓取(理论篇)
前言本文可能篇幅较长,但是绝对干货满满,提供了大量的学习资源和途径.达到让读者独立自主的编写基础网络爬虫的目标,这也是本文的主旨,输出有价值能够真正帮助到读者的知识,即授人以鱼不如授人以渔,让我们直 ...
Python爬虫入门实战之猫眼电影数据抓取（实战篇）
项目实战静态网页实战本节我们将为大家展现一个完整爬虫的大致过程,此次项目内容为提取猫眼电影TOP100榜中的所有电影信息并存储至CSV文件中,其首页地址为http://maoyan.com/boa ...
【Python】36氪(36kr)数据抓取 scrapy
1. 36氪(36kr)数据----写在前面今天抓取一个新闻媒体,36kr的文章内容,也是为后面的数据分析做相应的准备 36kr 让一部分人先看到未来,而你今天要做的事情确实要抓取它的过去. 网址 ...

基于Python猫眼票房TOP100电影数据抓取

基于Python猫眼票房TOP100电影数据抓取

使用requests库抓取数据

BeatifulSoup解析URL，将数据存入csv文件

处理数据，生成图表

基于Python猫眼票房TOP100电影数据抓取相关推荐

最新文章

热门文章