前言

爬取猫眼网排行榜，2020年到2021年影片的历史日票房数据，网售占比，日期

一、分析猫眼网榜单网页和票房明细网页

# 猫眼影片总票房排行榜
http://piaofang.maoyan.com/rankings/year?v=yes

#票房明细
#******是每个电影唯一的id
http://piaofang.maoyan.com/movie/******/boxshow?wkwebview=1

二、使用步骤

1.引入库

import requests
import re
import random
import time
import pandas as pd

2.获取并分析源码函数

代码如下

###获得页面源码
def get_page(url):# User_Agent列表user_agent_list = ["Mozilla/5.0 (Windows NT 6.1; WOW64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/39.0.2171.95 Safari/537.36 OPR/26.0.1656.60","Opera/8.0 (Windows NT 5.1; U; en)","Mozilla/5.0 (Windows NT 5.1; U; en; rv:1.8.1) Gecko/20061208 Firefox/2.0.0 Opera 9.50","Mozilla/4.0 (compatible; MSIE 6.0; Windows NT 5.1; en) Opera 9.50","Mozilla/5.0 (Windows NT 6.1; WOW64; rv:34.0) Gecko/20100101 Firefox/34.0","Mozilla/5.0 (X11; U; Linux x86_64; zh-CN; rv:1.9.2.10) Gecko/20100922 Ubuntu/10.10 (maverick) Firefox/3.6.10","Mozilla/5.0 (Windows NT 6.1; WOW64) AppleWebKit/534.57.2 (KHTML, like Gecko) Version/5.1.7 Safari/534.57.2","Mozilla/5.0 (Windows NT 6.1; WOW64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/39.0.2171.71 Safari/537.36","Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/537.11 (KHTML, like Gecko) Chrome/23.0.1271.64 Safari/537.11","Mozilla/5.0 (Windows; U; Windows NT 6.1; en-US) AppleWebKit/534.16 (KHTML, like Gecko) Chrome/10.0.648.133 Safari/534.16","Mozilla/5.0 (Windows NT 6.1; WOW64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/30.0.1599.101 Safari/537.36","Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/537.11 (KHTML, like Gecko) Chrome/23.0.1271.64 Safari/537.11","Mozilla/5.0 (Windows; U; Windows NT 6.1; en-US) AppleWebKit/534.16 (KHTML, like Gecko) Chrome/10.0.648.133 Safari/534.16","Mozilla/5.0 (Windows NT 6.1; WOW64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/30.0.1599.101 Safari/537.36","Mozilla/5.0 (Windows NT 6.1; WOW64; Trident/7.0; rv:11.0) like Gecko","Mozilla/5.0 (Windows NT 6.1; WOW64) AppleWebKit/536.11 (KHTML, like Gecko) Chrome/20.0.1132.11 TaoBrowser/2.0 Safari/536.11","Mozilla/5.0 (Windows NT 6.1; WOW64) AppleWebKit/537.1 (KHTML, like Gecko) Chrome/21.0.1180.71 Safari/537.1 LBBROWSER","Mozilla/5.0 (compatible; MSIE 9.0; Windows NT 6.1; WOW64; Trident/5.0; SLCC2; .NET CLR 2.0.50727; .NET CLR 3.5.30729; .NET CLR 3.0.30729; Media Center PC 6.0; .NET4.0C; .NET4.0E; LBBROWSER)","Mozilla/4.0 (compatible; MSIE 6.0; Windows NT 5.1; SV1; QQDownload 732; .NET4.0C; .NET4.0E; LBBROWSER)","Mozilla/5.0 (compatible; MSIE 9.0; Windows NT 6.1; WOW64; Trident/5.0; SLCC2; .NET CLR 2.0.50727; .NET CLR 3.5.30729; .NET CLR 3.0.30729; Media Center PC 6.0; .NET4.0C; .NET4.0E; QQBrowser/7.0.3698.400)","Mozilla/4.0 (compatible; MSIE 6.0; Windows NT 5.1; SV1; QQDownload 732; .NET4.0C; .NET4.0E)","Mozilla/5.0 (Windows NT 5.1) AppleWebKit/535.11 (KHTML, like Gecko) Chrome/17.0.963.84 Safari/535.11 SE 2.X MetaSr 1.0","Mozilla/4.0 (compatible; MSIE 7.0; Windows NT 5.1; Trident/4.0; SV1; QQDownload 732; .NET4.0C; .NET4.0E; SE 2.X MetaSr 1.0)","Mozilla/5.0 (Windows NT 6.1; WOW64) AppleWebKit/537.36 (KHTML, like Gecko) Maxthon/4.4.3.4000 Chrome/30.0.1599.101 Safari/537.36","Mozilla/5.0 (Windows NT 6.1; WOW64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/38.0.2125.122 UBrowser/4.0.3214.0 Safari/537.36","Mozilla/5.0 (iPhone; U; CPU iPhone OS 4_3_3 like Mac OS X; en-us) AppleWebKit/533.17.9 (KHTML, like Gecko) Version/5.0.2 Mobile/8J2 Safari/6533.18.5","Mozilla/5.0 (iPod; U; CPU iPhone OS 4_3_3 like Mac OS X; en-us) AppleWebKit/533.17.9 (KHTML, like Gecko) Version/5.0.2 Mobile/8J2 Safari/6533.18.5","Mozilla/5.0 (iPad; U; CPU OS 4_2_1 like Mac OS X; zh-cn) AppleWebKit/533.17.9 (KHTML, like Gecko) Version/5.0.2 Mobile/8C148 Safari/6533.18.5","Mozilla/5.0 (iPad; U; CPU OS 4_3_3 like Mac OS X; en-us) AppleWebKit/533.17.9 (KHTML, like Gecko) Version/5.0.2 Mobile/8J2 Safari/6533.18.5","Mozilla/5.0 (Linux; U; Android 2.2.1; zh-cn; HTC_Wildfire_A3333 Build/FRG83D) AppleWebKit/533.1 (KHTML, like Gecko) Version/4.0 Mobile Safari/533.1","Mozilla/5.0 (Linux; U; Android 2.3.7; en-us; Nexus One Build/FRF91) AppleWebKit/533.1 (KHTML, like Gecko) Version/4.0 Mobile Safari/533.1","MQQBrowser/26 Mozilla/5.0 (Linux; U; Android 2.3.7; zh-cn; MB200 Build/GRJ22; CyanogenMod-7) AppleWebKit/533.1 (KHTML, like Gecko) Version/4.0 Mobile Safari/533.1","Opera/9.80 (Android 2.3.4; Linux; Opera Mobi/build-1107180945; U; en-GB) Presto/2.8.149 Version/11.10","Mozilla/5.0 (Linux; U; Android 3.0; en-us; Xoom Build/HRI39) AppleWebKit/534.13 (KHTML, like Gecko) Version/4.0 Safari/534.13","Mozilla/5.0 (BlackBerry; U; BlackBerry 9800; en) AppleWebKit/534.1+ (KHTML, like Gecko) Version/6.0.0.337 Mobile Safari/534.1+","Mozilla/5.0 (hp-tablet; Linux; hpwOS/3.0.0; U; en-US) AppleWebKit/534.6 (KHTML, like Gecko) wOSBrowser/233.70 Safari/534.6 TouchPad/1.0","Mozilla/5.0 (SymbianOS/9.4; Series60/5.0 NokiaN97-1/20.0.019; Profile/MIDP-2.1 Configuration/CLDC-1.1) AppleWebKit/525 (KHTML, like Gecko) BrowserNG/7.1.18124","Mozilla/5.0 (compatible; MSIE 9.0; Windows Phone OS 7.5; Trident/5.0; IEMobile/9.0; HTC; Titan)","UCWEB7.0.2.37/28/999","NOKIA5700/ UCWEB7.0.2.37/28/999","Openwave/ UCWEB7.0.2.37/28/999"]# 产生一个随机User-Agentproxy_list = [{'http': '110.18.2.44:11176',},]proxies = random.choice(proxy_list)headers = {# 从上面列表中随机取出一个'User-Agent': random.choice(user_agent_list)}# response = requests.get(url, headers=headers, proxies=proxies,)response = requests.get(url, headers=headers,)# print(proxies)if response.status_code == 200:return response.textelse:return 'Crawl Failed'def parse_html_ID(html):# 获得电影ID和名称pattern = re.compile('<ul.*?hrefTo,href.*?/(\d+).">.*?<p class="first-line">(\w+)</p>.*?"second-line">(\d+).*?</p>', re.S)result = re.findall(pattern, html)return resultdef parse_html_detial(html):# 获得电影日票房,占比，日期pattern = re.compile('"box".*?(\d+).*?"onlineBoxRate".*?"(.*?)".*?"showDate".*?(\w+)', re.S)result = re.findall(pattern, html)# print(result)return result

该处使用的url网络请求的数据。

3.主函数部分

直接爬取有个问题就是，如果使用单一ip，会在连续爬取十个左右的数据时，被反爬，然后无法获得数据，所以推荐，使用id，单独爬取数据。

if __name__ == "__main__":#爬取每部电影的IDhtml_id = get_page('http://piaofang.maoyan.com/rankings/year?_v_=yes')data_id = parse_html_ID(html_id)# print(data)for id in data_id:#id, name, year# print(id)year = int(id[2])if(year == 2020 or year == 2021):detial_url = "http://piaofang.maoyan.com/movie/" + id[0] + "/boxshow?wkwebview=1"# print(year)print(id)# print(detial_url)html_detial = get_page(detial_url)name = re.findall('<div.*?"info-title-bar">(\w+)</div>', html_detial)name = str(name)# print(name)#票房，占比，日期date_ticket = parse_html_detial(html_detial)if len(date_ticket) == 0:print("##########数据爬取失败##########")break##读取数据写入csvbox_office = []percent = []date = []for item in date_ticket:# print(item)# print(item[0])# print(item[1])# print(item[2])box_office.append(int(item[0]))percent.append(item[1])date.append(item[2])dataframe = pd.DataFrame({'电影名称': name,'综合票房': box_office,'网售占比': percent,'日期': date,})dataframe.to_csv('data'+name+'.csv', index=False, sep=',', encoding='gbk')print('data'+name+'.csv')#随机延时，降低爬取速度time.sleep(random.random() * 3)

若需要额外的榜单外的电影，可以直接去猫眼查询电影名称，

然后查看网页源码，使用ctrl + f 搜索 movieid，就可以知道，电影对应的ID号，

直接在下面的代码中修改id，替换上面的主函数部分即可。

if __name__ == "__main__":# #爬取每部电影的ID# html_id = get_page('http://piaofang.maoyan.com/rankings/year?_v_=yes')# data_id = parse_html_ID(html_id)# # print(data)# for id in data_id:#     #id, name, year#     print(id)#     year = int(id[2])#     if(year == 2020 or year == 2021):# 一点就到家# id = '1331267'id = '1222234'detial_url = "http://piaofang.maoyan.com/movie/" + id + "/boxshow?wkwebview=1"print(id)print(detial_url)html_detial = get_page(detial_url)####获取电影名称name = re.findall('<div.*?"info-title-bar">(\w+)</div>', html_detial)name = str(name)print(name)#票房，占比，日期date_ticket = parse_html_detial(html_detial)if len(date_ticket) == 0:print("##########数据爬取失败##########")##读取数据写入csvbox_office = []percent = []date = []for item in date_ticket:# print(item)# print(item[0])# print(item[1])# print(item[2])box_office.append(int(item[0]))percent.append(item[1])date.append(item[2])dataframe = pd.DataFrame({'电影名称': name,'综合票房': box_office,'网售占比': percent,'日期': date,})dataframe.to_csv('data'+name+'.csv', index=False, sep=',', encoding='gbk')print('data'+name+'.csv')#连续爬取时，随机延时，降低爬取速度，防止ip被封# time.sleep(random.random() * 3)

总结

以上就是今天要讲的内容，本文仅仅简单介绍了猫眼网的票房数据爬取，其他网页的爬取，可以举一反三。

猫眼网历史日票房数据爬取相关推荐

python爬虫实战三：近十年中国电影票房数据爬取与分析
近十年中国电影票房数据爬取与分析前言爬取分析十年top10 年度top5 每年电影数每年总票房二八原则代码与数据前言这篇文章主要讲述的是近十年(2010-2019)中国电影票房数据的 ...
项目三：近10年来中国电影票房数据爬取分析
近10年来中国电影票房数据爬取分析前言数据采集与存储数据清洗和简单分析引入库,导入数据近10年top 年度top5 每年电影数每年总票房结论二八原则 end 点击跳转到总目录前言这 ...
python-东方财富网贴吧评论数据爬取
python-东方财富网贴吧文本数据爬取分享一下写论文时爬数据用到的代码,有什么问题或者改善的建议的话小伙伴们一起评论区讨论. def getzssz399001DATA(x,y): #爬取x至y页 ...
十年电影票房数据爬取与分析 | 免费数据教程
3月8日妇女节,我很期待的超级英雄电影<惊奇队长>上映了,票房表现很快过亿,但大众口碑却让人失望. 一个有趣且常见的现象是,隔壁获奖无数,口碑爆炸的<绿皮书>,票房却远远不如& ...
历史天气信息数据爬取
北京市2020年1月份历史天气爬取视频教程(讲的阔以的) 1 确定目标网址 1.1 目标网址 [2345天气] 1.2 目标数据北京2020年一月份 2 分析网页分析过程大家去看视频吧,讲的很好 ...
链家网杭州租房信息数据爬取+数据分析
参考https://mp.weixin.qq.com/s/vvZ2yBb2eMKP800LUPoAWg 需求分析¶ 过去一个月,全国热点城市的房租如脱缰野马.一线的房租同比涨了近20%.一夜醒来,无产 ...
中金所金融期货日统计数据爬取
基本思路构造URL下载每个交易日统计的附件csv文件,提取csv文件中的次月交割的股指期货统计数据代码块 from io import StringIO import csv import url ...
全国历史天气查询/历史天气预报查询——全国各月份数据爬取
全国历史天气查询/历史天气预报查询--全国各月份数据爬取效果图1 目标爬取数据图2 最终实验效果实验效果:最终可将官网已有的数据进行爬取整理,共363个城市,从2011年1月--至今数据已上 ...
大数据信息资料采集：文化公众号槽边往事历史文章搜集评论爬取
大数据信息资料采集:文化公众号槽边往事历史文章搜集评论爬取大数据信息资料采集公众号历史文章采集公众号评论爬取微信公众号历史文章导出抓取微信公众号所有文章. 公众号文章抓取工具抓取公众号所有 ...

猫眼网历史日票房数据爬取

文章目录

前言