python爬虫——Ajax-get

1.Ajax-get方法抓取豆瓣

~~~ 1.1 准备工作

搜索‘豆瓣电影’-》排行榜-》选择一个类别（此处笔者选的动作）-》向下滑动可以看见最右边的滚动条滑到底部会弹回上面，此页面中就含有Ajax接口

用fidder抓包看json格式的信息

url解读：

https://movie.douban.com/j/chart/top_list?type=5&interval_id=100%3A90&action=&start=20&limit=20

~~~ 1.2编写代码

import urllib.request
import urllib.parseurl = 'https://movie.douban.com/j/chart/top_list?type=5&interval_id=100%3A90&action=&'
page = int(input("请输入你想要第几页："))
number = 5  # 设定每一页5条数据
# 构建get参数
data = {'start': (page - 1) * number,'limit': number,
}
# 将字典转化为query_string
query_string = urllib.parse.urlencode(data)
# 修改url
url += query_string
# 伪装头部
headers = {'User-Agent': 'Mozilla/5.0 (Windows NT 10.0; WOW64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/74.0.3729.131 Safari/537.36',
}
# 构建请求对象
request = urllib.request.Request(url=url, headers=headers)
# 发出请求
response = urllib.request.urlopen(request)
print(response.read().decode())

抓取的5部电影和前面一样

[{"rating":["9.4","50"],"rank":1,"cover_url":"https://img3.doubanio.com\/view\/photo\/s_ratio_poster\/public\/p511118051.jpg","is_playable":true,"id":"1295644","types":["剧情","动作","犯罪"],"regions":["法国"],"title":"这个杀手不太冷","url":"https:\/\/movie.douban.com\/subject\/1295644\/","release_date":"1994-09-14","actor_count":20,"vote_count":1515720,"score":"9.4","actors":["让·雷诺","娜塔莉·波特曼","加里·奥德曼","丹尼·爱罗","彼得·阿佩尔","迈克尔·巴达鲁科","艾伦·格里尼","伊丽莎白·瑞根","卡尔·马图斯维奇","弗兰克·赛格","麦温","乔治·马丁","罗伯特·拉萨多","亚当·布斯奇","马里奥·托迪斯科","萨米·纳塞利","让·雨果·安格拉德","埃莱娜·卡多纳","大塚明夫","沈晓谦"],"is_watched":false},{"rating":["9.2","45"],"rank":2,"cover_url":"https://img3.doubanio.com\/view\/photo\/s_ratio_poster\/public\/p462657443.jpg","is_playable":true,"id":"1851857","types":["剧情","动作","科幻","犯罪","惊悚"],"regions":["美国","英国"],"title":"蝙蝠侠：黑暗骑士","url":"https:\/\/movie.douban.com\/subject\/1851857\/","release_date":"2008-07-14","actor_count":39,"vote_count":629899,"score":"9.2","actors":["克里斯蒂安·贝尔","希斯·莱杰","艾伦·艾克哈特","迈克尔·凯恩","玛吉·吉伦哈尔","加里·奥德曼","摩根·弗里曼","莫尼克·加布里埃拉·库尔内","罗恩·迪恩","基里安·墨菲","黄经汉","内斯特·卡博内尔","埃里克·罗伯茨","里特奇·科斯特","安东尼·迈克尔·豪尔","基斯·斯扎拉巴基克","柯林·麦克法兰","约书亚·哈尔托","美琳达·麦格劳","内森·甘宝","迈克尔·维约","迈克尔·斯托扬诺夫","威廉·斯米利","丹尼·高德林","迈克尔·加·怀特","马修·奥尼尔","威廉·菲克纳","欧鲁米基·欧拉乌米","格雷格·比姆","爱利克·赫尔曼","毕崔斯·罗森","文森佐·尼克力","陈冠希","罗南·萨莫尔斯","大卫·达斯马齐连","玛丽扎·卡布蕾拉","理查德·迪兰","利萨·麦克阿里斯特","Adam Kalesperis"],"is_watched":false},{"rating":["9.2","45"],"rank":3,"cover_url":"https://img3.doubanio.com\/view\/photo\/s_ratio_poster\/public\/p1910825503.jpg","is_playable":true,"id":"1291552","types":["剧情","动作","奇幻","冒险"],"regions":["美国","新西兰"],"title":"指环王3：王者无敌","url":"https:\/\/movie.douban.com\/subject\/1291552\/","release_date":"2004-03-12","actor_count":38,"vote_count":463426,"score":"9.2","actors":["维果·莫腾森","伊利亚·伍德","西恩·奥斯汀","丽芙·泰勒","伊恩·麦克莱恩","奥兰多·布鲁姆","凯特·布兰切特","米兰达·奥图","安迪·瑟金斯","雨果·维文","多米尼克·莫纳汉","比利·博伊德","马尔顿·索克斯","卡尔·厄本","克里斯托弗·李","约翰·瑞斯-戴维斯","阿兰·霍华德","伊安·霍姆","萨德文·布罗菲","布雷特·麦肯齐","劳伦斯·马克奥雷","Maisy McLeod-Riera","乔尔·托贝克","萨拉·贝克","杰德·布罗菲","姜广涛","彼得·杰克逊","张磊","伯纳德·希尔","约翰·贝西","Sarah McLeod","Noel Appleby","阿利斯泰尔·布朗宁","肖恩·宾","佩特·史密斯","大卫·文翰","约翰·诺贝尔","Harry Sinclair"],"is_watched":false},{"rating":["9.2","50"],"rank":4,"cover_url":"https://img3.doubanio.com\/view\/photo\/s_ratio_poster\/public\/p2215886505.jpg","is_playable":true,"id":"1295399","types":["动作","冒险","剧情"],"regions":["日本"],"title":"七武士","url":"https:\/\/movie.douban.com\/subject\/1295399\/","release_date":"1954-04-26","actor_count":20,"vote_count":118242,"score":"9.2","actors":["三船敏郎","志村乔","稻叶义男","宫口精二","千秋实","加东大介","木村功","本间文子","小杉义男","高木新平","中岛春雄","小川虎之助","东野英治郎","土屋嘉男","高堂国典","津岛惠子","藤原釜足","仲代达矢","左卜全","马野都留子"],"is_watched":false},{"rating":["9.1","45"],"rank":5,"cover_url":"https://img9.doubanio.com\/view\/photo\/s_ratio_poster\/public\/p909265336.jpg","is_playable":true,"id":"1291572","types":["剧情","动作","奇幻","冒险"],"regions":["美国","新西兰"],"title":"指环王2：双塔奇兵","url":"https:\/\/movie.douban.com\/subject\/1291572\/","release_date":"2003-04-25","actor_count":29,"vote_count":429017,"score":"9.1","actors":["伊利亚·伍德","西恩·奥斯汀","伊恩·麦克莱恩","维果·莫腾森","奥兰多·布鲁姆","克里斯托弗·李","丽芙·泰勒","安迪·瑟金斯","雨果·维文","卡尔·厄本","凯特·布兰切特","多米尼克·莫纳汉","大卫·文翰","比利·博伊德","布拉德·道里夫","伯纳德·希尔","约翰·瑞斯-戴维斯","米兰达·奥图","肖恩·宾","陆建艺","杰德·布罗菲","丹·汉纳","菲利普·格里夫","张磊","姜广涛","克雷格·帕克","约翰·诺贝尔","劳伦斯·马克奥雷","彼得·杰克逊"],"is_watched":false}]

python爬虫——Ajax-get相关推荐

Python爬虫——Ajax数据爬取
前言有时候我们在用requests抓取页面的时候,得到的结果可能和在浏览器中看到的不一样,在浏览器中可以看到正常显示的页面数据,但使用requests得到的结果并没有.这是因为在requests获得 ...
python 爬虫 ajax
在浏览器打开月光社武器大全按下F12或检查点击第一个选项卡双枪可以看到这个请求的方式是:GET X-Requested-With: XMLHttpRequest 只有ajax异步请求才会有 ...
Python 爬虫 ajax爬取马云爸爸微博内容
ajax爬取情况有时候我们在用 Requests 抓取页面的时候,得到的结果可能和在浏览器中看到的是不一样的,在浏览器中可以看到正常显示的页面数据,但是使用 Requests 得到的结果并没有,这其 ...
Python爬虫【urllib模块】
通用爬虫爬虫的一般流程 1 初始化一批URL,将这些URL放入队列 2 从队列中取出这些URL,通过DNS解析IP,对IP对应的网站下载HTML页面,保存到本地服务器中,爬取完的URL放到已爬取队列 ...
python爬虫动态加载页面_如何爬动态加载的页面？ajax爬虫你有必要掌握
通过前面几期Python爬虫的文章,不少童鞋已经可以随心所欲的爬取自己想要的数据,就算是一些页面很难分析,也可以用之前介绍的终极技能之「Selenium」+「Webdriver」解决相关问题,但无奈这 ...
python爬虫今日头条_python爬虫—分析Ajax请求对json文件爬取今日头条街拍美图
python爬虫-分析Ajax请求对json文件爬取今日头条街拍美图前言本次抓取目标是今日头条的街拍美图,爬取完成之后,将每组图片下载到本地并保存到不同文件夹下.下面通过抓取今日头条街拍美图讲解一 ...
Python爬虫4.2 — ajax(动态网页数据抓取)用法教程
Python爬虫4.2 - ajax[动态网页数据]用法教程综述 AJAX 介绍什么是AJAX 实例说明请求分析获取方式实例说明其他博文链接综述本系列文档用于对Python爬虫技术的学 ...
携程ajax,Python爬虫实战之爬取携程评论
一.分析数据源这里的数据源是指html网页?还是Aajx异步.对于爬虫初学者来说,可能不知道怎么判断,这里辰哥也手把手过一遍. 提示:以下操作均不需要登录(当然登录也可以) 咱们先在浏览器里面搜索携 ...
Python爬虫学习之API与模拟ajax请求
API的使用 --通过API向服务器请求数据优点:简单直接,高效率地批量爬取:解决动态网页的爬取. 缺点:通过API的缺点是有些网站会限制API调用的次数和频率,用户需要付费来升级成高级用户来获取更 ...
Python爬虫笔记——分析AJAX传递的JSON获取数据-初步分析动态网页
转载文章链接: Python爬虫:分析AJAX传递的JSON获取数据-初步分析动态网页(1) [4]实战:爬取动态网页的两种思路爬取新浪趣图(1) [5]实战:爬取动态网页的两种思路爬取新浪趣图(2) ...

python爬虫——Ajax-get

1.Ajax-get方法抓取豆瓣

~~~ 1.1 准备工作

~~~ 1.2编写代码

python爬虫——Ajax-get相关推荐

最新文章

热门文章