原文链接: scrapy 斗鱼 主播信息爬取

上一篇: scrapy 妹子图网站 全站图片爬取

下一篇: TensorFlow models 的slim 模块 使用预训练模型进行识别

api

http://capi.douyucdn.cn/api/v1/getVerticalRoom?limit=20&offect=0

创建项目

scrapy startproject douyu

创建爬虫

scrapy genspider  demo   capi.douyucdn.cn

项目结构

设置，由于api接口是手机端的，所以要设置agent

# 图片存放目录
IMAGES_STORE = 'd:/data/douyu/'#  agent 设为手机端
USER_AGENT = 'Mozilla/5.0 (iPhone; CPU iPhone OS 10_3_3 like Mac OS X) AppleWebKit/603.3.8 (KHTML, like Gecko) Mobile/14G60 MicroMessenger/6.5.18 NetType/WIFI Language/en'# 禁用robot
ROBOTSTXT_OBEY = False# 开启 pipe
ITEM_PIPELINES = {'douyu.pipelines.DouyuPipeline': 300,
}

爬虫文件 demo.py

import scrapy
import jsonclass DemoSpider(scrapy.Spider):name = 'demo'allowed_domains = ['capi.douyucdn.cn']start_urls = ['http://capi.douyucdn.cn/api/v1/getVerticalRoom?limit=20&offect=0']offset = 0def parse(self, response):data_list = json.loads(response.body)['data']if not len(data_list):returnfor data in data_list:item = {'nickname': data['nickname'],'vertical_src': data['vertical_src'],}yield itemself.offset += 20if self.offset > 1000:returnurl = f"http://capi.douyucdn.cn/api/v1/getVerticalRoom?limit=20&offect={self.offset}"print(url)yield scrapy.Request(url, callback=self.parse)

pipe文件 专用来下载图片，并且修改图片名称

# -*- coding: utf-8 -*-from scrapy.pipelines.images import ImagesPipeline
from scrapy import Request# 继承一个专用的图片处理类
class DouyuPipeline(ImagesPipeline):def get_media_requests(self, item, info):url = item['vertical_src']# print(url, '--------------------get')# yield Request(url)yield Request(url, meta={'item': item})def item_completed(self, results, item, info):# [(True, {'url': 'https://rpic.douyucdn.cn/asrpic/180809/95741_2313.jpg',# 'path': 'images/龍骑士丶我是传奇.jpg',# 'checksum': '4980363396792b4d8279b1a568713ea1'})]# 分别表示，下载成功，下载url，保存路径，和md5码# 查看下载了多少图片print(results, '---------')def file_path(self, request, response=None, info=None):item = request.meta['item']  # 通过上面的meta传递过来itemname = item['nickname']down_file_name = f"images/{name}.jpg"return down_file_name

运行

scrapy crawl demo --nolog

效果