Python爬取视频(其实是一篇福利)

窗外下着小雨，作为单身程序员的我逛着逛着发现一篇好东西，来自知乎你都用 Python 来做什么？的第一个高亮答案。

到上面去看了看，地址都是明文的，得，赶紧开始吧。

下载流式文件，requests库中请求的stream设为True就可以啦，文档在此。

先找一个视频地址试验一下：

# -*- coding: utf-8 -*-
import requestsdef download_file(url, path):with requests.get(url, stream=True) as r:chunk_size = 1024content_size = int(r.headers['content-length'])print '下载开始'with open(path, "wb") as f:for chunk in r.iter_content(chunk_size=chunk_size):f.write(chunk)if __name__ == '__main__':url = '就在原帖...'path = '想存哪都行'download_file(url, path)

遭遇当头一棒：

AttributeError: __exit__

这文档也会骗人的么！

看样子是没有实现上下文需要的__exit__方法。既然只是为了保证要让r最后close以释放连接池，那就使用contextlib的closing特性好了：

# -*- coding: utf-8 -*-
import requests
from contextlib import closingdef download_file(url, path):with closing(requests.get(url, stream=True)) as r:chunk_size = 1024content_size = int(r.headers['content-length'])print '下载开始'with open(path, "wb") as f:for chunk in r.iter_content(chunk_size=chunk_size):f.write(chunk)

程序正常运行了，不过我盯着这文件，怎么大小不见变啊，到底是完成了多少了呢？还是要让下好的内容及时存进硬盘，还能省点内存是不是：

# -*- coding: utf-8 -*-
import requests
from contextlib import closing
import osdef download_file(url, path):with closing(requests.get(url, stream=True)) as r:chunk_size = 1024content_size = int(r.headers['content-length'])print '下载开始'with open(path, "wb") as f:for chunk in r.iter_content(chunk_size=chunk_size):f.write(chunk)f.flush()os.fsync(f.fileno())

文件以肉眼可见的速度在增大，真心疼我的硬盘，还是最后一次写入硬盘吧，程序中记个数就好了：

def download_file(url, path):with closing(requests.get(url, stream=True)) as r:chunk_size = 1024content_size = int(r.headers['content-length'])print '下载开始'with open(path, "wb") as f:n = 1for chunk in r.iter_content(chunk_size=chunk_size):loaded = n*1024.0/content_sizef.write(chunk)print '已下载{0:%}'.format(loaded)n += 1

结果就很直观了：

已下载2.579129%
已下载2.581255%
已下载2.583382%
已下载2.585508%

心怀远大理想的我怎么会只满足于这一个呢，写个类一起使用吧：

# -*- coding: utf-8 -*-
import requests
from contextlib import closing
import timedef download_file(url, path):with closing(requests.get(url, stream=True)) as r:chunk_size = 1024*10content_size = int(r.headers['content-length'])print '下载开始'with open(path, "wb") as f:p = ProgressData(size = content_size, unit='Kb', block=chunk_size)for chunk in r.iter_content(chunk_size=chunk_size):f.write(chunk)p.output()class ProgressData(object):def __init__(self, block,size, unit, file_name='', ):self.file_name = file_nameself.block = block/1000.0self.size = size/1000.0self.unit = unitself.count = 0self.start = time.time()def output(self):self.end = time.time()self.count += 1speed = self.block/(self.end-self.start) if (self.end-self.start)>0 else 0self.start = time.time()loaded = self.count*self.blockprogress = round(loaded/self.size, 4)if loaded >= self.size:print u'%s下载完成\r\n'%self.file_nameelse:print u'{0}下载进度{1:.2f}{2}/{3:.2f}{4} 下载速度{5:.2%} {6:.2f}{7}/s'.\format(self.file_name, loaded, self.unit,\self.size, self.unit, progress, speed, self.unit)print '%50s'%('/'*int((1-progress)*50))

运行：

下载开始
下载进度10.24Kb/120174.05Kb 0.01% 下载速度4.75Kb/s
/
下载进度20.48Kb/120174.05Kb 0.02% 下载速度32.93Kb/s
/

看上去舒服多了。

下面要做的就是多线程同时下载了，主线程生产url放入队列，下载线程获取url：

# -*- coding: utf-8 -*-
import requests
from contextlib import closing
import time
import Queue
import hashlib
import threading
import osdef download_file(url, path):with closing(requests.get(url, stream=True)) as r:chunk_size = 1024*10content_size = int(r.headers['content-length'])if os.path.exists(path) and os.path.getsize(path)>=content_size:print '已下载'returnprint '下载开始'with open(path, "wb") as f:p = ProgressData(size = content_size, unit='Kb', block=chunk_size, file_name=path)for chunk in r.iter_content(chunk_size=chunk_size):f.write(chunk)p.output()class ProgressData(object):def __init__(self, block,size, unit, file_name='', ):self.file_name = file_nameself.block = block/1000.0self.size = size/1000.0self.unit = unitself.count = 0self.start = time.time()def output(self):self.end = time.time()self.count += 1speed = self.block/(self.end-self.start) if (self.end-self.start)>0 else 0self.start = time.time()loaded = self.count*self.blockprogress = round(loaded/self.size, 4)if loaded >= self.size:print u'%s下载完成\r\n'%self.file_nameelse:print u'{0}下载进度{1:.2f}{2}/{3:.2f}{4} {5:.2%} 下载速度{6:.2f}{7}/s'.\format(self.file_name, loaded, self.unit,\self.size, self.unit, progress, speed, self.unit)print '%50s'%('/'*int((1-progress)*50))queue = Queue.Queue()def run():while True:url = queue.get(timeout=100)if url is None:print u'全下完啦'breakh = hashlib.md5()h.update(url)name = h.hexdigest()path = 'e:/download/' + name + '.mp4'download_file(url, path)def get_url():queue.put(None)if __name__ == '__main__':get_url()for i in xrange(4):t = threading.Thread(target=run)t.daemon = Truet.start()

加了重复下载的判断，至于怎么源源不断的生产url，诸位摸索吧，保重身体！

Python爬取视频(其实是一篇福利)相关推荐

python可以爬取的内容有什么_Python爬取视频(其实是一篇福利)过程解析 Python爬虫可以爬取什么...
如何用python爬取视频网站的数据如何用python爬取js渲染加载的视频文件不是每个人都有资格说喜欢,也不是每个人都能选择伴你一生! 有哪位大神指导下,有些视频网站上的视频文件是通过 js 加载 ...
python爬取视频的工具_Python爬取视频(其实是一篇福利)
原博文 2018-01-09 00:14 − 窗外下着小雨,作为单身程序员的我逛着逛着发现一篇好东西,来自知乎你都用 Python 来做什么?的第一个高亮答案. 到上面去看了看,地址都是明文的,得, ...
python能爬视频吗_Python爬取视频(其实是一篇福利)
先找一个视频地址试验一下: # -*- coding: utf-8 -*- import requests def download_file(url, path): with requests.ge ...
Python爬取视频之爱情电影及解密TS文件和两种合并ts的方法
俗话说,兴趣所在,方能大展拳脚.so结合兴趣的学习才能事半功倍,更加努力专心,apparently本次任务是在视频网站爬取一些好看的小电影,地址不放(狗头保命)只记录过程. 实现功能: 从网站上爬取采 ...
如何用python爬取视频_介绍Python爬取哔哩哔哩视频
python视频教程栏目介绍如何爬取视频本篇文章主要给大家讲解下如实使用python 爬取哔哩哔哩中的视频,首先我是一名大数据开发工程师,爬虫只是我的一个业余爱好,喜欢爬虫的小伙伴可以一起交流.好了 ...
使用python爬取视频
写在前面:此博客仅用于参考学习,代码中部分网址已删准备工作 python burpsuite 视频爬取视频网址:https://v.qq.com/x/cover/mzc00200xnrv1u5.h ...
使用Python 爬取视频
本文章爬取视频以 ".ts" 文件为例,可从视频网站爬取相关视频碎片,最后拼接成一个完整的视频首先安装Python,安装方法可参考菜鸟教程(Python 环境搭建 | 菜鸟教程) ...
Python爬取视频之爱情电影及解密TS文件和两种合并ts！
俗话说,兴趣所在,方能大展拳脚.so结合兴趣的学习才能事半功倍,更加努力专心,apparently本次任务是在视频网站爬取一些好看的小电影,地址不放(狗头保命)只记录过程. 实现功能: 从网站上爬取采 ...
python爬取视频界面简介
一用到的库 requests库:众所周知,爬虫神器二 JSON是什么 JSON ( JavaScript Object Notation) ,是一种数据交互格式. 没有结束标签更短读写的速度更 ...

Python爬取视频(其实是一篇福利)

Python爬取视频(其实是一篇福利)相关推荐

最新文章

热门文章