python协程多线程HTMLParser下载mp3

使用HTMLParser解析html网页，顺便协程+多线程下载mp3文件

# coding=utf-8
import requests
import os
import chardet
import asyncio
import threading
from concurrent.futures import ThreadPoolExecutor
from enum import Enum
from html.parser import HTMLParsergPool = ThreadPoolExecutor(30)
gLock = threading.Lock()class EProcess(Enum):idle = 0start = 1end = 2class CParseHtml(HTMLParser):def __init__(self, url):HTMLParser.__init__(self)hName = os.path.basename(url)prefix = hName.strip(".html").strip("?")self.mlog = "log" + prefix + ".txt"if os.path.isfile(self.mlog):os.remove(self.mlog)self.mUrls = []self.mProcess = EProcess.idleself.mBasePrefix = "http://www.txxxxx.com"def pinfo(self, text, flag=False):if flag:with open(self.mlog, "a") as f:f.write(text + "\n")print(text)class CParsePageFirst(CParseHtml):def __init__(self, url):CParseHtml.__init__(self, url)self.mUl = Falsedef handle_starttag(self, tag, attrs):for kv in attrs:if len(kv) == 2:if kv[1] == "zaixianlianbo":self.mProcess = EProcess.startelif self.mProcess == EProcess.start and kv[0] == "href":ul = self.mBasePrefix + kv[1]self.mUrls.append(ul)def handle_endtag(self, tag):if self.mProcess == EProcess.start and tag == "ul":self.mProcess = EProcess.endclass CParsePageDown(CParseHtml):def __init__(self, url):CParseHtml.__init__(self, url)self.mUl = Falsedef handle_starttag(self, tag, attrs):if tag == "script":self.mProcess = EProcess.startdef handle_endtag(self, tag):if tag == "script":self.mProcess = EProcess.idledef handle_data(self, data):if (self.mProcess == EProcess.start):# print("data = {}".format(data))if data.find("getAspParas") != -1:httpPrefix = "http://mp3"httpSuffix = ".mp3"idxStart = data.find(httpPrefix)idxEnd = data.find(httpSuffix)if idxStart != -1 and idxEnd != -1:mp3Url = data[idxStart:idxEnd + len(httpSuffix)]self.mUrls.append(mp3Url)async def getHtml(url, loop):global gPoolresult = await loop.run_in_executor(gPool, requests.get, url)if (result.status_code == 200):en = chardet.detect(result.content)# print("en = {}".format(en))encode = en["encoding"]if encode.lower() == "gb2312":encode = "gb18030"content = result.content.decode(encode)# base = os.path.basename(url)# with open(base.strip("?"), "w") as f:#     f.write(content)return 200, contentreturn 404, ""async def dealOne(url, loop):code, content = await getHtml(url, loop)if code == 200:down = CParsePageDown(url)down.feed(content)if len(down.mUrls) != 1:# down.pinfo("downOne:error len(murls) == {}".len(down.mUrls))passelse:mp3Url = down.mUrls[0]saveName = os.path.basename(mp3Url)if not os.path.isfile(saveName):global gPoolresult = await loop.run_in_executor(gPool, requests.get, mp3Url)if (result.status_code == 200):with open(saveName, "wb") as f:f.write(result.content)global gLockgLock.acquire()print("down ok {}".format(saveName))gLock.release()return saveName, Truereturn url, Falseasync def dealAll(url, loop):code, content = await getHtml(url, loop)if code == 200:parseFirst = CParsePageFirst(url)parseFirst.feed(content)ts = []for ul in parseFirst.mUrls:task = asyncio.ensure_future(dealOne(ul, loop))ts.append(task)rs = await asyncio.gather(*ts)return rselse:print("error:dealAll")return []if __name__ == "__main__":loop = asyncio.get_event_loop()try:url = "http://www.txxxxxxm/books/15051.html"rs = loop.run_until_complete(dealAll(url, loop))okCount = 0failCount = 0for name, success in rs:if success:okCount += 1else:failCount += 1print("fail name = {}".format(name))print("成功下载{}， 失败下载{}".format(okCount, failCount))finally:loop.close()print("main:end")

python协程多线程HTMLParser下载mp3相关推荐

python 协程多线程_python进阶之多线程（简单介绍协程）
多线程线程:实现多任务的另一种方式一个进程中,也经常需要同时做多件事,就需要同时运行多个'子任务',这些子任务,就是线程线程又被称为轻量级进程(lightweight process),是更小的 ...
python爬虫之多线程threading、多进程multiprocessing、协程aiohttp 批量下载图片
一.单线程常规下载常规单线程执行脚本爬取壁纸图片,只爬取一页的图片. import datetime import re import requests from bs4 import Beauti ...
python多线程调用携程,Python 协程，Python携程
Python 协程,Python携程协程进程:操作系统中存在线程:操作系统中存在协程:是微线程模块(greenlet) 协程不是一个真实存在的东西,是由程序员创造出来的协程,是对一个线程分 ...
python 协程（概念+示例代码）
目录 1. 迭代器 1.1 迭代器应用:自定义列表 1.2 迭代器应用:斐波那契数列 2. 生成器 2.1 生成器案例:斐波那契数列 2.2 生成器注意事项 3. 协程 3.1 yield 简单实现协 ...
5分钟完全掌握Python协程
1. 协程相关的概念 1.1 进程和线程进程(Process)是应用程序启动的实例,拥有代码.数据和文件和独立的内存空间,是操作系统最小资源管理单元.每个进程下面有一个或者多个线程(Thread), ...
python 协程_Python 协程与 Go 协程的区别（一）
? "Python猫" ,一个值得加星标的公众号花下猫语:年关将近,不知各位过得怎样?我最近有些忙,收获也挺多,以后有机会分享下.吃饭时间,追了两部剧<了不起的麦瑟尔夫人& ...
python协程实时输出_python协程
不知道你有没有被问到过有没有使用过的python协程? 协程是什么? 协程是一种用户态轻量级,是实现并发编程的一种方式.说到并发,就能想到了多线程 / 多进程模型,是解决并发问题的经典模型之一. 但是 ...
python中协程与函数的区别_深入浅析python 协程与go协程的区别
进程.线程和协程进程的定义: 进程,是计算机中已运行程序的实体.程序本身只是指令.数据及其组织形式的描述,进程才是程序的真正运行实例. 线程的定义: 操作系统能够进行运算调度的最小单位.它被包含在进 ...
python 协程可以嵌套协程吗_Python线程、协程探究（2）——揭开协程的神秘面纱...
一.上集回顾在上一篇中我们主要研究了python的多线程困境,发现多核情况下由于GIL的存在,python的多线程程序无法发挥多线程该有的并行威力.在文章的结尾,我们提出如下需求: 既然python ...
c++ 协程_理解Python协程(Coroutine)
由于GIL的存在,导致Python多线程性能甚至比单线程更糟. GIL: 全局解释器锁(英语:Global Interpreter Lock,缩写GIL),是计算机程序设计语言解释器用于同步线程的一种 ...

python协程多线程HTMLParser下载mp3

python协程多线程HTMLParser下载mp3相关推荐

最新文章

热门文章