python爬虫笔记

记录时间：2021年2月1日

1、验证码识别

验证码是一种反爬机制

需要识别验证码图片中的数据，用于模拟登录操作

识别验证码的操作：

人工肉眼识别（不推荐）
第三方自动识别（推荐）

python验证码识别（例如使用尖叫数据）

import urllib.parse, urllib.request, sys
import rehost = 'http://apigateway.jianjiaoshuju.com'
path = '/api/v_1/yzmCustomized.html'
method = 'POST'
appcode = 'xxxxxxxxxxx'
appKey = 'xxxxxxxxxxx'
appSecret = 'xxxxxxxxxxx'
querys = ''
bodys = {}
url = host + path
# 图片的base64编码
bodys['v_pic'] = 'data:image/gif;base64,xxxxxxx'
bodys['pri_id'] = 'ne'
post_data = bytes(urllib.parse.urlencode(bodys), encoding="utf-8")
request = urllib.request.Request(url, post_data)
request.add_header('appcode', appcode)
request.add_header('appKey', appKey)
request.add_header('appSecret', appSecret)
request.add_header('Content-Type', 'application/x-www-form-urlencoded; charset=UTF-8')
response = urllib.request.urlopen(request).read()
if response:content = response.decode("utf-8")v_code = re.findall('"v_code":"(.*?)"', content)[0]print(v_code)

需求1：古诗文网的验证码识别

import requests
from lxml import etree
import urllib.parse, urllib.request
import re
import base64# 获取验证码的文本
def getCodeText(v_pic, pri_id):host = 'http://apigateway.jianjiaoshuju.com'path = '/api/v_1/yzmCustomized.html'method = 'POST'appcode = 'xxxxxxxxxxxx'appKey = 'xxxxxxxxxxxx'appSecret = 'xxxxxxxxxxxx'querys = ''bodys = {}url = host + pathv_code = '获取失败'bodys['v_pic'] = v_picbodys['pri_id'] = pri_idpost_data = bytes(urllib.parse.urlencode(bodys), encoding="utf-8")request = urllib.request.Request(url, post_data)request.add_header('appcode', appcode)request.add_header('appKey', appKey)request.add_header('appSecret', appSecret)request.add_header('Content-Type', 'application/x-www-form-urlencoded; charset=UTF-8')response = urllib.request.urlopen(request).read()if response:content = response.decode("utf-8")v_code = re.findall('"v_code":"(.*?)"', content)[0]# print(v_code)return v_code# 获取base64编码后的v_pic
def getv_pic(filepath):f = open(filepath, 'rb')  # 第一个参数图像路径img_base64 = base64.b64encode(f.read()).decode('utf-8')f.close()return img_base64if __name__ == '__main__':baseurl = 'https://so.gushiwen.org/user/login.aspx'headers = {'User-Agent': 'Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/88.0.4324.96 Safari/537.36 Edg/88.0.705.56'}html = requests.get(url=baseurl, headers=headers)tree = etree.HTML(html.text)imgCode_url = 'https://so.gushiwen.org' + tree.xpath('//*[@id="imgCode"]/@src')[0]imgCode_data = requests.get(url=imgCode_url, headers=headers).contentimgCode_path = './古诗文网模拟登录/imgCode.jpg'with open(imgCode_path, 'wb') as fp:fp.write(imgCode_data)CodeText = getCodeText(getv_pic(imgCode_path), 'ne')print(CodeText)

2、cookie

cookie：用来让服务器端记录客户端的相关状态。

手动处理：通过抓包工具获取cookie值，将该值封装到headers中。(不建议)
自动处理：
—cookie的来源是哪里?
—模拟登录post请求后，由服务器端创建。

session会话对象：
—作用:
1.可以进行请求的发送。
2.如果请求过程中产生了cookie，则该cookie会被自动存储/携带在该session对象中。

创建一个session对象——“session = requests.Session()”

使用session对象进行模拟登录post请求的发送(cookie就会被存储在session中)

session对象对个人主页对应的get请求进行发送(携带了cookie)

需求2：爬取豆瓣个人信息的相应数据（手动cookie处理）

import requests
from lxml import etree
import reif __name__ == '__main__':url = 'https://www.douban.com/'headers = {'Cookie': 'll="118316"; bid=0FIL6vjWxhw; douban-fav-remind=1; __yadk_uid=4Z8xtdej0WrqXqPer14U40sfi6Lo0xob; _vwo_uuid_v2=DC9368AFA6BB2FE1F1B553BB855F45CF2|7738f80fafd129733109fbe70c7e4bbe; _pk_ref.100001.8cb4=%5B%22%22%2C%22%22%2C1612173329%2C%22https%3A%2F%2Fwww.baidu.com%2Flink%3Furl%3DV6fFSEPFUqp8jF2t6jGBYVKdE4SOFTtJxVZxIGwCIXaX8NMPRGjkfdDgeAb0rIyA%26wd%3D%26eqid%3Ddcdce7840000177e000000046017d00b%22%5D; _pk_ses.100001.8cb4=*; __utma=30149280.1976354768.1552034909.1612004998.1612173337.14; __utmc=30149280; __utmz=30149280.1612173337.14.5.utmcsr=baidu|utmccn=(organic)|utmcmd=organic; ap_v=0,6.0; push_noty_num=0; push_doumail_num=0; __utmv=30149280.23190; __gads=ID=6040484a5bfb4e2e-2293a631e7c50037:T=1612173464:R:S=ALNI_MZAuEevqcWL_eMHTmRcwRdUm3ptFA; __utmt=1; dbcl2="231900772:KYdGpxBvCEI"; ck=Y91S; _pk_id.100001.8cb4=009a280458d2bfa8.1569556611.7.1612174391.1612170160.; __utmb=30149280.16.10.1612173337','User-Agent': 'Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/88.0.4324.96 Safari/537.36 Edg/88.0.705.56'}html = requests.get(url=url, headers=headers)tree = etree.HTML(html.text)span = tree.xpath('//*[@id="db-global-nav"]/div/div[1]/ul/li[2]/a/span[1]/text()')[0]# print(span)username = re.findall('(.*?)的帐号', span)[0]print(username)

需求3：古诗文网模拟登录（全过程，自动获取cookie）

import requests
from lxml import etree
import urllib.parse, urllib.request
import re
import base64# 获取验证码的文本
def getCodeText(v_pic, pri_id):host = 'http://apigateway.jianjiaoshuju.com'path = '/api/v_1/yzmCustomized.html'method = 'POST'appcode = 'xxxxxxxxxxxxxxxxxxx'appKey = 'xxxxxxxxxxxxxxxxxxx'appSecret = 'xxxxxxxxxxxxxxxxxxx'querys = ''bodys = {}url = host + pathv_code = '获取失败'bodys['v_pic'] = v_picbodys['pri_id'] = pri_idpost_data = bytes(urllib.parse.urlencode(bodys), encoding="utf-8")request = urllib.request.Request(url, post_data)request.add_header('appcode', appcode)request.add_header('appKey', appKey)request.add_header('appSecret', appSecret)request.add_header('Content-Type', 'application/x-www-form-urlencoded; charset=UTF-8')response = urllib.request.urlopen(request).read()if response:content = response.decode("utf-8")v_code = re.findall('"v_code":"(.*?)"', content)[0]# print(v_code)return v_code# 获取base64编码后的v_pic
def getv_pic(filepath):f = open(filepath, 'rb')  # 第一个参数图像路径img_base64 = base64.b64encode(f.read()).decode('utf-8')f.close()return img_base64if __name__ == '__main__':session = requests.Session()headers = {'User-Agent': 'Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/88.0.4324.96 Safari/537.36 Edg/88.0.705.56'}url = "https://so.gushiwen.org/user/login.aspx"# 获取表单登录令牌html = session.get(url=url, headers=headers)tree = etree.HTML(html.text)VIEWSTATE = tree.xpath('//*[@id="__VIEWSTATE"]/@value')[0]VIEWSTATEGENERATOR = tree.xpath('//*[@id="__VIEWSTATEGENERATOR"]/@value')[0]# 下载验证码html = session.get(url=url, headers=headers)tree = etree.HTML(html.text)imgCode_src = 'https://so.gushiwen.org' + tree.xpath('//*[@id="imgCode"]/@src')[0]# print(imgCode_src)imgCode_data = session.get(url=imgCode_src, headers=headers).contentimgCode_path = './古诗文网模拟登录/imgCode.jpg'with open(imgCode_path, 'wb') as fp:fp.write(imgCode_data)CodeText = getCodeText(getv_pic('./古诗文网模拟登录/imgCode.jpg'), 'ne')# print(CodeText)# 登录data = {'__VIEWSTATE': VIEWSTATE,'__VIEWSTATEGENERATOR': VIEWSTATEGENERATOR,'from': '','email': 'xxxxxxxxxxxxxxxxxxx','pwd': 'xxxxxxxxxxxxxxxxxxx','code': CodeText,'denglu': '登录'}html = session.post(url=url, headers=headers, data=data)with open('./古诗文网模拟登录/古诗文网.html', 'w', encoding='utf-8') as fp:fp.write(html.text)# 获取个人主页的个人信息（如：绑定手机的手机号）html = session.get(url='https://so.gushiwen.org/user/collect.aspx', headers=headers)tree = etree.HTML(html.text)phone = tree.xpath('//*[@id="mainSearch"]/div[3]/div[1]/div[3]/span/text()')[0]print(phone)

python爬虫笔记（3）相关推荐

Python爬虫笔记
Python爬虫笔记总体看过程赋予地址获取网页信息获取网页中所需的信息将得到所需的信息保存到硬盘总体看过程爬虫-从网络上爬取所需要的信息,怎么爬取?首先得知道你要去哪里获取(即地址,首先 ...
Python爬虫笔记（3）- 爬取丁香园留言
Python爬虫笔记(3)- 爬取丁香园留言爬取丁香园留言:主要用到了模拟登录爬取丁香园留言:主要用到了模拟登录 import requests, json, re, random,time fr ...
python爬虫笔记第二章
前言你好! 这是基于b站2021年路飞学成爬虫教程的python爬虫学习笔记,主要是方便作者复习和回顾课程内容. 已经发布了第一章和第二章,可以在主页中查看. python爬虫笔记(第二章) 前言 ...
Python爬虫笔记汇总
文中介绍的比较详细的有,requests库,urllib.request库,BeautifulSoup库,re库和正则表达式,Scrapy常用命令. 一.requests库课程小案例京东商品详情页 ...
Python爬虫笔记——解析json数据(以周杰伦歌单为例)及Headers
一.Network Network能够记录浏览器的所有请求.我们最常用的是:ALL(查看全部)/XHR(仅查看XHR)/Doc(Document,第0个请求一般在这里),有时候也会看看:Img(仅查看 ...
Python爬虫笔记——分析AJAX传递的JSON获取数据-初步分析动态网页
转载文章链接: Python爬虫:分析AJAX传递的JSON获取数据-初步分析动态网页(1) [4]实战:爬取动态网页的两种思路爬取新浪趣图(1) [5]实战:爬取动态网页的两种思路爬取新浪趣图(2) ...
Python爬虫笔记——解决python 写入文件中文乱码问题
原链接: python 爬虫写入文件中文乱码问题解决分析过程根据网上的一些建议,将文件的打开编码改为 utf-8 ,结果跟本不起作用奇怪的是用notepad 打开 csv 文件竟然显示正常,看来 ...
Python 爬虫笔记、多线程、xml解析、基础笔记（不定时更新）
1 Python学习网址:http://www.runoob.com/python/python-multithreading.html 注意高级中的xml解析和多线程 2 参考笔记虫师 ...
PYTHON 爬虫笔记七:Selenium库基础用法
知识点一:Selenium库详解及其基本使用什么是Selenium selenium 是一套完整的web应用程序测试系统,包含了测试的录制(selenium IDE),编写及运行(Selenium ...
请查收，一份让你年薪突破20W的Python爬虫笔记！
本次学习的教学视频来自嵩天老师的网络爬虫教学,主要学习内容有requests\BeautifulSoup\scrapy\re,目前除了scrapy其他刚好看完.并搬运实现了一些小项目如58同城租房信息 ...

python爬虫笔记（3）