python的requests使用

基本使用方法

import requestsurl = "https://www.baidu.com"
response = requests.get(url)
print(response.text) #打印源码的str类型的数据

但是我们发现运行程序之后会有乱码出现，原因是传输是test为byte类型，response.text 解释为 response.content.decode(“推测的编码类型”)

print(response.encoding)ISO-8859-1

import requestsurl = "https://www.baidu.com"
response = requests.get(url)
print(response.content.decode())

test和content区别
response.test
str类型，解码类型为自动http头部响应推测

response.content
byte类型，解码类型没指定
可以用decode指定编码，默认为utf-8

也可以手动设定编码

import requestsurl = "https://www.baidu.com"
response = requests.get(url)
response.encoding = "utf-8"
print(response.text)

响应对象参数

import requestsurl = "https://www.baidu.com"
response = requests.get(url)print(response.url) #响应的url
print(response.status_code) #状态码
print(response.headers) #响应对应的请求头部
print(response.request.headers) #响应对应的请求头
print(response.cookies) #响应的cookie jarhttps://www.baidu.com/
200
{'Cache-Control': 'private, no-cache, no-store, proxy-revalidate, no-transform', 'Connection': 'keep-alive', 'Content-Encoding': 'gzip', 'Content-Type': 'text/html', 'Date': 'Mon, 28 Nov 2022 12:13:28 GMT', 'Last-Modified': 'Mon, 23 Jan 2017 13:24:45 GMT', 'Pragma': 'no-cache', 'Server': 'bfe/1.0.8.18', 'Set-Cookie': 'BDORZ=27315; max-age=86400; domain=.baidu.com; path=/', 'Transfer-Encoding': 'chunked'}
{'User-Agent': 'python-requests/2.28.1', 'Accept-Encoding': 'gzip, deflate', 'Accept': '*/*', 'Connection': 'keep-alive'}
<RequestsCookieJar[<Cookie BDORZ=27315 for .baidu.com/>]>

可以看到这里的uesr-agent是以python模块去请求的

headers参数接受字典请求头

import requestsurl = "https://www.baidu.com"
head = {'User-Agent' :'Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/107.0.0.0 Safari/537.36'
}
response = requests.get(url,headers = head)
print(response.request.headers)

header中不仅仅用字典携带User-Agent，也可以携带cookie

带参数的请求

import requestsurl = "https://www.baidu.com/s?"data = {'wd' : 'python'
}
head = {'User-Agent' :'Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/107.0.0.0 Safari/537.36'
}response = requests.get(url,headers = head,params = data)
print(response.url)
with open("baidu.html",mode='wb') as b:b.write(response.content)https://wappass.baidu.com/static/captcha/tuxing.html

可以看到百度的安全认证，这里还被识别到了，但已经有一层伪装了。被拦截原因是没有携带cookie参数

headers携带cookie参数
携带cookie不会被拦截

import requestsurl = "https://www.baidu.com/s?"
data = {'wd' : 'python'
}
head = {'User-Agent' :'Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/107.0.0.0 Safari/537.36' ,'cookie' : "BIDUPSID=43D35F560169A9C65D8693129286F7D1; PSTM=1650964233; BAIDUID=43D35F560169A9C6DF7F57E2CBD0B4EA:FG=1; BDUSS=kZidjhuLXJvQjBwamdXbkdhWWh2MjBoMjJnYlJFbzBWdDcyamphZDQtSldqdnRpSVFBQUFBJCQAAAAAAAAAAAEAAABOUVn3stTT6rTzyMs1MTIwNQAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAFYB1GJWAdRic; BDUSS_BFESS=kZidjhuLXJvQjBwamdXbkdhWWh2MjBoMjJnYlJFbzBWdDcyamphZDQtSldqdnRpSVFBQUFBJCQAAAAAAAAAAAEAAABOUVn3stTT6rTzyMs1MTIwNQAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAFYB1GJWAdRic; BAIDUID_BFESS=43D35F560169A9C6DF7F57E2CBD0B4EA:FG=1; ZFY=pCNmkGR3rPGBgfDfqRoYcbGSRmmguuudGEcxgcMEtKQ:C; BDORZ=B490B5EBF6F3CD402E515D22BCDA1598; kleck=99e5b2a24ec5f84d72228354360e794a; H_PS_PSSID=36557_37550_37769_37139_37841_34813_37801_37761_37759_26350_37787; BA_HECTOR=0l21002k24agaka18k2002771hoapm41f"
}response = requests.get(url,headers = head,params = data)
print(response.url)
with open("baidu.html",mode='wb') as b:b.write(response.content)

cookie参数使用

import requestsurl = "https://www.baidu.com/s?"
data = {'wd' : 'python'
}
head = {'User-Agent' :'Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/107.0.0.0 Safari/537.36'
}
tmp = "BIDUPSID=43D35F560169A9C65D8693129286F7D1; PSTM=1650964233; BAIDUID=43D35F560169A9C6DF7F57E2CBD0B4EA:FG=1; BDUSS=kZidjhuLXJvQjBwamdXbkdhWWh2MjBoMjJnYlJFbzBWdDcyamphZDQtSldqdnRpSVFBQUFBJCQAAAAAAAAAAAEAAABOUVn3stTT6rTzyMs1MTIwNQAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAFYB1GJWAdRic; BDUSS_BFESS=kZidjhuLXJvQjBwamdXbkdhWWh2MjBoMjJnYlJFbzBWdDcyamphZDQtSldqdnRpSVFBQUFBJCQAAAAAAAAAAAEAAABOUVn3stTT6rTzyMs1MTIwNQAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAFYB1GJWAdRic; BAIDUID_BFESS=43D35F560169A9C6DF7F57E2CBD0B4EA:FG=1; ZFY=pCNmkGR3rPGBgfDfqRoYcbGSRmmguuudGEcxgcMEtKQ:C; BDORZ=B490B5EBF6F3CD402E515D22BCDA1598; kleck=99e5b2a24ec5f84d72228354360e794a; H_PS_PSSID=36557_37550_37769_37139_37841_34813_37801_37761_37759_26350_37787; BA_HECTOR=0l21002k24agaka18k2002771hoapm41f"
cookie_list = tmp.split(";")
cookies = { cookie.split("=")[0]:cookie.split("=")[-1] for cookie in cookie_list }response = requests.get(url,headers = head,params = data,cookies = cookies)
print(response.url)
with open("baidu.html",mode='wb') as b:b.write(response.content)

超时参数timeout
在有一些网站超时问题需要设置timeout

import requests
url = "https://www.baidu.com/s?"
response = requests.get(url,timeout=5)

代理参数使用

import requestsurl = "https://www.baidu.com/s?"
proxies = {"http" : "http://192.168.1.1:80" ,"https" : "https://192.168.1.1:443"
}#根据使用情况选择
response = requests.get(url,proxies=proxies)

verify参数使用
用于忽略CA证书

import requestsurl = "https://www.baidu.com/s?"
response = requests.get(url,verify=False)

python的requests使用相关推荐

Java 爬虫--类似Python的requests库--HttpClient， HttpAsyncClient--Maven
今天在找Java的爬虫的库,发现Java的爬虫框架挺多的,常见的有nutch,Heritrix,crawler4j等. 但我想要的是类似Python的requests库的Java库. 看到一个库叫Un ...
Python之Requests库的异常
Python之Requests库的异常参考文章: (1)Python之Requests库的异常 (2)https://www.cnblogs.com/BASE64/p/10285466.html 备 ...
关于python中requests模块导入问题-Python中requests模块的核心使用（1）
简介: requests模块的底层实现其实还是urllib,但是urllib并不常用 requests库简单易用快速上手 http://docs.python-requests.org/zh_CN/ ...
python中requests库的用途-数据爬虫（三）：python中requests库使用方法详解
有些网站访问时必须带有浏览器等信息,如果不传入headers就会报错,如下使用 Requests 模块,上传文件也是如此简单的,文件的类型会自动进行处理: 因为12306有一个错误证书,我们那它的网 ...
关于python中requests模块导入问题-python中requests模块的使用方法
本文实例讲述了python中requests模块的使用方法.分享给大家供大家参考.具体分析如下: 在HTTP相关处理中使用python是不必要的麻烦,这包括urllib2模块以巨大的复杂性代价获取综合 ...
python中requests库的用途-python中requests库session对象的妙用详解
在进行接口测试的时候,我们会调用多个接口发出多个请求,在这些请求中有时候需要保持一些共用的数据,例如cookies信息. 妙用1 requests库的session对象能够帮我们跨请求保持某些参数,也 ...
Py之requests：python的requests包的简介、安装、使用方法详细攻略
Py之requests:python的requests包的简介.安装.使用方法详细攻略目录 requests包的简介 requests包的安装 requests包的使用方法 requests包的简介 ...
python的requests模块功能_《Python数据可视化编程实战》—— 1.7　安装requests模块-阿里云开发者社区...
本节书摘来异步社区<Python数据可视化编程实战>一书中的第1章,第1.7节,作者:[爱尔兰]Igor Milovanović,更多章节内容可以访问云栖社区"异步社区" ...
在python中requests模块怎么安装_Python requests模块在Windows下安装
发现一个爬虫库太方便了,而且支持python3! 安装方法在http://docs.python-requests.org/en/latest/user/install/#install很详细只不过 ...
pythonunittest接口测试_基于python+unittest +requests接口测试
2019独角兽企业重金招聘Python工程师标准>>> 谈到接口测试,大家都不会感到陌生.接口测试的工具和实现方式也有很多,比如ant+jmeter+jemkins.postman. ...

python的requests使用

python的requests使用相关推荐

最新文章

热门文章