scrapy 报错401

新人学scrapy，最近在爬金华信义居的房屋信息新房 - 列表，楼盘详细信息都很顺利的爬取成功了，但抓楼盘单元就报401的错，抓破脑袋不知道咋搞！

百度了说是401 是需要验证用户信息，但具体应该怎么操作呢？？？

贴上代码

import datetime
import json
import pandas as pd
import scrapy
from jinhua.items import HouseItemclass ProjectSpider(scrapy.Spider):name = 'jhhouse'allowed_domains = ['https://www.jhtmsf.com']start_urls = ['https://www.jhtmsf.com/House/GetPageForRoom']spiderTime = datetime.datetime.now().strftime('%Y-%m-%d')def start_requests(self):df = pd.read_excel('../采集结果/py_jhtmsf_loupan.xlsx')for idx, row in df.iterrows():project_id = row['project_id']referer_url = 'https://www.jhtmsf.com/House/Room/' + str(project_id)yield scrapy.FormRequest(self.start_urls[0],headers={'Referer': referer_url},cookies={'__RequestVerificationToken': 'XpgK_gMXlG71JzgTKt27kPr9ZQE1Ptbm6DhRfN7Ol7OMuuS_p43T6XOKkwg48zNUlI5jSYlJA97oO_KoupElbXV5Zm-1ldmVCCjUkltPB8c1','ASP.NET_SessionId': 'vwtw2pdanxrkdnfbqimzol1u','Hm_lvt_88b265ab6b07373c61ffa7d36d6db2c3': '1634609773,1634711852,1634882161,1634883812','Hm_lpvt_88b265ab6b07373c61ffa7d36d6db2c3': '1634883832'},formdata={'eid': str(project_id), 'bulid': '', 'layer': '', 'status': '0', 'pageNumber': '1', 'pageSize': '15', 'sortName': 'StartDate', 'sortOrder': 'desc'},callback=self.parse,meta={'project_id': project_id, 'referer_url': referer_url})def parse(self, response):jsonBody = json.loads(response.body)page = jsonBody["TotalPage"]total = jsonBody["Total"]project_id = response.meta['project_id']referer_url = response.meta['referer_url']for pg in range(1, int(page) + 1):yield scrapy.FormRequest(self.start_urls[0], headers={'Referer': str(referer_url)},formdata={'eid': str(project_id), 'bulid': '', 'layer': '', 'status': "0", 'pageNumber': str(pg),'pageSize': "15", 'sortName': 'StartDate', 'sortOrder': 'desc'},callback=self.content_parse, meta={'project_id': project_id, 'total': total})def content_parse(self, response):jsonBody = json.loads(response.body)jrows = jsonBody["Rows"]if jrows:for row in jrows:item = HouseItem()item['project_id'] = response.meta['project_id']item['total'] = response.meta['total']item['area'] = row['Area']item['build_nb'] = row['Bulid']item['on_layer'] = row['Layer']item['price'] = row['Price']item['room_nb'] = row['RoomNO']item['start_time'] = row['StartDate']item['house_status'] = row['Status']item['spider_time'] = self.spiderTimeyield item

scrapy 报错401相关推荐

解决：building 'twisted.test.raiser' extension安装scrapy报错
解决:building 'twisted.test.raiser' extension error: Microsoft Visual C++ 14.0 is required. Get it ...
CMD终端关于pip报错,scrapy报错的一种处理方法
CMD终端关于pip报错,scrapy报错的一种处理方法如果在终端输入pip,或scrapy,报如下错误: Fatal error in launcher: Unable to create pro ...
安装scrapy报错
安装scrapy报错 win10 pycharm virtualenv python3.5 (env) C:\Users\lg\PycharmProjects\untitled>pip3 ins ...
安装scrapy报错failed with error code 1 in C:\Users\ADMINI~\AppData\Local\Temp\pip-install-4447ozxq\Twist
安装scrapy的时候报错安装scrapy报错failed with error code 1 in C:\Users\ADMINI~1\AppData\Local\Temp\pip-install- ...
Deepin下安装scrapy报错fatal error: Python.h: 没有那个文件或目录
Deepin下安装scrapy报错(亲测有效!) 问题描述在deepin操作系统上,使用python3.6,通过sudo pip3 install scrapy安装时候,出现以下错误: src/tw ...
cesium 地图无法加载报错401 缺少token
cesium 地图无法加载报错401 缺少token 首先,报错401 说明用户没有访问权限,需要进行身份认证,也就是cesium需要密钥,也是最近实行的吧,因为之前不用也能加载出地图. token ...
nexus配置禁止匿名访问后，maven导入依赖报错401问题解决方案
问题说明这个问题具体是,公司的nexus私服因为安全策略配置了禁止匿名访问,本地maven配置了所有依赖都只从私服下载,但是下载依赖时报错401. Failed to execute goal or ...
接口报错401 状态码 Unauthorized
接口报错401 状态码 Unauthorized 接口报错401 状态码 Unauthorized 报错原因状态码 401一般指认证发生错误,需要排查认证的参数及获取方式是否正确解决办法如果请求 ...
HTTP报错401和403详解及解决办法
一.401: 1. HTTP 401 错误 - 未授权: (Unauthorized) 您的Web服务器认为,客户端发送的 HTTP 数据流是正确的,但进入网址 (URL) 资源 , 需要用户身份验证 ...
anaconda安装scrapy报错解决办法
今天在用anaconda安装scrapy的时候遇见个坑,现在将解决办法发出来,供大家参考使用: 问题描述: anaconda安装scrapy,使用 conda install scrapy 命令.安装 ...

scrapy 报错401

scrapy 报错401相关推荐

最新文章

热门文章