运行scrapy后无结果,提示debug信息显示403

2023-01-19 09:51:35 [scrapy.utils.log] INFO: Scrapy 2.7.1 started (bot: tl)
2023-01-19 09:51:35 [scrapy.utils.log] INFO: Versions: lxml 4.5.0.0, libxml2 2.9.9, cssselect 1.2.0, parsel 1.7.0, w3lib 2.1.1, Twisted 22.4.0, Python 3.7.6 (default, Jan  8 2020, 20:23:39) [MSC v.1916 64 bit (AMD64)], pyOpenSSL 23.0.0 (OpenSSL 3.0.7 1 Nov 2022), cryptography 39.0.0, Platform Windows-10-10.0.22000-SP0
2023-01-19 09:51:35 [scrapy.crawler] INFO: Overridden settings:
{'BOT_NAME': 'tl','NEWSPIDER_MODULE': 'tl.spiders','REQUEST_FINGERPRINTER_IMPLEMENTATION': '2.7','SPIDER_MODULES': ['tl.spiders'],'TWISTED_REACTOR': 'twisted.internet.asyncioreactor.AsyncioSelectorReactor'}
2023-01-19 09:51:35 [asyncio] DEBUG: Using selector: SelectSelector
2023-01-19 09:51:35 [scrapy.utils.log] DEBUG: Using reactor: twisted.internet.asyncioreactor.AsyncioSelectorReactor
2023-01-19 09:51:35 [scrapy.utils.log] DEBUG: Using asyncio event loop: asyncio.windows_events._WindowsSelectorEventLoop
2023-01-19 09:51:35 [scrapy.extensions.telnet] INFO: Telnet Password: 14fefb2358fcdeed
2023-01-19 09:51:35 [scrapy.middleware] INFO: Enabled extensions:
['scrapy.extensions.corestats.CoreStats','scrapy.extensions.telnet.TelnetConsole','scrapy.extensions.logstats.LogStats']
2023-01-19 09:51:35 [scrapy.middleware] INFO: Enabled downloader middlewares:
['scrapy.downloadermiddlewares.httpauth.HttpAuthMiddleware','scrapy.downloadermiddlewares.downloadtimeout.DownloadTimeoutMiddleware','scrapy.downloadermiddlewares.defaultheaders.DefaultHeadersMiddleware','scrapy.downloadermiddlewares.useragent.UserAgentMiddleware','scrapy.downloadermiddlewares.retry.RetryMiddleware','scrapy.downloadermiddlewares.redirect.MetaRefreshMiddleware','scrapy.downloadermiddlewares.httpcompression.HttpCompressionMiddleware','scrapy.downloadermiddlewares.redirect.RedirectMiddleware','scrapy.downloadermiddlewares.cookies.CookiesMiddleware','scrapy.downloadermiddlewares.httpproxy.HttpProxyMiddleware','scrapy.downloadermiddlewares.stats.DownloaderStats']
2023-01-19 09:51:35 [scrapy.middleware] INFO: Enabled spider middlewares:
['scrapy.spidermiddlewares.httperror.HttpErrorMiddleware','scrapy.spidermiddlewares.offsite.OffsiteMiddleware','scrapy.spidermiddlewares.referer.RefererMiddleware','scrapy.spidermiddlewares.urllength.UrlLengthMiddleware','scrapy.spidermiddlewares.depth.DepthMiddleware']
2023-01-19 09:51:35 [scrapy.middleware] INFO: Enabled item pipelines:
[]
2023-01-19 09:51:35 [scrapy.core.engine] INFO: Spider opened
2023-01-19 09:51:35 [scrapy.extensions.logstats] INFO: Crawled 0 pages (at 0 pages/min), scraped 0 items (at 0 items/min)
2023-01-19 09:51:35 [scrapy.extensions.telnet] INFO: Telnet console listening on 127.0.0.1:6023
2023-01-19 09:51:35 [scrapy.core.engine] DEBUG: Crawled (403) <GET https://book.douban.com/top250> (referer: None)
2023-01-19 09:51:35 [scrapy.spidermiddlewares.httperror] INFO: Ignoring response <403 https://book.douban.com/top250>: HTTP status code is not handled or not allowed
2023-01-19 09:51:35 [scrapy.core.engine] INFO: Closing spider (finished)
2023-01-19 09:51:35 [scrapy.statscollectors] INFO: Dumping Scrapy stats:
{'downloader/request_bytes': 221,'downloader/request_count': 1,'downloader/request_method_count/GET': 1,'downloader/response_bytes': 309,'downloader/response_count': 1,'downloader/response_status_count/403': 1,'elapsed_time_seconds': 0.241555,'finish_reason': 'finished','finish_time': datetime.datetime(2023, 1, 19, 1, 51, 35, 777823),'httpcompression/response_bytes': 146,'httpcompression/response_count': 1,'httperror/response_ignored_count': 1,'httperror/response_ignored_status_count/403': 1,'log_count/DEBUG': 4,'log_count/INFO': 11,'response_received_count': 1,'scheduler/dequeued': 1,'scheduler/dequeued/memory': 1,'scheduler/enqueued': 1,'scheduler/enqueued/memory': 1,'start_time': datetime.datetime(2023, 1, 19, 1, 51, 35, 536268)}
2023-01-19 09:51:35 [scrapy.core.engine] INFO: Spider closed (finished)

修改user_agent后成功,但仍有红色提示

查找default_settings文件(输入from scrapy.settings import default_settings,然后按住Ctrl键,单击default_settings即可打开),修改其中的LOG_LEVEL的值为WARNING。

Scrapy 提示错误 DEBUG: Crawled (403) <GET https://book.douban.com/top250> (referer: None)相关推荐

  1. scrapy抓取豆瓣网信息时报错提醒INFO: Ignoring response <403 https://movie.douban.com/top250>: HTTP status code is

    自学python的爬虫scrapy,可能会遇到如下问题: 通过上文解释对豆瓣网进行抓取过程中出现报错如下: 2020-08-20 14:27:46 [scrapy.extensions.telnet] ...

  2. 爬取豆瓣电影TOP250的所有电影名称,网址为:https://movie.douban.com/top250

    2019独角兽企业重金招聘Python工程师标准>>> 所谓静态页面是指纯粹的HTML格式的页面,这样的页面在浏览器中展示的内容都在HTML源码中. 目标:爬取豆瓣电影TOP250的 ...

  3. Scrapy:[scrapy.core.engine] DEBUG: Crawled (200)解决尝试

    在学习Scrapy用法的实验过程中,拟"得到"某网站信息时,出现大量的[scrapy.core.engine] DEBUG和[scrapy.spidermiddlewares.ht ...

  4. 【PyCharm编辑器】之引用selenium包提示错误:Unresolved reference 'selenium' less... (Ctrl+F1)...

    一.现象还原: 当新建.py文件时,需要引用selenium中的方法时,报错,提示红波浪线: Unresolved reference 'selenium' less... (Ctrl+F1) Thi ...

  5. Python3安装turtle提示错误:Command python setup.py egg_info failed with error code 1

    Python3安装turtle提示错误:Command "python setup.py egg_info" failed with error code 1 Python3.5安 ...

  6. 爬虫中遇到的问题Crawled (404),[scrapy.spidermiddlewares.offsite] DEBUG: Filtered offsite request to

    1.错误1:url地址有误 Crawled (200) <GET http://www.itcast.cn/robots.txt> (referer: None) DEBUG: Crawl ...

  7. NuGet在Push的时候提示“远程服务器返回错误:(403)已禁用”问题解决

    在使用NuGet把包push到nuget官网的时候,提示了如下信息: Failed to process request. 'The specified API key is invalid or d ...

  8. 运行提示‘错误: 找不到或无法加载主类 Engine.Runner’;修改文件夹为package

    Q:配置run/debug configuration时 main class 报红,运行后提示"错误: 找不到或无法加载主类 Engine.Runner":如下图: A:排查得知 ...

  9. Navicat连接数据库成功,新建查询时提示错误“Cannot create file ……”

    Navicat连接数据库成功,新建查询时提示错误"Cannot create file --" 原因:编辑连接{高级}<设置位置>被修改,该oci.dll不正确 解决方 ...

最新文章

  1. 世界上有哪些代码量很少,但很牛逼很经典的算法或项目案例?
  2. android.database.cursorindexoutofboundsexception错误解决 及获取某行某列信息
  3. 3层,5层,7层,9层网络性能比较-0-2
  4. 如何避免贫穷和忙碌,在2018年你需要这样提升自己 2018年01月07日 00:00:00 2099 热文导读 | 点击标题阅读 Java和Android架构2017年总结:文章精选 吊炸天!74
  5. OpenCV在图像中寻找轮廓的实例(附完整代码)
  6. map容器的使用 逆序遍历map容器当中的数据
  7. 牛客网-数据结构笔试题目(八)-离子能力跃迁问题求解
  8. 常见位操作:获取,设置,清零
  9. 【HNOI2006】鬼谷子的钱袋
  10. struts-config message-resources配置问题总结
  11. 互联网是由多个计算机相互连接而成,计算机网络
  12. FreeSWITCH第三方库(视频)的简单介绍(二)
  13. PyTorch 深度学习:30分钟快速入门
  14. linux ftp命令参数全集
  15. Linux下maven安装
  16. Windwos Server 2012 R2 安装LLDP协议驱动程序
  17. 解决Excel装了excel link 加载宏之后,打开excel就自动打开matlab的方法
  18. 腾讯云短信发送php
  19. C++ 读取txt文件中数据并存入数组中
  20. 02自定义面板区附加功能模块及清单编写

热门文章

  1. 【设计模式系列19】状态模式原理分析及其和策略模式,责任链模式的区别
  2. 【python接口自动化】csdn涨薪技术- DDT数据驱动测试
  3. EPG开发《异常排查以及解决方案》
  4. 解决maven创建中的警告:No archetype found in remote catalog. Defaulting to internal catalog.问题
  5. SOLIDWORKS Flow Simulation卡门涡街模拟
  6. 英文打字速度180kpm
  7. 两个74HC595D驱动芯片控制三位数码管显示
  8. tensorflow使用object detection API训练自己的数据(个人总结)
  9. python第三方库安装有哪些要求,python第三方库的四种安装方法
  10. 曝光 兼职达人(深圳市青木网络科技)无耻、恶心