debug信息的认识

常见的debug信息

如果我们的爬取的url地址不在我们设置的allowed_domains即是被爬取网站的域名下面，会出现什么样的情况呢？

allowed_domains = ['sun0769debug.com']
start_urls = ['http://wz.sun0769.com/political/index/politicsNewest?id=1&page=1']

注意：为了使能够看到现象，必须要做好下面的工作

1.修改域名，使其与网站的url地址不一致

2.在settings.py中不要设置LOG_LEVEL这一参数，或者是将其值设置的很小很小

好了，现在万事俱备，启动爬虫
scrapy crawl sun
下面是完整的输出信息：

2020-10-11 20:58:59 [scrapy.utils.log] INFO: Scrapy 2.1.0 started (bot: Sun)
2020-10-11 20:58:59 [scrapy.utils.log] INFO: Versions: lxml 4.5.0.0, libxml2 2.9.5, cssselect 1.1.0, pars
el 1.5.2, w3lib 1.21.0, Twisted 20.3.0, Python 3.7.7 (tags/v3.7.7:d7c567b08f, Mar 10 2020, 10:41:24) [MSCv.1900 64 bit (AMD64)], pyOpenSSL 19.1.0 (OpenSSL 1.1.1g  21 Apr 2020), cryptography 2.9.2, Platform Win
dows-10-10.0.17134-SP0
2020-10-11 20:58:59 [scrapy.utils.log] DEBUG: Using reactor: twisted.internet.selectreactor.SelectReactor2020-10-11 20:58:59 [scrapy.crawler] INFO: Overridden settings:
{'BOT_NAME': 'Sun','NEWSPIDER_MODULE': 'Sun.spiders','ROBOTSTXT_OBEY': True,'SPIDER_MODULES': ['Sun.spiders'],'USER_AGENT': 'Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 ''(KHTML, like Gecko) Chrome/84.0.4147.89 Safari/537.36'}
2020-10-11 20:58:59 [scrapy.extensions.telnet] INFO: Telnet Password: effafb3a8e54b45d
2020-10-11 20:58:59 [scrapy.middleware] INFO: Enabled extensions:
['scrapy.extensions.corestats.CoreStats','scrapy.extensions.telnet.TelnetConsole','scrapy.extensions.logstats.LogStats']
2020-10-11 20:59:00 [scrapy.middleware] INFO: Enabled downloader middlewares:
['scrapy.downloadermiddlewares.robotstxt.RobotsTxtMiddleware','scrapy.downloadermiddlewares.httpauth.HttpAuthMiddleware','scrapy.downloadermiddlewares.downloadtimeout.DownloadTimeoutMiddleware','scrapy.downloadermiddlewares.defaultheaders.DefaultHeadersMiddleware','scrapy.downloadermiddlewares.useragent.UserAgentMiddleware','scrapy.downloadermiddlewares.retry.RetryMiddleware','scrapy.downloadermiddlewares.redirect.MetaRefreshMiddleware','scrapy.downloadermiddlewares.httpcompression.HttpCompressionMiddleware','scrapy.downloadermiddlewares.redirect.RedirectMiddleware','scrapy.downloadermiddlewares.cookies.CookiesMiddleware','scrapy.downloadermiddlewares.httpproxy.HttpProxyMiddleware','scrapy.downloadermiddlewares.stats.DownloaderStats']
2020-10-11 20:59:00 [scrapy.middleware] INFO: Enabled spider middlewares:
['scrapy.spidermiddlewares.httperror.HttpErrorMiddleware','scrapy.spidermiddlewares.offsite.OffsiteMiddleware','scrapy.spidermiddlewares.referer.RefererMiddleware','scrapy.spidermiddlewares.urllength.UrlLengthMiddleware','scrapy.spidermiddlewares.depth.DepthMiddleware']
2020-10-11 20:59:00 [scrapy.middleware] INFO: Enabled item pipelines:
['Sun.pipelines.SunPipeline']
2020-10-11 20:59:00 [scrapy.core.engine] INFO: Spider opened
2020-10-11 20:59:00 [scrapy.extensions.logstats] INFO: Crawled 0 pages (at 0 pages/min), scraped 0 items
(at 0 items/min)
2020-10-11 20:59:00 [scrapy.extensions.telnet] INFO: Telnet console listening on 127.0.0.1:6023
2020-10-11 20:59:01 [scrapy.core.engine] DEBUG: Crawled (200) <GET http://wz.sun0769.com/robots.txt> (ref
erer: None)
2020-10-11 20:59:01 [scrapy.core.engine] DEBUG: Crawled (200) <GET http://wz.sun0769.com/political/index/
politicsNewest?id=1&page=1> (referer: None)
2020-10-11 20:59:01 [scrapy.spidermiddlewares.offsite] DEBUG: Filtered offsite request to 'wz.sun0769.com
': <GET http://wz.sun0769.com/political/politics/index?id=475564>
2020-10-11 20:59:01 [scrapy.core.engine] INFO: Closing spider (finished)
2020-10-11 20:59:01 [scrapy.statscollectors] INFO: Dumping Scrapy stats:
{'downloader/request_bytes': 640,'downloader/request_count': 2,'downloader/request_method_count/GET': 2,'downloader/response_bytes': 9590,'downloader/response_count': 2,'downloader/response_status_count/200': 2,'elapsed_time_seconds': 0.626431,'finish_reason': 'finished','finish_time': datetime.datetime(2020, 10, 11, 12, 59, 1, 481002),'log_count/DEBUG': 3,'log_count/INFO': 10,'offsite/domains': 1,'offsite/filtered': 17,'request_depth_max': 1,'response_received_count': 2,'robotstxt/request_count': 1,'robotstxt/response_count': 1,'robotstxt/response_status_count/200': 1,'scheduler/dequeued': 1,'scheduler/dequeued/memory': 1,'scheduler/enqueued': 1,'scheduler/enqueued/memory': 1,'start_time': datetime.datetime(2020, 10, 11, 12, 59, 0, 854571)}
2020-10-11 20:59:01 [scrapy.core.engine] INFO: Spider closed (finished)

这个才是错误的关键部分

这个debug信息是什么意思呢？
我们要爬取的网站的url地址（也就是start_urls）被scrapy框架过滤掉了，因为不满足allowed_domains这个域名的条件，也就是说，url地址不在规定好的域名下面。

因为debug不是error，它的等级没有warn的等级高，所以如果我们设置LOG_LEVEL="WARN"，程序是不会出现任何的情况的。
那时候的情况应该是下面这样：

程序会一直执行，然后没有任何的结果输出，因为等级没有WARN高，所以根本显示不出来。

注意

start_urls不会经过allowed_domains的过滤

重要的debug信息解读

2020-10-11 20:58:59 [scrapy.utils.log] INFO: Scrapy 2.1.0 started (bot: Sun)
# scrapy框架开始启动了，版本为2.1.0，项目名称叫做Sun2020-10-11 20:58:59 [scrapy.utils.log] INFO: Versions: lxml 4.5.0.0, libxml2 2.9.5, cssselect 1.1.0, pars
el 1.5.2, w3lib 1.21.0, Twisted 20.3.0, Python 3.7.7 (tags/v3.7.7:d7c567b08f, Mar 10 2020, 10:41:24) [MSCv.1900 64 bit (AMD64)], pyOpenSSL 19.1.0 (OpenSSL 1.1.1g  21 Apr 2020), cryptography 2.9.2, Platform Win
dows-10-10.0.17134-SP0
# 这个是scrapy所安装的一些依赖的名称+版本2020-10-11 20:58:59 [scrapy.crawler] INFO: Overridden settings:
{'BOT_NAME': 'Sun','NEWSPIDER_MODULE': 'Sun.spiders','ROBOTSTXT_OBEY': True,'SPIDER_MODULES': ['Sun.spiders'],'USER_AGENT': 'Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 ''(KHTML, like Gecko) Chrome/84.0.4147.89 Safari/537.36'}
# 这个是我们对settings.py所做的一些设置和修改2020-10-11 20:58:59 [scrapy.crawler] INFO: Overridden settings:
{'BOT_NAME': 'Sun','NEWSPIDER_MODULE': 'Sun.spiders','ROBOTSTXT_OBEY': True,'SPIDER_MODULES': ['Sun.spiders'],'USER_AGENT': 'Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 ''(KHTML, like Gecko) Chrome/84.0.4147.89 Safari/537.36'}
# 自己设置的settings的信息2020-10-11 20:58:59 [scrapy.middleware] INFO: Enabled extensions:
['scrapy.extensions.corestats.CoreStats','scrapy.extensions.telnet.TelnetConsole','scrapy.extensions.logstats.LogStats']
# 启动的扩展，默认有一堆2020-10-11 20:59:00 [scrapy.middleware] INFO: Enabled downloader middlewares:
['scrapy.downloadermiddlewares.robotstxt.RobotsTxtMiddleware','scrapy.downloadermiddlewares.httpauth.HttpAuthMiddleware','scrapy.downloadermiddlewares.downloadtimeout.DownloadTimeoutMiddleware','scrapy.downloadermiddlewares.defaultheaders.DefaultHeadersMiddleware','scrapy.downloadermiddlewares.useragent.UserAgentMiddleware','scrapy.downloadermiddlewares.retry.RetryMiddleware','scrapy.downloadermiddlewares.redirect.MetaRefreshMiddleware','scrapy.downloadermiddlewares.httpcompression.HttpCompressionMiddleware','scrapy.downloadermiddlewares.redirect.RedirectMiddleware','scrapy.downloadermiddlewares.cookies.CookiesMiddleware','scrapy.downloadermiddlewares.httpproxy.HttpProxyMiddleware','scrapy.downloadermiddlewares.stats.DownloaderStats']
# 启动的下载中间件，默认有一堆2020-10-11 20:59:00 [scrapy.middleware] INFO: Enabled spider middlewares:
['scrapy.spidermiddlewares.httperror.HttpErrorMiddleware','scrapy.spidermiddlewares.offsite.OffsiteMiddleware','scrapy.spidermiddlewares.referer.RefererMiddleware','scrapy.spidermiddlewares.urllength.UrlLengthMiddleware','scrapy.spidermiddlewares.depth.DepthMiddleware']
# 启动的爬虫中间件，默认有一堆2020-10-11 20:59:00 [scrapy.middleware] INFO: Enabled item pipelines:
['Sun.pipelines.SunPipeline']
# 启动的item管道，保存数据的管道的那个类2020-10-11 20:59:00 [scrapy.extensions.telnet] INFO: Telnet console listening on 127.0.0.1:6023
# 爬虫运行的时候能够使用talent命令对爬虫做一些控制，比如暂停等等2020-10-11 20:59:01 [scrapy.statscollectors] INFO: Dumping Scrapy stats:
{'downloader/request_bytes': 640,'downloader/request_count': 2,'downloader/request_method_count/GET': 2,'downloader/response_bytes': 9590,'downloader/response_count': 2,'downloader/response_status_count/200': 2,'elapsed_time_seconds': 0.626431,'finish_reason': 'finished','finish_time': datetime.datetime(2020, 10, 11, 12, 59, 1, 481002),'log_count/DEBUG': 3,'log_count/INFO': 10,'offsite/domains': 1,'offsite/filtered': 17,'request_depth_max': 1,'response_received_count': 2,'robotstxt/request_count': 1,'robotstxt/response_count': 1,'robotstxt/response_status_count/200': 1,'scheduler/dequeued': 1,'scheduler/dequeued/memory': 1,'scheduler/enqueued': 1,'scheduler/enqueued/memory': 1,'start_time': datetime.datetime(2020, 10, 11, 12, 59, 0, 854571)}
# 爬虫结束时候的一些统计信息，比如请求响应数量等2020-10-11 20:59:01 [scrapy.core.engine] DEBUG: Crawled (200) <GET http://wz.sun0769.com/robots.txt> (ref
erer: None)
2020-10-11 20:59:01 [scrapy.core.engine] DEBUG: Crawled (200) <GET http://wz.sun0769.com/political/index/
politicsNewest?id=1&page=1> (referer: None)
2020-10-11 20:59:01 [scrapy.spidermiddlewares.offsite] DEBUG: Filtered offsite request to 'wz.sun0769.com
': <GET http://wz.sun0769.com/political/politics/index?id=475564>
# 每次yield item的时候返回提示item的内容以及这个item来自的url的地址

debug信息的认识相关推荐

cocos2d-x 输出debug信息
在Classes目录下添加文件AppDef.h #ifndef _APP_DEF_H_ #define _APP_DEF_H_ #include <android/log.h> #defi ...
C 预处理器 —— __DATE__ # __TIME__ # __FILE__ # __LINE__ # __STDC__ (预处理宏的使用 —— 打印debug信息：)
预定义宏: ANSI C 定义了许多宏.在编程中您可以使用这些宏,但是不能直接修改这些预定义的宏. 宏描述 __DATE__ 当前日期,一个以 "MMM DD YYYY&quo ...
log4j 配置，tomcat 启动或有后台操作时，控制台会显示很多 DEBUG 信息
log4j 配置,tomcat 启动或有后台操作时,控制台会显示很多 DEBUG 信息日志信息可以以文件形式显示,也可以在控制台输出,在 log4j.properties 文件设置. 控制台有很多 ...
Scrapy框架的学习(7. 了解Scrapy中的debug信息以及Scrapy shell的使用)
认识程序中的debug信息 https://blog.csdn.net/wei18791957243/article/details/86157707 这个博客里写了,怎么关闭这些debug信息因 ...
andorid 查看OpenCv Mat的Debug信息
在进行Android调试时,不能再Console显示Debug信息,只能在LogCat上显示,显示信息如下图: 代码段: public void printMat2Txt(Mat ElemM, Str ...
关于scrapy shell 在ipython环境下 TAB出现debug信息解决方案
在scrapy shell中调试的时候,按下tab总会出现debug信息,如图: 解决方案是禁止输出debug信息 1. 更新parso 控制台中输入 pip install --upgrade ...
Unity程序在VR一体机(Android)上卡死(闪退)后怎么办？——用adb查看android上某Unity app的debug信息
一.之前面临的困境 Unity的程序build到android一体机后,仿佛进入了一个黑箱子,你既看不到脚本的debug报错信息,也看不到任务管理器里的内存和CPU使用情况?如果黑屏.闪屏.花屏怎么办 ...
DWARF格式对于debug信息的支持
DWARF格式对于debug信息的支持 by Peter 4 YEARS AGO 4 MINUTEREAD http://tsecer.blog.163.com/blog/static/1501817 ...
Linux内核开启DebugFS以及查看debug信息（/sys/kernel/debug及debugfs）
一.Linux内核开启DebugFS 进入内核配置界面,查找DebugFS: 开启配置: 二.Linux内核挂载DebugFS mount -t debugfs none /sys/kernel/de ...

debug信息的认识

常见的debug信息

如果我们的爬取的url地址不在我们设置的allowed_domains即是被爬取网站的域名下面，会出现什么样的情况呢？

注意：为了使能够看到现象，必须要做好下面的工作

1.修改域名，使其与网站的url地址不一致

2.在settings.py中不要设置LOG_LEVEL这一参数，或者是将其值设置的很小很小

注意

重要的debug信息解读

debug信息的认识相关推荐

最新文章

热门文章