pycharm环境下配置scrap爬虫环境

【写在开头】

参考文章后面给出了备注信息，是在解决这个问题的时候，查找的比较有亮点的参考文章，如果本文章写的不太清楚的，可以去原文章进行查看。下面列举的四个文章有参考的成分也有验证的成分，解决办法重点参考的是文章4，前面3个只是对pycharm使用和安装包的便捷性上比较有亮点，这里还是进行引用。后面方便对参考文章进行引用，对文献进行编号。

【参考文章】：

1、https://www.cnblogs.com/zengsf/p/9801966.html

pycharm的设置选项中安装文件，只是作为了解，知道可以通过pycharm平台安装和卸载包，该文章也介绍了不能直接安装twisted，所以还是需要下载安装，我是直接从文献4中给的百度盘下载安装的，twisted安装成功之后，可以成功安装scrapy包，但是不能查看scrapy版本，提示scrapy不是内部命令，放弃了，直接看正文部分教程或者文章4。

2、http://www.manongjc.com/article/44529.html

提到的用豆瓣源进行加速安装的命令，对了一下，直接pip install scrapy和使用豆瓣源安装，后者的确很快，看了下https://pypi.douban.com/simple 源社区里面很多包，安装比较慢的时候可以考虑使用，作为了解。

3、https://www.cnblogs.com/llssx/p/8378832.html

给出安装scrapy时会遇到的一些错误提示，和安装scrapy时自下而上的顺序结构图，只是作为了解，文章没有给出解决方案。

4、https://blog.csdn.net/YiXiao1997/article/details/96345737

是一个完整的安装流程，并且给出Twisted的百度盘文件，很方便。文章最后对该教程与该文章进行了对比。

【正文部分】

1、由于入手python的时候就用的pycharm，所以学到scrapy的时候就想着在pycharm下进行一些配置，pycharm有个控制台，见图1，可以直接输入命令行进行安装操作，本来以为直接pip install scrapy就可以了，但事实并不是这样，下图2给出了提示的错误信息，实际提示的错误信息有将近一千多行，直接去最后比较重要的两行，大概意思就是：安装twisted出现错误，Microsoft Visual C++ 14.0文件缺失。（备注：错误提示内容在文献3提到了）

图1 pycharm控制台

图2 错误提示信息：安装twisted出现错误，Microsoft Visual C++ 14.0文件缺失

2、接着根据文献4的提示，下载好twisted的安装包，在pycharm的控制台进行安装程序，安装提示信息如下所示：

D:\zhuomian>pip install Twisted-19.2.0-cp37-cp37m-win_amd64.whl
Processing d:\zhuomian\twisted-19.2.0-cp37-cp37m-win_amd64.whl
Requirement already satisfied: constantly>=15.1 in c:\users\余生，别无他求\appdata\local\programs\python\python37\lib\site-packages (from Twiste
d==19.2.0) (15.1.0)
Requirement already satisfied: attrs>=17.4.0 in c:\users\余生，别无他求\appdata\local\programs\python\python37\lib\site-packages (from Twisted==
19.2.0) (19.1.0)
Requirement already satisfied: PyHamcrest>=1.9.0 in c:\users\余生，别无他求\appdata\local\programs\python\python37\lib\site-packages (from Twist
ed==19.2.0) (1.9.0)
Requirement already satisfied: zope.interface>=4.4.2 in c:\users\余生，别无他求\appdata\local\programs\python\python37\lib\site-packages (from T
wisted==19.2.0) (4.6.0)
Requirement already satisfied: incremental>=16.10.1 in c:\users\余生，别无他求\appdata\local\programs\python\python37\lib\site-packages (from Tw
isted==19.2.0) (17.5.0)
Requirement already satisfied: Automat>=0.3.0 in c:\users\余生，别无他求\appdata\local\programs\python\python37\lib\site-packages (from Twisted=
=19.2.0) (0.7.0)
Requirement already satisfied: hyperlink>=17.1.1 in c:\users\余生，别无他求\appdata\local\programs\python\python37\lib\site-packages (from Twist
ed==19.2.0) (19.0.0)
Requirement already satisfied: six in c:\users\余生，别无他求\appdata\local\programs\python\python37\lib\site-packages (from PyHamcrest>=1.9.0->
Twisted==19.2.0) (1.12.0)
Requirement already satisfied: setuptools in c:\users\余生，别无他求\appdata\local\programs\python\python37\lib\site-packages (from PyHamcrest>=
1.9.0->Twisted==19.2.0) (39.0.1)
Requirement already satisfied: idna>=2.5 in c:\users\余生，别无他求\appdata\local\programs\python\python37\lib\site-packages (from hyperlink>=17
.1.1->Twisted==19.2.0) (2.8)
Installing collected packages: Twisted
Successfully installed Twisted-19.2.0

3、安装好twisted之后，使用命令pip install scrapy继续安装scrapy，下面是控制台提示信息，成功安装。

D:\zhuomian>pip install scrapy
Collecting scrapyUsing cached https://files.pythonhosted.org/packages/a3/b1/d1ab5b3f84640097cf5ff642e2e357546781746d4fec2ebb40432904c57d/Scrapy-1.7.2-py2.py3-n
one-any.whl
Requirement already satisfied: lxml in c:\users\余生，别无他求\appdata\roaming\python\python37\site-packages (from scrapy) (4.3.4)
Requirement already satisfied: pyOpenSSL in c:\users\余生，别无他求\appdata\roaming\python\python37\site-packages (from scrapy) (19.0.0)
Requirement already satisfied: Twisted>=13.1.0; python_version != "3.4" in c:\users\余生，别无他求\appdata\local\programs\python\python37\lib\si
te-packages (from scrapy) (19.2.0)
Requirement already satisfied: queuelib in c:\users\余生，别无他求\appdata\roaming\python\python37\site-packages (from scrapy) (1.5.0)
Requirement already satisfied: six>=1.5.2 in c:\users\余生，别无他求\appdata\local\programs\python\python37\lib\site-packages (from scrapy) (1.1
2.0)
Requirement already satisfied: service-identity in c:\users\余生，别无他求\appdata\local\programs\python\python37\lib\site-packages (from scrapy
) (18.1.0)
Requirement already satisfied: cssselect>=0.9 in c:\users\余生，别无他求\appdata\roaming\python\python37\site-packages (from scrapy) (1.0.3)
Requirement already satisfied: w3lib>=1.17.0 in c:\users\余生，别无他求\appdata\roaming\python\python37\site-packages (from scrapy) (1.20.0)
Requirement already satisfied: PyDispatcher>=2.0.5 in c:\users\余生，别无他求\appdata\local\programs\python\python37\lib\site-packages (from scr
apy) (2.0.5)
Requirement already satisfied: parsel>=1.5 in c:\users\余生，别无他求\appdata\roaming\python\python37\site-packages (from scrapy) (1.5.1)
Requirement already satisfied: cryptography>=2.3 in c:\users\余生，别无他求\appdata\local\programs\python\python37\lib\site-packages (from pyOpe
nSSL->scrapy) (2.7)
Requirement already satisfied: PyHamcrest>=1.9.0 in c:\users\余生，别无他求\appdata\local\programs\python\python37\lib\site-packages (from Twist
ed>=13.1.0; python_version != "3.4"->scrapy) (1.9.0)
Requirement already satisfied: incremental>=16.10.1 in c:\users\余生，别无他求\appdata\local\programs\python\python37\lib\site-packages (from Tw
isted>=13.1.0; python_version != "3.4"->scrapy) (17.5.0)
Requirement already satisfied: hyperlink>=17.1.1 in c:\users\余生，别无他求\appdata\local\programs\python\python37\lib\site-packages (from Twist
ed>=13.1.0; python_version != "3.4"->scrapy) (19.0.0)
Requirement already satisfied: Automat>=0.3.0 in c:\users\余生，别无他求\appdata\local\programs\python\python37\lib\site-packages (from Twisted>
=13.1.0; python_version != "3.4"->scrapy) (0.7.0)
Requirement already satisfied: constantly>=15.1 in c:\users\余生，别无他求\appdata\local\programs\python\python37\lib\site-packages (from Twiste
d>=13.1.0; python_version != "3.4"->scrapy) (15.1.0)
Requirement already satisfied: zope.interface>=4.4.2 in c:\users\余生，别无他求\appdata\local\programs\python\python37\lib\site-packages (from T
wisted>=13.1.0; python_version != "3.4"->scrapy) (4.6.0)
Requirement already satisfied: attrs>=17.4.0 in c:\users\余生，别无他求\appdata\local\programs\python\python37\lib\site-packages (from Twisted>=
13.1.0; python_version != "3.4"->scrapy) (19.1.0)
Requirement already satisfied: pyasn1 in c:\users\余生，别无他求\appdata\local\programs\python\python37\lib\site-packages (from service-identity
->scrapy) (0.4.5)
Requirement already satisfied: pyasn1-modules in c:\users\余生，别无他求\appdata\local\programs\python\python37\lib\site-packages (from service-
identity->scrapy) (0.2.5)
Requirement already satisfied: cffi!=1.11.3,>=1.8 in c:\users\余生，别无他求\appdata\local\programs\python\python37\lib\site-packages (from cryp
tography>=2.3->pyOpenSSL->scrapy) (1.12.3)
Requirement already satisfied: asn1crypto>=0.21.0 in c:\users\余生，别无他求\appdata\local\programs\python\python37\lib\site-packages (from cryp
tography>=2.3->pyOpenSSL->scrapy) (0.24.0)
Requirement already satisfied: setuptools in c:\users\余生，别无他求\appdata\local\programs\python\python37\lib\site-packages (from PyHamcrest>=
1.9.0->Twisted>=13.1.0; python_version != "3.4"->scrapy) (39.0.1)
Requirement already satisfied: idna>=2.5 in c:\users\余生，别无他求\appdata\local\programs\python\python37\lib\site-packages (from hyperlink>=17
.1.1->Twisted>=13.1.0; python_version != "3.4"->scrapy) (2.8)
Requirement already satisfied: pycparser in c:\users\余生，别无他求\appdata\local\programs\python\python37\lib\site-packages (from cffi!=1.11.3,
>=1.8->cryptography>=2.3->pyOpenSSL->scrapy) (2.19)
Installing collected packages: scrapy
Successfully installed scrapy-1.7.2

4、进行验证，查看scrapy版本，没有提示scrapy不是内部命令，也给出了版本信息，说明安装成功

D:\zhuomian>scrapy
Scrapy 1.7.2 - no active projectUsage:scrapy <command> [options] [args]Available commands:bench         Run quick benchmark testfetch         Fetch a URL using the Scrapy downloadergenspider     Generate new spider using pre-defined templatesrunspider     Run a self-contained spider (without creating a project)settings      Get settings valuesshell         Interactive scraping consolestartproject  Create new projectversion       Print Scrapy versionview          Open URL in browser, as seen by Scrapy[ more ]      More commands available when run from project directoryUse "scrapy <command> -h" to see more info about a command

【写在最后】

　　就像前面说的，这篇文章是自己对在学习scrapy环境配置的时候，遇到的问题的一个记录，参考的还是网上的解决办法，但同时也是对网上的教程的一个验证，前面四篇文章的备注都作出了说明，对比一下第四篇参考文章，我这里没有安装wheel工具包，直接安装的twisted，也是可以成功安装的。

　　写到这里，只是进行了环境配置，还没有进行验证，是不是可以成功进行项目开发，下一篇进行讲解，scrapy简单的项目开发。

小编尊重原创，文章如有侵权，联系小编删帖，谢谢监督。转发请备注出处！

转载于:https://www.cnblogs.com/ljxh/p/11235079.html

pycharm环境下配置scrap爬虫环境相关推荐

Mac环境下配置Java开发环境（jdk+maven+tomcat+idea）
记录下在Mac环境下配置java开发环境,包括jdk,maven,tomcat和idea 下载及安装jdk 首先到官网下载jdk,这里就用目前官网最新的9.0.4 无脑双击安装就可以: 配置jdk 进 ...
Geany编辑器win7环境下配置Python编程环境
Geany是一款简单但功能强大的文本编辑器,可以为Python提供便利的编程环境. Geany官方下载地址:https://www.geany.org/Download/Releases,支持wind ...
ubuntu18.04环境下配置opencv c++环境（make实现cpp编译）
安装环境:ubuntu18.04 opencv包:opencv-3.4.15 1.确保你的ubuntu可以联网大多数如果初始不是ubuntu等linux系统,有可能本来支持windows,但是换了l ...
anaconda下配置R子环境并配置jupyter notebook的R Kernel
anaconda下配置R子环境并配置jupyter notebook的R Kernel #在anaconda下打开主环境的terminal: conda create --name r4-base # ...
Window环境下配置MySQL 5.6的主从复制、备份恢复
Window环境下配置MySQL 5.6的主从复制.备份恢复 1.环境准备 Windows 7 64位 MySQL 5.6 主库:192.168.103.207 从库:192.168.103.208 ...
Windows环境下配置环境变量
安装好MySQL后,在Windows环境下配置环境变量 1)新建MYSQL_HOME系统变量配置MySQL的安装路径:C:\Program Files\MySQL\MySQL Server 8.0 ...
Windows下配置单机Hadoop环境
Windows下配置单机Hadoop环境文章目录 Windows下配置单机Hadoop环境下载Hadoop 下载配置文件配置环境变量初始化HDFS 启动Hadoop 关闭Hadoop 常见错误 ...
Windows下配置IDEA开发环境
Windows下配置IDEA开发环境文章目录 Windows下配置IDEA开发环境安装JDK 查看JDK是否安装下载JDK 配置环境变量安装IDEA 下载IDEA 安装IDEA 初始化IDEA ...
JDK5.0环境下配置PKCS#11
JDK5.0环境下配置PKCS#11 发表日期:2008-1-5 新浪微博 QQ空间 QQ微博百度搜藏腾讯朋友 QQ收藏百度空间人人网开心网这篇文章不介绍具体的编程方法,而是针对PKCS# ...

pycharm环境下配置scrap爬虫环境

pycharm环境下配置scrap爬虫环境相关推荐

最新文章

热门文章