Python3.6+Twisted+Scrapy安装

一、首先检查是否有openssl-devel

这一步是必要的，可能会导致PIP命令报错，安装后需重新编译Python
检查命令：# rpm -aq|grep openssl
安装命令：# yum install openssl-devel -y

二、安装zlib

安装命令：# yum -y install zlib*

三、下载安装Python3.6

解压后进入目录
修改/Modules/Setup文件如下，文件名区分大小写

# -*- makefile -*-
# The file Setup is used by the makesetup script to construct the files
# Makefile and config.c, from Makefile.pre and config.c.in,
# respectively.  The file Setup itself is initially copied from
# Setup.dist; once it exists it will not be overwritten, so you can edit
# Setup to your heart's content.  Note that Makefile.pre is created
# from Makefile.pre.in by the toplevel configure script.# (VPATH notes: Setup and Makefile.pre are in the build directory, as
# are Makefile and config.c; the *.in and *.dist files are in the source
# directory.)# Each line in this file describes one or more optional modules.
# Modules enabled here will not be compiled by the setup.py script,
# so the file can be used to override setup.py's behavior.# Lines have the following structure:
#
# <module> ... [<sourcefile> ...] [<cpparg> ...] [<library> ...]
#
# <sourcefile> is anything ending in .c (.C, .cc, .c++ are C++ files)
# <cpparg> is anything starting with -I, -D, -U or -C
# <library> is anything ending in .a or beginning with -l or -L
# <module> is anything else but should be a valid Python
# identifier (letters, digits, underscores, beginning with non-digit)
#
# (As the makesetup script changes, it may recognize some other
# arguments as well, e.g. *.so and *.sl as libraries.  See the big
# case statement in the makesetup script.)
#
# Lines can also have the form
#
# <name> = <value>
#
# which defines a Make variable definition inserted into Makefile.in
#
# Finally, if a line contains just the word "*shared*" (without the
# quotes but with the stars), then the following modules will not be
# built statically.  The build process works like this:
#
# 1. Build all modules that are declared as static in Modules/Setup,
#    combine them into libpythonxy.a, combine that into python.
# 2. Build all modules that are listed as shared in Modules/Setup.
# 3. Invoke setup.py. That builds all modules that
#    a) are not builtin, and
#    b) are not listed in Modules/Setup, and
#    c) can be build on the target
#
# Therefore, modules declared to be shared will not be
# included in the config.c file, nor in the list of objects to be
# added to the library archive, and their linker options won't be
# added to the linker options. Rules to create their .o files and
# their shared libraries will still be added to the Makefile, and
# their names will be collected in the Make variable SHAREDMODS.  This
# is used to build modules as shared libraries.  (They can be
# installed using "make sharedinstall", which is implied by the
# toplevel "make install" target.)  (For compatibility,
# *noconfig* has the same effect as *shared*.)
#
# In addition, *static* explicitly declares the following modules to
# be static.  Lines containing "*static*" and "*shared*" may thus
# alternate throughout this file.# NOTE: As a standard policy, as many modules as can be supported by a
# platform should be present.  The distribution comes with all modules
# enabled that are supported by most platforms and don't require you
# to ftp sources from elsewhere.# Some special rules to define PYTHONPATH.
# Edit the definitions below to indicate which options you are using.
# Don't add any whitespace or comments!# Directories where library files get installed.
# DESTLIB is for Python modules; MACHDESTLIB for shared libraries.
DESTLIB=$(LIBDEST)
MACHDESTLIB=$(BINLIBDEST)# NOTE: all the paths are now relative to the prefix that is computed
# at run time!# Standard path -- don't edit.
# No leading colon since this is the first entry.
# Empty since this is now just the runtime prefix.
DESTPATH=# Site specific path components -- should begin with : if non-empty
SITEPATH=# Standard path components for test modules
TESTPATH=# Path components for machine- or system-dependent modules and shared libraries
MACHDEPPATH=:$(PLATDIR)
EXTRAMACHDEPPATH=COREPYTHONPATH=$(DESTPATH)$(SITEPATH)$(TESTPATH)$(MACHDEPPATH)$(EXTRAMACHDEPPATH)
PYTHONPATH=$(COREPYTHONPATH)# The modules listed here can't be built as shared libraries for
# various reasons; therefore they are listed here instead of in the
# normal order.# This only contains the minimal set of modules required to run the
# setup.py script in the root of the Python source tree.posix posixmodule.c     # posix (UNIX) system calls
errno errnomodule.c     # posix (UNIX) errno values
pwd pwdmodule.c         # this is needed to find out the user's home dir# if $HOME is not set
_sre _sre.c         # Fredrik Lundh's new regular expressions
_codecs _codecsmodule.c     # access to the builtin codecs and codec registry
_weakref _weakref.c     # weak references
_functools _functoolsmodule.c   # Tools for working with functions and callable objects
_operator _operator.c           # operator.add() and similar goodies
_collections _collectionsmodule.c # Container types
itertools itertoolsmodule.c    # Functions creating iterators for efficient looping
atexit atexitmodule.c      # Register functions to be run at interpreter-shutdown
_signal signalmodule.c
_stat _stat.c           # stat.h interface
time timemodule.c   # -lm # time operations and variables# access to ISO C locale support
_locale _localemodule.c  # -lintl# Standard I/O baseline
_io -I$(srcdir)/Modules/_io _io/_iomodule.c _io/iobase.c _io/fileio.c _io/bytesio.c _io/bufferedio.c _io/textio.c _io/stringio.c# The zipimport module is always imported at startup. Having it as a
# builtin module avoids some bootstrapping problems and reduces overhead.
zipimport zipimport.c# faulthandler module
faulthandler faulthandler.c# debug tool to trace memory blocks allocated by Python
_tracemalloc _tracemalloc.c hashtable.c# The rest of the modules listed in this file are all commented out by
# default.  Usually they can be detected and built as dynamically
# loaded modules by the new setup.py script added in Python 2.1.  If
# you're on a platform that doesn't support dynamic loading, want to
# compile modules statically into the Python binary, or need to
# specify some odd set of compiler switches, you can uncomment the
# appropriate lines below.# ======================================================================# The Python symtable module depends on .h files that setup.py doesn't track
_symtable symtablemodule.c# Uncommenting the following line tells makesetup that all following
# modules are to be built as shared libraries (see above for more
# detail; also note that *static* reverses this effect):#*shared*# GNU readline.  Unlike previous Python incarnations, GNU readline is
# now incorporated in an optional module, configured in the Setup file
# instead of by a configure script switch.  You may have to insert a
# -L option pointing to the directory where libreadline.* lives,
# and you may have to change -ltermcap to -ltermlib or perhaps remove
# it, depending on your system -- see the GNU readline instructions.
# It's okay for this to be a shared library, too.#readline readline.c -lreadline -ltermcap# Modules that should always be present (non UNIX dependent):#array arraymodule.c    # array objects
#cmath cmathmodule.c _math.c # -lm # complex math library functions
#math mathmodule.c _math.c # -lm # math library functions, e.g. sin()
#_struct _struct.c  # binary structure packing/unpacking
#_weakref _weakref.c    # basic weak reference support
#_testcapi _testcapimodule.c    # Python C API test module
#_random _randommodule.c    # Random number generator
#_elementtree -I$(srcdir)/Modules/expat -DHAVE_EXPAT_CONFIG_H -DUSE_PYEXPAT_CAPI _elementtree.c    # elementtree accelerator
#_pickle _pickle.c  # pickle accelerator
#_datetime _datetimemodule.c    # datetime accelerator
#_bisect _bisectmodule.c    # Bisection algorithms
#_heapq _heapqmodule.c  # Heap queue algorithm
#_asyncio _asynciomodule.c  # Fast asyncio Future#unicodedata unicodedata.c    # static Unicode character database# Modules with some UNIX dependencies -- on by default:
# (If you have a really backward UNIX, select and socket may not be
# supported...)#fcntl fcntlmodule.c    # fcntl(2) and ioctl(2)
#spwd spwdmodule.c      # spwd(3)
#grp grpmodule.c        # grp(3)
#select selectmodule.c  # select(2); not on ancient System V# Memory-mapped files (also works on Win32).
#mmap mmapmodule.c# CSV file helper
#_csv _csv.c# Socket module helper for socket(2)
_socket socketmodule.c# Socket module helper for SSL support; you must comment out the other
# socket line above, and possibly edit the SSL variable:
SSL=/usr/local/ssl
_ssl _ssl.c \-DUSE_SSL -I$(SSL)/include -I$(SSL)/include/openssl \-L$(SSL)/lib -lssl -lcrypto# The crypt module is now disabled by default because it breaks builds
# on many systems (where -lcrypt is needed), e.g. Linux (I believe).
#
# First, look at Setup.config; configure may have set this for you.#_crypt _cryptmodule.c # -lcrypt    # crypt(3); needs -lcrypt on some systems# Some more UNIX dependent modules -- off by default, since these
# are not supported by all UNIX systems:#nis nismodule.c -lnsl  # Sun yellow pages -- not everywhere
#termios termios.c  # Steen Lumholt's termios module
#resource resource.c    # Jeremy Hylton's rlimit interface#_posixsubprocess _posixsubprocess.c  # POSIX subprocess module helper# Multimedia modules -- off by default.
# These don't work for 64-bit platforms!!!
# #993173 says audioop works on 64-bit platforms, though.
# These represent audio samples or images as strings:#audioop audioop.c  # Operations on audio samples# Note that the _md5 and _sha modules are normally only built if the
# system does not have the OpenSSL libs containing an optimized version.# The _md5 module implements the RSA Data Security, Inc. MD5
# Message-Digest Algorithm, described in RFC 1321.#_md5 md5module.c# The _sha module implements the SHA checksum algorithms.
# (NIST's Secure Hash Algorithms.)
#_sha1 sha1module.c
#_sha256 sha256module.c
#_sha512 sha512module.c
#_sha3 _sha3/sha3module.c# _blake module
#_blake2 _blake2/blake2module.c _blake2/blake2b_impl.c _blake2/blake2s_impl.c# The _tkinter module.
#
# The command for _tkinter is long and site specific.  Please
# uncomment and/or edit those parts as indicated.  If you don't have a
# specific extension (e.g. Tix or BLT), leave the corresponding line
# commented out.  (Leave the trailing backslashes in!  If you
# experience strange errors, you may want to join all uncommented
# lines and remove the backslashes -- the backslash interpretation is
# done by the shell's "read" command and it may not be implemented on
# every system.# *** Always uncomment this (leave the leading underscore in!):
# _tkinter _tkinter.c tkappinit.c -DWITH_APPINIT \
# *** Uncomment and edit to reflect where your Tcl/Tk libraries are:
#   -L/usr/local/lib \
# *** Uncomment and edit to reflect where your Tcl/Tk headers are:
#   -I/usr/local/include \
# *** Uncomment and edit to reflect where your X11 header files are:
#   -I/usr/X11R6/include \
# *** Or uncomment this for Solaris:
#   -I/usr/openwin/include \
# *** Uncomment and edit for Tix extension only:
#   -DWITH_TIX -ltix8.1.8.2 \
# *** Uncomment and edit for BLT extension only:
#   -DWITH_BLT -I/usr/local/blt/blt8.0-unoff/include -lBLT8.0 \
# *** Uncomment and edit for PIL (TkImaging) extension only:
#     (See http://www.pythonware.com/products/pil/ for more info)
#   -DWITH_PIL -I../Extensions/Imaging/libImaging  tkImaging.c \
# *** Uncomment and edit for TOGL extension only:
#   -DWITH_TOGL togl.c \
# *** Uncomment and edit to reflect your Tcl/Tk versions:
#   -ltk8.2 -ltcl8.2 \
# *** Uncomment and edit to reflect where your X11 libraries are:
#   -L/usr/X11R6/lib \
# *** Or uncomment this for Solaris:
#   -L/usr/openwin/lib \
# *** Uncomment these for TOGL extension only:
#   -lGL -lGLU -lXext -lXmu \
# *** Uncomment for AIX:
#   -lld \
# *** Always uncomment this; X11 libraries to link with:
#   -lX11# Lance Ellinghaus's syslog module
#syslog syslogmodule.c      # syslog daemon interface# Curses support, requiring the System V version of curses, often
# provided by the ncurses library.  e.g. on Linux, link with -lncurses
# instead of -lcurses).
#
# First, look at Setup.config; configure may have set this for you.#_curses _cursesmodule.c -lcurses -ltermcap
# Wrapper for the panel library that's part of ncurses and SYSV curses.
#_curses_panel _curses_panel.c -lpanel -lncurses# Modules that provide persistent dictionary-like semantics.  You will
# probably want to arrange for at least one of them to be available on
# your machine, though none are defined by default because of library
# dependencies.  The Python module dbm/__init__.py provides an
# implementation independent wrapper for these; dbm/dumb.py provides
# similar functionality (but slower of course) implemented in Python.# The standard Unix dbm module has been moved to Setup.config so that
# it will be compiled as a shared library by default.  Compiling it as
# a built-in module causes conflicts with the pybsddb3 module since it
# creates a static dependency on an out-of-date version of db.so.
#
# First, look at Setup.config; configure may have set this for you.#_dbm _dbmmodule.c  # dbm(3) may require -lndbm or similar# Anthony Baxter's gdbm module.  GNU dbm(3) will require -lgdbm:
#
# First, look at Setup.config; configure may have set this for you.#_gdbm _gdbmmodule.c -I/usr/local/include -L/usr/local/lib -lgdbm# Helper module for various ascii-encoders
#binascii binascii.c# Fred Drake's interface to the Python parser
#parser parsermodule.c# Lee Busby's SIGFPE modules.
# The library to link fpectl with is platform specific.
# Choose *one* of the options below for fpectl:# For SGI IRIX (tested on 5.3):
#fpectl fpectlmodule.c -lfpe# For Solaris with SunPro compiler (tested on Solaris 2.5 with SunPro C 4.2):
# (Without the compiler you don't have -lsunmath.)
#fpectl fpectlmodule.c -R/opt/SUNWspro/lib -lsunmath -lm# For other systems: see instructions in fpectlmodule.c.
#fpectl fpectlmodule.c ...# Test module for fpectl.  No extra libraries needed.
#fpetest fpetestmodule.c# Andrew Kuchling's zlib module.
# This require zlib 1.1.3 (or later).
# See http://www.gzip.org/zlib/
zlib zlibmodule.c -I$(prefix)/include -L$(exec_prefix)/lib -lz# Interface to the Expat XML parser
#
# Expat was written by James Clark and is now maintained by a group of
# developers on SourceForge; see www.libexpat.org for more
# information.  The pyexpat module was written by Paul Prescod after a
# prototype by Jack Jansen.  Source of Expat 1.95.2 is included in
# Modules/expat/.  Usage of a system shared libexpat.so/expat.dll is
# not advised.
#
# More information on Expat can be found at www.libexpat.org.
#
#pyexpat expat/xmlparse.c expat/xmlrole.c expat/xmltok.c pyexpat.c -I$(srcdir)/Modules/expat -DHAVE_EXPAT_CONFIG_H -DUSE_PYEXPAT_CAPI# Hye-Shik Chang's CJKCodecs# multibytecodec is required for all the other CJK codec modules
#_multibytecodec cjkcodecs/multibytecodec.c#_codecs_cn cjkcodecs/_codecs_cn.c
#_codecs_hk cjkcodecs/_codecs_hk.c
#_codecs_iso2022 cjkcodecs/_codecs_iso2022.c
#_codecs_jp cjkcodecs/_codecs_jp.c
#_codecs_kr cjkcodecs/_codecs_kr.c
#_codecs_tw cjkcodecs/_codecs_tw.c# Example -- included for reference only:
# xx xxmodule.c# Another example -- the 'xxsubtype' module shows C-level subtyping in action
xxsubtype xxsubtype.c

安装命令：# ./configure –prefix=/usr/local/python3
编译命令：# make && make install
python软链接命令：# ln -s /usr/local/python3/bin/python3.6 /usr/local/bin/python3
pip软链接命令：# ln -s /usr/local/python3/bin/pip3.6 /usr/local/bin/pip
软链接是为了在终端中可以直接使用python3关键字，注意关键字要与软链接一致。
也可以在/etc/profile中修改环境变量，但是Linux系统自带Python在/usr/bin目录下，系统优先搜索此目录，有屏蔽作用，故不可使用python作为关键字。

四、安装Twisted

下载地址：https://pypi.python.org/packages/source/T/Twisted/Twisted-15.2.1.tar.bz2#md5=4be066a899c714e18af1ecfcb01cfef7
解压后进入目录运行：# python3 setup.py install

五、安装Scrapy

安装命令：# pip install Scrapy

安装至此完成，如有其他环境遗漏根据错误信息持续更新。