目标：爬取某一学习网站上课程信息，前期调试仅获取课程名称

爬虫文件：

import scrapy

from xtzx.items import XtzxItem

from scrapy.http import Request

class LessonSpider(scrapy.Spider):

name = 'lesson'

allowed_domains = ['xuetangx.com']

start_urls = ['http://www.xuetangx.com/courses/course-v1:TsinghuaX+80512073X+2018_T1/about']

'''

def start_requests(self):

ua={"User-Agent":"Mozilla/5.0 (Windows NT 10.0; WOW64; Trident/7.0; rv:11.0) like Gecko"}

yield Request("www.xuetangx.com/courses/course-v1:TsinghuaX+80512073X+2018_T1/about",headers=ua)

'''

def parse(self, response):

item=XtzxItem()

item["title"]=response.xpath("//div[@class='title_detail'/h3[@class='courseabout_title']/text()").extract()

print(item["title"])

执行日志：

2018-04-28 11:08:33 [scrapy.utils.log] INFO: Scrapy 1.5.0 started (bot: xtzx)

2018-04-28 11:08:33 [scrapy.utils.log] INFO: Versions: lxml 4.2.1.0, libxml2 2.9.7, cssselect 1.0.3, parsel 1.4.0, w3lib 1.19.0, Twisted 17.9.0, Python 3.5.4 (v3.5.4:3f56838, Aug 8 2017, 02:17:05) [MSC v.1900 64 bit (AMD64)], pyOpenSSL 17.5.0 (OpenSSL 1.1.0h 27 Mar 2018), cryptography 2.2.2, Platform Windows-10-10.0.16299-SP0

2018-04-28 11:08:33 [scrapy.crawler] INFO: Overridden settings: {'SPIDER_MODULES': ['xtzx.spiders'], 'BOT_NAME': 'xtzx', 'NEWSPIDER_MODULE': 'xtzx.spiders', 'USER_AGENT': 'Mozilla/5.0 (Windows NT 10.0; WOW64; Trident/7.0; rv:11.0) like Gecko'}

2018-04-28 11:08:33 [scrapy.middleware] INFO: Enabled extensions:

['scrapy.extensions.corestats.CoreStats',

'scrapy.extensions.telnet.TelnetConsole',

'scrapy.extensions.logstats.LogStats']

2018-04-28 11:08:34 [scrapy.middleware] INFO: Enabled downloader middlewares:

['scrapy.downloadermiddlewares.httpauth.HttpAuthMiddleware',

'scrapy.downloadermiddlewares.downloadtimeout.DownloadTimeoutMiddleware',

'scrapy.downloadermiddlewares.defaultheaders.DefaultHeadersMiddleware',

'scrapy.downloadermiddlewares.useragent.UserAgentMiddleware',

'scrapy.downloadermiddlewares.retry.RetryMiddleware',

'scrapy.downloadermiddlewares.redirect.MetaRefreshMiddleware',

'scrapy.downloadermiddlewares.httpcompression.HttpCompressionMiddleware',

'scrapy.downloadermiddlewares.redirect.RedirectMiddleware',

'scrapy.downloadermiddlewares.cookies.CookiesMiddleware',

'scrapy.downloadermiddlewares.httpproxy.HttpProxyMiddleware',

'scrapy.downloadermiddlewares.stats.DownloaderStats']

2018-04-28 11:08:34 [scrapy.middleware] INFO: Enabled spider middlewares:

['scrapy.spidermiddlewares.httperror.HttpErrorMiddleware',

'scrapy.spidermiddlewares.offsite.OffsiteMiddleware',

'scrapy.spidermiddlewares.referer.RefererMiddleware',

'scrapy.spidermiddlewares.urllength.UrlLengthMiddleware',

'scrapy.spidermiddlewares.depth.DepthMiddleware']

2018-04-28 11:08:34 [scrapy.middleware] INFO: Enabled item pipelines:

[]

2018-04-28 11:08:34 [scrapy.core.engine] INFO: Spider opened

----------好像从这开始出问题

2018-04-28 11:08:34 [scrapy.extensions.logstats] INFO: Crawled 0 pages (at 0 pages/min), scraped 0 items (at 0 items/min)

2018-04-28 11:08:34 [scrapy.extensions.telnet] DEBUG: Telnet console listening on 127.0.0.1:6023

2018-04-28 11:08:34 [scrapy.core.engine] DEBUG: Crawled (200) (referer: None)

2018-04-28 11:08:34 [scrapy.core.scraper] ERROR: Spider error processing (referer: None)

Traceback (most recent call last):

File "d:python3.5libsite-packagesparselselector.py", line 228, in xpath

**kwargs)

File "srclxmletree.pyx", line 1577, in lxml.etree._Element.xpath

File "srclxmlxpath.pxi", line 307, in lxml.etree.XPathElementEvaluator.__call__

File "srclxmlxpath.pxi", line 227, in lxml.etree._XPathEvaluatorBase._handle_result

lxml.etree.XPathEvalError: Invalid predicate

During handling of the above exception, another exception occurred:

Traceback (most recent call last):

File "d:python3.5libsite-packagestwistedinternetdefer.py", line 653, in _runCallbacks

current.result = callback(current.result, *args, **kw)

File "E:pythonxtzxxtzxspiderslesson.py", line 16, in parse

item["title"]=response.xpath("//div[@class='title_detail'/h3[@class='courseabout_title']/text()").extract()

File "d:python3.5libsite-packagesscrapyhttpresponsetext.py", line 119, in xpath

return self.selector.xpath(query, **kwargs)

File "d:python3.5libsite-packagesparselselector.py", line 232, in xpath

six.reraise(ValueError, ValueError(msg), sys.exc_info()[2])

File "d:python3.5libsite-packagessix.py", line 692, in reraise

raise value.with_traceback(tb)

File "d:python3.5libsite-packagesparselselector.py", line 228, in xpath

**kwargs)

File "srclxmletree.pyx", line 1577, in lxml.etree._Element.xpath

File "srclxmlxpath.pxi", line 307, in lxml.etree.XPathElementEvaluator.__call__

File "srclxmlxpath.pxi", line 227, in lxml.etree._XPathEvaluatorBase._handle_result

ValueError: XPath error: Invalid predicate in //div[@class='title_detail'/h3[@class='courseabout_title']/text()

2018-04-28 11:08:35 [scrapy.core.engine] INFO: Closing spider (finished)

2018-04-28 11:08:35 [scrapy.statscollectors] INFO: Dumping Scrapy stats:

{'downloader/request_bytes': 301,

'downloader/request_count': 1,

'downloader/request_method_count/GET': 1,

'downloader/response_bytes': 24409,

'downloader/response_count': 1,

'downloader/response_status_count/200': 1,

'finish_reason': 'finished',

'finish_time': datetime.datetime(2018, 4, 28, 3, 8, 35, 118088),

'log_count/DEBUG': 2,

'log_count/ERROR': 1,

'log_count/INFO': 7,

'response_received_count': 1,

'scheduler/dequeued': 1,

'scheduler/dequeued/memory': 1,

'scheduler/enqueued': 1,

'scheduler/enqueued/memory': 1,

'spider_exceptions/ValueError': 1,

'start_time': datetime.datetime(2018, 4, 28, 3, 8, 34, 418003)}

2018-04-28 11:08:35 [scrapy.core.engine] INFO: Spider closed (finished)

感觉程序很简单，但是就是不行，其他items都是常规的设置，pipelines里面没有添加新的内容，然后settings里面就修改了一下ROBOTSTXT_OBEY的值

网上查了很久这样的错误，都没找到相应的方法，也试过伪装浏览器爬取也没用，自学，没有老师，完全没辙了，求助各位.