目的：

使用scrapy框架进行租房信息（杭州地区）爬取，包括小区名称，位置，价格，面积，房间朝向、户型等，然后把爬取的信息保存到本地csv表格中。

分析：

某家的网站爬取不难，看一下页面，需要爬取的是小区名称，位置，价格，面积，房间朝向、户型，和房源维护时间，爬取完当前页面后再爬取下一页信息，这里主要是使用resquest和xpath方法进行爬取，然后使用pandas库进行数据保存，从而把数据保存到本地。

代码部分：

items.py

# -*- coding: utf-8 -*-# Define here the models for your scraped items
#
# See documentation in:
# https://docs.scrapy.org/en/latest/topics/items.htmlimport scrapyclass LianjiaItem(scrapy.Item):# define the fields for your item here like:# name = scrapy.Field()#passtitle = scrapy.Field()     #租房标题quyu = scrapy.Field()     #所在市区，如西湖区ziquyu = scrapy.Field()    #所在子市区，如西湖区的转塘house_name = scrapy.Field()  #小区名称mianji = scrapy.Field()   #mianjichaoxiang = scrapy.Field() #房子朝向huxing = scrapy.Field()    #房子户型price = scrapy.Field()  #租房价格date = scrapy.Field()  #维护日期

spiders.py

# -*- coding: utf-8 -*-
import scrapy
from lianjia.items import LianjiaItem
import urllibclass LianjiawangSpider(scrapy.Spider):name = 'lianjiawang'allowed_domains = ['hz.XXXXXX.com']start_urls = ['https://hz.XXXXX.com/zufang/']def parse(self, response):#passdiv_list = response.xpath('//div[@class="content__article"]/div[@class="content__list"]/div[@class="content__list--item"]')for div in div_list:item = LianjiaItem()item['title'] = div.xpath('./div/p[1]/a[@class="twoline"]/text()').extract_first()    #租房标题if item['title'] is not None:item['title'] = item['title'].strip()item['quyu'] = div.xpath('./div/p[2]/a[1]/text()').extract_first()    #所在市区，如西湖区item['ziquyu'] = div.xpath('./div/p[2]/a[2]/text()').extract_first()      #所在子市区，如西湖区的转塘item['house_name'] = div.xpath('./div/p[2]/a[3]/text()').extract_first()      #小区名称item['mianji']  = div.xpath('./div/p[2]/text()[5]').extract_first().strip()   #房子面积item['chaoxiang']  = div.xpath('./div/p[2]/text()[6]').extract_first()  #房子朝向if item['chaoxiang'] is not None:item['chaoxiang'] = item['chaoxiang'].strip()item['huxing'] = div.xpath('./div/p[2]/text()[7]').extract_first()   #房子户型if item['huxing'] is not None:item['huxing'] = item['huxing'].strip()item['price'] = div.xpath('./div/span/em/text()').extract_first() + '元/月'  #租房价格url = div.xpath('./div/p[1]/a[@class="twoline"]/@href').extract_first()  #详情页链接地址if url is not None:url =  'https://hz.XXXXXX.com/' + urlyield scrapy.Request(url,callback = self.parse_detail,meta={'item': item})# 下一页请求pages = list(range(2, 6))   #获取前5页的租房信息for page in pages:next_urls = 'https://hz.XXXXXX.com/zufang/pg{}/#contentList'.format(page)#print(next_urls)if next_urls is not None:next_urls = next_urlsyield scrapy.Request(next_urls,callback=self.parse,)def parse_detail(self, response):  # 处理详情页item = response.meta['item']item['date'] = response.xpath('//div[@class="content__subtitle"]/text()[1]').extract_first()  #房源日期if item['date'] is not  None:item['date'] = item['date'].strip()yield item

pipelines.py

# -*- coding: utf-8 -*-# Define your item pipelines here
#
# Don't forget to add your pipeline to the ITEM_PIPELINES setting
# See: https://docs.scrapy.org/en/latest/topics/item-pipeline.html# class LianjiaPipeline(object):
#     def process_item(self, item, spider):
#         return itemimport pandas as pd#下载保存数据，可用
class LianjiaPipeline(object):# def process_item(self, item, spider):#     #return item#     print("name:", item['name'])#     print("url:", item['url'])def __init__(self):  # 定义构造器，初始化要写入的文件self.mypd = pd.DataFrame(columns=['title', 'quyu','ziquyu','house_name','mianji','chaoxiang','huxing ','price','date'])def close_spider(self, spider):  # 重写close_spider回调方法self.mypd.to_csv("lianjia_data.csv")def process_item(self, item, spider):  # 添加数据到pandas中self.mypd = self.mypd.append({'title': item['title'], 'quyu': item['quyu'],'ziquyu':item['ziquyu'],'house_name':item['house_name'],'mianji':item['mianji'],'chaoxiang': item['chaoxiang'], 'huxing': item['huxing'], 'price': item['price'],'date': item['date']}, ignore_index=True)print(len(self.mypd))

settings.py

BOT_NAME = 'lianjia'SPIDER_MODULES = ['lianjia.spiders']
NEWSPIDER_MODULE = 'lianjia.spiders'LOG_LEVEL = "WARNING"
ROBOTSTXT_OBEY = True#随机使用User-Agent，一种反扒手段
USER_AGENTS_LIST = ["Mozilla/5.0 (Windows NT 6.1; WOW64) AppleWebKit/537.1 (KHTML, like Gecko) Chrome/22.0.1207.1 Safari/537.1","Mozilla/5.0 (X11; CrOS i686 2268.111.0) AppleWebKit/536.11 (KHTML, like Gecko) Chrome/20.0.1132.57 Safari/536.11","Mozilla/5.0 (Windows NT 6.1; WOW64) AppleWebKit/536.6 (KHTML, like Gecko) Chrome/20.0.1092.0 Safari/536.6","Mozilla/5.0 (Windows NT 6.2) AppleWebKit/536.6 (KHTML, like Gecko) Chrome/20.0.1090.0 Safari/536.6","Mozilla/5.0 (Windows NT 6.2; WOW64) AppleWebKit/537.1 (KHTML, like Gecko) Chrome/19.77.34.5 Safari/537.1","Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/536.5 (KHTML, like Gecko) Chrome/19.0.1084.9 Safari/536.5","Mozilla/5.0 (Windows NT 6.0) AppleWebKit/536.5 (KHTML, like Gecko) Chrome/19.0.1084.36 Safari/536.5","Mozilla/5.0 (Windows NT 6.1; WOW64) AppleWebKit/536.3 (KHTML, like Gecko) Chrome/19.0.1063.0 Safari/536.3","Mozilla/5.0 (Windows NT 5.1) AppleWebKit/536.3 (KHTML, like Gecko) Chrome/19.0.1063.0 Safari/536.3","Mozilla/5.0 (Macintosh; Intel Mac OS X 10_8_0) AppleWebKit/536.3 (KHTML, like Gecko) Chrome/19.0.1063.0 Safari/536.3","Mozilla/5.0 (Windows NT 6.2) AppleWebKit/536.3 (KHTML, like Gecko) Chrome/19.0.1062.0 Safari/536.3","Mozilla/5.0 (Windows NT 6.1; WOW64) AppleWebKit/536.3 (KHTML, like Gecko) Chrome/19.0.1062.0 Safari/536.3","Mozilla/5.0 (Windows NT 6.2) AppleWebKit/536.3 (KHTML, like Gecko) Chrome/19.0.1061.1 Safari/536.3","Mozilla/5.0 (Windows NT 6.1; WOW64) AppleWebKit/536.3 (KHTML, like Gecko) Chrome/19.0.1061.1 Safari/536.3","Mozilla/5.0 (Windows NT 6.1) AppleWebKit/536.3 (KHTML, like Gecko) Chrome/19.0.1061.1 Safari/536.3","Mozilla/5.0 (Windows NT 6.2) AppleWebKit/536.3 (KHTML, like Gecko) Chrome/19.0.1061.0 Safari/536.3","Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/535.24 (KHTML, like Gecko) Chrome/19.0.1055.1 Safari/535.24","Mozilla/5.0 (Windows NT 6.2; WOW64) AppleWebKit/535.24 (KHTML, like Gecko) Chrome/19.0.1055.1 Safari/535.24"]
USER_AGENT = random.choice(USER_AGENTS_LIST)
DEFAULT_REQUEST_HEADERS = {'Accept': 'text/html,application/xhtml+xml,application/xml;q=0.9,*/*;q=0.8','Accept-Language': 'en',# 'User-Agent':"Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/58.0.3029.110 Safari/537.36",'User-Agent':USER_AGENT
}ITEM_PIPELINES = {'lianjia.pipelines.LianjiaPipeline': 300,
}

运行结果：

scrapy某家租房信息爬取相关推荐

基于scrapy下的租房信息爬取与数据展示工具的设计与实现
环境:python 3.6.0 Anaconda custom 64bit 4.3.0 Pycharm x64 专业版 2018.1.2 Web strom x64 专业版 2018.1.3 scra ...
scrapy 斗鱼主播信息爬取
原文链接: scrapy 斗鱼主播信息爬取上一篇: scrapy 妹子图网站全站图片爬取下一篇: TensorFlow models 的slim 模块使用预训练模型进行识别 api http ...
爬虫实战：链家租房数据爬取，实习僧网站数据爬取
前面已经进行了爬虫基础部分的学习,于是自己也尝试爬了一些网站数据,用的策略都是比较简单,可能有些因素没有考虑到,但是也爬取到了一定的数据,下面介绍两个爬过的案例. 爬虫实战链家网站爬取实习僧网站爬 ...
爬虫系列之链家的信息爬取及数据分析
关于链家的数据爬取和分析已经实现 1.房屋数据爬取并下载 2.房屋按区域分析 3.房屋按经纪人分析 4.前十经纪人 5.经纪人最有可能的位置分析 6.实现以地区划分房屋目前存在的问题: 1.多线程 ...
链家武汉租房信息爬取并可视化数据分析
爬取的信息内容:区域,标题,二级区域,小区名字,大小,朝向,户型,租金项目下载链接:https://pan.baidu.com/s/15LcBtBARBlqs5xuJW4TgWQ 提取码:njp8 ...
链家网页爬虫_爬虫实战1-----链家二手房信息爬取
经过一段机器学习之后,发现实在是太枯燥了,为了增添一些趣味性以及熟练爬虫,在之后会不定时的爬取一些网站旨在熟悉网页结构--尤其是HTML的元素,ajax存储,json:熟练使用pyspider,sc ...
PyQt5+Python+Excel链家二手房信息爬取、可视化以及数据存取
成果图: 第一步运行代码searsh.py,效果如下第二步选择你所需要爬取数据的城市,如湖北-武汉然后搜索,结果如下如果你想爬取更多信息,只需要点击下一页即可第三步,保存数据.可以将所显示的所 ...
python爬虫分析——广州租房信息爬取并分析
一.在某网站上爬取相关数据爬取到的数据情况如下: 二.数据的读取和预处理(爬取下来的数据没有缺失值和异常值) import pandas as pd import numpy as np impor ...
Python——安居客租房信息爬取（以南昌为例）
前言: 提前安装好所需要的库. 本代码的输入仅需要某个城市的租房地址首页即可,其他自会生成. 使用前请创建所需的目录,或者为代码添加os.makedir() 支持断点重爬,重行运行即可. header ...

scrapy某家租房信息爬取

目的：

分析：

代码部分：

items.py

spiders.py

pipelines.py

settings.py

运行结果：

scrapy某家租房信息爬取相关推荐

最新文章

热门文章