Scrapy爬取贝壳网并存入csv

一、分析目标站点

本文以爬取宁波地区的房产信息为例

1、宁波整个市页面

宁波地区的网址都是以上面的格式，再加上各个区的拼音，即可进入各个区的网页。

各个区的名称：

areas = {'海曙区': 'haishuqu1', '江北区': 'jiangbeiqu1','镇海区': 'zhenhaiqu1', '北仑区': 'beilunqu1','鄞州区': 'yinzhouqu2', '余姚市': 'yuyaoshi','慈溪市': 'cixishi', '奉化区': 'fenghuaqu','象山县': 'xiangshanxian', '宁海县': 'ninghaixian'}

2、解析各个区的页面

分析海曙区的页面发现随着页数的增加，页面的变化规律为 pg + 当页的页数

而各个区页面的页数是不相同的，比如海曙区的有100页，而宁海县的确只有一页。

通过查看源代码发现，各个区的最大页数在totalPage属性中。

接下我们使用requests库与BeautifulSoup库提取出出各个区的最大页数

import requests
from bs4 import BeautifulSoupurl = 'https://nb.ke.com/ershoufang/haishuqu1/pg2/'
response = requests.get(url)
max_num = BeautifulSoup(response.text, 'lxml').find(class_='house-lst-page-box')
min = max_num.attrs['page-data'].find(":")
max = max_num.attrs['page-data'].find(",")
page = int(max_num.attrs['page-data'][min+1:max])
print(page)

再利用for循环迭代各个区的最大页数，将其与各区url连接起来便可得到各个区所有页面的url

for i in range(1, page+1):print('https://nb.ke.com/ershoufang/haishuqu1/pg{}/'.format(i))

找出当页每个房源信息的url

url = 'https://nb.ke.com/ershoufang/haishuqu1/pg2/'
response = requests.get(url)
soups = BeautifulSoup(response.text, 'lxml').find_all('li', class_='clear')
for soup in soups:every_url = soup.find(class_='title').a.attrs['href']print(every_url)

3、解析各个房源的页面

在这里我们取小区、单价、总价和以下信息。

房源基本信息都在base属性下的li标签里

房源基本信息中所要提取的信息使用正则匹配，因为这几个信息可能会不完全而且位置有时候会发生变化。

url = 'https://nb.ke.com/ershoufang/19102810810100142243.html'response = requests.get(url)soup = BeautifulSoup(response.text, 'lxml')
li = soup.find(class_='base').find_all('li')
content = u'{}'.format(soup.find(class_='base'))  # re匹配的对象
totalPrice = soup.find(class_='total').get_text().strip()  # 总价
unitPrice = soup.find(class_='unitPrice').get_text().strip()   # 单价
houseName = soup.find(class_='communityName').a.get_text().strip()  # 小区楼盘try:description = re.search('<li>.*?房屋户型</span>(.*?)</li>', content, re.S).group(1)  # 房子描述floor = re.search('<li>.*?所在楼层</span>(.*?)</li>', content, re.S).group(1)
# 楼层信息area = re.search('<li>.*?建筑面积</span>(.*?)</li>', content, re.S).group(1)
# 面积orientation = re.search('<li>.*?房屋朝向</span>(.*?)</li>', content, re.S).group(1)  # 朝向propertyAge = re.search('<li>.*?产权年限</span>(.*?)</li>', content, re.S).group(1)   # 房产年限decoration = re.search('<li>.*?装修情况</span>(.*?)</li>', content, re.S).group(1)    # 装修情况elevator = re.search('<li>.*?梯户比例</span>(.*?)</li>', content, re.S).group(1)
# 梯户比例
except:elevator = '无电梯'description = ' 'floor = ' 'area = ' 'orientation = ' 'propertyAge = ' 'decoration = ' 'print(houseName, description, floor, orientation, totalPrice, unitPrice, propertyAge, area, elevator, propertyAge)

二、整合代码

1、创建项目

用这行代码开始一个scrapy项目

2、创建spider

进入创建好的文件，然后输入想要的文件名称和目标网址

创建好的文件

items.py

这个脚本主要是用来定义我们要获取的字段。

from scrapy import Item, Fieldclass BeikeItem(Item):# define the fields for your item here like:# name = scrapy.Field()url = Field()houseName = Field()     # 小区楼盘position = Field()      # 房子所属区totalPrice = Field()    # 总价unitPrice = Field()     # 单价description = Field()   # 房子描述floor = Field()         # 楼层信息area = Field()          # 总面积orientation = Field()   # 朝向elevator = Field()      # 梯户比例propertyAge = Field()   # 房产年限decoration = Field()    # 装修情况

spider.py

# -*- coding: utf-8 -*-
import re
import time
import random
import scrapy
from bs4 import BeautifulSoup
from scrapy.http import Request
from Beike.items import BeikeItemclass BeikeSpider(scrapy.Spider):name = 'beike'allowed_domains = ['nb.ke.com']base_url = 'https://nb.ke.com/ershoufang/'areas = {'海曙区': 'haishuqu1', '江北区': 'jiangbeiqu1','镇海区': 'zhenhaiqu1', '北仑区': 'beilunqu1','鄞州区': 'yinzhouqu2', '余姚市': 'yuyaoshi','慈溪市': 'cixishi', '奉化区': 'fenghuaqu','象山县': 'xiangshanxian', '宁海县': 'ninghaixian'}def start_requests(self):for area in self.areas:url = self.base_url + self.areas[area]  # areas的value，也就是拼音的字段 yield Request(url, callback=self.page,meta={'url': url, 'area': area})  # 将url和地区名字传送到page函数中去# 获得宁波所有区的页面def page(self, response):url = response.meta['url']area = response.meta['area']soup = BeautifulSoup(response.text, 'lxml').find(class_='house-lst-page-box').attrs['page-data']# 每一个去的最大页数包含在 house-lst-page-box 内# 包含最大页数标签 {"totalPage":6,"curPage":1}min = soup.find(":")max = soup.find(",")max_num = soup[min + 1:max]pages = int(max_num)    # 最大页数是字符串类型，将它转换成文本数值类型for page in range(1, pages+1):page_url = url + "/pg{}/".format(page)print("*" * 50)print('正在爬取{}房产的第{}页'.format(area, page))ran = random.randint(2, 5)time.sleep(ran)# 来识别正在爬哪一区的哪一页yield Request(page_url, callback=self.every_url,meta={'area': area})  # 将地区传到下面去def every_url(self, response):soups = BeautifulSoup(response.text, 'lxml').find_all('li', class_='clear')for soup in soups:every_url = soup.find(class_='title').a.attrs['href']# print(every_url)ran = random.randint(2, 3)time.sleep(ran)     # 先暂停个2、3秒避免给目标站点服务器太大压力yield Request(every_url, callback=self.parse, meta={'area': response.meta['area']})# 找出本页所有房产的url，并请求该urldef parse(self, response):url = response.url                              # 本页面的urlposition = response.meta['area']                # 所在的小区soup = BeautifulSoup(response.text, 'lxml')content = u'{}'.format(soup.find(class_='base'))             # re匹配的对象totalPrice = soup.find(class_='total').get_text().strip()           # 总价unitPrice = soup.find(class_='unitPrice').get_text().strip()        # 单价houseName = soup.find(class_='communityName').a.get_text().strip()  # 小区楼盘try:description = re.search('<li>.*?房屋户型</span>(.*?)</li>', content, re.S).group(1)     # 房子描述floor = re.search('<li>.*?所在楼层</span>(.*?)</li>', content, re.S).group(1)           # 楼层信息area = re.search('<li>.*?建筑面积</span>(.*?)</li>', content, re.S).group(1)            # 面积orientation = re.search('<li>.*?房屋朝向</span>(.*?)</li>', content, re.S).group(1)     # 朝向propertyAge = re.search('<li>.*?产权年限</span>(.*?)</li>', content, re.S).group(1)     # 房产年限decoration = re.search('<li>.*?装修情况</span>(.*?)</li>', content, re.S).group(1)      # 装修情况elevator = re.search('<li>.*?梯户比例</span>(.*?)</li>', content, re.S).group(1)        # 梯户比例except:elevator = '无电梯'description = ' 'floor = ' 'area = ' 'orientation = ' 'propertyAge = ' 'decoration = ' '# 有些房源的基本信息可能没有，所以用空白字符代替item = BeikeItem()item['url'] = urlitem['position'] = positionitem['houseName'] = houseNameitem['description'] = descriptionitem['floor'] = flooritem['orientation'] = orientationitem['totalPrice'] = totalPriceitem['unitPrice'] = unitPriceitem['area'] = areaitem['elevator'] = elevatoritem['propertyAge'] = propertyAgeitem['decoration'] = decorationyield item# yield是返回一个可迭代的对象

setting.py

# -*- coding: utf-8 -*-# Scrapy settings for Beike project
#
# For simplicity, this file contains only settings considered important or
# commonly used. You can find more settings consulting the documentation:
#
#     https://doc.scrapy.org/en/latest/topics/settings.html
#     https://doc.scrapy.org/en/latest/topics/downloader-middleware.html
#     https://doc.scrapy.org/en/latest/topics/spider-middleware.htmlBOT_NAME = 'Beike'SPIDER_MODULES = ['Beike.spiders']
NEWSPIDER_MODULE = 'Beike.spiders'# Crawl responsibly by identifying yourself (and your website) on the user-agent
#USER_AGENT = 'Beike (+http://www.yourdomain.com)'# Obey robots.txt rules
ROBOTSTXT_OBEY = False   # 是否遵循爬取协议，不的话FALSE# Configure maximum concurrent requests performed by Scrapy (default: 16)
#CONCURRENT_REQUESTS = 32# Configure a delay for requests for the same website (default: 0)
# See https://doc.scrapy.org/en/latest/topics/settings.html#download-delay
# See also autothrottle settings and docs
#DOWNLOAD_DELAY = 3
# The download delay setting will honor only one of:
#CONCURRENT_REQUESTS_PER_DOMAIN = 16
#CONCURRENT_REQUESTS_PER_IP = 16# Disable cookies (enabled by default)
#COOKIES_ENABLED = False# Disable Telnet Console (enabled by default)
#TELNETCONSOLE_ENABLED = False# Override the default request headers:
DEFAULT_REQUEST_HEADERS = {'Accept': 'text/html,application/xhtml+xml,application/xml;q=0.9,image/webp,image/apng,*/*;q=0.8,application/signed-exchange;v=b3;q=0.9','Accept-Language': 'zh-CN,zh;q=0.9','User-Agent': 'Mozilla/5.0 (Windows NT 10.0; WOW64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/80.0.3987.116 Safari/537.36','Host': 'nb.ke.com'     # 加header头，模仿成浏览器
}
# Enable or disable spider middlewares
# See https://doc.scrapy.org/en/latest/topics/spider-middleware.html
#SPIDER_MIDDLEWARES = {
#    'Beike.middlewares.BeikeSpiderMiddleware': 543,
#}# Enable or disable downloader middlewares
# See https://doc.scrapy.org/en/latest/topics/downloader-middleware.html
#DOWNLOADER_MIDDLEWARES = {
#    'Beike.middlewares.BeikeDownloaderMiddleware': 543,
#}# Enable or disable extensions
# See https://doc.scrapy.org/en/latest/topics/extensions.html
#EXTENSIONS = {
#    'scrapy.extensions.telnet.TelnetConsole': None,
#}# Configure item pipelines
# See https://doc.scrapy.org/en/latest/topics/item-pipeline.html
ITEM_PIPELINES = {'Beike.pipelines.BeikePipeline': 300,   # 开启这个后pipeline下载组件才会运行
}
# Enable and configure the AutoThrottle extension (disabled by default)
# See https://doc.scrapy.org/en/latest/topics/autothrottle.html
#AUTOTHROTTLE_ENABLED = True
# The initial download delay
#AUTOTHROTTLE_START_DELAY = 5
# The maximum download delay to be set in case of high latencies
#AUTOTHROTTLE_MAX_DELAY = 60
# The average number of requests Scrapy should be sending in parallel to
# each remote server
#AUTOTHROTTLE_TARGET_CONCURRENCY = 1.0
# Enable showing throttling stats for every response received:
#AUTOTHROTTLE_DEBUG = False
HTTPERROR_ALLOWED_CODES = [301]  # 防止重定向
# Enable and configure HTTP caching (disabled by default)
# See https://doc.scrapy.org/en/latest/topics/downloader-middleware.html#httpcache-middleware-settings
HTTPCACHE_ENABLED = True
HTTPCACHE_EXPIRATION_SECS = 0
HTTPCACHE_DIR = 'httpcache'
HTTPCACHE_IGNORE_HTTP_CODES = []
HTTPCACHE_STORAGE = 'scrapy.extensions.httpcache.FilesystemCacheStorage'
# 这几行注释的作用是，Scrapy会缓存你有的Requests!
# 当你再次请求时，如果存在缓存文档则返回缓存文档，
# 而不是去网站请求，这样既加快了本地调试速度，
# 也减轻了 网站的压力。一举多得

pipeline.py

import csv
import osclass BeikePipeline(object):def __init__(self):self.header = ['url', 'position', 'houseName', 'description', 'floor', 'orientation', 'totalPrice', 'unitPrice']# csv文件的位置,无需事先创建store_file = os.path.dirname(__file__) + '/spiders/data1.csv'# #获取当前运行脚本的绝对路径# 打开(创建)文件self.file = open(store_file, 'w+', newline='')# w+以读写的方式写入文件，后面的参数是为了解决插入一行数据后产生空白行的问题# csv写法self.writer = csv.writer(self.file)def process_item(self, item, spider):# 判断字段值不为空再以行的方式写入文件self.writer.writerow((item['url'], item['position'], item['houseName'], item['description'],item['floor'], item['orientation'], item['totalPrice'], item['unitPrice'],item['area'], item['elevator'], item['propertyAge'], item['decoration']))return itemdef close_spider(self, spider):# 关闭爬虫时顺便将文件保存退出self.file.close()

爬完了，一共13270条数据

Scrapy爬取贝壳网并存入csv相关推荐

Scrapy爬取豆瓣图书详情存入CSV并导入MySQL
目录前言一.新建爬虫工程二.修改settings.py文件三.编写items.py 四.编写pipelines.py 五.编写book.py 六.存储到MySQL 七.爬取结果八.后言前言 ...
python 爬取贝壳网小区名称_Python爬虫实战：爬取贝壳网二手房40000条数据
前言本文的文字及图片来源于网络,仅供学习.交流使用,不具有任何商业用途,版权归原作者所有,如有问题请及时联系我们以作处理以下文章来源于啤酒就辣条 ,作者啤酒就辣条一.网页分析爬取贝壳网石家庄二 ...
BeautifulSoup爬取贝壳网成都二手房源信息（附：完整源代码）
BeautifulSoup爬取贝壳网成都二手房源信息最近想出一套二手房,刚好在学习爬虫,那就顺手爬一下贝壳上成都各区二手房信息了解一下行情,本篇先介绍数据爬取,数据分析稍后有时间再写.仅用于技术交流 ...
Python爬虫实战：爬取贝壳网二手房40000条数据
前言本文的文字及图片来源于网络,仅供学习.交流使用,不具有任何商业用途,版权归原作者所有,如有问题请及时联系我们以作处理以下文章来源于啤酒就辣条 ,作者啤酒就辣条一.网页分析爬取贝壳网石家庄二 ...
python爬虫scrapy爬取新闻标题及链接_python爬虫框架scrapy爬取梅花网资讯信息
原标题:python爬虫框架scrapy爬取梅花网资讯信息一.介绍本例子用scrapy-splash爬取梅花网(http://www.meihua.info/a/list/today)的资讯信息, ...
Scrapy爬取当当网图书销售前100
scrapy爬取当当网图书畅销榜一.采集任务爬取当当网图书畅销榜信息,获取热销图书前500相关数据. 二.网页解析 1. 打开当当网,按照图书榜>图书畅销榜进入当当网图书畅销榜[http: ...
Scrapy爬取当当网的商品信息存到MySQL数据库
Scrapy爬取当当网的商品信息存到MySQL数据库 Scrapy 是一款十分强大的爬虫框架,能够快速简单地爬取网页,存到你想要的位置.经过两天的摸索,终于搞定了一个小任务,将当当网的商品信息爬下来存 ...
python实战|用scrapy爬取当当网数据
本文的文字及图片来源于网络,仅供学习.交流使用,不具有任何商业用途,版权归原作者所有,如有问题请及时联系我们以作处理以下文章来源于腾讯云作者:Python进击者 ( 想要学习Python?Pyth ...
利用python爬取贝壳网租房信息
前言本文的文字及图片来源于网络,仅供学习.交流使用,不具有任何商业用途,如有问题请及时联系我们以作处理. PS:如有需要Python学习资料的小伙伴可以加点击下方链接自行获取 python免费学习资 ...