思路：

step 1：爬取单个详情页的信息(getInfo(url)函数)

step 2：在一个列表页爬取所有详情页的链接，再调用getInfo(url)函数，得到该列表页中所有详情页的信息(onePageInfo(list_url)函数)

step 3：通过观察每个列表页网址，发现网址组成规律，调用onePageInfo(list_url)爬取信息

难点：

判断房东性别

结果展示：

总结：

房东性别通过房东图片右下角的小图片来判断，小图片class属性能区分男女，存在的问题是有些房东没设置性别，无法判断，这里默认为男性。通过字符串format()方法嵌套if-else实现

发现得到的address后面跟了\n和一段空格，这里用字符串rstrip()方法去除

房间图片的属性为style，这里用列表切片得到需要信息

列表推导式：

urls = ['{}'.format(str(i)) for i in range(1,8)] 正确

urls = ['{}'.format(str(i) for i in range(1,8))] 错误

我的代码：

frombs4importBeautifulSoup

importrequests

#获得单个详情页的信息

defgetInfo(url):

wb_data = requests.get(url)

soup = BeautifulSoup(wb_data.text,'lxml')

titles = soup.select('body > div.wrap.clearfix.con_bg > div.con_l > div.pho_info > h4 > em')

addresses = soup.select('body > div.wrap.clearfix.con_bg > div.con_l > div.pho_info > p > span')

rentals = soup.select('#pricePart > div.day_l > span')

house_images = soup.select('#imgMouseCusor')

lord_images = soup.select('#floatRightBox > div.js_box.clearfix > div.member_pic > a > img')

lord_names = soup.select('#floatRightBox > div.js_box.clearfix > div.w_240 > h6 > a')

lord_sexs = soup.select('#floatRightBox > div.js_box.clearfix > div.member_pic > div')

fortitle,address,rental,house_image,lord_image,lord_name,lord_sexinzip(titles,addresses,rentals,house_images,lord_images,lord_names,lord_sexs):

data = {

'title': title.get_text(),

'address': address.get_text().rstrip(),

'rental': rental.get_text(),

'house_image': house_image.get('style')[16:-2],

'lord_image': lord_image.get('src'),

'lord_name': lord_name.get_text(),

'lord_sex':"{}".format('female'iflord_sex['class'][0] =="member_ico1"else'male')

}

print(data)

#得到一整页

defonePageInfo(list_url):

page_list = requests.get(list_url)

page_list_soup = BeautifulSoup(page_list.text,'lxml')

house_list = page_list_soup.select('#page_list > ul > li > a')

'''

house_link = []

for i in house_list:

house_link.append(i['href'])

print(house_link)

'''

foriinhouse_list:

getInfo(i['href'])

urls = ['http://bj.xiaozhu.com/search-duanzufang-p{}-0/'.format(str(i))foriinrange(1,8)]

#urls = ['http://bj.xiaozhu.com/search-duanzufang-p{}-0/'.format(str(i) for i in range(1,8))]错误

forurlinurls:

onePageInfo(url)