文章目录

前言
一、分析url
二、拼接url
- 1.实例化chrome
- 2.获取首字符、page
- 3.拼接url
三、获取房源数据

前言

本文全面解析了链家新房源数据，爬取了全部城市的房源信息，共两万六千条数据

提示：以下是本篇文章正文内容，下面案例可供参考

一、分析url

这个url的构成很简单，https://后面是城市的首字母 pg后表示页数。我们为了拿到所有城市的房源数据，需要构造出准确无误的所有url 有了url利用request进行访问

现在我们有两个部分是必须要获取的一个是拿下所有城市的首字母，一个是拿下对应城市的页数

https://bj.fang.lianjia.com/loupan/pg1/

这样的话问题就来了，去哪里获取城市的首字符和对应的页数呢，我们打开他的首页看一看，点击圈中的按钮之后就会出现所选的城市，我们可以通过这些城市拿到对应的首字母

我们右键检查元素，每个城市对应的就是每个城市房源数据的url，我们拿到他们的url提取出首字符就可以了

现在有了首字符，还缺对应城市的页数，我的方法是利用selenium访问每个url然后拿到他们的总页数

每个页数对应一个a标签，当存在两个及两个以上页数时我们那他倒数第二个a标签（有下一页按钮）
当仅仅只有一页时我们就直接让他等于1（没有下一页按钮)

二、拼接url

根据上文我们有了思路现在我们就去实现代码

1.实例化chrome

基本配置

代码如下（示例）：

chrome_options = Options()
chrome_options.add_argument('--headless')
chrome_options.add_argument('--disable-gpu')  #设置无头浏览器
prefs = {'profile.managed_default_content_settings.images': 2}
chrome_options.add_experimental_option('prefs', prefs) # 禁止图片加载，加快速度
bro = webdriver.Chrome(options=chrome_options)#实例化对象
})

2.获取首字符、page

先顺顺思路，我们进入页面之后要干啥

#mermaid-svg-g1RV9GqxTrslu7BS .label{font-family:'trebuchet ms', verdana, arial;font-family:var(--mermaid-font-family);fill:#333;color:#333}#mermaid-svg-g1RV9GqxTrslu7BS .label text{fill:#333}#mermaid-svg-g1RV9GqxTrslu7BS .node rect,#mermaid-svg-g1RV9GqxTrslu7BS .node circle,#mermaid-svg-g1RV9GqxTrslu7BS .node ellipse,#mermaid-svg-g1RV9GqxTrslu7BS .node polygon,#mermaid-svg-g1RV9GqxTrslu7BS .node path{fill:#ECECFF;stroke:#9370db;stroke-width:1px}#mermaid-svg-g1RV9GqxTrslu7BS .node .label{text-align:center;fill:#333}#mermaid-svg-g1RV9GqxTrslu7BS .node.clickable{cursor:pointer}#mermaid-svg-g1RV9GqxTrslu7BS .arrowheadPath{fill:#333}#mermaid-svg-g1RV9GqxTrslu7BS .edgePath .path{stroke:#333;stroke-width:1.5px}#mermaid-svg-g1RV9GqxTrslu7BS .flowchart-link{stroke:#333;fill:none}#mermaid-svg-g1RV9GqxTrslu7BS .edgeLabel{background-color:#e8e8e8;text-align:center}#mermaid-svg-g1RV9GqxTrslu7BS .edgeLabel rect{opacity:0.9}#mermaid-svg-g1RV9GqxTrslu7BS .edgeLabel span{color:#333}#mermaid-svg-g1RV9GqxTrslu7BS .cluster rect{fill:#ffffde;stroke:#aa3;stroke-width:1px}#mermaid-svg-g1RV9GqxTrslu7BS .cluster text{fill:#333}#mermaid-svg-g1RV9GqxTrslu7BS div.mermaidTooltip{position:absolute;text-align:center;max-width:200px;padding:2px;font-family:'trebuchet ms', verdana, arial;font-family:var(--mermaid-font-family);font-size:12px;background:#ffffde;border:1px solid #aa3;border-radius:2px;pointer-events:none;z-index:100}#mermaid-svg-g1RV9GqxTrslu7BS .actor{stroke:#ccf;fill:#ECECFF}#mermaid-svg-g1RV9GqxTrslu7BS text.actor>tspan{fill:#000;stroke:none}#mermaid-svg-g1RV9GqxTrslu7BS .actor-line{stroke:grey}#mermaid-svg-g1RV9GqxTrslu7BS .messageLine0{stroke-width:1.5;stroke-dasharray:none;stroke:#333}#mermaid-svg-g1RV9GqxTrslu7BS .messageLine1{stroke-width:1.5;stroke-dasharray:2, 2;stroke:#333}#mermaid-svg-g1RV9GqxTrslu7BS #arrowhead path{fill:#333;stroke:#333}#mermaid-svg-g1RV9GqxTrslu7BS .sequenceNumber{fill:#fff}#mermaid-svg-g1RV9GqxTrslu7BS #sequencenumber{fill:#333}#mermaid-svg-g1RV9GqxTrslu7BS #crosshead path{fill:#333;stroke:#333}#mermaid-svg-g1RV9GqxTrslu7BS .messageText{fill:#333;stroke:#333}#mermaid-svg-g1RV9GqxTrslu7BS .labelBox{stroke:#ccf;fill:#ECECFF}#mermaid-svg-g1RV9GqxTrslu7BS .labelText,#mermaid-svg-g1RV9GqxTrslu7BS .labelText>tspan{fill:#000;stroke:none}#mermaid-svg-g1RV9GqxTrslu7BS .loopText,#mermaid-svg-g1RV9GqxTrslu7BS .loopText>tspan{fill:#000;stroke:none}#mermaid-svg-g1RV9GqxTrslu7BS .loopLine{stroke-width:2px;stroke-dasharray:2, 2;stroke:#ccf;fill:#ccf}#mermaid-svg-g1RV9GqxTrslu7BS .note{stroke:#aa3;fill:#fff5ad}#mermaid-svg-g1RV9GqxTrslu7BS .noteText,#mermaid-svg-g1RV9GqxTrslu7BS .noteText>tspan{fill:#000;stroke:none}#mermaid-svg-g1RV9GqxTrslu7BS .activation0{fill:#f4f4f4;stroke:#666}#mermaid-svg-g1RV9GqxTrslu7BS .activation1{fill:#f4f4f4;stroke:#666}#mermaid-svg-g1RV9GqxTrslu7BS .activation2{fill:#f4f4f4;stroke:#666}#mermaid-svg-g1RV9GqxTrslu7BS .mermaid-main-font{font-family:"trebuchet ms", verdana, arial;font-family:var(--mermaid-font-family)}#mermaid-svg-g1RV9GqxTrslu7BS .section{stroke:none;opacity:0.2}#mermaid-svg-g1RV9GqxTrslu7BS .section0{fill:rgba(102,102,255,0.49)}#mermaid-svg-g1RV9GqxTrslu7BS .section2{fill:#fff400}#mermaid-svg-g1RV9GqxTrslu7BS .section1,#mermaid-svg-g1RV9GqxTrslu7BS .section3{fill:#fff;opacity:0.2}#mermaid-svg-g1RV9GqxTrslu7BS .sectionTitle0{fill:#333}#mermaid-svg-g1RV9GqxTrslu7BS .sectionTitle1{fill:#333}#mermaid-svg-g1RV9GqxTrslu7BS .sectionTitle2{fill:#333}#mermaid-svg-g1RV9GqxTrslu7BS .sectionTitle3{fill:#333}#mermaid-svg-g1RV9GqxTrslu7BS .sectionTitle{text-anchor:start;font-size:11px;text-height:14px;font-family:'trebuchet ms', verdana, arial;font-family:var(--mermaid-font-family)}#mermaid-svg-g1RV9GqxTrslu7BS .grid .tick{stroke:#d3d3d3;opacity:0.8;shape-rendering:crispEdges}#mermaid-svg-g1RV9GqxTrslu7BS .grid .tick text{font-family:'trebuchet ms', verdana, arial;font-family:var(--mermaid-font-family)}#mermaid-svg-g1RV9GqxTrslu7BS .grid path{stroke-width:0}#mermaid-svg-g1RV9GqxTrslu7BS .today{fill:none;stroke:red;stroke-width:2px}#mermaid-svg-g1RV9GqxTrslu7BS .task{stroke-width:2}#mermaid-svg-g1RV9GqxTrslu7BS .taskText{text-anchor:middle;font-family:'trebuchet ms', verdana, arial;font-family:var(--mermaid-font-family)}#mermaid-svg-g1RV9GqxTrslu7BS .taskText:not([font-size]){font-size:11px}#mermaid-svg-g1RV9GqxTrslu7BS .taskTextOutsideRight{fill:#000;text-anchor:start;font-size:11px;font-family:'trebuchet ms', verdana, arial;font-family:var(--mermaid-font-family)}#mermaid-svg-g1RV9GqxTrslu7BS .taskTextOutsideLeft{fill:#000;text-anchor:end;font-size:11px}#mermaid-svg-g1RV9GqxTrslu7BS .task.clickable{cursor:pointer}#mermaid-svg-g1RV9GqxTrslu7BS .taskText.clickable{cursor:pointer;fill:#003163 !important;font-weight:bold}#mermaid-svg-g1RV9GqxTrslu7BS .taskTextOutsideLeft.clickable{cursor:pointer;fill:#003163 !important;font-weight:bold}#mermaid-svg-g1RV9GqxTrslu7BS .taskTextOutsideRight.clickable{cursor:pointer;fill:#003163 !important;font-weight:bold}#mermaid-svg-g1RV9GqxTrslu7BS .taskText0,#mermaid-svg-g1RV9GqxTrslu7BS .taskText1,#mermaid-svg-g1RV9GqxTrslu7BS .taskText2,#mermaid-svg-g1RV9GqxTrslu7BS .taskText3{fill:#fff}#mermaid-svg-g1RV9GqxTrslu7BS .task0,#mermaid-svg-g1RV9GqxTrslu7BS .task1,#mermaid-svg-g1RV9GqxTrslu7BS .task2,#mermaid-svg-g1RV9GqxTrslu7BS .task3{fill:#8a90dd;stroke:#534fbc}#mermaid-svg-g1RV9GqxTrslu7BS .taskTextOutside0,#mermaid-svg-g1RV9GqxTrslu7BS .taskTextOutside2{fill:#000}#mermaid-svg-g1RV9GqxTrslu7BS .taskTextOutside1,#mermaid-svg-g1RV9GqxTrslu7BS .taskTextOutside3{fill:#000}#mermaid-svg-g1RV9GqxTrslu7BS .active0,#mermaid-svg-g1RV9GqxTrslu7BS .active1,#mermaid-svg-g1RV9GqxTrslu7BS .active2,#mermaid-svg-g1RV9GqxTrslu7BS .active3{fill:#bfc7ff;stroke:#534fbc}#mermaid-svg-g1RV9GqxTrslu7BS .activeText0,#mermaid-svg-g1RV9GqxTrslu7BS .activeText1,#mermaid-svg-g1RV9GqxTrslu7BS .activeText2,#mermaid-svg-g1RV9GqxTrslu7BS .activeText3{fill:#000 !important}#mermaid-svg-g1RV9GqxTrslu7BS .done0,#mermaid-svg-g1RV9GqxTrslu7BS .done1,#mermaid-svg-g1RV9GqxTrslu7BS .done2,#mermaid-svg-g1RV9GqxTrslu7BS .done3{stroke:grey;fill:#d3d3d3;stroke-width:2}#mermaid-svg-g1RV9GqxTrslu7BS .doneText0,#mermaid-svg-g1RV9GqxTrslu7BS .doneText1,#mermaid-svg-g1RV9GqxTrslu7BS .doneText2,#mermaid-svg-g1RV9GqxTrslu7BS .doneText3{fill:#000 !important}#mermaid-svg-g1RV9GqxTrslu7BS .crit0,#mermaid-svg-g1RV9GqxTrslu7BS .crit1,#mermaid-svg-g1RV9GqxTrslu7BS .crit2,#mermaid-svg-g1RV9GqxTrslu7BS .crit3{stroke:#f88;fill:red;stroke-width:2}#mermaid-svg-g1RV9GqxTrslu7BS .activeCrit0,#mermaid-svg-g1RV9GqxTrslu7BS .activeCrit1,#mermaid-svg-g1RV9GqxTrslu7BS .activeCrit2,#mermaid-svg-g1RV9GqxTrslu7BS .activeCrit3{stroke:#f88;fill:#bfc7ff;stroke-width:2}#mermaid-svg-g1RV9GqxTrslu7BS .doneCrit0,#mermaid-svg-g1RV9GqxTrslu7BS .doneCrit1,#mermaid-svg-g1RV9GqxTrslu7BS .doneCrit2,#mermaid-svg-g1RV9GqxTrslu7BS .doneCrit3{stroke:#f88;fill:#d3d3d3;stroke-width:2;cursor:pointer;shape-rendering:crispEdges}#mermaid-svg-g1RV9GqxTrslu7BS .milestone{transform:rotate(45deg) scale(0.8, 0.8)}#mermaid-svg-g1RV9GqxTrslu7BS .milestoneText{font-style:italic}#mermaid-svg-g1RV9GqxTrslu7BS .doneCritText0,#mermaid-svg-g1RV9GqxTrslu7BS .doneCritText1,#mermaid-svg-g1RV9GqxTrslu7BS .doneCritText2,#mermaid-svg-g1RV9GqxTrslu7BS .doneCritText3{fill:#000 !important}#mermaid-svg-g1RV9GqxTrslu7BS .activeCritText0,#mermaid-svg-g1RV9GqxTrslu7BS .activeCritText1,#mermaid-svg-g1RV9GqxTrslu7BS .activeCritText2,#mermaid-svg-g1RV9GqxTrslu7BS .activeCritText3{fill:#000 !important}#mermaid-svg-g1RV9GqxTrslu7BS .titleText{text-anchor:middle;font-size:18px;fill:#000;font-family:'trebuchet ms', verdana, arial;font-family:var(--mermaid-font-family)}#mermaid-svg-g1RV9GqxTrslu7BS g.classGroup text{fill:#9370db;stroke:none;font-family:'trebuchet ms', verdana, arial;font-family:var(--mermaid-font-family);font-size:10px}#mermaid-svg-g1RV9GqxTrslu7BS g.classGroup text .title{font-weight:bolder}#mermaid-svg-g1RV9GqxTrslu7BS g.clickable{cursor:pointer}#mermaid-svg-g1RV9GqxTrslu7BS g.classGroup rect{fill:#ECECFF;stroke:#9370db}#mermaid-svg-g1RV9GqxTrslu7BS g.classGroup line{stroke:#9370db;stroke-width:1}#mermaid-svg-g1RV9GqxTrslu7BS .classLabel .box{stroke:none;stroke-width:0;fill:#ECECFF;opacity:0.5}#mermaid-svg-g1RV9GqxTrslu7BS .classLabel .label{fill:#9370db;font-size:10px}#mermaid-svg-g1RV9GqxTrslu7BS .relation{stroke:#9370db;stroke-width:1;fill:none}#mermaid-svg-g1RV9GqxTrslu7BS .dashed-line{stroke-dasharray:3}#mermaid-svg-g1RV9GqxTrslu7BS #compositionStart{fill:#9370db;stroke:#9370db;stroke-width:1}#mermaid-svg-g1RV9GqxTrslu7BS #compositionEnd{fill:#9370db;stroke:#9370db;stroke-width:1}#mermaid-svg-g1RV9GqxTrslu7BS #aggregationStart{fill:#ECECFF;stroke:#9370db;stroke-width:1}#mermaid-svg-g1RV9GqxTrslu7BS #aggregationEnd{fill:#ECECFF;stroke:#9370db;stroke-width:1}#mermaid-svg-g1RV9GqxTrslu7BS #dependencyStart{fill:#9370db;stroke:#9370db;stroke-width:1}#mermaid-svg-g1RV9GqxTrslu7BS #dependencyEnd{fill:#9370db;stroke:#9370db;stroke-width:1}#mermaid-svg-g1RV9GqxTrslu7BS #extensionStart{fill:#9370db;stroke:#9370db;stroke-width:1}#mermaid-svg-g1RV9GqxTrslu7BS #extensionEnd{fill:#9370db;stroke:#9370db;stroke-width:1}#mermaid-svg-g1RV9GqxTrslu7BS .commit-id,#mermaid-svg-g1RV9GqxTrslu7BS .commit-msg,#mermaid-svg-g1RV9GqxTrslu7BS .branch-label{fill:lightgrey;color:lightgrey;font-family:'trebuchet ms', verdana, arial;font-family:var(--mermaid-font-family)}#mermaid-svg-g1RV9GqxTrslu7BS .pieTitleText{text-anchor:middle;font-size:25px;fill:#000;font-family:'trebuchet ms', verdana, arial;font-family:var(--mermaid-font-family)}#mermaid-svg-g1RV9GqxTrslu7BS .slice{font-family:'trebuchet ms', verdana, arial;font-family:var(--mermaid-font-family)}#mermaid-svg-g1RV9GqxTrslu7BS g.stateGroup text{fill:#9370db;stroke:none;font-size:10px;font-family:'trebuchet ms', verdana, arial;font-family:var(--mermaid-font-family)}#mermaid-svg-g1RV9GqxTrslu7BS g.stateGroup text{fill:#9370db;fill:#333;stroke:none;font-size:10px}#mermaid-svg-g1RV9GqxTrslu7BS g.statediagram-cluster .cluster-label text{fill:#333}#mermaid-svg-g1RV9GqxTrslu7BS g.stateGroup .state-title{font-weight:bolder;fill:#000}#mermaid-svg-g1RV9GqxTrslu7BS g.stateGroup rect{fill:#ECECFF;stroke:#9370db}#mermaid-svg-g1RV9GqxTrslu7BS g.stateGroup line{stroke:#9370db;stroke-width:1}#mermaid-svg-g1RV9GqxTrslu7BS .transition{stroke:#9370db;stroke-width:1;fill:none}#mermaid-svg-g1RV9GqxTrslu7BS .stateGroup .composit{fill:white;border-bottom:1px}#mermaid-svg-g1RV9GqxTrslu7BS .stateGroup .alt-composit{fill:#e0e0e0;border-bottom:1px}#mermaid-svg-g1RV9GqxTrslu7BS .state-note{stroke:#aa3;fill:#fff5ad}#mermaid-svg-g1RV9GqxTrslu7BS .state-note text{fill:black;stroke:none;font-size:10px}#mermaid-svg-g1RV9GqxTrslu7BS .stateLabel .box{stroke:none;stroke-width:0;fill:#ECECFF;opacity:0.7}#mermaid-svg-g1RV9GqxTrslu7BS .edgeLabel text{fill:#333}#mermaid-svg-g1RV9GqxTrslu7BS .stateLabel text{fill:#000;font-size:10px;font-weight:bold;font-family:'trebuchet ms', verdana, arial;font-family:var(--mermaid-font-family)}#mermaid-svg-g1RV9GqxTrslu7BS .node circle.state-start{fill:black;stroke:black}#mermaid-svg-g1RV9GqxTrslu7BS .node circle.state-end{fill:black;stroke:white;stroke-width:1.5}#mermaid-svg-g1RV9GqxTrslu7BS #statediagram-barbEnd{fill:#9370db}#mermaid-svg-g1RV9GqxTrslu7BS .statediagram-cluster rect{fill:#ECECFF;stroke:#9370db;stroke-width:1px}#mermaid-svg-g1RV9GqxTrslu7BS .statediagram-cluster rect.outer{rx:5px;ry:5px}#mermaid-svg-g1RV9GqxTrslu7BS .statediagram-state .divider{stroke:#9370db}#mermaid-svg-g1RV9GqxTrslu7BS .statediagram-state .title-state{rx:5px;ry:5px}#mermaid-svg-g1RV9GqxTrslu7BS .statediagram-cluster.statediagram-cluster .inner{fill:white}#mermaid-svg-g1RV9GqxTrslu7BS .statediagram-cluster.statediagram-cluster-alt .inner{fill:#e0e0e0}#mermaid-svg-g1RV9GqxTrslu7BS .statediagram-cluster .inner{rx:0;ry:0}#mermaid-svg-g1RV9GqxTrslu7BS .statediagram-state rect.basic{rx:5px;ry:5px}#mermaid-svg-g1RV9GqxTrslu7BS .statediagram-state rect.divider{stroke-dasharray:10,10;fill:#efefef}#mermaid-svg-g1RV9GqxTrslu7BS .note-edge{stroke-dasharray:5}#mermaid-svg-g1RV9GqxTrslu7BS .statediagram-note rect{fill:#fff5ad;stroke:#aa3;stroke-width:1px;rx:0;ry:0}:root{--mermaid-font-family: '"trebuchet ms", verdana, arial';--mermaid-font-family: "Comic Sans MS", "Comic Sans", cursive}#mermaid-svg-g1RV9GqxTrslu7BS .error-icon{fill:#522}#mermaid-svg-g1RV9GqxTrslu7BS .error-text{fill:#522;stroke:#522}#mermaid-svg-g1RV9GqxTrslu7BS .edge-thickness-normal{stroke-width:2px}#mermaid-svg-g1RV9GqxTrslu7BS .edge-thickness-thick{stroke-width:3.5px}#mermaid-svg-g1RV9GqxTrslu7BS .edge-pattern-solid{stroke-dasharray:0}#mermaid-svg-g1RV9GqxTrslu7BS .edge-pattern-dashed{stroke-dasharray:3}#mermaid-svg-g1RV9GqxTrslu7BS .edge-pattern-dotted{stroke-dasharray:2}#mermaid-svg-g1RV9GqxTrslu7BS .marker{fill:#333}#mermaid-svg-g1RV9GqxTrslu7BS .marker.cross{stroke:#333}:root { --mermaid-font-family: "trebuchet ms", verdana, arial;} #mermaid-svg-g1RV9GqxTrslu7BS {color: rgba(0, 0, 0, 0.75);font: ;}

获取首字符

进入界面

点击城市按钮

点击每一个城市

点击筛选

获取页数

我们在点击城市按钮后出现了城市列表获取城市首字符，在我们点击第三步筛选后我们获取最大页数

有了这个思路现在我们写代码

打开网页定位所有城市列表

bro.get('https://bj.fang.lianjia.com/')#打开网页
#城市列表
li_list=bro.find_elements_by_xpath("/html/body/div[3]/div[3]//a")

定位每个元素循环点击，点击 --》点击筛选 --》退回 --》点击

因为每次退回主页页面重新的刷新，所以每次循环最后在从新获取li_list

获取最大页数的代码位置一定要在指定的位置，只有切换到房源页是才能找到最大页数

for i in range(len(li_list)): bro.find_element_by_xpath("/html/body/div[1]/div/div[1]/a[2]").click()  # 点击城市按钮li_list[i].click() #点击城市#在这获取页面首字符bro.find_element_by_xpath('//div[@data-search-position="search_result"]').click() #点击筛选ws = bro.window_handles # 当前所有页面bro.switch_to.window(ws[1])  # 切换新页面,房源页#在这获取最大页数，并保存数据bro.close() #关闭房源页bro.switch_to.window(ws[0]) # 回到主页li_list=bro.find_elements_by_xpath('/html/body/div[3]/div[3]//a')

页面首字符代码，我还多加了一个城市名称

 dic["city"]=li_list[i].get_attribute("href").replace(".fang.lianjia.com/","").replace("https://","") # 城市首字母dic["city_name"]=li_list[i].text #每个城市

最大页数、保存数据代码

当数据存在多个页数时是有“下一页”这个按钮的，当只有一页时是不存在“下一页”按钮，以此判断页数，最后边爬边保存

 if "下一页" in bro.page_source:dic["page"]=bro.find_element_by_xpath('//div[@class="page-box"]/a[last()-1]').text #最多页数else:dic["page"]="1"with open(".//city_page.csv", "a", encoding="utf-8") as f:writer = csv.DictWriter(f, dic.keys())writer.writerow(dic)

3.拼接url

我们获取了最大page、和首字符现在开始拼接url

names=["city","name","page"]
df=pd.read_csv("city_page.csv",names=names)  #读入数据
df.page=df.page.map(lambda x:re.findall("(\d+)",str(x))[0])  #提取出页数
for a, b in zip(df.city, df.page):for i in range(int(b)):url = "https://%s.fang.lianjia.com/loupan/pg%d/" % (a, i+1)  #拼接url#保存url为txtwith open("urls.txt","a",encoding="utf-8") as f:f.write(url)f.write("\n")

到此为止，所有城市的url全部拼接完毕，一共有2669条url，有了这些我们就可以用requests多线程快速的爬取数据了

三、获取房源数据

有了url，现在我们来解析一下页面数据,我们要拿下名称、地址、价格、类型等等数据

右键检查元素我们知道，房源数据都在ul标签的li标签里，每一个li对应一个房源

这样我们就可以利用Xpath定位获取数据，代码没有加try 应该加上try 因为有的数据在列表中是不存在的

li_list=html.xpath('//ul[@class="resblock-list-wrapper"]/li')
for li in li_list:dic={}dic["title"]=li.xpath('./div//div[@class="resblock-name"]/a/text()')[0]dic["type"]=li.xpath('./div//span[@class="resblock-type"]/text()')[0]dic["status"]=li.xpath('./div//span[@class="sale-status"]/text()')[0]dic["location"]=''.join([x for x in li.xpath('./div//div[@class="resblock-location"]//text()') if '\n' not in  x])dic["room"]=''.join([x for x in li.xpath('./div//a[@class="resblock-room"]//text()') if '\n' not in  x])dic["area"]=li.xpath('.//div[@class="resblock-area"]/span/text()')[0]dic["tag"]=''.join([x for x in li.xpath('./div//div[@class="resblock-tag"]//text()') if '\n' not in  x])dic["main_price"]=li.xpath('./div//div[@class="main-price"]/span[@class="number"]/text()')[0]dic["price"]=li.xpath('./div//div[@class="resblock-price"]/div[@class="second"]/text')[0]dic["img_link"]=li.xpath('./a//img[@class="lj-lazy"]/@src')[0]

location、area、tag三个数据获取到的都是列表所以在获取时剔除掉无用字符，并且连接为字符串

''.join([x for x in li.xpath('./div//div[@class="resblock-tag"]//text()') if '\n' not in  x])list=[]
for x in li.xpath('./div//div[@class="resblock-tag"]//text()'):if "\n" not in x:list.append(x)
''.join(list)

读取url 因为我们上步已经保存了url为txt 现在我们读入url

urls = open("urls.txt", "r", encoding="utf-8").read()
urls=urls.split("\n")[:-1] #根据换行符分割 转换为列表

保存数据，这个应放在循环li_list中，边爬边存

with open(".//fangyuan.csv", "a", encoding="utf-8") as f:writer = csv.DictWriter(f, dic.keys())writer.writerow(dic)

最后可以开启线程池快速的爬取数据

 pool = Pool(4)pool.map(parser_data,urls)

采集完成之后看一下效果，有5MB多

去重之后数据有将近3W条，10个字段

到这里这篇文章就结束了，如果你对爬虫有兴趣，可以关注我的主页近期再写一些爬虫项目，主页中已经更新了多个爬虫。

本文章用到的代码在公众号“阿虚学Python”中回复“链家”获取，还有更多爬虫项目等你来看哦

谢谢大家的观看，如果觉得这篇文章不错就点个赞支持一下吧

爬虫项目六：用Python爬下链家新房所有城市近三万条数据相关推荐

python爬取链家新房_Python爬虫项目--爬取链家热门城市新房
本次实战是利用爬虫爬取链家的新房(声明: 内容仅用于学习交流, 请勿用作商业用途) 环境 win8, python 3.7, pycharm 正文 1. 目标网站分析通过分析, 找出相关url, 确 ...
python爬取链家新房_Python爬虫实战：爬取链家网二手房数据
前言本文的文字及图片来源于网络,仅供学习.交流使用,不具有任何商业用途,如有问题请及时联系我们以作处理. 买房装修,是每个人都要经历的重要事情之一.相对于新房交易市场来说,如今的二手房交易市场一点也 ...
python爬取链家新房数据_Python爬虫实战：爬取链家网二手房数据
前言本文的文字及图片来源于网络,仅供学习.交流使用,不具有任何商业用途,如有问题请及时联系我们以作处理. 买房装修,是每个人都要经历的重要事情之一.相对于新房交易市场来说,如今的二手房交易市场一点也 ...
python爬取链家新房数据
没有搜索到关于python爬虫,所以自己写一个 from bs4 import BeautifulSoup import requests import time import pandas as p ...
Python 爬取链家新房并生成物业类型对比图
1,爬取网站新房内容 from bs4 import BeautifulSoup import requests import time import pandas as pdheaders = {' ...
python爬房源信息_用python爬取链家网的二手房信息
题外话:这几天用python做题,算是有头有尾地完成了.这两天会抽空把我的思路和方法,还有代码贴出来,供python的初学者参考.我python的实战经历不多,所以代码也是简单易懂的那种.当然过程中还 ...
python 爬取链家数据_用python爬取链家网的二手房信息
题外话:这几天用python做题,算是有头有尾地完成了.这两天会抽空把我的思路和方法,还有代码贴出来,供python的初学者参考.我python的实战经历不多,所以代码也是简单易懂的那种.当然过程中还 ...
python爬取链家网的房屋数据
python爬取链家网的房屋数据爬取内容爬取源网站爬取内容爬取思路爬取的数据代码获取房屋url 获取房屋具体信息爬取内容爬取源网站北京二手房 https://bj.lianjia. ...
python关于二手房的课程论文_基于python爬取链家二手房信息代码示例
基本环境配置 python 3.6 pycharm requests parsel time 相关模块pip安装即可确定目标网页数据哦豁,这个价格..................看到都觉得脑阔 ...

爬虫项目六：用Python爬下链家新房所有城市近三万条数据