python实现携程网站爬取

一爬虫及相关知识介绍

1 网络爬虫（Crawler）又称网络蜘蛛，或者网络机器人（Robots）。它是一种按照一定的规则，自动地抓取万维网信息的程序或者脚本。它可以根据网页的链接地址自动获取网页内容。如果把互联网比做一个大蜘蛛网，它里面有许许多多的网页，网络蜘蛛可以获取所有网页的内容。网络爬虫的优势在于不需要用户轮班值守，能自动化运行，自动判断网页数据的状态，及时更新。

2 网络爬虫应用主要有三个步骤：

爬取：通过网页链接获取网页内容：urllib、requests等库；

解析：对获得的内容进行处理：正则表达式或者BeautifulSoup、lxml等库；

存储：将数据信息保存本地以便后续处理，txt、 JSON文件，MySQL数据库。

3 Requests库是一个简洁且简单的处理HTTP请求的第三方库。编写过程接近正常URL访问过程。建立在urllib3库的基础上。urllib库功能和request很像，request库更加简洁易用。

4 用requests库获取HTML页面并转换成字符串后，需要进一步解析HTML页面格式，从网页中提取有用的信息。正则表达式：构造复杂易出错。BeautifulSoup库：简单。安装pip install beautifulsoup4。BeautifulSoup库的名字取自刘易斯 ·卡罗尔在《爱丽丝梦游仙境》里的同名诗歌。

二任务描述

1旅游网站爬虫分析；

2要求：抓取携程上的旅游景点信息并分析；

3功能：获取10个国家的景点信息，根据受欢迎程度做出每个国家的景点排名。设计GUI界面可以展示每个景点的代表照片、特色的文字介绍和根据用户评价生成的worldcloud。

三实验环境

1实验软件 pycharm

2实验工具包 wordcloud pyside2 imageio jieba requests bs4

3实验编译器 python3.8

四系统性能评估

1 本系统可以查询10个国家受欢迎程度最高的前15个景点的数据,显示每个景点的4张图片，以及景点介绍，评论，词云图，景点周围的购物场所。

2 景点转换时延时稍长；按钮仅仅是装饰作用，点击不会有响应。

3 本代码换电脑运行时需要注意，在图片爬取函数中需要修改图片保存路径，代码中为C:\\Users\\Admin\\Desktop\\chpicture。图片保存的文件夹自己新建即可。

4读者需要自己找一张图片作为词云图背景图（在当前目录），以及中文停词用表（在当前目录），和新建一个comment.txt保存评论（在当前目录）。

下图为系统初始界面

下面是爬取的景点示例

1德国的勃兰登堡门

2英国的大英博物馆

3内蒙古的呼伦贝尔大草原

4美国的好莱坞环球影城

五代码实现

# -*- codeing = utf-8 -*-
# @Time : 2021/4/4 23:36
# @Author : 陈浩
# @File : chenhao2018212503.py
# @software: PyCharmfrom bs4 import BeautifulSoup
import requests
from PySide2.QtWidgets import *
from PySide2 import QtGui,QtCore
import jieba
import imageio
import wordcloud
import os
lsls1=[]       #景点网址
lsls = []      #景点名称
def getbrief(url):try:kv={"User-Agent": "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/90.0.4430.93 Safari/537.36 Edg/90.0.818.56"}r=requests.get(url,headers=kv,timeout=30)r.raise_for_status()r=r.textsoup=BeautifulSoup(r,'html.parser')s = soup.find_all('div', attrs={'class': 'rdetailbox'})lsls.clear()lsls1.clear()for i in s:ss = i.find_all('a')lsls.append(ss[0].string)b = soup.find_all('div', attrs={'class': 'list_mod2'})for i in b:bb=i.find('div', attrs={'class': 'leftimg'})bbb=bb.find('a', attrs={'target': '_blank'})p='https://you.ctrip.com'+bbb['href']lsls1.append(p)except:return 'get error'def getcomment(url):kv = {"User-Agent": "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/90.0.4430.93 Safari/537.36 Edg/90.0.818.56"}r = requests.get(url, headers=kv, timeout=30)r.raise_for_status()r = r.textsoup = BeautifulSoup(r, 'html.parser')s = soup.find('div', attrs={'class': 'moduleWrap'})ss = s.find('div', attrs={'class': 'LimitHeightText'})sss = ss.find('p', attrs={'class': 'inset-p'})text1.clear()text2.clear()try:# text1.insertPlainText(sss.string)text1.insertPlainText(str(sss.contents).replace('<br/>,', '').strip('[\'\']'))except AttributeError:sss = ss.find('div')text1.insertPlainText(sss.string)# except TypeError:#     text1.insertPlainText(str(sss.contents).replace('<br/>,', ''))# if sss=='':#     sss=ss.find('div')#     text1.insertPlainText(sss.string)# elif sss!='':#     text1.insertPlainText(sss.string)a = soup.find_all('div', attrs={'class': 'commentDetail'})ls = []for i in a:ls.append(i.string)text2.insertPlainText(i.string+'\n'+'\n')def getpicture(url):kv = {"User-Agent": "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/90.0.4430.93 Safari/537.36 Edg/90.0.818.56"}r = requests.get(url, headers=kv, timeout=30)r.raise_for_status()r = r.textsoup = BeautifulSoup(r, 'html.parser')list1 = []list2 = []global flagflag=0b = soup.find('div', attrs={'class': 'swiper-wrapper'})for i in b:list1.append(i.attrs['style'])for j in list1:ss = j.replace('background-image:url(', '')ss = ss.replace(')', '')list2.append(ss)for k in list2:filename = str(flag)+'.jpg'response = requests.get(k, headers=kv)with open('C:\\Users\\Admin\\Desktop\\chpicture\\' + filename, 'wb') as f:f.write(response.content)flag=flag+1f=[]g=os.walk('C:\\Users\\Admin\\Desktop\\chpicture')for p,d,filelist in g:for filename in filelist:if '.jpg' in filename:f.append(filename)label2.setPixmap(QtGui.QPixmap('C:\\Users\\Admin\\Desktop\\chpicture\\'+f[0]))label2.setFrameStyle(QFrame.Panel | QFrame.Sunken)label2.setScaledContents(True)label3.setPixmap(QtGui.QPixmap('C:\\Users\\Admin\\Desktop\\chpicture\\' + f[1]))label3.setFrameStyle(QFrame.Panel | QFrame.Sunken)label3.setScaledContents(True)label4.setPixmap(QtGui.QPixmap('C:\\Users\\Admin\\Desktop\\chpicture\\' + f[2]))label4.setFrameStyle(QFrame.Panel | QFrame.Sunken)label4.setScaledContents(True)label5.setPixmap(QtGui.QPixmap('C:\\Users\\Admin\\Desktop\\chpicture\\' + f[3]))label5.setFrameStyle(QFrame.Panel | QFrame.Sunken)label5.setScaledContents(True)def getciyuntu(url):kv = {"User-Agent": "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/90.0.4430.93 Safari/537.36 Edg/90.0.818.56"}r = requests.get(url, headers=kv, timeout=30)r.raise_for_status()r = r.textsoup = BeautifulSoup(r, 'html.parser')a = soup.find_all('div', attrs={'class': 'commentDetail'})list1 = []for i in a:list1.append(i.string)with open('comment.txt', mode='w', encoding='utf-8') as f:for c in list1:f.write(c.replace('\n', ''))f.write('\n')img = imageio.imread('1.jpg')f = open('comment.txt', mode='r', encoding='utf-8')txt = f.read()txtlist = jieba.lcut(txt)string1 = ' '.join(txtlist)wc = wordcloud.WordCloud(width=1000,height=1000,background_color='black',font_path='msyh.ttc',mask=img,scale=15,stopwords=set([line.strip() for line in open('中文停用词表.txt', mode='r', encoding='utf-8').readlines()]))wc.generate(string1)wc.to_file('1out.png')label1.setPixmap(QtGui.QPixmap('1out.png'))label1.setFrameStyle(QFrame.Panel | QFrame.Sunken)label1.setScaledContents(True)def getshop(url):try:kv={"User-Agent": "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/90.0.4430.93 Safari/537.36 Edg/90.0.818.56"}r=requests.get(url,headers=kv,timeout=30)r.raise_for_status()r=r.textsoup=BeautifulSoup(r,'html.parser')text3.clear()s = soup.find_all('div', attrs={'class': 'nearbyList'})ls=[]for i in s:ss= i.find_all('div', attrs={'class': 'contentBox'})for j in ss:sss=j.find_all('div', attrs={'class': 'contentTitle'})ls.append(sss[0].string)for k in ls:text3.insertPlainText(k+'\n')except:return 'get error'def myactivated1(s):combo2.clear()if s=='英国':getbrief('https://you.ctrip.com/sightlist/UnitedKingdom20354.html?ordertype=11')for j in lsls:combo2.addItem(j)elif s=='葡萄牙':getbrief("https://you.ctrip.com/sightlist/Portugal100106.html?ordertype=11")for j in lsls:combo2.addItem(j)elif s=='内蒙古':getbrief('https://you.ctrip.com/sightlist/InnerMongolia100062.html?ordertype=11')for j in lsls:combo2.addItem(j)elif s == '德国':getbrief('https://you.ctrip.com/sightlist/Germany100025.html?ordertype=11')for j in lsls:combo2.addItem(j)elif s == '荷兰':getbrief('https://you.ctrip.com/sightlist/Netherlands100028.html?ordertype=11')for j in lsls:combo2.addItem(j)elif s == '美国':getbrief('https://you.ctrip.com/sightlist/UnitedStates100047.html?ordertype=11')for j in lsls:combo2.addItem(j)elif s == '俄罗斯':getbrief('https://you.ctrip.com/sightlist/Russia100083.html?ordertype=11')for j in lsls:combo2.addItem(j)elif s == '意大利':getbrief('https://you.ctrip.com/sightlist/Italy100026.html?ordertype=11')for j in lsls:combo2.addItem(j)elif s == '澳大利亚':getbrief('https://you.ctrip.com/sightlist/Australia100048.html?ordertype=11')for j in lsls:combo2.addItem(j)elif s == '瑞士':getbrief('https://you.ctrip.com/sightlist/Switzerland100050.html?ordertype=11')for j in lsls:combo2.addItem(j)def myactivated2(s):if s==0:getcomment(lsls1[0])getpicture(lsls1[0])getciyuntu(lsls1[0])getshop(lsls1[0])elif s==1:getcomment(lsls1[1])getpicture(lsls1[1])getciyuntu(lsls1[1])getshop(lsls1[1])elif s==2:getcomment(lsls1[2])getpicture(lsls1[2])getciyuntu(lsls1[2])getshop(lsls1[2])elif s==3:getcomment(lsls1[3])getpicture(lsls1[3])getciyuntu(lsls1[3])getshop(lsls1[3])elif s==4:getcomment(lsls1[4])getpicture(lsls1[4])getciyuntu(lsls1[4])getshop(lsls1[4])elif s==5:getcomment(lsls1[5])getpicture(lsls1[5])getciyuntu(lsls1[5])getshop(lsls1[5])elif s==6:getcomment(lsls1[6])getpicture(lsls1[6])getciyuntu(lsls1[6])getshop(lsls1[6])elif s==7:getcomment(lsls1[7])getpicture(lsls1[7])getciyuntu(lsls1[7])getshop(lsls1[7])elif s==8:getcomment(lsls1[8])getpicture(lsls1[8])getciyuntu(lsls1[8])getshop(lsls1[8])elif s==9:getcomment(lsls1[9])getpicture(lsls1[9])getciyuntu(lsls1[9])getshop(lsls1[9])elif s==10:getcomment(lsls1[10])getpicture(lsls1[10])getciyuntu(lsls1[10])getshop(lsls1[10])elif s==11:getcomment(lsls1[11])getpicture(lsls1[11])getciyuntu(lsls1[11])getshop(lsls1[11])elif s==12:getcomment(lsls1[12])getpicture(lsls1[12])getciyuntu(lsls1[12])getshop(lsls1[12])elif s==13:getcomment(lsls1[13])getpicture(lsls1[13])getciyuntu(lsls1[13])getshop(lsls1[13])elif s==14:getcomment(lsls1[14])getpicture(lsls1[14])getciyuntu(lsls1[14])getshop(lsls1[14])app = QApplication([])window = QMainWindow()
window.resize(1400, 850)
window.move(30, 30)
window.setWindowTitle('携程')button1 = QPushButton('景点介绍', window)
button1.move(20,120)
# button1.clicked.connect(clickclick1)button2 = QPushButton('词云图', window)
button2.move(20,260)
# button2.clicked.connect(clickclick4)button3 = QPushButton('景点图片', window)
button3.move(460,260)
# button3.clicked.connect(clickclick4)button4 = QPushButton('景点评论', window)
button4.move(830,20)
# button4.clicked.connect(clickclick4)button5 = QPushButton('附近购物场所', window)
button5.move(20,710)text1 = QPlainTextEdit(window)
text1.move(200,120)
text1.resize(700,100)text2 = QPlainTextEdit(window)
text2.move(950,20)
text2.resize(440,800)text3 = QPlainTextEdit(window)
text3.move(20,750)
text3.resize(400,70)label1 = QLabel(window)
label1.move(20,300)
label1.resize(400,400)
# label1.setPixmap(QtGui.QPixmap('1out.png'))
# label1.setFrameStyle(QFrame.Panel | QFrame.Sunken)
# label1.setScaledContents(True)label2 = QLabel(window)
label2.move(460,300)
label2.resize(200,200)
# label2.setPixmap(QtGui.QPixmap('111.jpg'))
# label2.setFrameStyle(QFrame.Panel | QFrame.Sunken)
# label2.setScaledContents(True)label3 = QLabel(window)
label3.move(700,300)
label3.resize(200,200)label4 = QLabel(window)
label4.move(460,540)
label4.resize(200,200)label5 = QLabel(window)
label5.move(700,540)
label5.resize(200,200)clab1=QLabel('国家',window)
clab1.move(20,20)
combo1=QComboBox(window)
combo1.move(60,20)
combo1.activated[str].connect(myactivated1)
combo1.addItem('选择国家')
combo1.addItem('英国')
combo1.addItem('葡萄牙')
combo1.addItem('内蒙古')
combo1.addItem('德国')
combo1.addItem('荷兰')
combo1.addItem('美国')
combo1.addItem('俄罗斯')
combo1.addItem('意大利')
combo1.addItem('澳大利亚')
combo1.addItem('瑞士')clab2=QLabel('景点',window)
clab2.move(180,20)
combo2=QComboBox(window)
combo2.move(220,20)
combo2.activated[int].connect(myactivated2)
combo2.addItem('选择景点')window.show()
app.exec_()

python实现携程网站爬取相关推荐

python爬虫：Selenium爬取B站视频标题、播放量、发布时间
上次尝试了利用Ajax机制爬取B站视频播放量等数据(链接在下方),但是发现响应的JSON数据中没有发布时间的数据,这次决定用Selenium试一下. python爬虫:Ajax爬取B站视频标题.播放量 ...
Python爬虫---影评的爬取
Python爬虫-影评的爬取介绍:爬虫练手,使用Requests库进行豆瓣影评的爬虫,做成词云图,写文章主要做一些问题解决的记录. 运行环境:python 3.8, Pycharm 关于在豆瓣爬取影 ...
【Python爬虫】从零开始爬取Sci-Hub上的论文(串行爬取)
[Python爬虫]从零开始爬取Sci-Hub上的论文(串行爬取) 维护日志项目简介步骤与实践 STEP1 获取目标内容的列表 STEP2 利用开发者工具进行网页调研 2.1 提取文章链接和分页链 ...
Python的Requests来爬取今日头条的图片和文章
Python的Requests来爬取今日头条的图片和文章并且存入mongo config.py MONGO_HOST = 'localhost' MONGO_PORT = 27017 MONGO_DB ...
python 爬虫实例电影-Python爬虫教程-17-ajax爬取实例（豆瓣电影）
Python爬虫教程-17-ajax爬取实例(豆瓣电影) ajax: 简单的说,就是一段js代码,通过这段代码,可以让页面发送异步的请求,或者向服务器发送一个东西,即和服务器进行交互对于ajax: ...
每日一练：Python国内疫情数据爬取与地图绘制
Python 国内疫情数据爬取与地图绘制效果图累计确诊疫情地图绘制 ① 时时数据抓取 ② 获取省份疫情数据 ③ 视觉配置项分段颜色数据设置 ④ 累计确诊疫情地图绘制现存确诊疫情地图绘制 ① 获取 ...
广州python平均薪资_爬取广州的python和Java薪资，为什么Python 高于Java（有代码）...
标签: 小编不知道这个是不是属于那个啥,要看的赶紧看,说不定小编的号要被封了. 用python爬取了某某某直聘网站,直聘Python和Java的招聘信息,比较了两个方向的发展前(钱)途 ,主要是本科生 ...
Python学习笔记：爬取网页图片
Python学习笔记:爬取网页图片上次我们利用requests与BeautifulSoup爬取了豆瓣<下町火箭>短评,这次我们来学习爬取网页图片. 比如想爬取下面这张网页的所有图片.网址 ...
python实战1.0——爬取知乎某问题下的回复
python实战1.0--爬取知乎某问题下的回复确定问题爬取进行简单筛选保存数据 # 获取问题下的回复总数 def get_number():url = 'https://www.zhihu. ...
Python+scrapy+mysql实现爬取磁力链接
Python+scrapy+mysql实现爬取磁力链接作为老司机中的一员,所以试试爬取磁力链接,看看效果咋样. 直接上代码: class torrentSpider(scrapy.Spider):n ...

python实现携程网站爬取

python实现携程网站爬取相关推荐

最新文章

热门文章