Python爬虫实战示例-51job和豆瓣电影

2018年7月16日笔记

1.conda常用命令

1.1 列出当前环境的所有库

命令：conda list
在cmd中运行命令如下图所示：

图片.png-36.6kB

1.2 管理环境

创建环境

命令:conda create -n {} python={}第一对大括号替换为环境的命名，第二对大括号替换为python的版本号
例如:conda create -n python27 python=2.7 这个命令就是创建一个python版本为2.7的环境，并命名为python27

列出所有环境

命令:conda info -e

进入环境

activate {}，大括号替换为虚拟环境名

环境添加库

conda install {}，大括号替换为要安装库的库名

环境删除库

conda remove {}，大括号替换为要安装库的库名

删除环境

conda remove -n {} -all，大括号替换为要删除库的库名

2. 爬虫示例

爬取豆瓣钱排名前250条信息,即下图这个网页的信息。

图片.png-340.8kB

下面的sql语句用来创建数据库的表

drop database if exists douban;
create database douban;
use douban;
DROP TABLE IF EXISTS `top250`;
CREATE TABLE `top250` (`director` varchar(100) DEFAULT NULL,`role` varchar(100) DEFAULT NULL,`year` varchar(100) DEFAULT NULL,`area` varchar(20) DEFAULT NULL,`genre` varchar(100) DEFAULT NULL,`title` varchar(255)  DEFAULT NULL
) ENGINE=InnoDB DEFAULT CHARSET=utf8;

把豆瓣排名前250的电影信息导入mysql数据库中
下面一段代码能够成功运行的前提有两个：
1.安装库requests：pip install requests
安装库pymysql:pip install pymysql
2.修改下面代码中进入mysql数据库的用户名和密码，即修改下面这一句：
conn = pymysql.connect(host='localhost', user='root', passwd='...your password', db='douban',charset="utf8")

import requests
from bs4 import BeautifulSoup as bs
import pymysqlif __name__ == "__main__":movieInfos = []  # 用于保存所有的电影信息baseUrl = 'https://movie.douban.com/top250?start={}&filter='for startIndex in range(0, 226, 25):url = baseUrl.format(startIndex)# 爬取网页r = requests.get(url)# 获取html内容htmlContent = r.text# 用BeautifulSoup加载html文本内容进行处理soup = bs(htmlContent, "lxml")# 获取到页面中索引的class名为info的标签(应该有25个)movieList = soup.find_all("div", attrs={"class": "info"})# 遍历25条电影信息for movieItem in movieList:movieInfo = {}  # 创建空字典,保存电影信息# 获取到名为class名为hd的div标签内容hd_div = movieItem.find("div", attrs={"class": "hd"})# 通过bd_div获取到里面第一个span标签内容hd_infos = hd_div.find("span").get_text().strip().split("\n")# < span  class ="title" > 天堂电影院 < / span >movieInfo['title'] = hd_infos[0]# 获取到class名为bd的div标签内容bd_div = movieItem.find("div", attrs={"class": "bd"})# print(bd_div)# 通过bd_div获取到里面第一个p标签内容infos = bd_div.find("p").get_text().strip().split("\n")# print(infos)   #包含了两行电影信息的列表# 获取导演和主演infos_1 = infos[0].split("\xa0\xa0\xa0")if len(infos_1) == 2:# 获取导演,只获取排在第一位的导演名字director = infos_1[0][4:].rstrip("...").split("/")[0]movieInfo['director'] = director# 获取主演role = infos_1[1][4:].rstrip("...").rstrip("/").split("/")[0]movieInfo['role'] = roleelse:movieInfo['director'] = NonemovieInfo['role'] = None# 获取上映的时间/地区/电影类型infos_2 = infos[1].lstrip().split("\xa0/\xa0")# 获取上映时间year = infos_2[0]movieInfo['year'] = year# 获取电影地区area = infos_2[1]movieInfo['area'] = area# 获取类型genre = infos_2[2]movieInfo['genre'] = genreprint(movieInfo)movieInfos.append(movieInfo)conn = pymysql.connect(host='localhost', user='root', passwd='...your password', db='douban',charset="utf8")# 获取游标对象cursor = conn.cursor()# 查看结果print('添加了{}条数据'.format(cursor.rowcount))for movietiem in movieInfos:director = movietiem['director']role = movietiem['role']year = movietiem['year']area = movietiem['area']genre = movietiem['genre']title = movietiem['title']sql = 'INSERT INTO top250 values("%s","%s","%s","%s","%s","%s")' % (director, role, year, area, genre, title)# 执行sqlcursor.execute(sql)# 提交conn.commit()print('添加了{}条数据'.format(cursor.rowcount))

插入数据库成功截图如下：

图片.png-67.2kB

2018年7月17日笔记

3.HTTP理解

3.1 HTTP请求格式

当浏览器向Web服务器发出请求时，它向服务器传递了一个数据块，也就是请求信息，HTTP请求信息由3部分组成：
1.请求方法URL协议/版本；2.请求头;3.请求体内容

图片.png-149.8kB

3.2 HTTP请求方式

常见的http请求方式有get和post
Get是比较简单的http请求，直接会将发送给web服务器的数据放在请求地址的后面，即在请求地址后使用?key1=value1&ke2=value2形式传递数据，只适合数据量少，且没有安全性的请求
Post是需要发送给web服务器的数据经过编码放到请求体中，可以传递大量数据，并且有一定安全性，常用于表单提交

4.爬取51job网站信息

爬取51job网站信息并将数据持久化为excel文件

import requests
from bs4 import BeautifulSoup as bs
import re
from urllib import parse
import pandas as pddef cssFind(soup,cssSelector,nth=1):if len(soup.select(cssSelector)) >= nth:return soup.select(cssSelector)[nth-1].textelse:return 0def getSoup(url):response = requests.get(url)response.encoding = 'gbk'soup = bs(response.text,'lxml')return soupdef getMaxPageNumber(url):soup = getSoup(url)maxPageNumberBefore = cssFind(soup,"span.td")pattern = "共(\d*)页"maxPageNumber = re.findall(pattern,maxPageNumberBefore)[0]return int(maxPageNumber)def getJobList(url):soup = getSoup(url)webpage_job_list = soup.select("div.dw_table div.el")[1:]job_list = []for item in webpage_job_list:job = {}job['职位名'] = cssFind(item,"a").strip()job['公司名'] = cssFind(item,"span.t2")job['工作地点'] = cssFind(item,"span.t3")job['薪资'] = cssFind(item,"span.t4")job['发布时间'] = cssFind(item,"span.t5")job_list.append(job)return job_listdef getUrl(job,page):url_before = "https://search.51job.com/list/020000,000000,0000,00,9,99,{},2," \"{}.html?lang=c&stype=1&postchannel=0000&workyear=99&cotype=99&" \"degreefrom=99&jobterm=99&companysize=99&lonlat=0%2C0&radius=-1&" \"ord_field=0&confirmdate=9&dibiaoid=0&specialarea=00"url = url_before.format(parse.quote(job),page)return urlif __name__ == "__main__":job = "人工智能"firstPage_url = getUrl(job,1)maxPageNumber = getMaxPageNumber(firstPage_url)job_list = []for i in range(1,maxPageNumber+1):print("共有%d页，正在获取第%d页" %(maxPageNumber,i))url = getUrl(job,i)job_list.extend(getJobList(url))df = pd.DataFrame(job_list,columns=job_list[0].keys())excel_name = "51job_{}.xlsx".format(job)df.to_excel(excel_name)print("finished!")

5.爬取豆瓣排名前250电影信息

下面一段代码只需要修改连接mysql数据库的密码就可以运行。
sql语句写在代码中，所以代码比较长。

# coding=utf-8
from bs4 import BeautifulSoup as bs
import requests
import re
import pymysqldef cssFind(movie,cssSelector,nth=1):if len(movie.select(cssSelector)) >= nth:return movie.select(cssSelector)[nth-1].text.strip()else:return ''def reFind(pattern,sourceStr,nth=1):if len(re.findall(pattern,sourceStr)) >= nth:return re.findall(pattern,sourceStr)[nth-1]else:return ''def getConn(database ="pydb"):args = dict(host = 'localhost',user = 'root',passwd = '... your password',charset = 'utf8',db = database)return pymysql.connect(**args)if __name__ == "__main__":#连接数据库conn = getConn("douban")cursor = conn.cursor()#解析网页并将每条电影信息插入mysql数据库url_before = "https://movie.douban.com/top250?start={}"flag = Truefor i in range(0,250,25):url = url_before.format(i)response = requests.get(url)response.encoding = 'utf-8'soup = bs(response.text, 'lxml')movie_list = soup.select("ol.grid_view li")for movie in movie_list:item = {}item['title_zh'] = cssFind(movie, "span.title")  #提取标题item['title2'] = cssFind(movie, "span.title", 2).lstrip('/').strip() #提取item['title_other'] = cssFind(movie, "span.other").lstrip('/').strip()details = cssFind(movie, "div.bd p")pattern_director = "导演: (.*)主"item['director'] = reFind(pattern_director, details).strip('/...').strip()if item['director'] == "":item['director'] = reFind("导演: (.*)", details).strip('/...').strip()pattern_actor = "主演: (.*)"item['actor'] = reFind(pattern_actor, details).strip('/...').strip()detail2 = details.split('\n')[1]item['year'] = detail2.split('/')[0].strip()item['country'] = detail2.split('/')[1].strip()item['genre'] = detail2.split('/')[2].strip()item['rating_grade'] = cssFind(movie, "span.rating_num")item['rating_number'] = cssFind(movie, "div.star span", 4).rstrip("人评价")item['summary'] = cssFind(movie, "span.inq")if flag:drop_sql = "drop table if exists movie"cursor.execute(drop_sql)conn.commit()table_movie = ','.join(['`%s` varchar(200)'%key for key in item.keys()])create_sql = "create table movie(`id` int primary key auto_increment,%s)" %table_moviecursor.execute(create_sql)conn.commit()flag = Falsetable_field = ','.join(['`%s`'%key for key in item.keys()])table_row = ','.join(['"%s"'%value for value in item.values()])insert_sql = "insert into movie(%s) values(%s)"%(table_field, table_row)print(insert_sql)cursor.execute(insert_sql)conn.commit()conn.close()