关于2020年中国统计年鉴（教育部分）的数据采集

一、准备阶段

1. 网站结构观察

目标网站： http://www.stats.gov.cn/tjsj/ndsj/2020/indexch.htm

观察结果：具体数据以图片的形式展现,在html中留有一个网址，需要一次请求获取图片

*2.搭建Scrapy环境

scrapy startproject statistical_yearbook
scrapy genspider main http://www.stats.gov.cn/tjsj/ndsj/2020/indexch.htm

*3. 代码逻辑

1.利用Scrapy请求网站获得采集目标的图片网址以及图片标题，保存在StatisticalYearbookItem中
2.在pipelines.py中建立MyImagePipeline类继承于ImagesPipeline父类，配合采集到的网址和标题下载图片并命名，保存在images文件夹中

二、代码实现

main.py

import scrapy
from ..items import StatisticalYearbookItemclass MainSpider(scrapy.Spider):name = 'main'# allowed_domains = ['baidu']start_urls = ['http://www.stats.gov.cn/tjsj/ndsj/2020/indexch.htm']def parse(self, response):url = response.xpath('/html/frameset/frameset/frame[1]/@src').extract()[0]yield scrapy.Request(response.urljoin(url),callback= self.parse_one)def parse_one(self,response):#教育部分sel = response.xpath('//*[@id="foldinglist"][21]')item = StatisticalYearbookItem()item['name'] = []item['url'] = []for i in list(zip(sel.css('li a::text').extract()[1:-1],sel.css('li a::attr(href)').extract()[1:-1])):item['name'].append(i[0])item['url'].append(response.urljoin(i[1]))yield item

items.py

import scrapyclass StatisticalYearbookItem(scrapy.Item):# define the fields for your item here like:# name = scrapy.Field()name = scrapy.Field()url = scrapy.Field()

pipelines.py

from itemadapter import ItemAdapter
from scrapy.pipelines.images import ImagesPipeline
import scrapyclass MyImagePipeline(ImagesPipeline):def get_media_requests(self, item, info):# return [Request(x) for x in item.get(self.images_urls_field, [])]for i in zip(item['url'],item['name']):yield scrapy.Request(url=i[0],meta = {'name':i[1]})def file_path(self, request, response=None,info=None,item = None):img_name = request.meta['name'] + '.jpg'return img_name  # 返回文件名def item_completed(self, results, item, info):return item  # 返回给下一个即将被执行的管道类

结果：

三、对结果的进一步处理

从上诉中，我们采集到的结果是一张张jpg格式的图片，无法作为数据分析的数据源，因此需要对采集结果进行格式转换。

1.观察采集结果

发现：第一，图片不是完整的表格结构，只在标题上有表格结构，这增加了提取数据的难度。第二，图片中的数据以表格的形式排列。

2.代码逻辑

1.利用OpenCv对图片进行解析，为图片画上表格线。

① 对图片进行水平投影，根据得到的投影数据可以为表格画上横线
②找到每张图片的通用特征，利用找图得到只有标题表格竖线没有其余任何文字干扰的那一列像素排列，利用OpenCV画上竖线
③为每张图片画上边缘（因为会用到百度ocr识别表格，边缘没有画线会导致识别不准确）
④将处理好的数据上传至百度ocr服务器，获得识别结果，数据提取完毕。

四、图片转Excel代码实现

为图片画出表格

import cv2
import numpy as np
import osdef match_img(img_big,img_small,value = 0.91):h,w,l = cv2.imread(img_small).shapeim = cv2.imread(img_big)img_gray = cv2.cvtColor(im,cv2.COLOR_BGR2GRAY)template = cv2.imread(img_small,0)res = cv2.matchTemplate(img_gray,template,cv2.TM_CCOEFF_NORMED)loc = np.where(res >= value)top_left = (loc[1][0],loc[0][0])return top_left,(top_left[0] + w, top_left[1] + h)
#21-12 22
address_all = "./statistical_yearbook/images"for iii in os.listdir(address_all):address = address_all +'/' + iiiprint(address)address3 = './statistical_yearbook/muban3.png'address1 = './statistical_yearbook/muban1.png'address2 = './statistical_yearbook/muban2.png'imgg = cv2.imread(address)h1, w1,z1 = imgg.shapeimg = cv2.imread(address,0)ret,img1=cv2.threshold(img,100,255,cv2.THRESH_BINARY)h,w=img1.shapea , b = match_img(address,address1)c , d = match_img(address,address3)e , f = match_img(address,address2)x = int((d[0] - c[0])/2) + c[0]y = int((d[1] - c[1])/2) + c[1]x1 = int((b[0] - a[0])/2) + a[0]y1 = int((b[1] - a[1])/2) + a[1]y_s = img1[e[1] ]# y_s = img1[c[1]]s_res = []for i, ii in enumerate(y_s):if ii == 0:s_res.append(i)#划横线（得到值）h,w=img1.shapea=[0 for z in range(0,h)]for i in range(0,h):          #遍历每一行for j in range(0,w):      #遍历每一列if img1[i,j]==0:      #判断该点是否为黑点，0代表黑点a[i]+=1           #该行的计数器加一img1[i,j]=255     #将其改为白点，即等于255a  = [i if i > 3 else 0 for i in a ]b = []for i in range(len(a)):if  a[i] != 0 and a[i - 1] != 0 and a[i + 1] == 0:b.append(i)c = []for i in b:if y <i and i < y1:c.append(i)c = [i + 5 for i in c]'''划横线'''for i in c:cv2.line(imgg,(0,i),(w,i),(0,0,0),1)'''划竖线'''# for i in aaa:#     cv2.line(imgg,(i + x,y),(i + x,y1),(0,0,0),1)for i in s_res:cv2.line(imgg, (i, y - 5), (i, y1), (0, 0, 0), 1)'''划四周'''cv2.rectangle(imgg, (0, 0), (w1, h1), (0, 0, 0), 2)address4 = f'./statistical_yearbook/images_excel/{iii}'cv2.imwrite(address4,imgg)print(iii + '保存完毕')print('-' * 20)

结果：

百度OCR部分：

import os
import sys
import requests
import time
import tkinter as tk
from tkinter import filedialog
from aip import AipOcr# 定义常量
APP_ID = 'nide id'
API_KEY = 'nide key'
SECRET_KEY = 'nide secretkey'
# 初始化AipFace对象
client = AipOcr(APP_ID, API_KEY, SECRET_KEY)# 读取图片
def get_file_content(filePath):with open(filePath, 'rb') as fp:return fp.read()#文件下载函数
def file_download(url, file_path):r = requests.get(url)with open(file_path, 'wb') as f:f.write(r.content)if __name__ == "__main__":root = tk.Tk()root.withdraw()data_dir = filedialog.askdirectory(title='请选择图片文件夹') + '/'result_dir = filedialog.askdirectory(title='请选择输出文件夹') + '/'num = 0for name in os.listdir(data_dir):print ('{0} : {1} 正在处理：'.format(num+1, name.split('.')[0]))image = get_file_content(os.path.join(data_dir, name))res = client.tableRecognitionAsync(image)# print ("res:", res)if 'error_code' in res.keys():print ('Error! error_code: ', res['error_code'])sys.exit()req_id = res['result'][0]['request_id']    #获取识别ID号for count in range(1, 20):    #OCR识别也需要一定时间，设定10秒内每隔1秒查询一次res = client.getTableRecognitionResult(req_id)    #通过ID获取表格文件XLS地址print(res['result']['ret_msg'])if res['result']['ret_msg'] == '已完成':break    #云端处理完毕，成功获取表格文件下载地址，跳出循环else:time.sleep(1)url = res['result']['result_data']xls_name = name.split('.')[0] + '.xls'file_download(url, os.path.join(result_dir, xls_name))num += 1print ('{0} : {1} 下载完成。'.format(num, xls_name))time.sleep(1)

结果：

5.结束

这次爬取总共花了两天时间，其中三分之二的时间都在思考为图片画上表格，对OpenCv的感悟深了一步，对数据采集的手段又多了一份理解！