二战企*查*查-企业-数-据爬虫

近期收到上头新的需求，又是企业数据，公司电话地址这些，so我又来了

需求：

根据已知企业名称，爬取相应的企业名称、企业状态、法人代表、注册资本、成立日期、联系方式、地址

知识点涉及：

1、requests + bs4的登录爬虫实战

2、openpyxl 对excel操作练习

看出来没？不登录爬取不到手机号，所以下面的爬虫是在已登录情况下实现(很简单，保存登录后的cookie)

===============================================================================================

单账号插曲：我的号在爬取到2500条数据后被封了QAQ，此处给各位分享一波，经过测试每条爬取间隔设置3,-5秒，100条左右会出现滑动验证码需要手动进入浏览器账号内解决，不然程序就卡了，后我把时间调为3-8秒，稳定运行至2500条数据封账号（非永久封，但具体解封时间没算过）

所以爬企查查建议堆个cookie池，几十个账号轮换可以适当减少延时，不叨叨了，该上原码了

# -*- coding: UTF-8 -*-
import random
import time
import re
import requests
from bs4 import BeautifulSoup
from urllib import parse
# 证书忽略
import urllib3
urllib3.disable_warnings()
from openpyxl import load_workbookCOOKIE = ["",
]USER_AGENT = ['Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/76.0.3809.132 Safari/537.36','',
]class Spider(object):def __init__(self,key):key = parse.quote(key)self.url = "https://www.qcc.com/search?key={}".format(key)self.headers = {'accept': 'text/html,application/xhtml+xml,application/xml;q=0.9,image/webp,image/apng,*/*;q=0.8,application/signed-exchange;v=b3','accept-encoding': 'gzip, deflate, br','accept-language': 'zh-CN,zh;q=0.9,en;q=0.8','cache-control': 'max-age=0','referer':'https://www.qcc.com/search?key={}'.format(key),  # 插入的为url编码的公司名称'sec-fetch-mode': 'navigate','sec-fetch-site': 'none','sec-fetch-user': '?1','upgrade-insecure-requests': '1','cookie': COOKIE[random.randint(0, 1)],  # 数量根据你们存的cookie量定'user-agent': USER_AGENT[random.randint(0, 1)],  # 数量根据你们存的useranget定}passdef run(self):response = requests.get(self.url, headers=self.headers, verify=False)html = response.text# print(html)bs = BeautifulSoup(html, "lxml")tr_list = bs.find_all('tr')if len(tr_list) != 0:data_list = []for tr in tr_list:# print(tr)item = {}p = tr.find_all('p')span = tr.find_all('span', attrs={'class': 'm-l'})# 公司名title = tr.find('a', attrs={'class': 'ma_h1'}).get_text()title = ''.join(title)print(title)# print(type(title))item['name'] = titletry:# 状态state = tr.find('span', attrs={'class': 'nstatus text-success-lt m-l-xs'}).get_text()state = ''.join(state)print(state)item['state'] = stateexcept Exception as e:item['state'] = ''# # 法定代表人try:author = tr.find('a', attrs={'class': 'text-primary'}).get_text()author = ''.join(author)print(author)# print type(author)item['people'] = authorexcept Exception as e:author = str(p[0])aaa = re.findall('法定代表人：(.*?)<span', author.replace(' ', ''), re.DOTALL)author = aaa[0].strip()item['people'] = author# # 注册资本money = tr.find('span', attrs={'class': 'm-l'}).get_text()money = ''.join(money)print(money)# print type(money)item['money'] = money# 成立日期date = span[1].get_text()date = ''.join(date)print(date)# print type(date)item['time'] = date# 邮箱电话email = p[1].get_text()email = ''.join(email)email = re.sub(r'n', '', email).strip().replace(' ', '')num = re.findall('电话：.*', email, re.DOTALL)num = num[0].strip()print(num)# print type(num)item['phone'] = ''.join(num)# 地址site = p[2].get_text()site = re.sub(r'n', '', site).strip()print(site)# print type(site)item['add'] = sitedata_list.append(item)breakreturn data_listif __name__ == '__main__':# 打开文件wb = load_workbook(r'E:\python3\Job_Spider\QCC_TYC_Spider\datas_new.xlsx')  # 此处修改为你们需要爬取的公司excelws = wb.activedatas = ws['F1913':'F{}'.format(ws.max_row)]  # 按行读取excel单列信息aqsj_list = []a = 1913  # 改值控制爬取数据在新excel里的存储位置for cell in datas:for data in cell:print(a)keyword = data.valuespider = Spider(keyword)data_list = spider.run()if data_list:# aqsj_list.append(keyword)# print(data_list)name = data_list[0]['name']state = data_list[0]['state']people = data_list[0]['people']money = data_list[0]['money']timea = data_list[0]['time']phone = data_list[0]['phone']add = data_list[0]['add']wb = load_workbook(r'E:\python3\Job_Spider\QCC_TYC_Spider\cont.xlsx')  # 打开另存的excelws = wb.active# 直接根据位置进行赋值ws['A%s'% a] = namews['B%s'% a] = statews['C%s'% a] = peoplews['D%s'% a] = moneyws['E%s'% a] = timeaws['F%s'% a] = phonews['G%s'% a] = add# 保存文件wb.save(r'E:\python3\Job_Spider\QCC_TYC_Spider\cont.xlsx')# breaka+=1time.sleep(random.randint(3,8))

代码拉下来需要修改的地方有 COOKIE(换成你们自己的cookie) ，USER_AGENT(多加几个总是好的，省的人家以为你一台电脑好几个人用)，还有尾部打开的第一个文件为需要爬取的企业excel(可以自己手动创建一个.xlsx，代码内格式为一行一个公司名)，第二个打开的文件为爬取后的数据存储(需要新建一个.xlsx)，最后a = 1913(意为存储在文件二的A1913)，F1913(读取文件一的F1913格数据)

好了我该下班了~~~