【爬虫实战】 58同城爬虫-商铺信息采集

58同城商铺信息采集器-含商家电话-含ui

结果展示

程序展示

程序技术点

requests -用于向页面发送请求
queue - 队列用来存放爬虫任务、爬虫结果
threading - 线程库用来实现多线程采集
bs4 - 解析得到的html页面数据
openpyxl - 将数据写入excel 相对于xlwt 可写入行数更多 xlwt只能写入6w+行数据不能满足需求
mysql -数据库
pyqt5 解决ui需求

主要代码

程序UI

class FormLayout(QtWidgets.QWidget):def __init__(self):QtWidgets.QWidget.__init__(self)self.province_list = {'全国': ['全国        '],'安徽': ['合肥-hf', '芜湖-wuhu', '蚌埠-bengbu', '阜阳-fy', '淮南-hn', '安庆-anqing', '宿州-suzhou', '六安-la', '淮北-huaibei', '滁州-chuzhou', '马鞍山-mas', '铜陵-tongling', '宣城-xuancheng', '亳州-bozhou', '黄山-huangshan', '池州-chizhou', '巢湖-ch', '和县-hexian', '霍邱-hq', '桐城-tongcheng', '宁国-ningguo', '天长-tianchang', '东至-dongzhi', '无为-wuweixian'],'福建': ['福州-fz', '厦门-xm', '泉州-qz', '莆田-pt', '漳州-zhangzhou', '宁德-nd', '三明-sm', '南平-np', '龙岩-ly', '武夷山-wuyishan', '石狮-shishi', '晋江-jinjiangshi', '南安-nananshi', '龙海-longhai', '上杭-shanghangxian', '福安-fuanshi', '福鼎-fudingshi', '安溪-anxixian', '永春-yongchunxian', '永安-yongan', '漳浦-zhangpu'],'广东': ['深圳-sz', '广州-gz', '东莞-dg', '佛山-fs', '中山-zs', '珠海-zh', '惠州-huizhou', '江门-jm', '汕头-st', '湛江-zhanjiang', '肇庆-zq', '茂名-mm', '揭阳-jy', '梅州-mz', '清远-qingyuan', '阳江-yj', '韶关-sg', '河源-heyuan', '云浮-yf','汕尾-sw', '潮州-chaozhou', '台山-taishan', '阳春-yangchun', '顺德-sd', '惠东-huidong', '博罗-boluo', '海丰-haifengxian', '开平-kaipingshi', '陆丰-lufengshi'],'广西': ['南宁-nn', '柳州-liuzhou', '桂林-gl', '玉林-yulin', '梧州-wuzhou', '北海-bh', '贵港-gg', '钦州-qinzhou', '百色-baise', '河池-hc', '来宾-lb', '贺州-hezhou', '防城港-fcg', '崇左-chongzuo', '桂平-guipingqu', '北流-beiliushi', '博白-bobaixian', '岑溪-cenxi'],'贵州': ['贵阳-gy', '遵义-zunyi', '黔东南-qdn', '黔南-qn', '六盘水-lps', '毕节-bijie', '铜仁-tr', '安顺-anshun', '黔西南-qxn', '仁怀-renhuaishi', '清镇-qingzhen'],'甘肃': ['兰州-lz', '天水-tianshui', '白银-by', '庆阳-qingyang', '平凉-pl', '酒泉-jq', '张掖-zhangye', '武威-wuwei', '定西-dx', '金昌-jinchang', '陇南-ln', '临夏-linxia', '嘉峪关-jyg', '甘南-gn', '敦煌-dunhuang'],'海南': ['海口-haikou', '三亚-sanya', '五指山-wzs', '三沙-sansha', '琼海-qh', '文昌-wenchang', '万宁-wanning', '屯昌-tunchang', '琼中-qiongzhong', '陵水-lingshui', '东方-df', '定安-da', '澄迈-cm', '保亭-baoting', '白沙-baish', '儋州-danzhou'],'河南': ['郑州-zz', '洛阳-luoyang', '新乡-xx', '南阳-ny', '许昌-xc', '平顶山-pds', '安阳-ay', '焦作-jiaozuo', '商丘-sq', '开封-kaifeng', '濮阳-puyang', '周口-zk', '信阳-xy', '驻马店-zmd', '漯河-luohe', '三门峡-smx', '鹤壁-hb', '济源-jiyuan', '明港-mg','鄢陵-yanling', '禹州-yuzhou', '长葛-changge', '灵宝-lingbaoshi','杞县-qixianqu', '汝州-ruzhou', '项城-xiangchengshi','偃师-yanshiqu', '长垣-changyuan', '滑县-huaxian', '林州-linzhou', '沁阳-qinyang', '孟州-mengzhou', '温县-wenxian', '尉氏-weishixian', '兰考-lankaoxian', '通许-tongxuxian', '新安-lyxinan', '伊川-yichuan', '孟津-mengjinqu', '宜阳-lyyiyang','舞钢-wugang', '永城-yongcheng', '睢县-suixian', '鹿邑-luyi', '渑池-yingchixian', '沈丘-shenqiu', '太康-taikang', '商水-shangshui', '淇县-qixianq', '浚县-junxian', '范县-fanxian', '固始-gushixian', '淮滨-huaibinxian', '邓州-dengzhou', '新野-xinye'],'黑龙江': ['哈尔滨-hrb', '大庆-dq', '齐齐哈尔-qqhr', '牡丹江-mdj', '绥化-suihua', '佳木斯-jms', '鸡西-jixi', '双鸭山-sys', '鹤岗-hegang', '黑河-heihe', '伊春-yich', '七台河-qth', '大兴安岭-dxal', '安达-shanda', '肇东-shzhaodong', '肇州-zhaozhou'],'湖北': ['武汉-wh', '宜昌-yc', '襄阳-xf', '荆州-jingzhou', '十堰-shiyan', '黄石-hshi', '孝感-xiaogan', '黄冈-hg', '恩施-es', '荆门-jingmen', '咸宁-xianning', '鄂州-ez', '随州-suizhou', '潜江-qianjiang', '天门-tm', '仙桃-xiantao', '神农架-snj', '宜都-yidou', '汉川-hanchuan', '枣阳-zaoyang','武穴-wuxueshi', '钟祥-zhongxiangshi', '京山-jingshanxian', '沙洋-shayangxian', '松滋-songzi', '广水-guangshuishi', '赤壁-chibishi', '老河口-laohekou', '谷城-gucheng', '宜城-yichengshi', '南漳-nanzhang', '云梦-yunmeng', '安陆-anlu', '大悟-dawu', '孝昌-xiaochang', '当阳-dangyang', '枝江-zhijiang', '嘉鱼-jiayuxian', '随县-suixia'],'湖南': ['长沙-cs', '株洲-zhuzhou', '益阳-yiyang', '常德-changde', '衡阳-hy', '湘潭-xiangtan', '岳阳-yy', '郴州-chenzhou', '邵阳-shaoyang', '怀化-hh', '永州-yongzhou', '娄底-ld', '湘西-xiangxi', '张家界-zjj', '醴陵-liling', '澧县-lixian', '桂阳-czguiyang', '资兴-zixing', '永兴-yongxing', '常宁-changningshi','祁东-qidongxian', '衡东-hengdong', '冷水江-lengshuijiangshi', '涟源-lianyuanshi', '双峰-shuangfengxian', '邵阳县-shaoyangxian', '邵东-shaodongxian', '沅江-yuanjiangs', '南县-nanxian', '祁阳-qiyang', '湘阴-xiangyin', '华容-huarong', '慈利-cilixian', '攸县-zzyouxian'],'河北': ['石家庄-sjz', '保定-bd', '唐山-ts', '廊坊-lf', '邯郸-hd', '秦皇岛-qhd', '沧州-cangzhou', '邢台-xt', '衡水-hs', '张家口-zjk', '承德-chengde', '定州-dingzhou', '馆陶-gt', '张北-zhangbei', '赵县-zx', '正定-zd', '迁安市-qianan', '任丘-renqiu', '三河-sanhe', '武安-wuan','雄安新区-xionganxinqu', '燕郊-lfyanjiao', '涿州-zhuozhou', '河间-hejian', '黄骅-huanghua', '沧县-cangxian', '磁县-cixian', '涉县-shexian', '霸州-bazhou', '香河-xianghe', '固安-lfguan', '遵化市-zunhua', '迁西-qianxixian', '玉田-yutianxian', '滦南-luannanxian', '沙河-shaheshi'],'江苏': ['苏州-su', '南京-nj', '无锡-wx', '常州-cz', '徐州-xz', '南通-nt', '扬州-yz', '盐城-yancheng', '淮安-ha', '连云港-lyg', '泰州-taizhou', '宿迁-suqian', '镇江-zj', '沭阳-shuyang', '大丰-dafeng', '如皋-rugao', '启东-qidong', '溧阳-liyang', '海门-haimen', '东海-donghai', '扬中-yangzhong', '兴化-xinghuashi', '新沂-xinyishi','泰兴-taixing', '如东-rudong', '邳州-pizhou', '沛县-xzpeixian', '靖江-jingjiang', '建湖-jianhu', '海安-haian', '东台-dongtai', '丹阳-danyang', '宝应县-baoyingx', '灌南-guannan', '灌云-guanyun', '姜堰-jiangyan', '金坛-jintan','昆山-szkunshan', '泗洪-sihong', '泗阳-siyang', '句容-jurong', '射阳-sheyang', '阜宁-funingxian', '响水-xiangshui', '盱眙-xuyi', '金湖-jinhu', '江阴-jiangyins'],'江西': ['南昌-nc', '赣州-ganzhou', '九江-jj', '宜春-yichun', '吉安-ja', '上饶-sr', '萍乡-px', '抚州-fuzhou', '景德镇-jdz', '新余-xinyu', '鹰潭-yingtan', '永新-yxx', '乐平-lepingshi', '进贤-jinxian', '分宜-fenyi', '丰城-fengchengshi', '樟树-zhangshu', '高安-gaoan', '余江-yujiang', '南城-nanchengx', '浮梁-fuliangxian'],'吉林': ['长春-cc', '吉林-jl', '四平-sp', '延边-yanbian', '松原-songyuan', '白城-bc', '通化-th', '白山-baishan', '辽源-liaoyuan', '公主岭-gongzhuling', '梅河口-meihekou', '扶余-fuyuxian', '长岭-changlingxian', '桦甸-huadian', '磐石-panshi', '梨树县-lishu'],'辽宁': ['沈阳-sy', '大连-dl', '鞍山-as', '锦州-jinzhou', '抚顺-fushun', '营口-yk', '盘锦-pj', '朝阳-cy', '丹东-dandong', '辽阳-liaoyang', '本溪-benxi', '葫芦岛-hld', '铁岭-tl', '阜新-fx', '庄河-pld', '瓦房店-wfd', '灯塔-dengta', '凤城-fengcheng', '北票-beipiao', '开原-kaiyuan'],'宁夏': ['银川-yinchuan', '吴忠-wuzhong', '石嘴山-szs', '中卫-zw', '固原-guyuan'],'内蒙古': ['呼和浩特-hu', '包头-bt', '赤峰-chifeng', '鄂尔多斯-erds', '通辽-tongliao', '呼伦贝尔-hlbe', '巴彦淖尔市-bycem', '乌兰察布-wlcb', '锡林郭勒-xl', '兴安盟-xam', '乌海-wuhai', '阿拉善盟-alsm', '海拉尔-hlr'],'青海': ['西宁-xn', '海西-hx', '海北-haibei', '果洛-guoluo', '海东-haidong', '黄南-huangnan', '玉树-ys', '海南-hainan', '格尔木-geermushi'],'山东': ['青岛-qd', '济南-jn', '烟台-yt', '潍坊-wf', '临沂-linyi', '淄博-zb', '济宁-jining', '泰安-ta', '聊城-lc', '威海-weihai', '枣庄-zaozhuang','德州-dz', '日照-rizhao', '东营-dy', '菏泽-heze', '滨州-bz', '莱芜-lw', '章丘-zhangqiu', '垦利-kl', '诸城-zc', '寿光-shouguang', '龙口-longkou', '曹县-caoxian', '单县-shanxian', '肥城-feicheng', '高密-gaomi', '广饶-guangrao', '桓台-huantaixian', '莒县-juxian','莱州-laizhou', '蓬莱-penglai', '青州-qingzhou', '荣成-rongcheng', '乳山-rushan', '滕州-tengzhou', '新泰-xintai', '招远-zhaoyuan', '邹城-zoucheng', '邹平-zouping', '临清-linqing', '茌平-chiping', '郓城-hzyc', '博兴-boxing', '东明-dongming', '巨野-juye', '无棣-wudi', '齐河-qihe', '微山-weishan','禹城-yuchengshi', '临邑-linyixianq', '乐陵-leling', '莱阳-laiyang', '宁津-ningjin', '高唐-gaotang', '莘县-shenxian', '阳谷-yanggu', '冠县-guanxian', '平邑-pingyi', '郯城-tancheng', '沂源-yiyuanxian', '汶上-wenshang', '梁山-liangshanx', '利津-lijin', '沂南-yinanxian', '栖霞-qixia', '宁阳-ningyang', '东平-dongping','昌邑-changyishi', '安丘-anqiu', '昌乐-changle', '临朐-linqu', '鄄城-juancheng'],'山西': ['太原-ty', '临汾-linfen', '大同-dt', '运城-yuncheng', '晋中-jz', '长治-changzhi', '晋城-jincheng', '阳泉-yq', '吕梁-lvliang', '忻州-xinzhou', '朔州-shuozhou', '临猗-linyixian', '清徐-qingxu', '柳林-liulin', '高平-gaoping', '泽州-zezhou', '襄垣-xiangyuanxian', '孝义-xiaoyi'],'陕西': ['西安-xa', '咸阳-xianyang', '宝鸡-baoji', '渭南-wn', '汉中-hanzhong', '榆林-yl', '延安-yanan', '安康-ankang', '商洛-sl', '铜川-tc', '神木-shenmu', '韩城-hancheng', '府谷-fugu', '靖边-jingbian', '定边-dingbian'],'四川': ['成都-cd', '绵阳-mianyang', '德阳-deyang', '南充-nanchong', '宜宾-yb', '自贡-zg', '乐山-ls', '泸州-luzhou', '达州-dazhou', '内江-scnj', '遂宁-suining', '攀枝花-panzhihua', '眉山-ms', '广安-ga', '资阳-zy', '凉山-liangshan', '广元-guangyuan', '雅安-ya', '巴中-bazhong', '阿坝-ab', '甘孜-ganzi','安岳-anyuexian', '广汉-guanghanshi', '简阳-jianyangshi', '仁寿-renshouxian', '射洪-shehongxian', '大竹-dazu', '宣汉-xuanhan', '渠县-qux', '长宁-changningx'],'新疆': ['乌鲁木齐-xj', '昌吉-changji', '巴音郭楞-bygl', '伊犁-yili', '阿克苏-aks', '喀什-ks', '哈密-hami', '克拉玛依-klmy', '博尔塔拉-betl', '吐鲁番-tlf', '和田-ht', '石河子-shz', '克孜勒苏-kzls', '阿拉尔-ale', '五家渠-wjq', '图木舒克-tmsk', '库尔勒-kel', '阿勒泰-alt', '塔城-tac'],'西藏': ['拉萨-lasa', '日喀则-rkz', '山南-sn', '林芝-linzhi', '昌都-changdu', '那曲-nq', '阿里-al', '日土-rituxian', '改则-gaizexian'],'云南': ['昆明-km', '曲靖-qj', '大理-dali', '红河-honghe', '玉溪-yx', '丽江-lj', '文山-ws', '楚雄-cx', '西双版纳-bn', '昭通-zt', '德宏-dh', '普洱-pe', '保山-bs', '临沧-lincang', '迪庆-diqing', '怒江-nujiang', '弥勒-milexian', '安宁-anningshi', '宣威-xuanwushi'],'浙江': ['杭州-hz', '宁波-nb', '温州-wz', '金华-jh', '嘉兴-jx', '台州-tz', '绍兴-sx', '湖州-huzhou', '丽水-lishui', '衢州-quzhou', '舟山-zhoushan', '乐清-yueqingcity', '瑞安-ruiancity', '义乌-yiwu', '余姚-yuyao', '诸暨-zhuji', '象山-xiangshanxian', '温岭-wenling', '桐乡-tongxiang', '慈溪-cixi','长兴-changxing', '嘉善-jiashanx', '海宁-haining', '德清-deqing', '东阳-dongyang', '安吉-anji', '苍南-cangnanxian', '临海-linhai', '永康-yongkang', '玉环-yuhuan', '平湖-pinghushi', '海盐-haiyan', '武义县-wuyix', '嵊州-shengzhou', '新昌-xinchang', '江山-jiangshanshi', '平阳-pingyangxian'],'其他': ['中国香港-hk', '中国澳门-am', '中国台湾-tw'],'海外': ['洛杉矶-gllosangeles', '旧金山-glsanfrancisco', '纽约-glnewyork', '多伦多-gltoronto', '温哥华-glvancouver', '伦敦-glgreaterlondon', '莫斯科-glmoscow', '首尔-glseoul', '东京-gltokyo', '新加坡-glsingapore', '曼谷-glbangkok', '清迈-glchiangmai', '迪拜-gldubai', '奥克兰-glauckland', '悉尼-glsydney', '墨尔本-glmelbourne', '其他海外城市-city']}# self.mysql_obj = connect_mysql()# 设置标题self.setWindowTitle('58商铺采集')self.all_info = []self.resize(600, 600)# 添加行编辑器# 选择框self.province = QComboBox()  # 省份self.district = QComboBox()  # 地区self.district.resize(20, 30)# self.district.setMinimumWidth(20)# self.district.setMaximumHeight(20)self.page_num = QLineEdit()  # 页码self.shengyu_Q = QLineEdit()  # 页码self.chushi_Q = QLineEdit()  # 页码# self.page_num.resize(20, 10)self.quchong = QCheckBox("去重采集")  # 是否去重self.loginBtn = QPushButton("开始采集")# 展示框self.save_Btn = QPushButton("存入excel")self.start_Btn = QPushButton("写入Mysql")self.model = QStandardItemModel(6, 6)# 设置水平方向四个头标签文本内容self.model.setHorizontalHeaderLabels(['标题', '姓名', '手机号', '平方', '价格', "基本信息"])self.tableView = QTableView()self.tableView.setModel(self.model)self.tableView.horizontalHeader().setStretchLastSection(True)# #水平方向，表格大小拓展到适当的尺寸self.tableView.horizontalHeader().setSectionResizeMode(QHeaderView.Stretch)# 添加表单布局self.gridlayout1 = QFormLayout()# gridlayout2 = QFormLayout()self.gridlayout3 = QFormLayout()self.gridlayout4 = QFormLayout()self.gridlayout5 = QFormLayout()self.gridlayout6 = QFormLayout()self.gridlayout7 = QFormLayout()self.gridlayout8 = QFormLayout()# 展示框self.gridlayout10 = QFormLayout()self.gridlayout11 = QFormLayout()# 添加水平布局self.hbox = QHBoxLayout()# 添加垂直布局self.vbox = QHBoxLayout()# 添加布局self.vlayout = QVBoxLayout()  # 整个程序的灵魂,将QVBoxLayout改成QHBoxLayout可以改变hbox和vbox的布局从垂直布局到水平布局# self.loginBtn.clicked.connect(lambda :self.Spider())  # 提交按钮self.loginBtn.clicked.connect(self.start_login)  # 提交按钮self.save_Btn.clicked.connect(lambda :self.processing_data())  # 保存按钮self.start_Btn.clicked.connect(lambda :self.save_to_mysql())  # 保存按钮self.initui()self.thread = None  # 初始化线程def initui(self):# 设置参数self.province.addItems(['全国', '安徽', '福建', '广东', '广西', '贵州', '甘肃', '海南', '河南', '黑龙江', '湖北', '湖南', '河北', '江苏', '江西', '吉林', '辽宁', '宁夏', '内蒙古', '青海', '山东', '山西', '陕西', '四川', '新疆', '西藏', '云南', '浙江', '其他', '海外'])self.district.addItems(self.province_list['全国'])self.gridlayout1.addRow("省份", self.province)self.gridlayout3.addRow("地区", self.district)self.gridlayout4.addRow("页数", self.page_num)self.gridlayout5.addRow("剩余页数", self.shengyu_Q)self.gridlayout6.addRow("初始页数", self.chushi_Q)self.gridlayout7.addRow(self.quchong)  # 去重self.gridlayout8.addRow(self.loginBtn)  ## self.gridlayout9.addRow(self.textbox)self.gridlayout10.addRow(self.save_Btn)self.gridlayout11.addRow(self.start_Btn)# 横向布局添加self.hbox.addLayout(self.gridlayout1)self.hbox.addLayout(self.gridlayout3)self.hbox.addLayout(self.gridlayout4)self.hbox.addLayout(self.gridlayout5)self.hbox.addLayout(self.gridlayout6)self.hbox.addLayout(self.gridlayout7)self.hbox.addLayout(self.gridlayout8)# 垂直布局self.vbox.addLayout(self.gridlayout11)self.vbox.addLayout(self.gridlayout10)# 总布局# self.vlayout.addStretch(1)self.vlayout.addLayout(self.hbox)self.vlayout.addWidget(self.tableView)self.vlayout.addLayout(self.vbox)# self.vlayout.addStretch(1)self.setLayout(self.vlayout)# 区域站点绑定self.province.currentIndexChanged[int].connect(self.province_changed)def start_login(self):# 创建线程province = self.province.currentText()district = self.district.currentText()page_num = self.page_num.text()quchong = self.quchong.isChecked()self.thread = Runthread(province, district, page_num, quchong)# 连接信号self.thread._signal.connect(self.call_backlog)  # 进程连接回传到GUI的事件# 开始线程self.thread.start()def call_backlog(self, row, column, item, hhh_flag, shengyu, chushi):if hhh_flag == 5:print("哇哈哈哈，", hhh_flag)self.tanchuang()return 0item = QStandardItem(item)self.model.setItem(int(row), int(column), item)self.shengyu_Q.setText(str(shengyu))self.chushi_Q.setText(str(chushi))def province_changed(self, province_idx):# 省下拉框改变，先清空市下拉框，然后添加市数据self.district.clear()self.district.addItems(self.province_list[self.province.currentText()])def processing_data(self):# 创建一个workbook 设置编码wb = Workbook()  # 创建文件对象ws = wb.active  # 获取第一个sheetfor hh in all_info:ws.append(hh)  # 写入多个单元格if not os.path.isdir("58商铺数据"):os.mkdir("58商铺数据")  # 定位time_now = time.time()timeArray = time.localtime(time_now)time_now = time.strftime("%Y--%m--%d-%H-%M-%S", timeArray)wb.save('58商铺数据/商铺信息%s.xlsx' % time_now)self.excel_tip()def save_to_mysql(self):for x in all_info:t = tuple(x)mysql_obj.insert_mysql(t)self.mysql_tip()def tanchuang(self):QMessageBox.information(self,"消息框标题","采集完成！！！",QMessageBox.Yes)def mysql_tip(self):QMessageBox.information(self,"消息框标题","mysql写入完成",QMessageBox.Yes)def excel_tip(self):QMessageBox.information(self,"消息框标题","excel写入完成",QMessageBox.Yes)

爬虫代码片段

class Thread_crawl(threading.Thread):def __init__(self,page_queue, data_queue):#初始化threading.Thread.__init__(self)#拿到任务的队列self.page_queue = page_queueself.data_queue = data_queue# self.name = nameself.headers = {"user-agent": "user-agent: Mozilla/5.0 (Windows NT 10.0; WOW64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/74.0.3729.169 Safari/537.36","referer": "https://www.58.com/"}def run(self):global zantingbiaojiwhile True:if self.page_queue.empty():zantingbiaoji += 1breakelse:# print('将要从队列中取任务')url = self.page_queue.get()self.spider_info(url=url)# print('完成任务的页码是：', url)def send_url(self, url):while True:tunnel = "快代理路径"# 用户名密码方式username = "快代理账号"password = "快代理密码"proxies = {# "http": "http://%(user)s:%(pwd)s@%(proxy)s/" % {"user": username, "pwd": password, "proxy": tunnel},"https": "http://%(user)s:%(pwd)s@%(proxy)s/" % {"user": username, "pwd": password, "proxy": tunnel}}# try:try:content = requests.get(url=url, headers=self.headers, proxies=proxies, verify=False)print(content.status_code)except:time.sleep(1)print("出错了")continueif content.status_code != 429:return contentelse:print("我暂停了一秒")time.sleep(1)# time.sleep(0.5)def spider_info(self,url):# global proxy_listurl = url[:url.find("&leg")]biaoji_2 = 0while True:print("----->>>>>>", url)content = self.send_url(url)soup = BeautifulSoup(content.text, "lxml")try:uls = soup.find("div", class_="content-side-left").find("ul", id="house-list-wrap")# print("我错了 在请求一次")lis = uls.find_all("li")for li in lis:title = li.find("h2").find('a').text.replace("\n", "")href_info = li.find("h2").find('a')["href"]baseinfo = li.find("p", class_='baseinfo').text.strip().replace("\n", "")area = li.find("div", class_="area").find("p", class_="num").text.replace("\n", "")money = li.find("div", class_="price").find("p", class_="up").text.replace("\n", "")# if title not in self.title_list:href_info = href_info[:href_info.find("&lego")]name, phone_num = self.get_detail(href_info)if phone_num == "1" or name == "2":continuein_list = [title, name, phone_num, area, money, baseinfo, href_info]self.data_queue.put(in_list)print(in_list)breakexcept:biaoji_2 += 1if biaoji_2 == 5:breakprint("没找到li")continue

程序主要难点

pyqt5打包时很多朋友会遇到运行失败的问题很大原因是因为你的pyqt5版本有点过高或者过低

只需将pyqt5改成5.9.x版本就好解决方法我也是百度出来的原链接已经找不到了抱歉
58同城会有ip检测所以需要用到代理ip 这里我选择了快代理（快代理打钱！！！）推荐使用隧道代理每次请求更换新的ip 一开始使用的是私密代理发现成本有点大（一天2000个ip 血亏）所以改为隧道代理
pyqt5阻塞问题只需要加一个信号槽就好

class Runthread(QtCore.QThread):#  通过类成员对象定义信号对象_signal = pyqtSignal(str, str, str, int, int, int)def __init__(self,province, district, page_num, quchong):super(Runthread, self).__init__()def __del__(self):self.wait()def run():pass# 传递值self._signal.emit("1", "1", "1", 5, 0, 0)
# 主线程中的函数 用来接收联系线程传递的值
def call_backlog(self, row, column, item, hhh_flag, shengyu, chushi):if hhh_flag == 5:print("哇哈哈哈，", hhh_flag)self.tanchuang()return 0item = QStandardItem(item)self.model.setItem(int(row), int(column), item)self.shengyu_Q.setText(str(shengyu))self.chushi_Q.setText(str(chushi))

至此文章结束
欢迎来技术交流
源码共526行没有全部贴出来
技术交流QQ
欢迎技术交流有爬虫需求也可联系

【爬虫实战】 58同城爬虫-商铺信息采集相关推荐

Python3 爬虫实战 — 58同城武汉出租房【加密字体对抗】
爬取时间:2019-10-21 爬取难度:★★★☆☆☆ 请求链接:https://wh.58.com/chuzu/ 爬取目标:58同城武汉出租房的所有信息涉及知识:网站加密字体的攻克.请求库 req ...
Python爬虫实战--58同城二手商品爬虫
呜呜~~本来说今天就把代码上传上来,可惜了,还是有点差错,今天估计赶不上啦!明天加油吧! 今天我们一起来好好分析一下,看看我们该如何去爬去58二手商品. 这里我们分成四步来完成本次任务~ 目标站点分析 ...
python爬取58同城房子发布的手机号码_爬虫(成都58同城所有房价,Python实现)
Python实现爬虫(爬取58同城所有房价) 程序发布日期2018-9-25(如果以后不能使用了,就需要更改解析方式.) github博客传送门 csdn博客传送门爬虫文件(Python实现)-爬取 ...
爬虫(成都58同城所有房价,Python实现)
Python实现爬虫(爬取58同城所有房价) 程序发布日期2018-9-25(如果以后不能使用了,就需要更改解析方式.) github博客传送门博客园传送门爬虫文件(Python实现)-爬取的数据 ...
爬虫实战-直聘-爬虫岗位分析
爬虫岗位数据分析一数据集 # boss直聘网址:https://www.zhipin.com/web/geek/job?query=%E7%88%AC%E8%99%AB&city=1000 ...
【Web_接口爬虫_Python3_58同城_requestosetreeproxies】58同城，商铺出租，爬取标题、内容、链接地址，保存文本_20200401
目录 [爬虫和数据挖掘] [绕过代理] [创建文件夹] [request请求] [response解析] [记录文件] [运行脚本] #!/usr/bin/env/python3 # -*- codi ...
Python爬虫(一)——58同城租房信息
代码: 1 # coding=utf-82 import sys3 import csv4 import requests5 from bs4 import BeautifulSoup6 7 relo ...
利用python爬取58同城简历数据_python爬虫程序 58同城二手交易信息爬取
本脚本分为5部分: spider_main 主程序 url_manager url管理器 html_downloader 网页下载器 html_parser 网页解析器 htm ...
python爬虫实战测评_Python 爬虫实战入门（上）
之前为了工作需要,写了不少爬虫小程序,和同事们聊得过程中发现,很多人对爬虫还是停留在听说过的阶段,基于这个原因,特意写点适合小白看的,希望大家能对爬虫有个小认知. 工欲善其事必先利其器,既然要写爬虫, ...
python爬虫实战案例-Python爬虫实战案例：手机APP抓包爬虫
1. items.pyclass DouyuspiderItem(scrapy.Item): name = scrapy.Field()# 存储照⽚的名字 imagesUrls = scrapy.Fi ...

【爬虫实战】 58同城爬虫-商铺信息采集

【爬虫实战】 58同城爬虫-商铺信息采集相关推荐

最新文章

热门文章