Python爬虫踩坑记录 _pickle.PicklingError: Can‘t pickle ＜class＞

做大作业老师要求帮他们组运行一个爬虫程序，下载源码后在Anaconda里运行，发现了奇怪的报错。

Traceback (most recent call last):File "ccf_crawler.py", line 118, in <module>save_dblp_papers()File "ccf_crawler.py", line 102, in save_dblp_paperswatcher_process.start()File "E:\jupter\lib\multiprocessing\process.py", line 121, in startself._popen = self._Popen(self)File "E:\jupter\lib\multiprocessing\context.py", line 224, in _Popenreturn _default_context.get_context().Process._Popen(process_obj)File "E:\jupter\lib\multiprocessing\context.py", line 326, in _Popenreturn Popen(process_obj)File "E:\jupter\lib\multiprocessing\popen_spawn_win32.py", line 93, in __init__reduction.dump(process_obj, to_child)File "E:\jupter\lib\multiprocessing\reduction.py", line 60, in dumpForkingPickler(file, protocol).dump(obj)
_pickle.PicklingError: Can't pickle <class 'paper_crawler.crawler_manager.CrawlerManager'>: it's not the same object as paper_crawler.crawler_manager.CrawlerManager
Traceback (most recent call last):File "<string>", line 1, in <module>File "E:\jupter\lib\multiprocessing\spawn.py", line 116, in spawn_mainexitcode = _main(fd, parent_sentinel)File "E:\jupter\lib\multiprocessing\spawn.py", line 126, in _mainself = reduction.pickle.load(from_parent)
EOFError: Ran out of input

是python pickle报错，pickle是一个简单的持久化功能，将对象序列化后以文件的形式保存。为什么在windows里运行不了呢？查阅资料，发现关于pickle和进程

原来这个爬虫是用进程编写的，而在windows系统中，进程使用socket对象，而socket对象是不可序列化的；在linux系统中，进程使用的是fork对象，因此可以被序列化。
所以解决方案有二：

重构代码，用线程而不是进程编写，对性能不会造成大的影响，这样就可以在windows上运行
使用linux系统运行即可

因此，打开了ubuntu虚拟机，配置环境，安装mongoDB，运行代码，成功！

没内存啦，唉

插播：Ubuntu 14.04系统安装MongoDB的报错
安装网络教程安装MongoDB，后出现报错

 mongod --dbpath data/db
Mon Dec  7 19:03:14.143 [initandlisten] MongoDB starting : pid=129519 port=27017 dbpath=data/db 64-bit host=ubuntu
Mon Dec  7 19:03:14.143 [initandlisten] db version v2.4.9
Mon Dec  7 19:03:14.143 [initandlisten] git version: nogitversion
Mon Dec  7 19:03:14.143 [initandlisten] build info: Linux orlo 3.2.0-58-generic #88-Ubuntu SMP Tue Dec 3 17:37:58 UTC 2013 x86_64 BOOST_LIB_VERSION=1_54
Mon Dec  7 19:03:14.143 [initandlisten] allocator: tcmalloc
Mon Dec  7 19:03:14.143 [initandlisten] options: { dbpath: "data/db" }
Mon Dec  7 19:03:14.149 [initandlisten] journal dir=data/db/journal
Mon Dec  7 19:03:14.149 [initandlisten] recover : no journal files present, no recovery needed
Mon Dec  7 19:03:14.149 [initandlisten]
Mon Dec  7 19:03:14.149 [initandlisten] ERROR: Insufficient free space for journal files
Mon Dec  7 19:03:14.149 [initandlisten] Please make at least 3379MB available in data/db/journal or use --smallfiles
Mon Dec  7 19:03:14.149 [initandlisten]
Mon Dec  7 19:03:14.150 [initandlisten] exception in initAndListen: 15926 Insufficient free space for journals, terminating
Mon Dec  7 19:03:14.150 dbexit:
Mon Dec  7 19:03:14.150 [initandlisten] shutdown: going to close listening sockets...
Mon Dec  7 19:03:14.150 [initandlisten] shutdown: going to flush diaglog...
Mon Dec  7 19:03:14.150 [initandlisten] shutdown: going to close sockets...
Mon Dec  7 19:03:14.150 [initandlisten] shutdown: waiting for fs preallocator...
Mon Dec  7 19:03:14.150 [initandlisten] shutdown: lock for final commit...
Mon Dec  7 19:03:14.150 [initandlisten] shutdown: final commit...
Mon Dec  7 19:03:14.150 [initandlisten] shutdown: closing all files...
Mon Dec  7 19:03:14.150 [initandlisten] closeAllFiles() finished
Mon Dec  7 19:03:14.150 [initandlisten] journalCleanup...
Mon Dec  7 19:03:14.150 [initandlisten] removeJournalFiles
Mon Dec  7 19:03:14.150 [initandlisten] shutdown: removing fs lock...
Mon Dec  7 19:03:14.150 dbexit: really exiting now

是因为内存不够了啊，orz，没办法，虚拟机太小了，只能使用替代命令–smallfiles运行了
注意空格之类的格式（这几天空格踩雷好多，疲惫）（要注意细节啊），
运行命令

 mongod --dbpath data/db --smallfiles

成功！

Mon Dec  7 19:03:59.144 [initandlisten] MongoDB starting : pid=129886 port=27017 dbpath=data/db 64-bit host=ubuntu
Mon Dec  7 19:03:59.145 [initandlisten] db version v2.4.9
Mon Dec  7 19:03:59.145 [initandlisten] git version: nogitversion
Mon Dec  7 19:03:59.145 [initandlisten] build info: Linux orlo 3.2.0-58-generic #88-Ubuntu SMP Tue Dec 3 17:37:58 UTC 2013 x86_64 BOOST_LIB_VERSION=1_54
Mon Dec  7 19:03:59.145 [initandlisten] allocator: tcmalloc
Mon Dec  7 19:03:59.145 [initandlisten] options: { dbpath: "data/db", smallfiles: true }
Mon Dec  7 19:03:59.148 [initandlisten] journal dir=data/db/journal
Mon Dec  7 19:03:59.148 [initandlisten] recover : no journal files present, no recovery needed
Mon Dec  7 19:03:59.215 [FileAllocator] allocating new datafile data/db/local.ns, filling with zeroes...
Mon Dec  7 19:03:59.215 [FileAllocator] creating directory data/db/_tmp
Mon Dec  7 19:03:59.217 [FileAllocator] done allocating datafile data/db/local.ns, size: 16MB,  took 0 secs
Mon Dec  7 19:03:59.218 [FileAllocator] allocating new datafile data/db/local.0, filling with zeroes...
Mon Dec  7 19:03:59.219 [FileAllocator] done allocating datafile data/db/local.0, size: 16MB,  took 0 secs
Mon Dec  7 19:03:59.223 [initandlisten] waiting for connections on port 27017
Mon Dec  7 19:03:59.225 [websvr] admin web console waiting for connections on port 28017Mon Dec  7 19:07:59.266 [PeriodicTask::Runner] task: DBConnectionPool-cleaner took: 8ms
Mon Dec  7 19:48:17.641 [PeriodicTask::Runner] task: WriteBackManager::cleaner took: 5ms

Python爬虫踩坑记录 _pickle.PicklingError: Can‘t pickle ＜class＞相关推荐

如何运行开源游戏？八分音符酱python版踩坑记录
如何运行开源游戏?八分音符酱python版踩坑记录如何从零到一搭建一个python游戏?这篇博客将以八分音符酱介绍详细过程. 下载源代码源代码搭建python环境参考此网站此游戏运行基于pyt ...
python kazoo 踩坑记录
问题场景:python 启动rpc服务挂载到zookeeper上供javai调用问题现象:zookeeper的节点总是掉,但是服务没有挂,监听进行重连也会再次挂掉问题原因:zookeeper服务端 ...
Python打包工具Pyintealler打包py文件为windows exe文件过程及踩坑记录+实战例子
Python打包工具Pyintealler打包py文件为windows exe文件过程及踩坑记录+实战例子目录 Python打包工具Pyintealler打包py文件为windows exe文件过程 ...
使用Java读取 “Python写入redis” 的数据踩坑记录
https://my.oschina.net/u/2338224/blog/3061507 使用Java读取 "Python写入redis" 的数据踩坑记录 https://seg ...
VS Code Python 代码智能提示（自动补全）编译环境设置踩坑记录
VS Code Python 编译环境设置踩坑记录本菜鸡非常喜欢VS code的简洁风格,而且跟原来再用的VS2017相比简直不要太快,前段时间电脑重装之后又要在windows系统上重新搭建整套环境 ...
TVM: Deep Learning模型的优化编译器(强烈推荐, 附踩坑记录)
本文作者是阿莱克西斯,原载于知乎,雷锋网(公众号:雷锋网)获得授权转载. (前排提醒,本文的人文内容部分稍稍带有艺术加工,请保持一定的幽默感进行阅读) 关注我最近想法的同学应该知道我最近都在把玩 TV ...
【踩坑记录】Tensorflow在Windows下使用
[踩坑记录]Tensorflow在Windows下使用 TensorFlow 是一个端到端开源机器学习平台安装 pip3 install tensorflow 使用时报错如下 2021-04-21 ...
Detectron2 win10踩坑记录
Detectron2 win10踩坑记录安装CUDA clone到本地修改cocoeval.cpp第483行(不同版本行数不一样) 创建虚拟环境安装pycocotools 安装VS2019或者生 ...
日常踩坑记录-汇总版
开发踩坑记录,不定时更新心得 RTFM 严谨的去思考问题,处理问题严格要求自己的代码编写习惯与风格注意单词拼写 20200207 mybatis plus 自带insert插入异常 sql i ...

Python爬虫踩坑记录 _pickle.PicklingError: Can‘t pickle ＜class＞

Python爬虫踩坑记录 _pickle.PicklingError: Can‘t pickle ＜class＞相关推荐

最新文章

热门文章