现象

multiprocessing.Pool传递一个普通方法(不在class中定义的)时, 能正常工作.

from multiprocessing import Poolp = Pool(3)
def f(x):return x*xp.map(f, [1,2,3])

但在class中定义的方法使用multiprocessing.Pool会报pickling error错误.

报错代码

# coding: utf8
import multiprocessingclass MyTask(object):def task(self, x):return x*xdef run(self):pool = multiprocessing.Pool(processes=3)a = [1, 2, 3]pool.map(self.task, a)if __name__ == '__main__':t = MyTask()t.run()

会出现如下异常:

cPickle.PicklingError: Can't pickle <type 'instancemethod'>: attribute lookup __builtin__.instancemethod failed

原因:

stackoverflow上的解释:
Pool methods all use a queue.Queue to pass tasks to the worker processes. Everything that goes through the queue.Queue must be pickable. So, multiprocessing can only transfer Python objects to worker processes which can be pickled. Functions are only picklable if they are defined at the top-level of a module, bound methods are not picklable.

pool方法都使用了queue.Queue将task传递给工作进程。multiprocessing必须将数据序列化以在进程间传递。方法只有在模块的顶层时才能被序列化，跟类绑定的方法不能被序列化，就会出现上面的异常。

解决方法:

用线程替换进程
可以使用copy_reg来规避上面的异常.
dill 或pathos.multiprocesssing ：use pathos.multiprocesssing, instead of multiprocessing. pathos.multiprocessing is a fork of multiprocessing that uses dill. dill can serialize almost anything in python, so you are able to send a lot more around in parallel.

正确代码1

 # coding: utf8
from multiprocessing.pool import ThreadPool as Poolclass MyTask(object):def task(self, x):return x*xdef run(self):pool = Pool(3)a = [1, 2, 3]ret = pool.map(self.task, a)print retif __name__ == '__main__':t = MyTask()t.run()

正确代码2:

# coding: utf8
import multiprocessing
import types
import copy_regdef _pickle_method(m):if m.im_self is None:return getattr, (m.im_class, m.im_func.func_name)else:return getattr, (m.im_self, m.im_func.func_name)copy_reg.pickle(types.MethodType, _pickle_method)class MyTask(object):def __init__(self):self.__result = []def task(self, x):return x * xdef result_collector(self, result):self.__result.append(result)def run(self):pool = multiprocessing.Pool(processes=3)a = [1, 2, 3]ret = pool.map(self.task, a)print retif __name__ == '__main__':t = MyTask()t.run()

multiprocessing.Pool报pickling error相关推荐

python class函数报错_Python multiprocess pool模块报错pickling error问题解决方法分析
本文实例讲述了Python multiprocess pool模块报错pickling error问题解决方法.分享给大家供大家参考,具体如下: 问题之前在调用class内的函数用multiproc ...
使用 Multiprocessing.Pool.map_async 报错 attribute lookup ＜lambda＞ on __main__ failed 的解决
使用 Multiprocessing.Pool.map_async 报错 attribute lookup on __main__ failed 的解决创作背景问题再现解决方法十六 char ...
python进程池win出错_解决windows下python3使用multiprocessing.Pool出现的问题
例如: from multiprocessing import Pool def f(x): return x*x pool = Pool(processes=4) r=pool.map(f, ran ...
mysql报错ERROR 1206 (HY000): The total number of locks exceeds the lock table size的解决方法...
1. 问题背景 InnoDB是新版MySQL(v5.5及以后)默认的存储引擎,之前版本的默认引擎为MyISAM,因此,低于5.5版本的mysql配置文件.my.cnf中,关于InnoD ...
python 多进程 multiprocessing.Queue()报错：The freeze_support() line can be omitted if the program
运行以下多进程测试代码时报错: # -*- coding: utf-8 -*- """ @File : test_191205_测试多进程Multiprocessing_ ...
multiprocessing.pool详解
由于python有全局锁限制,如果想利用多核,就需要使用多进程模块,但该模块有很多坑,本篇文章记录一下其用法以及踩过的坑. 一.map.apply.apply_async对比先贴一个对比图,引自mu ...
在windows下python3使用multiprocessing.Pool时出现的问题
例如: from multiprocessing import Pool def f(x): return x*x pool = Pool(processes=4) r=pool.map(f, ran ...
一、multiprocessing.pool.RemoteTraceback
遇到如下问题多半时数据有问题`. // A code block var foo = 'bar'; multiprocessing.pool.RemoteTraceback: "" ...
Flex报错Error #2048: 安全沙箱冲突
Flex+JPA架构,JPA程序迁移,从A服务器到B.其他一切没变,唯一变的就是IP. 前端Flex也就是swf报错Error #2048: 安全沙箱冲突:http://xxx.swf 不能从 xxx ...

multiprocessing.Pool报pickling error

现象

报错代码

原因:

解决方法:

正确代码1

正确代码2:

multiprocessing.Pool报pickling error相关推荐

最新文章

热门文章