python学习记录一：关于分布式进程执行报错以及解决方案

源代码内容如下：（来自廖雪峰官方网站-python教程-进程和线程-分布式进程）

# test_07.py
import random, queue
from multiprocessing.managers import BaseManager# 发送任务的队列：
task_queue = queue.Queue()
# 接收结果的队列
result_queue = queue.Queue()# 从BaseManager继承的QueueManager：
class QueueManager(BaseManager):pass# 把两个Queue都注册到网络上，callable参数关联了Queue对象
QueueManager.register('get_task_queue', callable=lambda: task_queue)
QueueManager.register('get_result_queue', callable=lambda: result_queue)
# 绑定端口5000，设置验证码‘abc’
manager = QueueManager(address=('', 5000), authkey=b'abc')
# 启动Queue：
manager.start()
# 获得通过网络访问的Queue对象
task = manager.get_task_queue()
result = manager.get_result_queue()
# 放几个任务进去：
for i in range(10):
n = random.randint(0, 10000)
print('Put task %d...' % n)
task.put(n)
# 从result队列读取结果：
print('Try get result...')
for i in range(10):
r = result.get(timeout=10)
print('Result: %s' % r)
# 关闭
manager.shutdown()
print('master exit.')

报错（一）：
执行报错“_pickle.PicklingError: Can’t pickle <function at 0x0000021EED298438>: attribute lookup on main failed”，具体报错内容如下：

"D:\Program Files\python-3.7.9\python.exe" D:/py/test-three/test_07.py
Traceback (most recent call last):File "D:/py/test-three/test_07.py", line 22, in <module>manager.start()File "D:\Program Files\python-3.7.9\lib\multiprocessing\managers.py", line 563, in startself._process.start()File "D:\Program Files\python-3.7.9\lib\multiprocessing\process.py", line 112, in startself._popen = self._Popen(self)File "D:\Program Files\python-3.7.9\lib\multiprocessing\context.py", line 322, in _Popenreturn Popen(process_obj)File "D:\Program Files\python-3.7.9\lib\multiprocessing\popen_spawn_win32.py", line 89, in __init__reduction.dump(process_obj, to_child)File "D:\Program Files\python-3.7.9\lib\multiprocessing\reduction.py", line 60, in dumpForkingPickler(file, protocol).dump(obj)
_pickle.PicklingError: Can't pickle <function <lambda> at 0x0000021EED298438>: attribute lookup <lambda> on __main__ failed

首先查看报错内容，寻找错误源头‘manager.start()’，无法启动Queue，根据“_pickle.PicklingError: Can’t pickle <function at 0x0000021EED298438>: attribute lookup on main failed”，lambda表达式无法序列化，参考文章【https://blog.csdn.net/tpc4289/article/details/79280659/】，自定义函数，修改部分代码如下：

# 自定义函数return_tast_queue
def return_task_queue():global task_queuereturn task_queue# 自定义函数return_result_queue
def return_result_queue():global result_queuereturn result_queue# 把两个Queue都注册到网络上，callable参数关联了Queue对象
QueueManager.register('get_task_queue', callable=return_task_queue)
QueueManager.register('get_result_queue', callable=return_result_queue)

执行修改后的代码，再次报错，报错内容变化，说明上一个问题解决，新增其他问题。
报错（二）：
is not going to be frozen to produce an executable.‘’')……具体报错内容为：

"D:\Program Files\python-3.7.9\python.exe" D:/py/test-three/test_07.py
Traceback (most recent call last):File "<string>", line 1, in <module>File "D:\Program Files\python-3.7.9\lib\multiprocessing\spawn.py", line 105, in spawn_mainexitcode = _main(fd)File "D:\Program Files\python-3.7.9\lib\multiprocessing\spawn.py", line 114, in _mainprepare(preparation_data)File "D:\Program Files\python-3.7.9\lib\multiprocessing\spawn.py", line 225, in prepare_fixup_main_from_path(data['init_main_from_path'])File "D:\Program Files\python-3.7.9\lib\multiprocessing\spawn.py", line 277, in _fixup_main_from_pathrun_name="__mp_main__")File "D:\Program Files\python-3.7.9\lib\runpy.py", line 263, in run_pathpkg_name=pkg_name, script_name=fname)File "D:\Program Files\python-3.7.9\lib\runpy.py", line 96, in _run_module_codemod_name, mod_spec, pkg_name, script_name)File "D:\Program Files\python-3.7.9\lib\runpy.py", line 85, in _run_codeexec(code, run_globals)File "D:\py\test-three\test_07.py", line 32, in <module>manager.start()File "D:\Program Files\python-3.7.9\lib\multiprocessing\managers.py", line 563, in startself._process.start()File "D:\Program Files\python-3.7.9\lib\multiprocessing\process.py", line 112, in startself._popen = self._Popen(self)File "D:\Program Files\python-3.7.9\lib\multiprocessing\context.py", line 322, in _Popenreturn Popen(process_obj)File "D:\Program Files\python-3.7.9\lib\multiprocessing\popen_spawn_win32.py", line 46, in __init__prep_data = spawn.get_preparation_data(process_obj._name)File "D:\Program Files\python-3.7.9\lib\multiprocessing\spawn.py", line 143, in get_preparation_data_check_not_importing_main()File "D:\Program Files\python-3.7.9\lib\multiprocessing\spawn.py", line 136, in _check_not_importing_mainis not going to be frozen to produce an executable.''')
RuntimeError: An attempt has been made to start a new process before thecurrent process has finished its bootstrapping phase.This probably means that you are not using fork to start yourchild processes and you have forgotten to use the proper idiomin the main module:if __name__ == '__main__':freeze_support()...The "freeze_support()" line can be omitted if the programis not going to be frozen to produce an executable.

这个报错内容很明显，需要添加“if name == ‘main’:”，具体原理可参考文章【https://blog.csdn.net/shenfuli/article/details/103969964】

报错（三）：

"D:\Program Files\python-3.7.9\python.exe" D:/py/test-three/test_08.py
Traceback (most recent call last):File "D:/py/test-three/test_08.py", line 36, in <module>task = manager.get_task_queue()File "D:\Program Files\python-3.7.9\lib\multiprocessing\managers.py", line 724, in temptoken, exp = self._create(typeid, *args, **kwds)File "D:\Program Files\python-3.7.9\lib\multiprocessing\managers.py", line 607, in _createconn = self._Client(self._address, authkey=self._authkey)File "D:\Program Files\python-3.7.9\lib\multiprocessing\connection.py", line 492, in Clientc = SocketClient(address)File "D:\Program Files\python-3.7.9\lib\multiprocessing\connection.py", line 620, in SocketClients.connect(address)
OSError: [WinError 10049] 在其上下文中，该请求的地址无效。进程已结束,退出代码1

此处就是照着源代码输入时，丢失ip，需要填入自己的IP，cmd输入‘ipconfig’查看本机ip，填入此段代码：manager = QueueManager(address=(‘此处填入本机IP’, 5000), authkey=b’abc’)

修改后的正确代码如下：

import random, queue
from multiprocessing.managers import BaseManager# 发送任务的队列：
task_queue = queue.Queue()
# 接收结果的队列
result_queue = queue.Queue()def return_task_queue():global task_queuereturn task_queuedef return_result_queue():global result_queuereturn result_queue# 从BaseManager继承的QueueManager：
class QueueManager(BaseManager):passif __name__ == '__main__':# 把两个Queue都注册到网络上，callable参数关联了Queue对象QueueManager.register('get_task_queue', callable=return_task_queue)QueueManager.register('get_result_queue', callable=return_result_queue)# 绑定端口5000，设置验证码‘abc’manager = QueueManager(address=('本机ip', 5000), authkey=b'abc')# 启动Queue：manager.start()# 获得通过网络访问的Queue对象task = manager.get_task_queue()result = manager.get_result_queue()# 放几个任务进去：for i in range(10):n = random.randint(0, 10000)print('Put task %d...' % n)task.put(n)# 从result队列读取结果：print('Try get result...')for i in range(10):r = result.get(timeout=10)print('Result: %s' % r)# 关闭manager.shutdown()print('master exit.')

总结报错1和2是Windows系统特有，由于python在Windows运行，有些使用的方法与其他系统不一样，导致学习过程中提供代码无法正常运行，需要自行排查。

python学习记录一：关于分布式进程执行报错以及解决方案相关推荐

python学习记录三：关于ImageFont.truetype(‘Arial.ttf‘, 36)执行报错以及解决方案
代码如下: from PIL import Image, ImageFilter, ImageFont, ImageDrawimport random# 随机字母 def rndChar():retu ...
学习记录438@MySQL group by having 报错1054 - Unknown column ‘type’ in ‘having clause’
问题引出在执行以下sql语句的时候报错:1054 - Unknown column 'type' in 'having clause' 原因 having 后跟的筛选条件所包含的字段必须是selec ...
关于Windows7 64位系统下regsvr32执行报错的解决方案
转载自:http://www.jb51.net/os/windows/182610.html 感谢分享. 问:我使用的win7 64位旗舰版系统,最近在运行regsvr32.exe的时候会提示&quo ...
Python学习记录day6-反射、常用模块
Python学习记录day6-反射.常用模块 @(学习)[python] Python学习记录day6-反射常用模块反射常用模块 1 sys System-specific parameters ...
Python学习记录day3
2019独角兽企业重金招聘Python工程师标准>>> Python学习记录 day3 今天是银角大王武sir讲课.先回顾了上节课所学,然后讲到了面向对象思想. set set是一个 ...
TCP与UDP协议初步学习——网络环境中分布式进程通信的基本概念
TCP与UDP协议初步学习--网络环境中分布式进程通信的基本概念一.单机系统中进程通信方法进程和进程通信是操作系统中最基本的概念,首先通过回忆操作系统课程中,关于单击系统中进程和进程通信的问题描述 ...
python学习记录——容器篇
容器字符串下标(索引) # 下表也称为是索引,是一个整型数字,可以是正数,也可以是负数 # 正数下标是从0开始的,表示第一个字符,-1表示最后一个字符 my_str = 'hello'h e l ...
Opencv+Python学习记录9：掩膜（掩码）的使用（内附详细代码）
一,基本概念 OpenCV中的很多函数都会指定一个掩模,也被称为掩码,例如: 计算结果=cv2.add(参数1,参数2,掩模) 当使用掩模参数时,操作只会在掩模值为非空的像素点上执行,并将其他像素点的 ...
蜗牛一步一步地往上爬的Python学习记录
Python学习记录前言简介基础知识类型变量高级特性函数式编程高阶函数偏函数装饰器decorator 匿名函数前言第一,绝对不断更: 第二,绝对认真写: 第三,绝对撰写的公正漂亮 ...

python学习记录一：关于分布式进程执行报错以及解决方案

python学习记录一：关于分布式进程执行报错以及解决方案相关推荐

最新文章

热门文章