一、multiprocessing.pool.RemoteTraceback

遇到如下问题多半时数据有问题`。

// A code block
var foo = 'bar';

multiprocessing.pool.RemoteTraceback:
"""
Traceback (most recent call last):File "/home/unaguo/anaconda3/envs/pt1.3-py3.6/lib/python3.6/multiprocessing/pool.py", line 119, in workerresult = (True, func(*args, **kwds))File "/home/unaguo/anaconda3/envs/pt1.3-py3.6/lib/python3.6/site-packages/mxnet/gluon/data/dataloader.py", line 429, in _worker_fnbatch = batchify_fn([_worker_dataset[i] for i in samples])File "/home/unaguo/anaconda3/envs/pt1.3-py3.6/lib/python3.6/site-packages/mxnet/gluon/data/dataloader.py", line 429, in <listcomp>batch = batchify_fn([_worker_dataset[i] for i in samples])File "/data2/enducation/paper_recog_total/train-paper-recog/line_detect/data/paper_dataset.py", line 375, in __getitem__data_dict = self._transforms(data_dict)File "/data2/enducation/paper_recog_total/train-paper-recog/line_detect/data/transforms_paper.py", line 13, in __call__args = trans(args)File "/data2/enducation/paper_recog_total/train-paper-recog/line_detect/data/transforms_paper.py", line 468, in __call__dst_points = np.array([[rdw(), rdh()], [w-1-rdw(), rdh()], [w-1-rdw(), h-1-rdh()], [rdw(), h-1-rdh()]])File "/data2/enducation/paper_recog_total/train-paper-recog/line_detect/data/transforms_paper.py", line 466, in <lambda>rdh = lambda: np.random.randint(0, self.max_affine_xy_ratio * h)File "mtrand.pyx", line 746, in numpy.random.mtrand.RandomState.randintFile "_bounded_integers.pyx", line 1254, in numpy.random._bounded_integers._rand_int64
ValueError: low >= high
"""The above exception was the direct cause of the following exception:Traceback (most recent call last):File "/data2/enducation/paper_recog_total/train-paper-recog/line_detect/scripts/train_gluon_testpaper.py", line 233, in <module>for batch_cnt, data_batch in enumerate(tqdm.tqdm(train_loader)):File "/home/unaguo/anaconda3/envs/pt1.3-py3.6/lib/python3.6/site-packages/tqdm/std.py", line 1178, in __iter__for obj in iterable:File "/home/unaguo/anaconda3/envs/pt1.3-py3.6/lib/python3.6/site-packages/mxnet/gluon/data/dataloader.py", line 484, in __next__batch = pickle.loads(ret.get(self._timeout))File "/home/unaguo/anaconda3/envs/pt1.3-py3.6/lib/python3.6/multiprocessing/pool.py", line 644, in getraise self._value
ValueError: low >= high

解决思路：
将mx.gluon.data.DataLoader中修改thread_pool=True，什么意思呢？
If True, use threading pool instead of multiprocessing pool. Using threadpool can avoid shared memory usage. If DataLoader is more IO bounded or GIL is not a killing problem, threadpool version may achieve better performance than multiprocessing.
翻译：如果True，则使用线程池而不是多处理池。使用线程池可以避免共享内存的使用。如果“DataLoader”的IO范围更大，或者GIL不是致命的问题是，线程池版本可能实现比多处理更好的性能。

train_loader = mx.gluon.data.DataLoader(train_dataset, batch_size=config.TRAIN.batch_size,shuffle=True, num_workers=2, thread_pool=True,last_batch="discard", batchify_fn=batch_fn)

一、multiprocessing.pool.RemoteTraceback相关推荐

python3多进程 pool manager_python多进程multiprocessing Pool相关问题
python多进程想必大部分人都用到过,可以充分利用多核CPU让代码效率更高效. 我们看看multiprocessing.pool.Pool.map的官方用法 map(func, iterable[, ...
python进程池multiprocessing.Pool运行错误：The freeze_support() line can be omitted if the program is not g
测试代码如下: 原文:https://blog.csdn.net/xiemanr/article/details/71700531 # -*- coding: utf-8 -*- import mul ...
python3多进程 pool manager_Python多进程multiprocessing.Pool
Multiprocessing.Pool可以提供指定数量的进程供用户调用,当有新的请求提交到pool中时,如果池还没有满,那么就会创建一个新的进程用来执行该请求:但如果池中的进程数已经达到规定最大值, ...
python子进程关闭fd_如果创建了multiprocessing.Pool,Python子进程wait()将失败
在一个使用subprocess到gzip输出的简单脚本中(使用subprocess.PIPE到外部命令的stdin),如果在创建子进程和关闭进程的stdin之间创建了multiprocessing.P ...
python process pool_python multiprocessing.Process，multiprocessing.Pool区别（不同之处）
我们知道GIL Lock禁用了Python中的多线程功能.在Python中multiprocessing提供了两个用于多进程的类,即Process和Pool类.在以下各节中,我已对使用pool和pro ...
python 多进程调用模块内函数_Python进程池multiprocessing.Pool的用法
一.multiprocessing模块 multiprocessing模块提供了一个Process类来代表一个进程对象,multiprocessing模块像线程一样管理进程,这个是multiproce ...
python processpoolexector 释放内存_使用Python的multiprocessing.pool,内存使用量不断增长...
这是程序: #!/usr/bin/python import multiprocessing def dummy_func(r): pass def worker(): pass if __name_ ...
python进程池：multiprocessing.pool
阅读目录例1:使用进程池例2:使用进程池(阻塞) 例3:使用进程池,并关注结果例4:使用多个进程池在利用Python进行系统管理的时候,特别是同时操作多个文件目录,或者远程控制多台主机,并行操 ...
Python 进程池 multiprocessing.Pool - Python零基础入门教程
目录一.Python 进程池 multiprocessing.Pool 介绍二.Python 进程池 multiprocessing.Pool 使用三.猜你喜欢零基础 Python 学习路线推 ...

一、multiprocessing.pool.RemoteTraceback

一、multiprocessing.pool.RemoteTraceback相关推荐

最新文章

热门文章