RuntimeError: cuda runtime error (3) : initialization error at /pytorch/aten/src/THC/THCGeneral.cpp:

第三个大坑：在python多进程中使用pytorch加载模型时报错：

THCudaCheck FAIL file=/pytorch/aten/src/THC/THCGeneral.cpp line=50 error=3 : initialization error
Process Process-12:
Traceback (most recent call last):File "/usr/lib/python3.6/multiprocessing/process.py", line 258, in _bootstrapself.run()File "/usr/lib/python3.6/multiprocessing/process.py", line 93, in runself._target(*self._args, **self._kwargs)File "/home/data/user1/intergration_test/recog/recognition.py", line 410, in recog_outputseg_net = infer.load_model(net_arc=args.seg_network, pre_model=args.seg_model)File "/home/data/user1/intergration_test/recog/recognition.py", line 115, in load_modelreturn torch.nn.DataParallel(net, device_ids=range(args.ngpu)).cuda()File "/home/user1/.local/lib/python3.6/site-packages/torch/nn/modules/module.py", line 305, in cudareturn self._apply(lambda t: t.cuda(device))File "/home/user1/.local/lib/python3.6/site-packages/torch/nn/modules/module.py", line 202, in _applymodule._apply(fn)File "/home/user1/.local/lib/python3.6/site-packages/torch/nn/modules/module.py", line 202, in _applymodule._apply(fn)File "/home/user1/.local/lib/python3.6/site-packages/torch/nn/modules/module.py", line 202, in _applymodule._apply(fn)File "/home/user1/.local/lib/python3.6/site-packages/torch/nn/modules/module.py", line 224, in _applyparam_applied = fn(param)File "/home/user1/.local/lib/python3.6/site-packages/torch/nn/modules/module.py", line 305, in <lambda>return self._apply(lambda t: t.cuda(device))File "/home/user1/.local/lib/python3.6/site-packages/torch/cuda/__init__.py", line 193, in _lazy_inittorch._C._cuda_init()
RuntimeError: cuda runtime error (3) : initialization error at /pytorch/aten/src/THC/THCGeneral.cpp:50

或者是下面这个：

THCudaCheck FAIL file=../aten/src/THC/THCGeneral.cpp line=51 error=3 : initialization error
Process Process-4:
Traceback (most recent call last):File "/home/data/conda_envs/thpj/lib/python3.6/multiprocessing/process.py", line 258, in _bootstrapself.run()File "/home/data/conda_envs/thpj/lib/python3.6/multiprocessing/process.py", line 93, in runself._target(*self._args, **self._kwargs)File "/home/pj/frm_v3/frmwork3_all_q/frmwork3_all_q/recog_v3.py", line 445, in recog_outputseg_net = infer.load_model(net_arc=args.seg_network, pre_model=args.seg_model)File "/home/pj/frm_v3/frmwork3_all_q/frmwork3_all_q/recog_v3.py", line 124, in load_modelreturn torch.nn.DataParallel(net, device_ids=range(args.ngpu)).cuda()File "/home/data/conda_envs/thpj/lib/python3.6/site-packages/torch/nn/modules/module.py", line 265, in cudareturn self._apply(lambda t: t.cuda(device))File "/home/data/conda_envs/thpj/lib/python3.6/site-packages/torch/nn/modules/module.py", line 193, in _applymodule._apply(fn)File "/home/data/conda_envs/thpj/lib/python3.6/site-packages/torch/nn/modules/module.py", line 193, in _applymodule._apply(fn)File "/home/data/conda_envs/thpj/lib/python3.6/site-packages/torch/nn/modules/module.py", line 193, in _applymodule._apply(fn)File "/home/data/conda_envs/thpj/lib/python3.6/site-packages/torch/nn/modules/module.py", line 199, in _applyparam.data = fn(param.data)File "/home/data/conda_envs/thpj/lib/python3.6/site-packages/torch/nn/modules/module.py", line 265, in <lambda>return self._apply(lambda t: t.cuda(device))File "/home/data/conda_envs/thpj/lib/python3.6/site-packages/torch/cuda/__init__.py", line 163, in _lazy_inittorch._C._cuda_init()
RuntimeError: cuda runtime error (3) : initialization error at ../aten/src/THC/THCGeneral.cpp:51

原因（个人判断）：theano不能和pytorch运行在同一个gpu上。可以让他们分别运行在两个gpu上，或者一个在gpu，另一个在cpu

可能的解决办法：

1，gpu充足的情况下，将theano和pytorch分别运行在不同的GPU上有可能解决这个问题。如果硬件不允许，则将theano运行在cpu, pytorch运行在gpu。

2，在运行过程中，先使用pytorch加载模型，再使用theano加载模型，保证这个先后顺序有可能能解决问题

3，其他可能的解决方法见参考github链接2。

4，这是一个bug。这个问题在pytorch 1.2及之后的版本中修复了？参考github链接1。

参考：

1 https://github.com/pytorch/pytorch/issues/17357

2 https://github.com/pytorch/pytorch/issues/15734

RuntimeError: cuda runtime error (3) : initialization error at /pytorch/aten/src/THC/THCGeneral.cpp:相关推荐

RuntimeError: cuda runtime error (11) : invalid argument at /pytorch/aten/src/THC/THCGeneral.cpp
RuntimeError: cuda runtime error (11) : invalid argument at /pytorch/aten/src/THC/THCGeneral.cpp cud ...
[debug]THCudaCheck FAIL file=/pytorch/aten/src/THC/THCGeneral.cpp line=405 error=11 : invalid argume
解决 THCudaCheck FAIL file=/pytorch/aten/src/THC/THCGeneral.cpp line=405 error=11 : invalid argument 完 ...
解决RuntimeError: cuda runtime error (30) : unknown error at /pytorch/aten/src/THC/THCGeneral.cpp:70N
解决两个问题: (1)RuntimeError: cuda runtime error (30) : unknown error at /pytorch/aten/src/THC/THCGeneral ...
RuntimeError: cuda runtime error (30) : unknown error at /pytorch/aten/src/THC/THCGeneral.cpp:50
RuntimeError: cuda runtime error (30) : unknown error at /pytorch/aten/src/THC/THCGeneral.cpp:50 在运行 ...
pytorch在调用GPU的时候出现cuda runtime error (2) : out of memory at ..\aten\src\THC\THCGeneral.cpp:50
这是因为GPU中的显存占满了,在任务管理器中关闭程序即可 @[TOC](pytorch在调用GPU的时候出现cuda runtime error (2) : out of memory at -\at ...
python出现THCudaCheck FAIL file=/pytorch/aten/src/THC/ThCGeneral.cpp line=405 error=11 : 情况
python出现THCudaCheck FAIL file=/pytorch/aten/src/THC/ThCGeneral.cpp line=405 error=11 : invalid argum ...
pytorch 报错“THCudaCheck FAIL file=/pytorch/aten/src/THC/THCGeneral.cpp line=663 error=11“解决方案
今天使用pysot测试数据,pytorch出现下面报错: THCudaCheck FAIL file=/opt/conda/conda-bld/pytorch_1535493744281/work/a ...
file=/pytorch/aten/src/THC/THCGeneral.cpp line=383 error=8 : invalid device
这个错误通常出现在使用CUDA加速的PyTorch代码中.这个错误表示在GPU设备上执行的某个函数不是一个有效的设备函数.其中的错误代码"error=8"是CUDA runtime ...
PyTorch报错：RuntimeError: CUDA error: device-side assert triggered at /pytorch/aten/src/THC/generic
训练模型报错 RuntimeError: cuda runtime error (710) : device-side assert triggered at /pytorch/aten/src/TH ...

RuntimeError: cuda runtime error (3) : initialization error at /pytorch/aten/src/THC/THCGeneral.cpp:

RuntimeError: cuda runtime error (3) : initialization error at /pytorch/aten/src/THC/THCGeneral.cpp:相关推荐

最新文章

热门文章