Reducer buckets have been rebuilt in this iteration.

在跑torch多GPU报错

“Reducer buckets have been rebuilt in this iteration.”原因是torch版本问题， torch1.7以上的distributed.py发生更改导致报错：

这玩意是distributed.py里的前向函数报错

def forward(self, inputs, *kwargs):           if self.ddp_join_enabled:               ones = torch.ones(                   1, device=self.device               )               work = dist.all_reduce(ones, group=self.process_group, async_op=True)               self.reducer._set_forward_pass_work_handle(                   work, self.ddp_join_divide_by_initial_world_size               )
# Calling _rebuild_buckets before forward compuation,# It may allocate new buckets before deallocating old buckets# inside _rebuild_buckets. To save peak memory usage,# call _rebuild_buckets before the peak memory usage increases# during forward computation.# This should be called only once during whole training period.if self.reducer._rebuild_buckets():logging.info("Reducer buckets have been rebuilt in this iteration.")if self.require_forward_param_sync:self._sync_params()if self.ddp_join_enabled:# Notify joined ranks whether they should sync in backwards pass or not.self._check_global_requires_backward_grad_sync(is_joined_rank=False)# ！！！if self.device_ids:inputs, kwargs = self.scatter(inputs, kwargs, self.device_ids)if len(self.device_ids) == 1:output = self.module(*inputs[0], **kwargs[0])else:# 单进程多线程多卡的情况outputs = self.parallel_apply(self._module_copies[:len(inputs)], inputs, kwargs)output = self.gather(outputs, self.output_device)else:output = self.module(*inputs, **kwargs)if torch.is_grad_enabled() and self.require_backward_grad_sync:self.require_forward_param_sync = True# We'll return the output object verbatim since it is a freeform# object. We need to find any tensors in this object, though,# because we need to figure out which parameters were used during# this forward pass, to ensure we short circuit reduction for any# unused parameters. Only if `find_unused_parameters` is set.if self.find_unused_parameters:# 当DDP参数 find_unused_parameter 为 true 时，其会在 forward 结束时，启动一个回溯，标记出所有没被用到的 parameter，提前把这些设定为 ready，这样 backward 就可以在一个 subgraph 进行，但这样会牺牲一部分时间。self.reducer.prepare_for_backward(list(_find_tensors(output)))else:self.reducer.prepare_for_backward([])else:self.require_forward_param_sync = Falsereturn output

解决思路：
1、对torch进行降级，构建torch1.6环境
torch1.6 cuda10 torchvision 0.7.0

pip install torch==1.6.0 torchvision==0.7.0 # CUDA 10.2
pip install torch==1.6.0 torchvision==0.7.0# CUDA 10.1
pip install torch==1.6.0+cu101 torchvision==0.7.0+cu101 -f https://download.pytorch.org/whl/torch_stable.html# CUDA 9.2
pip install torch==1.6.0+cu92 torchvision==0.7.0+cu92 -f https://download.pytorch.org/whl/torch_stable.html

修改代码
已解决，适用我的问题
我的代码：

predicts, loss, loss_statics = model(data)

结构整体改为：

loss, loss_statics = model(data)

对于distributed.py DistributedDataParallel来说，它的forward只接受关于 Loss的返回值，predicits就不可以加入；之前torch1.6版本没有问题；

Reducer buckets have been rebuilt in this iteration.相关推荐

Geometry-guided Kernel Transformer ——GKT代码复现
1,下载代码:https://github.com/hustvl/GKT git clone https://github.com/hustvl/GKT 2,创建环境: conda create -n ...
IDE Framework之mmdetection使用记录20200707-
文章目录 mmdetection代码阅读 Git版本控制时的commit message mmdet_v2.7 MMDet v2.7.0 MMDet v2.3.0 {Config}与{ConfigDi ...
民谣女神唱流行，基于AI人工智能so-vits库训练自己的音色模型(叶蓓/Python3.10)
流行天后孙燕姿的音色固然是极好的,但是目前全网都是她的声音复刻,听多了难免会有些审美疲劳,在网络上检索了一圈,还没有发现民谣歌手的音色模型,人就是这样,得不到的永远在骚动,本次我们自己构建训练集,来打 ...
【CV】深度学习中Epoch, Batch, Iteration的含义
Epoch 使用训练集的全部数据样本进行一次训练,称为一次epoch,即所有训练集的样本都在神经网络中进行了一次正向传播和一次反向传播神经网络中需要有多次epoch,每次epoch中会进行一次更新权 ...
ADPRL - 近似动态规划和强化学习 - Note 8 - 近似策略迭代 (Approximate Policy Iteration)
Note 8 近似策略迭代 Approximate Policy Iteration 近似策略迭代 Note 8 近似策略迭代 Approximate Policy Iteration 8.1 通用框 ...
ADPRL - 近似动态规划和强化学习 - Note 4 - Policy Iteration Algorithms
Note 4 - Policy Iteration Algorithms 4. Policy Iteration Algorithms 补充:范数的性质 4.1 贪婪诱导策略的特性 (Properti ...
vue/require-v-for-key]Elements in iteration expect to have ‘v-bind:key‘ directives
报错内容:[vue/require-v-for-key]Elements in iteration expect to have 'v-bind:key' directives.解决:加上v-bind ...
深度学习 | 三个概念：Epoch, Batch, Iteration
Epoch:使用训练集的全部数据对模型进行一次完整训练,被称之为"一代训练". Batch:使用训练集中的一小部分样本对模型权重进行一次反向传摇的参数更新,这一小部分样本被称为&q ...
迭代var（）内置函数的时候出现RuntimeError: dictionary changed size during iteration的解决办法...
下午看了Mr Seven的教学视频,其中有一段讲全局变量的视频,迭代输出全局变量的时候报错了. 视频中的做法: for k,v in vars().items():print(k) 打印结果 for ...

Reducer buckets have been rebuilt in this iteration.

Reducer buckets have been rebuilt in this iteration.相关推荐

最新文章

热门文章