resnet50代码_13、SOTA论文实践-学习ResNet（80行代码搞定残差backbone网络）

0、论文

Camera Distance-aware Top-down Approach for 3D Multi-person Pose Estimation from a Single RGB Image. ICCV 19.arxiv.org

1、项目地址

mks0601/3DMPPE_POSENET_RELEASEgithub.com

PoseNet部分是生成root-relative pose的部分。

2、my notes on the paper

banana16314：10. 3DMPPE: multi-person 3D pose estimationzhuanlan.zhihu.com

3、The Code

3.1 指定backbone残差网络类型和结构

作为残差网络的backbone网络的结构图：

Figure1. ResNet as backbone network

class ResNetBackbone(nn.Module):def __init__(self, resnet_type):resnet_spec = {18: (BasicBlock, [2, 2, 2, 2], [64, 64, 128, 256, 512], 'resnet18'),34: (BasicBlock, [3, 4, 6, 3], [64, 64, 128, 256, 512], 'resnet34'),50: (Bottleneck, [3, 4, 6, 3], [64, 256, 512, 1024, 2048], 'resnet50'),101: (Bottleneck, [3, 4, 23, 3], [64, 256, 512, 1024, 2048], 'resnet101'),152: (Bottleneck, [3, 8, 36, 3], [64, 256, 512, 1024, 2048], 'resnet152')}block, layers, channels, name = resnet_spec[resnet_type]self.name = nameself.inplanes = 64super(ResNetBackbone, self).__init__()self.conv1 = nn.Conv2d(3, 64, kernel_size=7, stride=2, padding=3,bias=False)self.bn1 = nn.BatchNorm2d(64)self.relu = nn.ReLU(inplace=True)self.maxpool = nn.MaxPool2d(kernel_size=3, stride=2, padding=1)self.layer1 = self._make_layer(block, 64, layers[0])self.layer2 = self._make_layer(block, 128, layers[1], stride=2)self.layer3 = self._make_layer(block, 256, layers[2], stride=2)self.layer4 = self._make_layer(block, 512, layers[3], stride=2)for m in self.modules():if isinstance(m, nn.Conv2d):# nn.init.kaiming_normal_(m.weight, mode='fan_out', nonlinearity='relu')nn.init.normal_(m.weight, mean=0, std=0.001)elif isinstance(m, nn.BatchNorm2d):nn.init.constant_(m.weight, 1)nn.init.constant_(m.bias, 0)

实验部分采用ResNet 50. batch_size为32.

Conv1:输入3通道的256x256大小的RGB图像，输出64通道，特征图尺寸64x64.

之后的每层，都是通过_make_layer()函数实现：

def _make_layer(self, block, planes, blocks, stride=1):downsample = Noneif stride != 1 or self.inplanes != planes * block.expansion:downsample = nn.Sequential(nn.Conv2d(self.inplanes, planes * block.expansion,kernel_size=1, stride=stride, bias=False),nn.BatchNorm2d(planes * block.expansion),)layers = []layers.append(block(self.inplanes, planes, stride, downsample))self.inplanes = planes * block.expansionfor i in range(1, blocks):layers.append(block(self.inplanes, planes))return nn.Sequential(*layers)

向前传播，提取深层特征，即下采样层和各层的输出尺寸：

def forward(self, x):# 32 3 256 256x = self.conv1(x)x = self.bn1(x)x = self.relu(x)x0 = self.maxpool(x)#print(x.size())# 32 64 64 64x1 = self.layer1(x0)#print(x.size())# 32 256 64 64 x2 = self.layer2(x1)#print(x.size()) # 32 512 32 32x3 = self.layer3(x2)#print(x.size())# 32 1024 16 16 x4 = self.layer4(x3)# print(x.size())# 32 2048 8 8return x4

最深层的特征图尺寸8x8，包含深层语义特征。

网络权重的初始化，从Model_zoo里面获取：

def init_weights(self):org_resnet = torch.utils.model_zoo.load_url(url = model_urls[self.name])# drop orginal resnet fc layer, add 'None' in case of no fc layer, that will raise errororg_resnet.pop('fc.weight', None)org_resnet.pop('fc.bias', None)self.load_state_dict(org_resnet)print("Initialize resnet from model zoo")

整个backbone残差网络，80行代码搞定！

3.2 构建头部网络-headnet

头部网络headnet是对于backbone的特征图进行处理，获取各个3d坐标。

首先，8x8的特征图进行上采样恢复到64x64，这里采用的是 nn.ConvTranspose2d()转置卷积操作，相比nn.Upsample()，转置卷积需要学习参数；

最后，从特征图生成关节热力图，实现是由1x1卷积；

再从热力图获取坐标。

特征图的上采样：

        self.deconv_layers = self._make_deconv_layer(3)
...def _make_deconv_layer(self, num_layers):layers = []for i in range(num_layers):layers.append(nn.ConvTranspose2d(in_channels=self.inplanes,out_channels=self.outplanes,kernel_size=4,stride=2,padding=1,output_padding=0,bias=False))layers.append(nn.BatchNorm2d(self.outplanes))layers.append(nn.ReLU(inplace=True))self.inplanes = self.outplanesreturn nn.Sequential(*layers)

8x8的特征图，经过3次转置卷积，尺寸恢复为64x64.

然后，1x1卷积生成热力图：

    self.final_layer = nn.Conv2d(in_channels=self.inplanes,out_channels=joint_num * cfg.depth_dim,kernel_size=1,stride=1,padding=0)

定义头部网络headnet的forward：

    def forward(self, x):x = self.deconv_layers(x)x = self.final_layer(x)return x

3.3 backbone和head结合为PoseNet

class ResPoseNet(nn.Module):def __init__(self, backbone, head, joint_num):super(ResPoseNet, self).__init__()self.backbone = backboneself.head = headself.joint_num = joint_numdef forward(self, input_img, target=None):fm = self.backbone(input_img)hm = self.head(fm)coord = soft_argmax(hm, self.joint_num)if target is None:return coordelse:target_coord = target['coord']target_vis = target['vis']target_have_depth = target['have_depth']## coordinate lossloss_coord = torch.abs(coord - target_coord) * target_visloss_coord = (loss_coord[:,:,0] + loss_coord[:,:,1] + loss_coord[:,:,2] * target_have_depth)/3.return loss_coorddef get_pose_net(cfg, is_train, joint_num):backbone = ResNetBackbone(cfg.resnet_type)head_net = HeadNet(joint_num)if is_train:backbone.init_weights()head_net.init_weights()model = ResPoseNet(backbone, head_net, joint_num)return model

其中，从热力图获取坐标，采取的操作是soft-argmax/integral operation. 原理可以参考：https://zhuanlan.zhihu.com/p/92417329

soft-argmax:

def soft_argmax(heatmaps, joint_num):heatmaps = heatmaps.reshape((-1, joint_num, cfg.depth_dim*cfg.output_shape[0]*cfg.output_shape[1]))heatmaps = F.softmax(heatmaps, 2)heatmaps = heatmaps.reshape((-1, joint_num, cfg.depth_dim, cfg.output_shape[0], cfg.output_shape[1]))accu_x = heatmaps.sum(dim=(2,3))accu_y = heatmaps.sum(dim=(2,4))accu_z = heatmaps.sum(dim=(3,4))accu_x = accu_x * torch.cuda.comm.broadcast(torch.arange(1,cfg.output_shape[1]+1).type(torch.cuda.FloatTensor), devices=[accu_x.device.index])[0]accu_y = accu_y * torch.cuda.comm.broadcast(torch.arange(1,cfg.output_shape[0]+1).type(torch.cuda.FloatTensor), devices=[accu_y.device.index])[0]accu_z = accu_z * torch.cuda.comm.broadcast(torch.arange(1,cfg.depth_dim+1).type(torch.cuda.FloatTensor), devices=[accu_z.device.index])[0]accu_x = accu_x.sum(dim=2, keepdim=True) -1accu_y = accu_y.sum(dim=2, keepdim=True) -1accu_z = accu_z.sum(dim=2, keepdim=True) -1coord_out = torch.cat((accu_x, accu_y, accu_z), dim=2)return coord_out

整个模型的构建117行代码。