ConvNeXt

论文地址：https://arxiv.org/abs/2201.03545

一、改进点

随着技术的不断发展，各种新的架构及优化策略促使Transformer拥有更好的效果
相同策略训练卷积神经网络

以ResNet-50为基准

1、Macro design

（1）Swin-T的比例是1：1：3：1 Swin-L的比例是1：1：9：1
堆叠次数由(3, 4, 6, 3)调整为(3, 3, 9, 3)
（2）最初的下采样模块为stem，例如ResNet中stem是7×7卷积核3×3最大池化组成
将ResNet中stem换成卷积核为4，stride为4的卷积层（参考swim-transformer）

2、ResNetXt

（1）使用group convolution
depthwise convolution组卷积的group数和输入层的channel数相等
depthwise convolution——对于每个通道输入图像，对应卷积核进行操作

（2）增大特征层的channel，将每个stage的channel设置与swin-transformer的channel保持一致

3、Inverted bottleneck

ResNet提出bottleneck结构(两头粗，中间细)，MobileNetV2提出Inverted Bottleneck(两头细，中间粗)

4、Large Kerner size

（1）moving up depthwise conv layer，将depthwise conv模块上移
原：1×1 conv → depthwise conv → 1×1 conv
现：depthwise conv → 1×1 conv → 1×1 conv
原因：depthwise conv layer类似Multi-head attention
（2）Increasing the kernel size，将depthwise conv卷积核大小由3×3改成7×7 （7与Swin-Transformer的窗口大小一致）

5、Various layer-wise Micro designs

Replacing ReLU with GELU——准确率没有变化
Fewer activation functions ——Swin Transformer Block仅在1×1卷积后有GELU
Fewer normalization layers
Substituting BN with LN ——Transformer使用LN
Separate downsampling layers

二、网络结构图及每层数据

不同网络的参数

ConvNeXt-T： C=(96, 192, 384, 768)， B=(3, 3, 9, 3)

ConvNeXt-S： C=(96, 192, 384, 768)， B=(3, 3, 27, 3)

ConvNeXt-B： C=(128, 256, 512, 1024)， B=(3, 3, 27, 3)

ConvNeXt-L： C=(192, 384, 768, 1536)， B=(3, 3, 27, 3)

ConvNeXt-XL：C=(256, 512, 1024, 2048)， B=(3, 3, 27, 3)

ConvNeXt-T

4×4, 96, stride 4
[d7×7， 96 1×1, 384 1×1, 192] × 3
[d7×7， 192 1×1, 768 1×1, 192] × 3
[d7×7， 384 1×1, 1536 1×1, 384] × 9
[d7×7， 768 1×1, 3072 1×1, 768] × 3

网络结构图

三、ConvNeXt网络代码

"""
original code from facebook research:
https://github.com/facebookresearch/ConvNeXt
"""import torch
import torch.nn as nn
import torch.nn.functional as Fdef drop_path(x, drop_prob: float = 0., training: bool = False):"""Drop paths (Stochastic Depth) per sample (when applied in main path of residual blocks).This is the same as the DropConnect impl I created for EfficientNet, etc networks, however,the original name is misleading as 'Drop Connect' is a different form of dropout in a separate paper...See discussion: https://github.com/tensorflow/tpu/issues/494#issuecomment-532968956 ... I've opted forchanging the layer and argument names to 'drop path' rather than mix DropConnect as a layer name and use'survival rate' as the argument."""if drop_prob == 0. or not training:return xkeep_prob = 1 - drop_probshape = (x.shape[0],) + (1,) * (x.ndim - 1)  # work with diff dim tensors, not just 2D ConvNetsrandom_tensor = keep_prob + torch.rand(shape, dtype=x.dtype, device=x.device)random_tensor.floor_()  # binarizeoutput = x.div(keep_prob) * random_tensorreturn outputclass DropPath(nn.Module):"""Drop paths (Stochastic Depth) per sample  (when applied in main path of residual blocks)."""def __init__(self, drop_prob=None):super(DropPath, self).__init__()self.drop_prob = drop_probdef forward(self, x):return drop_path(x, self.drop_prob, self.training)class LayerNorm(nn.Module):r""" LayerNorm that supports two data formats: channels_last (default) or channels_first.The ordering of the dimensions in the inputs. channels_last corresponds to inputs withshape (batch_size, height, width, channels) while channels_first corresponds to inputswith shape (batch_size, channels, height, width)."""# channels_first (batch_size, channels, height, width)  pytorch官方默认使用# channels_last  (batch_size, height, width, channels)def __init__(self, normalized_shape, eps=1e-6, data_format="channels_last"):super().__init__()self.weight = nn.Parameter(torch.ones(normalized_shape), requires_grad=True)  # weight bias对应γ βself.bias = nn.Parameter(torch.zeros(normalized_shape), requires_grad=True)self.eps = epsself.data_format = data_formatif self.data_format not in ["channels_last", "channels_first"]:raise ValueError(f"not support data format '{self.data_format}'")self.normalized_shape = (normalized_shape,)def forward(self, x: torch.Tensor) -> torch.Tensor:if self.data_format == "channels_last":return F.layer_norm(x, self.normalized_shape, self.weight, self.bias, self.eps)elif self.data_format == "channels_first":# [batch_size, channels, height, width]# 对channels 维度求均值mean = x.mean(1, keepdim=True)# 方差var = (x - mean).pow(2).mean(1, keepdim=True)# 减均值，除以标准差的操作x = (x - mean) / torch.sqrt(var + self.eps)x = self.weight[:, None, None] * x + self.bias[:, None, None]return x# ConvNeXt Block
class Block(nn.Module):r""" ConvNeXt Block. There are two equivalent implementations:(1) DwConv -> LayerNorm (channels_first) -> 1x1 Conv -> GELU -> 1x1 Conv; all in (N, C, H, W)(2) DwConv -> Permute to (N, H, W, C); LayerNorm (channels_last) -> Linear -> GELU -> Linear; Permute backWe use (2) as we find it slightly faster in PyTorchArgs:dim (int): Number of input channels.drop_rate (float): Stochastic depth rate. Default: 0.0layer_scale_init_value (float): Init value for Layer Scale. Default: 1e-6."""def __init__(self, dim, drop_rate=0., layer_scale_init_value=1e-6):super().__init__()self.dwconv = nn.Conv2d(dim, dim, kernel_size=7, padding=3, groups=dim)  # depthwise convself.norm = LayerNorm(dim, eps=1e-6, data_format="channels_last")self.pwconv1 = nn.Linear(dim, 4 * dim)  # pointwise/1x1 convs, implemented with linear layersself.act = nn.GELU()self.pwconv2 = nn.Linear(4 * dim, dim)# gamma 针对layer scale的操作self.gamma = nn.Parameter(layer_scale_init_value * torch.ones((dim,)),requires_grad=True) if layer_scale_init_value > 0 else Noneself.drop_path = DropPath(drop_rate) if drop_rate > 0. else nn.Identity()  # nn.Identity() 恒等映射def forward(self, x: torch.Tensor) -> torch.Tensor:shortcut = xx = self.dwconv(x)x = x.permute(0, 2, 3, 1)  # [N, C, H, W] -> [N, H, W, C]x = self.norm(x)x = self.pwconv1(x)x = self.act(x)x = self.pwconv2(x)if self.gamma is not None:x = self.gamma * xx = x.permute(0, 3, 1, 2)  # [N, H, W, C] -> [N, C, H, W]x = shortcut + self.drop_path(x)return xclass ConvNeXt(nn.Module):r""" ConvNeXtA PyTorch impl of : `A ConvNet for the 2020s`  -https://arxiv.org/pdf/2201.03545.pdfArgs:in_chans (int): Number of input image channels. Default: 3num_classes (int): Number of classes for classification head. Default: 1000depths (tuple(int)): Number of blocks at each stage. Default: [3, 3, 9, 3]dims (int): Feature dimension at each stage. Default: [96, 192, 384, 768]drop_path_rate (float): Stochastic depth rate. Default: 0.layer_scale_init_value (float): Init value for Layer Scale. Default: 1e-6.head_init_scale (float): Init scaling value for classifier weights and biases. Default: 1."""def __init__(self, in_chans: int = 3, num_classes: int = 1000, depths: list = None,dims: list = None, drop_path_rate: float = 0., layer_scale_init_value: float = 1e-6,head_init_scale: float = 1.):super().__init__()# 最初下采样部分self.downsample_layers = nn.ModuleList()  # stem and 3 intermediate downsampling conv layers# Conv2d k4, s4# LayerNormstem = nn.Sequential(nn.Conv2d(in_chans, dims[0], kernel_size=4, stride=4),LayerNorm(dims[0], eps=1e-6, data_format="channels_first"))self.downsample_layers.append(stem)# 对应stage2-stage4前的3个downsamplefor i in range(3):downsample_layer = nn.Sequential(LayerNorm(dims[i], eps=1e-6, data_format="channels_first"),nn.Conv2d(dims[i], dims[i + 1], kernel_size=2, stride=2))self.downsample_layers.append(downsample_layer)self.stages = nn.ModuleList()  # 4 feature resolution stages, each consisting of multiple blocks# 等差数列，初始值0，到drop path rate，总共depths个数dp_rates = [x.item() for x in torch.linspace(0, drop_path_rate, sum(depths))]cur = 0# 构建每个stage中堆叠的blockfor i in range(4):stage = nn.Sequential(*[Block(dim=dims[i], drop_rate=dp_rates[cur + j], layer_scale_init_value=layer_scale_init_value)for j in range(depths[i])])self.stages.append(stage)cur += depths[i]self.norm = nn.LayerNorm(dims[-1], eps=1e-6)  # final norm layerself.head = nn.Linear(dims[-1], num_classes)self.apply(self._init_weights)self.head.weight.data.mul_(head_init_scale)self.head.bias.data.mul_(head_init_scale)def _init_weights(self, m):if isinstance(m, (nn.Conv2d, nn.Linear)):nn.init.trunc_normal_(m.weight, std=0.2)nn.init.constant_(m.bias, 0)def forward_features(self, x: torch.Tensor) -> torch.Tensor:for i in range(4):x = self.downsample_layers[i](x)x = self.stages[i](x)return self.norm(x.mean([-2, -1]))  # global average pooling, (N, C, H, W) -> (N, C)def forward(self, x: torch.Tensor) -> torch.Tensor:x = self.forward_features(x)x = self.head(x)return xdef convnext_tiny(num_classes: int):# https://dl.fbaipublicfiles.com/convnext/convnext_tiny_1k_224_ema.pthmodel = ConvNeXt(depths=[3, 3, 9, 3],dims=[96, 192, 384, 768],num_classes=num_classes)return modeldef convnext_small(num_classes: int):# https://dl.fbaipublicfiles.com/convnext/convnext_small_1k_224_ema.pthmodel = ConvNeXt(depths=[3, 3, 27, 3],dims=[96, 192, 384, 768],num_classes=num_classes)return modeldef convnext_base(num_classes: int):# https://dl.fbaipublicfiles.com/convnext/convnext_base_1k_224_ema.pth# https://dl.fbaipublicfiles.com/convnext/convnext_base_22k_224.pthmodel = ConvNeXt(depths=[3, 3, 27, 3],dims=[128, 256, 512, 1024],num_classes=num_classes)return modeldef convnext_large(num_classes: int):# https://dl.fbaipublicfiles.com/convnext/convnext_large_1k_224_ema.pth# https://dl.fbaipublicfiles.com/convnext/convnext_large_22k_224.pthmodel = ConvNeXt(depths=[3, 3, 27, 3],dims=[192, 384, 768, 1536],num_classes=num_classes)return modeldef convnext_xlarge(num_classes: int):# https://dl.fbaipublicfiles.com/convnext/convnext_xlarge_22k_224.pthmodel = ConvNeXt(depths=[3, 3, 27, 3],dims=[256, 512, 1024, 2048],num_classes=num_classes)return model

PyTorch深度学习（25）网络结构ConvNeXt相关推荐

pytorch深度学习入门笔记
Pytorch 深度学习入门笔记作者:梅如你学习来源: 公众号: 阿力阿哩哩.土堆碎念 B站视频:https://www.bilibili.com/video/BV1hE411t7RN? 中国大学 ...
【Pytorch深度学习实践】B站up刘二大人课程笔记——目录与索引（已完结）
从有代码的课程开始讨论 [Pytorch深度学习实践]B站up刘二大人之LinearModel -代码理解与实现(1/9) [Pytorch深度学习实践]B站up刘二大人之 Gradient Desc ...
PyTorch实现 | 车牌OCR识别，《PyTorch深度学习之目标检测》
注:本文选自中国水利水电出版社出版<PyTorch深度学习之目标检测>一书,有改动福利!免费寄送图书!! 公众号[机器学习与AI生成创作]后台回复:168.即可参与免费寄送图书活动,活动 ...
pytorch | 深度学习分割网络U-net的pytorch模型实现
原文:https://blog.csdn.net/u014722627/article/details/60883185 pytorch | 深度学习分割网络U-net的pytorch模型实现这个是 ...
pytorch深度学习_深度学习和PyTorch的推荐系统实施
pytorch深度学习 The recommendation is a simple algorithm that works on the principle of data filtering. ...
pytorch深度学习_用于数据科学家的深度学习的最小pytorch子集
pytorch深度学习 PyTorch has sort of became one of the de facto standards for creating Neural Networks no ...
《PyTorch 深度学习实践》第10讲卷积神经网络（基础篇）
文章目录 1 卷积层 1.1 torch.nn.Conv2d相关参数 1.2 填充:padding 1.3 步长:stride 2 最大池化层 3 手写数字识别该专栏内容为对该视频的学习记录:[&l ...
Pytorch深度学习实战教程：UNet语义分割网络
1 前言本文属于Pytorch深度学习语义分割系列教程. 该系列文章的内容有: Pytorch的基本使用语义分割算法讲解本文的开发环境如下: 开发环境:Windows 开发语言:Python3. ...
【PyTorch深度学习实践】P9 kaggle otto商品分类作业（含注释）
<PyTorch深度学习实践>-刘二大人 Otto Group Product Classification作业将商品进行十分类,输入为93个特征10个类别的商品数据集,输出为预测数据集 ...
人工智能：PyTorch深度学习框架介绍
目录 1.PyTorch 2.PyTorch常用的工具包 3.PyTorch特点 4.PyTorch不足之处今天给大家讲解一下PyTorch深度学习框架的一些基础知识,希望对大家理解PyTorch有 ...

PyTorch深度学习（25）网络结构ConvNeXt