中山大学提出SimAM：无参Attention！助力分类/检测/分割涨点！

http://proceedings.mlr.press/v139/yang21o.html

code: https://github.com/ZjjConan/SimAM

在正式介绍本文所提注意力模块之前，我们先对现有代表性注意力模块(比如SE、CBAM、GC)进行简要总结；然后，我们再引出本文所提完全不同架构的注意力模块。

Overview of Existing Attention Modules

上图a与b列出了现有两种类型的注意力模块：

通道注意力：1D注意力，它对不同通道区别对待，对所有位置同等对待；
空域注意力：2D注意力，它对不同位置区别对待，对所有通道同等对待。

以下图为例，SE缺失了关于"grey_whale"的某些重要成分。我们认为3D注意力比1D和2D更佳，进而提出了上图c的3D注意力模块。

现有注意力模块的另一个重要影响因素：权值生成方法。现有注意力往往采用额外的子网络生成注意力权值，比如SE的GAP+FC+ReLU+FC+Sigmoid。更多注意力模块的操作、参数量可参考下表。总而言之，现有注意力的结构设计需要大量的工程性实验。我们认为：注意力机制的实现应当通过神经科学中的某些统一原则引导设计。

Our Attention Module

已有研究BAM、CBAM分别将空域注意力与通道注意力进行并行或串行组合。然而，人脑的两种注意力往往是协同工作，因此，我们提出了统一权值的注意力模块。

为更好的实现注意力，我们需要评估每个神经元的重要性。在神经科学中，信息丰富的神经元通常表现出与周围神经元不同的放电模式。而且，激活神经元通常会抑制周围神经元，即空域抑制。换句话说，具有空域抑制效应的神经元应当赋予更高的重要性。最简单的寻找重要神经元的方法：度量神经元之间的线性可分性。因此，我们定义了如下能量函数：

其中，。最小化上述公式等价于训练同一通道内神经元t与其他神经元之间的线性可分性。为简单起见，我们采用二值标签，并添加正则项，最终的能量函数定义如下：

理论上，每个通道有个能量函数。幸运的是，上述公式具有如下解析解：

其中，。因此，最小能量可以通过如下公式得到：

上述公式意味着：能量越低，神经元t与周围神经元的区别越大，重要性越高。因此，神经元的重要性可以通过得到。

到目前为止，我们推导了能量函数并挖掘了神经元的重要性。按照注意力机制的定义，我们需要对特征进行增强处理：

下图给出了SimAM的pytorch风格实现code。

Experiments

我把最后一层改为320，模型6.21m，荣耀9 上64*64 25ms。

测试代码：

import functoolsimport torch
from torch import nn
from torch import Tensor
# from .utils import load_state_dict_from_url
from typing import Callable, Any, Optional, Listimport torch
import torch.nn as nnclass Simam_module(torch.nn.Module):def __init__(self, channels = None, e_lambda = 1e-4):super(Simam_module, self).__init__()self.activaton = nn.Sigmoid()self.e_lambda = e_lambdadef __repr__(self):s = self.__class__.__name__ + '('s += ('lambda=%f)' % self.e_lambda)return s@staticmethoddef get_module_name():return "simam"def forward(self, x):b, c, h, w = x.size()n = w * h - 1x_minus_mu_square = (x - x.mean(dim=[2,3], keepdim=True)).pow(2)y = x_minus_mu_square / (4 * (x_minus_mu_square.sum(dim=[2,3], keepdim=True) / n + self.e_lambda)) + 0.5return x * self.activaton(y)def _make_divisible(v: float, divisor: int, min_value: Optional[int] = None) -> int:"""This function is taken from the original tf repo.It ensures that all layers have a channel number that is divisible by 8It can be seen here:https://github.com/tensorflow/models/blob/master/research/slim/nets/mobilenet/mobilenet.py:param v::param divisor::param min_value::return:"""if min_value is None:min_value = divisornew_v = max(min_value, int(v + divisor / 2) // divisor * divisor)# Make sure that round down does not go down by more than 10%.if new_v < 0.9 * v:new_v += divisorreturn new_vclass ConvBNActivation(nn.Sequential):def __init__(self, in_planes: int, out_planes: int, kernel_size: int = 3, stride: int = 1, groups: int = 1,norm_layer: Optional[Callable[..., nn.Module]] = None,activation_layer: Optional[Callable[..., nn.Module]] = None,attention_module: Optional[Callable[..., nn.Module]] = None, ) -> None:padding = (kernel_size - 1) // 2if norm_layer is None:norm_layer = nn.BatchNorm2dif activation_layer is None:activation_layer = nn.ReLU6if attention_module is not None:if type(attention_module) == functools.partial:module_name = attention_module.func.get_module_name()else:module_name = attention_module.get_module_name()if module_name == "simam":super(ConvBNReLU, self).__init__(nn.Conv2d(in_planes, out_planes, kernel_size, stride, padding, groups=groups, bias=False),Simam_module(e_lambda=0.1), norm_layer(out_planes), activation_layer(inplace=True))else:super(ConvBNReLU, self).__init__(nn.Conv2d(in_planes, out_planes, kernel_size, stride, padding, groups=groups, bias=False),norm_layer(out_planes), activation_layer(inplace=True))else:super(ConvBNReLU, self).__init__(nn.Conv2d(in_planes, out_planes, kernel_size, stride, padding, groups=groups, bias=False),norm_layer(out_planes), activation_layer(inplace=True))# necessary for backwards compatibility
ConvBNReLU = ConvBNActivationclass InvertedResidual(nn.Module):def __init__(self, inp: int, oup: int, stride: int, expand_ratio: int,norm_layer: Optional[Callable[..., nn.Module]] = None,attention_module: Optional[Callable[..., nn.Module]] = None) -> None:super(InvertedResidual, self).__init__()self.stride = strideassert stride in [1, 2]if norm_layer is None:norm_layer = nn.BatchNorm2dhidden_dim = int(round(inp * expand_ratio))self.use_res_connect = self.stride == 1 and inp == ouplayers: List[nn.Module] = []if expand_ratio != 1:# pwlayers.append(ConvBNReLU(inp, hidden_dim, kernel_size=1, norm_layer=norm_layer))layers.extend([# dwConvBNReLU(hidden_dim, hidden_dim, stride=stride, groups=hidden_dim, norm_layer=norm_layer,attention_module=attention_module), # pw-linearnn.Conv2d(hidden_dim, oup, 1, 1, 0, bias=False), norm_layer(oup), ])if attention_module is not None:if type(attention_module) == functools.partial:module_name = attention_module.func.get_module_name()else:module_name = attention_module.get_module_name()if module_name != "simam":# print(attention_module)layers.append(attention_module(oup))self.conv = nn.Sequential(*layers)def forward(self, x: Tensor) -> Tensor:if self.use_res_connect:return x + self.conv(x)else:return self.conv(x)class MobileNetV2(nn.Module):def __init__(self, num_classes: int = 1000, width_mult: float = 1.0,inverted_residual_setting: Optional[List[List[int]]] = None, round_nearest: int = 8,attention_module: Optional[Callable[..., nn.Module]] = None) -> None:super(MobileNetV2, self).__init__()block = InvertedResidualnorm_layer = nn.BatchNorm2dinput_channel = 32last_channel = 320if inverted_residual_setting is None:inverted_residual_setting = [# t, c, n, s[1, 16, 1, 1], [6, 32, 3, 2], [6, 64, 4, 2], [6, 96, 3, 1], [6, 160, 2, 2],[6, 320, 1, 1], ]# only check the first element, assuming user knows t,c,n,s are requiredif len(inverted_residual_setting) == 0 or len(inverted_residual_setting[0]) != 4:raise ValueError("inverted_residual_setting should be non-empty ""or a 4-element list, got {}".format(inverted_residual_setting))# building first layerinput_channel = _make_divisible(input_channel * width_mult, round_nearest)self.last_channel = _make_divisible(last_channel * max(1.0, width_mult), round_nearest)features: List[nn.Module] = [ConvBNReLU(3, input_channel, stride=2, norm_layer=norm_layer)]# building inverted residual blocksfor t, c, n, s in inverted_residual_setting:output_channel = _make_divisible(c * width_mult, round_nearest)for i in range(n):stride = s if i == 0 else 1features.append(block(input_channel, output_channel, stride, expand_ratio=t, norm_layer=norm_layer,attention_module=attention_module))input_channel = output_channel# building last several layersfeatures.append(ConvBNReLU(input_channel, self.last_channel, kernel_size=1, norm_layer=norm_layer))# make it nn.Sequentialself.features = nn.Sequential(*features)# building classifierself.classifier = nn.Sequential(# nn.Dropout(0.2),nn.Linear(self.last_channel, num_classes), )# weight initializationfor m in self.modules():if isinstance(m, nn.Conv2d):nn.init.kaiming_normal_(m.weight, mode='fan_out')if m.bias is not None:nn.init.zeros_(m.bias)elif isinstance(m, (nn.BatchNorm2d, nn.GroupNorm)):nn.init.ones_(m.weight)nn.init.zeros_(m.bias)elif isinstance(m, nn.Linear):nn.init.normal_(m.weight, 0, 0.01)if m.bias is not None:nn.init.zeros_(m.bias)def _forward_impl(self, x: Tensor) -> Tensor:# This exists since TorchScript doesn't support inheritance, so the superclass method# (this one) needs to have a name other than `forward` that can be accessed in a subclassx = self.features(x)# Cannot use "squeeze" as batch-size can be 1 => must use reshape with x.shape[0]x = nn.functional.adaptive_avg_pool2d(x, (1, 1)).reshape(x.shape[0], -1)x = self.classifier(x)return xdef forward(self, x: Tensor) -> Tensor:return self._forward_impl(x)if __name__ == '__main__':kwargs = {}kwargs["num_classes"] = 6kwargs["attention_module"] = Simam_module(e_lambda=0.1)model = MobileNetV2(**kwargs)size = 64# model.cuda()model.eval()model_path = "dicenet.pth"torch.save(model.state_dict(), model_path)import osimport timefsize = os.path.getsize(model_path)fsize = fsize / float(1024 * 1024)print(f"model size {round(fsize, 2)} m")input = torch.rand(2, 3, size, size)#.cuda()for i in range(15):t1 = time.time()loc = model(input)cnt = time.time() - t1print(cnt, loc.size())