SqueezeNet网络模型详解

前言：

SqueezeNet模型原理

Fire Module

SqueezeNet模型结构

SqueezeNet模型特点

SqueezeNet网络实现

前言：

CNN网络模型一直在追求识别成功率，从AlexNet到VGGNet模型，识别精度不断提高，TOP-5错误率从15.3%下降到7.3%，但参数数量也越来越多，但参数数量也越来越多，从60M增加到140M。过多的参数降低了分布式训练的效率，也给数据传输所需的网络带宽造成很大的负担。如何在保证识别精度的情况下，对网络参数进行压缩是需要进一步研究的方向。

本篇介绍的SqueezeNet模型（压缩模型）就是为了解决这一问题而提出的网路模型。本篇先阐述SqueezeNet模型的基本理论，然后解读Caffe的实现，最后介绍Caffe环境下的训练方法。

SqueezeNet模型原理

SqueezeNet设计目标不是为了得到最佳的CNN识别精度，而是希望在简化网络复杂度的同时保证网络模型的识别精度。

SqeezeNet模型的设计用以下三个方法简化网络复杂度：

（1）替换3x3的卷积核为1x1的卷积核。从AlexNet模型发展到现在，因为设计上的简洁和有效性，卷积核的大小都选择3x3.SqueezeNet模型用1x1的卷积核替换3x3的卷积核可以让网络参数缩小9倍。但是为了不影响识别精度，只做了部分替换。

（2）减少输入3x3卷积的输入特征数量。将卷积层分解为squeeze层以及expand层，并封装为一个Fire Module。

（3）在整个网络后期进行下采样，使得卷积层有较大的activation maps。

Fire Module

Fire Module是SqueezeNet的核心构件，其思想非常简单。即将一个卷积层分解为一个Squeeze层和一个expand层，并各自带上Relu激活曾。squeeze层包含全部都是1x1的卷积核，共有S个。expand层包含1x1核3x3的卷积核，其中1x1的卷积核有E1个，3x3的卷积核有E3个，要求满足S<(E1+E3)。如下图FireModule结构：

SqueezeNet模型结构

SqueezeNet的网络模型结构如图：

SqueezeNet模型共有九层Fire Module，中间穿插了三个MAX pooling层，最后一层用Average Pooling层替换全连接层是的参数大量减少。SqueezeNet网络模型在最上层核下层各保留了一个卷积层，这样做的目的是保证输入输出的大小可掌握。其他参数细节设置在下面caffe实现中会详细介绍。下表给出了每层的维度：

SqueezeNet模型特点

（1）SqueezeNet比AlexNet的参数减少的50倍，模型大小只有4.8M，在性能好的FPGA上可以运行起来，并且能带来与AlexNet相当的识别精度。

（2）SqueezeNet证明了小的神经网络也能达到很好的识别精度，这使得未来将嵌入式设备或移动设备植入神经网络成为一种可能。

SqueezeNet网络实现

# please cite:
# @article{SqueezeNet,
#     Author = {Forrest N. Iandola and Matthew W. Moskewicz and Khalid Ashraf and Song Han and William J. Dally and Kurt Keutzer},
#     Title = {SqueezeNet: AlexNet-level accuracy with 50x fewer parameters and $<$1MB model size},
#     Journal = {arXiv:1602.07360},
#     Year = {2016}
# }
layer {name: "data"       //data层type: "Data"top: "data"top: "label"include {phase: TRAIN}transform_param {crop_size: 227mean_value: 104mean_value: 117mean_value: 123}data_param {source: "examples/imagenet/ilsvrc12_train_lmdb" //训练数据集batch_size: 32backend: LMDB}
}
layer {name: "data"type: "Data"top: "data"top: "label"include {phase: TEST}transform_param {crop_size: 227mean_value: 104mean_value: 117mean_value: 123}data_param {source: "examples/imagenet/ilsvrc12_val_lmdb"batch_size: 25 #not *iter_sizebackend: LMDB}
}
layer {                 //第一个卷积层，缩小输入图像，提取96维特征name: "conv1"type: "Convolution"bottom: "data"top: "conv1"convolution_param {num_output: 64kernel_size: 3stride: 2weight_filler {type: "xavier"}}
}
layer {                  //RELU层name: "relu_conv1"type: "ReLU"bottom: "conv1"top: "conv1"
}
layer {name: "pool1"          //第一个Max Pooling层，降采样，缩小一半type: "Pooling"bottom: "conv1"top: "pool1"pooling_param {pool: MAXkernel_size: 3stride: 2}
}
layer {                  //第一个fire模块，模块内先用squeeze层减少通道数，再用expand层增加通道数name: "fire2/squeeze1x1"type: "Convolution"bottom: "pool1"top: "fire2/squeeze1x1"convolution_param {num_output: 16kernel_size: 1weight_filler {type: "xavier"}}
}
layer {name: "fire2/relu_squeeze1x1"type: "ReLU"bottom: "fire2/squeeze1x1"top: "fire2/squeeze1x1"
}
layer {name: "fire2/expand1x1"type: "Convolution"bottom: "fire2/squeeze1x1"top: "fire2/expand1x1"convolution_param {num_output: 64kernel_size: 1weight_filler {type: "xavier"}}
}
layer {name: "fire2/relu_expand1x1"type: "ReLU"bottom: "fire2/expand1x1"top: "fire2/expand1x1"
}
layer {name: "fire2/expand3x3"type: "Convolution"bottom: "fire2/squeeze1x1"top: "fire2/expand3x3"convolution_param {num_output: 64pad: 1              //增加一个像素边界，是的1x1和3x3filter对齐kernel_size: 3weight_filler {type: "xavier"}}
}
layer {name: "fire2/relu_expand3x3"type: "ReLU"bottom: "fire2/expand3x3"top: "fire2/expand3x3"
}
layer {name: "fire2/concat"type: "Concat"bottom: "fire2/expand1x1"bottom: "fire2/expand3x3"top: "fire2/concat"
}
layer {name: "fire3/squeeze1x1"type: "Convolution"bottom: "fire2/concat"top: "fire3/squeeze1x1"convolution_param {num_output: 16kernel_size: 1weight_filler {type: "xavier"}}
}
layer {name: "fire3/relu_squeeze1x1"type: "ReLU"bottom: "fire3/squeeze1x1"top: "fire3/squeeze1x1"
}
layer {name: "fire3/expand1x1"type: "Convolution"bottom: "fire3/squeeze1x1"top: "fire3/expand1x1"convolution_param {num_output: 64kernel_size: 1weight_filler {type: "xavier"}}
}
layer {name: "fire3/relu_expand1x1"type: "ReLU"bottom: "fire3/expand1x1"top: "fire3/expand1x1"
}
layer {name: "fire3/expand3x3"type: "Convolution"bottom: "fire3/squeeze1x1"top: "fire3/expand3x3"convolution_param {num_output: 64pad: 1kernel_size: 3weight_filler {type: "xavier"}}
}
layer {name: "fire3/relu_expand3x3"type: "ReLU"bottom: "fire3/expand3x3"top: "fire3/expand3x3"
}
layer {name: "fire3/concat"type: "Concat"bottom: "fire3/expand1x1"bottom: "fire3/expand3x3"top: "fire3/concat"
}
layer {name: "pool3"type: "Pooling"bottom: "fire3/concat"top: "pool3"pooling_param {pool: MAXkernel_size: 3stride: 2}
}
layer {name: "fire4/squeeze1x1"type: "Convolution"bottom: "pool3"top: "fire4/squeeze1x1"convolution_param {num_output: 32kernel_size: 1weight_filler {type: "xavier"}}
}
layer {name: "fire4/relu_squeeze1x1"type: "ReLU"bottom: "fire4/squeeze1x1"top: "fire4/squeeze1x1"
}
layer {name: "fire4/expand1x1"type: "Convolution"bottom: "fire4/squeeze1x1"top: "fire4/expand1x1"convolution_param {num_output: 128kernel_size: 1weight_filler {type: "xavier"}}
}
layer {name: "fire4/relu_expand1x1"type: "ReLU"bottom: "fire4/expand1x1"top: "fire4/expand1x1"
}
layer {name: "fire4/expand3x3"type: "Convolution"bottom: "fire4/squeeze1x1"top: "fire4/expand3x3"convolution_param {num_output: 128pad: 1kernel_size: 3weight_filler {type: "xavier"}}
}
layer {name: "fire4/relu_expand3x3"type: "ReLU"bottom: "fire4/expand3x3"top: "fire4/expand3x3"
}
layer {name: "fire4/concat"type: "Concat"bottom: "fire4/expand1x1"bottom: "fire4/expand3x3"top: "fire4/concat"
}
layer {name: "fire5/squeeze1x1"type: "Convolution"bottom: "fire4/concat"top: "fire5/squeeze1x1"convolution_param {num_output: 32kernel_size: 1weight_filler {type: "xavier"}}
}
layer {name: "fire5/relu_squeeze1x1"type: "ReLU"bottom: "fire5/squeeze1x1"top: "fire5/squeeze1x1"
}
layer {name: "fire5/expand1x1"type: "Convolution"bottom: "fire5/squeeze1x1"top: "fire5/expand1x1"convolution_param {num_output: 128kernel_size: 1weight_filler {type: "xavier"}}
}
layer {name: "fire5/relu_expand1x1"type: "ReLU"bottom: "fire5/expand1x1"top: "fire5/expand1x1"
}
layer {name: "fire5/expand3x3"type: "Convolution"bottom: "fire5/squeeze1x1"top: "fire5/expand3x3"convolution_param {num_output: 128pad: 1kernel_size: 3weight_filler {type: "xavier"}}
}
layer {name: "fire5/relu_expand3x3"type: "ReLU"bottom: "fire5/expand3x3"top: "fire5/expand3x3"
}
layer {name: "fire5/concat"type: "Concat"bottom: "fire5/expand1x1"bottom: "fire5/expand3x3"top: "fire5/concat"
}
layer {name: "pool5"type: "Pooling"bottom: "fire5/concat"top: "pool5"pooling_param {pool: MAXkernel_size: 3stride: 2}
}
layer {name: "fire6/squeeze1x1"type: "Convolution"bottom: "pool5"top: "fire6/squeeze1x1"convolution_param {num_output: 48kernel_size: 1weight_filler {type: "xavier"}}
}
layer {name: "fire6/relu_squeeze1x1"type: "ReLU"bottom: "fire6/squeeze1x1"top: "fire6/squeeze1x1"
}
layer {name: "fire6/expand1x1"type: "Convolution"bottom: "fire6/squeeze1x1"top: "fire6/expand1x1"convolution_param {num_output: 192kernel_size: 1weight_filler {type: "xavier"}}
}
layer {name: "fire6/relu_expand1x1"type: "ReLU"bottom: "fire6/expand1x1"top: "fire6/expand1x1"
}
layer {name: "fire6/expand3x3"type: "Convolution"bottom: "fire6/squeeze1x1"top: "fire6/expand3x3"convolution_param {num_output: 192pad: 1kernel_size: 3weight_filler {type: "xavier"}}
}
layer {name: "fire6/relu_expand3x3"type: "ReLU"bottom: "fire6/expand3x3"top: "fire6/expand3x3"
}
layer {name: "fire6/concat"type: "Concat"bottom: "fire6/expand1x1"bottom: "fire6/expand3x3"top: "fire6/concat"
}
layer {name: "fire7/squeeze1x1"type: "Convolution"bottom: "fire6/concat"top: "fire7/squeeze1x1"convolution_param {num_output: 48kernel_size: 1weight_filler {type: "xavier"}}
}
layer {name: "fire7/relu_squeeze1x1"type: "ReLU"bottom: "fire7/squeeze1x1"top: "fire7/squeeze1x1"
}
layer {name: "fire7/expand1x1"type: "Convolution"bottom: "fire7/squeeze1x1"top: "fire7/expand1x1"convolution_param {num_output: 192kernel_size: 1weight_filler {type: "xavier"}}
}
layer {name: "fire7/relu_expand1x1"type: "ReLU"bottom: "fire7/expand1x1"top: "fire7/expand1x1"
}
layer {name: "fire7/expand3x3"type: "Convolution"bottom: "fire7/squeeze1x1"top: "fire7/expand3x3"convolution_param {num_output: 192pad: 1kernel_size: 3weight_filler {type: "xavier"}}
}
layer {name: "fire7/relu_expand3x3"type: "ReLU"bottom: "fire7/expand3x3"top: "fire7/expand3x3"
}
layer {name: "fire7/concat"type: "Concat"bottom: "fire7/expand1x1"bottom: "fire7/expand3x3"top: "fire7/concat"
}
layer {name: "fire8/squeeze1x1"type: "Convolution"bottom: "fire7/concat"top: "fire8/squeeze1x1"convolution_param {num_output: 64kernel_size: 1weight_filler {type: "xavier"}}
}
layer {name: "fire8/relu_squeeze1x1"type: "ReLU"bottom: "fire8/squeeze1x1"top: "fire8/squeeze1x1"
}
layer {name: "fire8/expand1x1"type: "Convolution"bottom: "fire8/squeeze1x1"top: "fire8/expand1x1"convolution_param {num_output: 256kernel_size: 1weight_filler {type: "xavier"}}
}
layer {name: "fire8/relu_expand1x1"type: "ReLU"bottom: "fire8/expand1x1"top: "fire8/expand1x1"
}
layer {name: "fire8/expand3x3"type: "Convolution"bottom: "fire8/squeeze1x1"top: "fire8/expand3x3"convolution_param {num_output: 256pad: 1kernel_size: 3weight_filler {type: "xavier"}}
}
layer {name: "fire8/relu_expand3x3"type: "ReLU"bottom: "fire8/expand3x3"top: "fire8/expand3x3"
}
layer {name: "fire8/concat"type: "Concat"bottom: "fire8/expand1x1"bottom: "fire8/expand3x3"top: "fire8/concat"
}
layer {name: "fire9/squeeze1x1"type: "Convolution"bottom: "fire8/concat"top: "fire9/squeeze1x1"convolution_param {num_output: 64kernel_size: 1weight_filler {type: "xavier"}}
}
layer {name: "fire9/relu_squeeze1x1"type: "ReLU"bottom: "fire9/squeeze1x1"top: "fire9/squeeze1x1"
}
layer {name: "fire9/expand1x1"type: "Convolution"bottom: "fire9/squeeze1x1"top: "fire9/expand1x1"convolution_param {num_output: 256kernel_size: 1weight_filler {type: "xavier"}}
}
layer {name: "fire9/relu_expand1x1"type: "ReLU"bottom: "fire9/expand1x1"top: "fire9/expand1x1"
}
layer {name: "fire9/expand3x3"type: "Convolution"bottom: "fire9/squeeze1x1"top: "fire9/expand3x3"convolution_param {num_output: 256pad: 1kernel_size: 3weight_filler {type: "xavier"}}
}
layer {name: "fire9/relu_expand3x3"type: "ReLU"bottom: "fire9/expand3x3"top: "fire9/expand3x3"
}
layer {name: "fire9/concat"type: "Concat"bottom: "fire9/expand1x1"bottom: "fire9/expand3x3"top: "fire9/concat"
}
layer {name: "drop9"           //最后一个fire模块后，增加一个Dropout层type: "Dropout"bottom: "fire9/concat"top: "fire9/concat"dropout_param {dropout_ratio: 0.5   //丢弃率为50%}
}
layer {                  //第二个卷积层，为图的每个像素预测1000个分类name: "conv10"type: "Convolution"bottom: "fire9/concat"top: "conv10"convolution_param {num_output: 1000kernel_size: 1weight_filler {type: "gaussian"mean: 0.0std: 0.01}}
}
layer {name: "relu_conv10"type: "ReLU"bottom: "conv10"top: "conv10"
}
layer {                //average pooling层得到1000类name: "pool10"type: "Pooling"bottom: "conv10"top: "pool10"pooling_param {pool: AVEglobal_pooling: true}
}
layer {               //softmax层，使用softmax函数归一化为概率name: "loss"type: "SoftmaxWithLoss"bottom: "pool10"bottom: "label"top: "loss"#include {#  phase: TRAIN#}
}
layer {name: "accuracy"type: "Accuracy"bottom: "pool10"bottom: "label"top: "accuracy"#include {#  phase: TEST#}
}
layer {name: "accuracy_top5"type: "Accuracy"bottom: "pool10"bottom: "label"top: "accuracy_top5"#include {#  phase: TEST#}accuracy_param {top_k: 5}
}