二值网络训练--A Empirical Study of Binary Neural Networks' Optimisation

A Empirical Study of Binary Neural Networks’ Optimisation
ICLR2019
https://github.com/mi-lad/studying-binary-neural-networks

本文得到的几个结论如下：

ADAM for optimising the objective, (2) not using early stopping, (3) splitting the training into two stages, (4) removing gradient and weight clipping in the first stage and (5) reducing the averaging rate in Batch Normalisation layers in the second stage

在二值网络训练的时候有的用到了下面两个裁剪：
Gradient clipping 梯度裁剪梯度超过一定范围就丢弃
Weight clipping 权重裁剪让权重值保持在一定范围

forward path (and at the end of the training):

STE with gradient clipping provides an estimate for the gradient of this operation:

上图（a）中的二值卷积核实怎么得到的？二值卷积核是通过对 full-precision proxy 进行二值化（sign函数）得到，对应右图前向。那么这个 full-precision proxy 又是怎么来的了？通过 STE estimator 学习得到的，对应右图反向

3.1 Impact of Optimiser 优化器的影响

A possible hypothesis is that early stages of training binary models require more averaging for the optimiser to proceed in presence of binarisaton operation. On the other hand, in the late stages of the training, we rely on noisier sources to increase exploration power of the optimiser.

总体上来说 ADAM 更有优势

3.2 Impact of gradient and weight clipping
梯度裁剪和权重裁剪对于二值网络的精度影响不是很大，对于训练网络收敛速度有一定影响

the well-known observation that training a binary model is often notably slower than its non-binary counter-part

The slow down is mainly caused by the commonly applied gradient and weight clipping, as they keep parameters within the
{-1,1} range at all times during training

weight and gradient clipping help achieve better accuracy

We tested this hypothesis by training a binary model in two stages: (1) using vanilla STE in the first stage with higher learning rates and (2) turning clippings back on when the accuracy stops improving by reducing learning rate.

二值网络训练--A Empirical Study of Binary Neural Networks' Optimisation相关推荐

二值网络训练--Training Competitive Binary Neural Networks from Scratch
Training Competitive Binary Neural Networks from Scratch https://github.com/hpi-xnor/BMXNet-v2 MXNet ...
深度学习【6】二值网络（Binarized Neural Networks）学习与理解
http://blog.csdn.net/linmingan/article/details/51008830 Binarized Neural Networks: Training Neural N ...
二值网络--Structured Binary Neural Networks for Accurate Image Classification and Semantic Segmentation
Structured Binary Neural Networks for Accurate Image Classification and Semantic Segmentation https: ...
二值网络--Bi-Real Net: Enhancing the Performance of 1-bit CNNs
Bi-Real Net: Enhancing the Performance of 1-bit CNNs With Improved Representational Capability and A ...
二值网络--Binarized Neural Networks
Binarized Neural Networks NIPS2016 https://github.com/MatthieuCourbariaux/BinaryNet 本文针对二值网络提出了训练方法 ...
二值网络--XNOR-Net: ImageNet Classification Using Binary Convolutional Neural Networks
XNOR-Net: ImageNet Classification Using Binary Convolutional Neural Networks ECCV2016 http://allenai ...
14.7倍推理加速、18.9倍存储节省！北航、商汤、UCSD提出首个点云二值网络 | ICLR 2021...
允中编辑整理量子位报道 | 公众号 QbitAI 编者按: 无论是在自动驾驶场景中,还是在手持移动设备上,基于点云的深度学习模型应用越来越广泛. 但这些离线边缘场景自身的限制,给模型的推理.存储 ...
二值网络——开启小而快神经网络时代
摘要:这种使用浮点计算的神经网络要求的大存储空间和大计算量,严重阻碍了其在手机.手表和移动机器人等设备上的应用.二值神经网络设法让计算主要在正1或负1间进行,几十倍地降低了网络大小和计算量,但一直以来 ...
二值网络--Training Binary Weight Networks via Semi-Binary Decomposition
Training Binary Weight Networks via Semi-Binary Decomposition ECCV2018 CNN模型的压缩或加速总体上分为三类: pruning-b ...

二值网络训练--A Empirical Study of Binary Neural Networks' Optimisation

二值网络训练--A Empirical Study of Binary Neural Networks' Optimisation相关推荐

最新文章

热门文章