三值网络--Trained Ternary Quantization

Trained Ternary Quantization
ICLR 2017
https://github.com/TropComplique/trained-ternary-quantization pytorch
https://github.com/buaabai/Ternary-Weights-Network pytorch

传统的二值网络将权重 W 量化为 +1、-1；三值网络 TWN (Ternary weight networks) 将权重W 量化为 {−W_l ,0,+W_l }

阈值的计算公式如下所示

本文提出了新的三值网络

positive and negative weights，三个不同的值用于表示三值网络，这个正负权值是通过网络学习得到的
对应的梯度计算如下

本文的阈值选择采用：

set t to 0.05 in experiments on CIFAR-10 and ImageNet dataset

The quantization roughly proceeds as follows.

Train a model of your choice as usual (or take a trained model).
Copy all full precision weights that you want to quantize. Then do the initial quantization:
in the model replace them by ternary values {-1, 0, +1} using some heuristic.
Repeat until convergence:
1). Make the forward pass with the quantized model. 使用量化后的网络进行前向计算
2). Compute gradients for the quantized model. 对量化网络进行梯度计算
3). Preprocess the gradients and apply them to the copy of full precision weights. 使用梯度更新网络模型的权重
4). Requantize the model using the changed full precision weights. 对新的权重进行量化
Throw away the copy of full precision weights and use the quantized model.

三值网络--Trained Ternary Quantization相关推荐

权值简化（1）：三值神经网络（Ternary Weight Networks）
三值神经网络: 本文是对中科院两位科研工作者的文章的理解翻译: 就目前而言,深度神经网络的层数和训练时间变得越来越多.一方面,人们想要在处理器上面下功夫,但是处理器的处理速度也十分有限.造成目前一个比 ...
【知识星球】模型量化从1bit到8bit，二值到三值
欢迎大家来到<知识星球>专栏,这里是网络结构1000变小专题,模型量化是当前工业界的核心技术,我们最近在知识星球更新相关的内容. 作者&编辑 | 言有三 1 1bit量化-二值网络 ...
二值网络--Optimize Deep Convolutional Neural Network with Ternarized Weights and High Accuracy
Optimize Deep Convolutional Neural Network with Ternarized Weights and High Accuracy IEEE Winter Con ...
模型量化从1bit到8bit，二值到三值
转自:https://blog.csdn.net/u012101561/article/details/80868352 版权声明:本文为博主原创文章,遵循 CC 4.0 BY-SA 版权协议,转载请 ...
二值网络--Training Binary Weight Networks via Semi-Binary Decomposition
Training Binary Weight Networks via Semi-Binary Decomposition ECCV2018 CNN模型的压缩或加速总体上分为三类: pruning-b ...
二值网络--Structured Binary Neural Networks for Accurate Image Classification and Semantic Segmentation
Structured Binary Neural Networks for Accurate Image Classification and Semantic Segmentation https: ...
二值网络--Binarized Neural Networks
Binarized Neural Networks NIPS2016 https://github.com/MatthieuCourbariaux/BinaryNet 本文针对二值网络提出了训练方法 ...
二值网络--XNOR-Net: ImageNet Classification Using Binary Convolutional Neural Networks
XNOR-Net: ImageNet Classification Using Binary Convolutional Neural Networks ECCV2016 http://allenai ...
神经网络为什么可以实现分类？---三分类网络0，1，2与弹性振子力学系统
本文制作了一个三分类的网络来分类mnist数据集的0,1,2.并同时制作了一个力学模型,用来模拟这个三分类的过程,并用这个模型解释分类的原理. 上图可以用下列方程描述只要ωx0,ωx1,ωx2,ωx ...

三值网络--Trained Ternary Quantization

三值网络--Trained Ternary Quantization相关推荐

最新文章

热门文章