Paper之DL之BP：《Understanding the difficulty of training deep feedforward neural networks》

2024-06-04 19:57:13

Paper之DL之BP：《Understanding the difficulty of training deep feedforward neural networks》

目录

原文解读

文章内容以及划重点

结论

原文解读

原文：Understanding the difficulty of training deep feedforward neural networks

文章内容以及划重点

Sigmoid的四层局限

sigmoid函数的test loss和training loss要经过很多轮数一直为0.5，后再有到0.1的差强人意的变化。

We hypothesize that this behavior is due to the combinationof random initialization and the fact that an hidden unitoutput of 0 corresponds to a saturated sigmoid. Note that deep networks with sigmoids but initialized from unsupervisedpre-training (e.g. from RBMs) do not suffer fromthis saturation behavior.

tanh、softsign的五层局限

换为tanh函数，就会很好很快的收敛

结论

1、The normalization factor may therefore be important when initializing deep networks because of the multiplicative effect through layers, and we suggest the following initialization procedure to approximately satisfy our objectives of maintaining activation variances and back-propagated gradients variance as one moves up or down the network. We call it the normalized initialization

2、结果可知分布更加均匀

Activation values normalized histograms with hyperbolic tangent activation, with standard (top) vs normalized initialization (bottom). Top: 0-peak increases for higher layers.
Several conclusions can be drawn from these error curves:
(1)、The more classical neural networks with sigmoid or hyperbolic tangent units and standard initialization fare rather poorly, converging more slowly and apparently towards ultimately poorer local minima.
(2)、The softsign networks seem to be more robust to the initialization procedure than the tanh networks, presumably because of their gentler non-linearity.
(3)、For tanh networks, the proposed normalized initialization can be quite helpful, presumably because the layer-to-layer transformations maintain magnitudes of activations (flowing upward) and gradients (flowing backward).
3、Sigmoid 5代表有5层，N代表正则化，可得出预训练会得到更小的误差

相关文章
Understanding the difficulty of training deep feedforward neural networks 本文作者为：Xavier Glorot与Yoshua Bengio。

Paper之DL之BP：《Understanding the difficulty of training deep feedforward neural networks》相关推荐

Paper：Xavier参数初始化之《Understanding the difficulty of training deep feedforward neural networks》的翻译与解读
Paper:Xavier参数初始化之<Understanding the difficulty of training deep feedforward neural networks>的 ...
经典DL论文研读(part1)--Understanding the difficulty of training deep feedforward neural networks
学习笔记,仅供参考,有错必纠文章目录 Understanding the difficulty of training deep feedforward neural networks Abstra ...
【Deep Learning】笔记：Understanding the difficulty of training deep feedforward neural networks
这几天读了这篇论文,在这里将大致内容写在这里. Abstract 介绍这篇论文的主要内容就是尝试更好的理解为什么使用"标准随机初始化"来计算使用标准梯度下降的网络效果通常来讲都不是 ...
Understanding the difficulty of training deep feedforward neural networks
1.作者 Xavier Glorot Yoshua Bengio这就是非常出名的Xavier初始化的作者蒙特利尔大学加拿大 2.文章目的本篇文章的主题在于 1.理解为什么随机初始化在很深的神经 ...
论文解析-《Understanding the difficulty of training deep feedforward neural networks》
这篇论文详细解析了深度网络中参数xavier初始化方法,这里做一下读书笔记,同时记录一下自己的理解. 1 引言经典前馈神经网络其实很早就有了(Remelhart et al.,1986),近年来对深 ...
《Understanding the Effective Receptive Field in Deep Convolutional Neural Networks》理解深卷积神经网络中的有效感受野
前言论文推荐来自B站UP-启释科技,该up对感受野和卷积有深刻的理解推荐感受野视频讲解: 深度理解感受野(一) 深度理解感受野(二) 深度理解感受野(三) 深度理解感受野(四) 深度理解感受野(五 ...
有效感受野--Understanding the Effective Receptive Field in Deep Convolutional Neural Networks
Understanding the Effective Receptive Field in Deep Convolutional Neural Networks NIPS 2016 本文主要分析了 ...
论文阅读：Understanding the Effective Receptive Field in Deep Convolutional Neural Networks
文章目录论文总述 2.感受野的定义 3.理论感受野大小的计算 4.3种增加感受野的操作 5.感受野中心像素的值对输出feature map 的response影响比边缘的像素更大 6.如何知道fea ...
经典DL论文研读(part4)--ImageNet Classification with Deep Convolutional Neural Networks
学习笔记,仅供参考,有错必纠文章目录 ImageNet Classification with Deep Convolutional Neural Networks 摘要 Introduction ...

最新文章

热门文章