What is the theory behind the concatenation vs. summation of 2 tensors in deep learning? How does this empirically relate to information passed?

2 Answers

Farshid Rayhan, studied at University of Manchester

Updated Jul 16

The summation of tensors are the signature of ResNet and concatenations of tensors were first used in Inception nets and later in DenseNet.

The identity matrix summation speeds up the training process and improves gradient flow since the skip connections are taken from previous conv operations. Thus the backpropagation can effectively transfer error corrections to earlier layers much easily. This addresses the vanishing gradient problem.

The channel wise concatenation was used in inception net to concatenate the feature maps generated by different filter sizes so that the user wont have to choose an effective filter size. DenseNet concatenates feature maps of previous layers, similar to Inception nets, so that the next layer can choose to work on either the features maps generated by the immediate earlier layer or it can work on the feature maps of the conv operation before the immediate earlier layer.

DenseNet procedure makes the model much more thick as channels are being joined after each conv operation while that doesnt happen in ResNet as it just sums the tensor. DenseNet paper argued that the summation harms the gradient flow as it sums up the values.

Both of the process are very state of the art. I personally prefer the channel concatenation process because I believe the summing up tensor regardless pollutes the feature map of both, the immediate conv operation and the source of the skip connection. Since deep nets are by nature a very strong learner this is why this ResNet process works quite well. Plus I also believe in the strategy of DenseNet where the goal is to make a smaller network with can do equivalent of very deep net like ResNet.

concatenation 和 element-wise summation 该选哪个？相关推荐

关于element的select多选选择器，数据回显的问题
关于element的select多选,数据回显的问题在工作中遇到这样一个问题,新建表单时用element的select多选以后,在编辑的时候打开表单发现其他数据能正常显示,多选却无法正常回显.在网上 ...
vue element el-table表格勾选求和 el-table 勾选获取表格某几列数据求和
vue element el-table 表格勾选获取几列数据求和 el-table表格勾选求和先看案例如下图所示:el-table 勾选后求出支付金额数据,计算出总金额数据. 我们来一步步分析方便 ...
Element表格嵌入复选框以及单选框
1,element 表格嵌入CheckBox 效果图如下: 2,element结合checkBox实现单选效果如下: html代码: <template><div><p ...
element ui 表格全选与否
虽然element ui 的接口提供了全选与反选,可是当我有多条数据时, 选中了某一条,再点全选/反选按钮时,会出现异常情况.下面是解决办法, template部分: <template> ...
Element之checkBox复选框
一.基础用法 <el-checkbox v-model="checked">选项一</el-checkbox> 注:在<el-checkbox> ...
element怎么设置复选框属性_【JavaScript 教程】DOM——属性的操作
作者 | 阮一峰 HTML 元素包括标签名和若干个键值对,这个键值对就称为"属性"(attribute). <a id="test" href=" ...
element表格取消全选_ElementUi 表格取消全选框，用文字表示
Echarts ElementUi 表格取消全选框,用文字表示 1.先看看实现的图一. 添加添加复选框列二.给方法(这个方法主要就是给表格所需要的列添加类名.)具体可以看ElementUi的表格 ...
vue复选框组件自定义对勾_vue+element:树级复选框组件使用
handleCheck(checkedKeys,checkedNodes,halfCheckedKeys,halfCheckedNodes){var len = 0,num = 0;var _alen ...
element表格默认勾选
使用表格的时候,偶尔会需要回勾上一次勾选的数据(默认勾选): 主要是用到了默认提供的方法:toggleRowSelection(row,boolean) this.allData.forEach(it ...

concatenation 和 element-wise summation 该选哪个？

一直不知道 concatenation 和 element-wise summation 该选哪个？这个回答很不错。

[from] https://www.quora.com/What-is-the-theory-behind-the-concatenation-vs-summation-of-2-tensors-in-deep-learning-How-does-this-empirically-relate-to-information-passed

What is the theory behind the concatenation vs. summation of 2 tensors in deep learning? How does this empirically relate to information passed?

concatenation 和 element-wise summation 该选哪个？相关推荐

最新文章

热门文章