caffe中的layers
自从半年前开始进入418实验室研究深度学习,可以说caffe这一深度学习开源框架,一直伴随着我。经过了大半年的学习,虽然还有许多东西尚待完善,但起码对于caffe已经有了一个比较感性的认识。互联网上,关于深度学习和caffe的资料汗牛充栋,但质量往往参差不齐。我搜集了一些资料,加入自己的理解,总结在这里,也是对自己的一个督促吧。
本文主要总结了caffe框架中的各种layer。layer是神经网络搭建的脚手架,理解了layer,才能更好地理解NN(Neural Networks)。
下面这张图是我在网络上搜集的一张关于layer的思维导图,我觉得总结的挺好。有心人可以看完全文后,再回过头来看这张图,相信一定会有新的理解。
1. Outline
1.1. layer.hpp
<codeclass="hljs avrasm has-numbering" style="display: block; padding: 0px; color: inherit; box-sizing: border-box; font-family: 'Source Code Pro', monospace;font-size:undefined; white-space: pre; border-radius: 0px; word-wrap: normal; background: transparent;">layer<span class="hljs-preprocessor" style="color: rgb(68, 68, 68); box-sizing: border-box;">.hpp</span>common_layers<span class="hljs-preprocessor" style="color: rgb(68, 68, 68); box-sizing: border-box;">.hpp</span> data_layers<span class="hljs-preprocessor" style="color: rgb(68, 68, 68); box-sizing: border-box;">.hpp</span> loss_layers<span class="hljs-preprocessor" style="color: rgb(68, 68, 68); box-sizing: border-box;">.hpp</span> neuron_layers<span class="hljs-preprocessor" style="color: rgb(68, 68, 68); box-sizing: border-box;">.hpp</span> vision_layers<span class="hljs-preprocessor" style="color: rgb(68, 68, 68); box-sizing: border-box;">.hpp</span></code><ul class="pre-numbering" style="box-sizing: border-box; position: absolute;width: 50px; top: 0px; left: 0px; margin: 0px; padding: 6px 0px 40px; border-right-width: 1px; border-right-style: solid; border-right-color: rgb(221, 221, 221); list-style: none; text-align: right; background-color: rgb(238, 238, 238);"><li style="box-sizing: border-box; padding: 0px 5px;">1</li><li style="box-sizing: border-box; padding: 0px 5px;">2</li><li style="box-sizing: border-box; padding: 0px 5px;">3</li><li style="box-sizing: border-box; padding: 0px 5px;">4</li><li style="box-sizing: border-box; padding: 0px 5px;">5</li><li style="box-sizing: border-box; padding: 0px 5px;">6</li><li style="box-sizing: border-box; padding: 0px 5px;">7</li></ul><ul class="pre-numbering" style="box-sizing: border-box; position: absolute;width: 50px; top: 0px; left: 0px; margin: 0px; padding: 6px 0px 40px; border-right-width: 1px; border-right-style: solid; border-right-color: rgb(221, 221, 221); list-style: none; text-align: right; background-color: rgb(238, 238, 238);"><li style="box-sizing: border-box; padding: 0px 5px;">1</li><li style="box-sizing: border-box; padding: 0px 5px;">2</li><li style="box-sizing: border-box; padding: 0px 5px;">3</li><li style="box-sizing: border-box; padding: 0px 5px;">4</li><li style="box-sizing: border-box; padding: 0px 5px;">5</li><li style="box-sizing: border-box; padding: 0px 5px;">6</li><li style="box-sizing: border-box; padding: 0px 5px;">7</li></ul>
其中layer.hpp是抽象出来的基类,其他都是在其基础上的继承,也即剩下的五个头文件和上图中的五个部分。在layer.hpp头文件里,包含了这几个头文件:
<codeclass="hljs ruleslanguage has-numbering" style="display: block; padding: 0px; color: inherit; box-sizing: border-box; font-family: 'Source Code Pro', monospace;font-size:undefined; white-space: pre; border-radius: 0px; word-wrap: normal; background: transparent;"><span class="hljs-array" style="box-sizing: border-box;">#include </span><span class="hljs-string" style="color: rgb(0, 136, 0); box-sizing: border-box;">"caffe/blob.hpp"</span> <span class="hljs-array" style="box-sizing: border-box;">#include </span><span class="hljs-string" style="color: rgb(0, 136, 0); box-sizing: border-box;">"caffe/common.hpp"</span> <span class="hljs-array" style="box-sizing: border-box;">#include </span><span class="hljs-string" style="color: rgb(0, 136, 0); box-sizing: border-box;">"caffe/proto/caffe.pb.h"</span> <span class="hljs-array" style="box-sizing: border-box;">#include </span><span class="hljs-string" style="color: rgb(0, 136, 0); box-sizing: border-box;">"caffe/util/device_alternate.hpp"</span></code><ulclass="pre-numbering" style="box-sizing: border-box; position: absolute; width: 50px; top: 0px; left: 0px; margin: 0px; padding: 6px 0px 40px; border-right-width: 1px; border-right-style: solid; border-right-color: rgb(221, 221, 221); list-style: none; text-align: right; background-color: rgb(238, 238, 238);"><li style="box-sizing: border-box; padding: 0px 5px;">1</li><li style="box-sizing: border-box; padding: 0px 5px;">2</li><li style="box-sizing: border-box; padding: 0px 5px;">3</li><li style="box-sizing: border-box; padding: 0px 5px;">4</li></ul><ul class="pre-numbering" style="box-sizing: border-box; position: absolute; width: 50px; top: 0px; left: 0px; margin: 0px; padding: 6px 0px 40px; border-right-width: 1px; border-right-style: solid; border-right-color: rgb(221, 221, 221); list-style: none; text-align: right; background-color: rgb(238, 238, 238);"><li style="box-sizing: border-box; padding: 0px 5px;">1</li><li style="box-sizing: border-box; padding: 0px 5px;">2</li><li style="box-sizing: border-box; padding: 0px 5px;">3</li><li style="box-sizing: border-box; padding: 0px 5px;">4</li></ul>
在device_alternate.hpp中,通过#ifdef CPU_ONLY定义了一些宏来取消GPU的调用:
<codeclass="hljs cs has-numbering" style="display: block; padding: 0px; color: inherit; box-sizing: border-box; font-family: 'Source Code Pro', monospace;font-size:undefined; white-space: pre; border-radius: 0px; word-wrap: normal; background: transparent;"><span class="hljs-preprocessor" style="color: rgb(68, 68, 68); box-sizing: border-box;">#<span class="hljs-keyword" style="box-sizing: border-box;">define</span> STUB_GPU(classname)</span> <span class="hljs-preprocessor" style="color: rgb(68, 68, 68); box-sizing: border-box;">#<span class="hljs-keyword" style="box-sizing: border-box;">define</span> STUB_GPU_FORWARD(classname, funcname)</span> <span class="hljs-preprocessor" style="color: rgb(68, 68, 68); box-sizing: border-box;">#<span class="hljs-keyword" style="box-sizing: border-box;">define</span> STUB_GPU_BACKWARD(classname, funcname)</span></code><ul class="pre-numbering" style="box-sizing: border-box; position: absolute;width: 50px; top: 0px; left: 0px; margin: 0px; padding: 6px 0px 40px; border-right-width: 1px; border-right-style: solid; border-right-color: rgb(221, 221, 221); list-style: none; text-align: right; background-color: rgb(238, 238, 238);"><li style="box-sizing: border-box; padding: 0px 5px;">1</li><li style="box-sizing: border-box; padding: 0px 5px;">2</li><li style="box-sizing: border-box; padding: 0px 5px;">3</li></ul><ul class="pre-numbering" style="box-sizing: border-box; position: absolute;width: 50px; top: 0px; left: 0px; margin: 0px; padding: 6px 0px 40px; border-right-width: 1px; border-right-style: solid; border-right-color: rgb(221, 221, 221); list-style: none; text-align: right; background-color: rgb(238, 238, 238);"><li style="box-sizing: border-box; padding: 0px 5px;">1</li><li style="box-sizing: border-box; padding: 0px 5px;">2</li><li style="box-sizing: border-box; padding: 0px 5px;">3</li></ul>
<codeclass="hljs cpp has-numbering" style="display: block; padding: 0px; color: inherit; box-sizing: border-box; font-family: 'Source Code Pro', monospace;font-size:undefined; white-space: pre; border-radius: 0px; word-wrap: normal; background: transparent;">LayerParameter layer_param_; <span class="hljs-comment" style="color: rgb(136, 0, 0); box-sizing: border-box;">// 这个是protobuf文件中存储的layer参数</span> <span class="hljs-stl_container" style="box-sizing: border-box;"><span class="hljs-built_in" style="color: rgb(102, 0,102); box-sizing: border-box;">vector</span><share_ptr<Blob<Dtype></span>>>blobs_; <span class="hljs-comment" style="color: rgb(136, 0, 0); box-sizing: border-box;">// 这个存储的是layer的参数,在程序中用的</span> <span class="hljs-stl_container" style="box-sizing: border-box;"><span class="hljs-built_in" style="color: rgb(102, 0,102); box-sizing: border-box;">vector</span><<span class="hljs-keyword" style="color: rgb(0, 0, 136); box-sizing: border-box;">bool</span>></span> param_propagate_down_; <span class="hljs-comment" style="color: rgb(136, 0, 0); box-sizing: border-box;">// 这个bool表示是否计算各个blob参数的diff,即传播误差</span></code><ul class="pre-numbering" style="box-sizing: border-box; position: absolute;width: 50px; top: 0px; left: 0px; margin: 0px; padding: 6px 0px 40px; border-right-width: 1px; border-right-style: solid; border-right-color: rgb(221, 221, 221); list-style: none; text-align: right; background-color: rgb(238, 238, 238);"><li style="box-sizing: border-box; padding: 0px 5px;">1</li><li style="box-sizing: border-box; padding: 0px 5px;">2</li><li style="box-sizing: border-box; padding: 0px 5px;">3</li></ul><ul class="pre-numbering" style="box-sizing: border-box; position: absolute;width: 50px; top: 0px; left: 0px; margin: 0px; padding: 6px 0px 40px; border-right-width: 1px; border-right-style: solid; border-right-color: rgb(221, 221, 221); list-style: none; text-align: right; background-color: rgb(238, 238, 238);"><li style="box-sizing: border-box; padding: 0px 5px;">1</li><li style="box-sizing: border-box; padding: 0px 5px;">2</li><li style="box-sizing: border-box; padding: 0px 5px;">3</li></ul>
<codeclass="hljs cpp has-numbering" style="display: block; padding: 0px; color: inherit; box-sizing: border-box; font-family: 'Source Code Pro', monospace;font-size:undefined; white-space: pre; border-radius: 0px; word-wrap: normal; background: transparent;"><span class="hljs-keyword" style="color: rgb(0, 0, 136); box-sizing: border-box;">virtual</span> <span class="hljs-keyword" style="color: rgb(0, 0, 136); box-sizing: border-box;">void</span> SetUp(<span class="hljs-keyword" style="color: rgb(0, 0, 136); box-sizing: border-box;">const</span> <span class="hljs-stl_container" style="box-sizing: border-box;"><span class="hljs-built_in" style="color: rgb(102, 0, 102); box-sizing: border-box;">vector</span><Blob<Dtype></span>*>&bottom, <span class="hljs-stl_container" style="box-sizing: border-box;"><span class="hljs-built_in" style="color: rgb(102, 0,102); box-sizing: border-box;">vector</span><Blob<Dtype></span>*>*top) <span class="hljs-keyword" style="color: rgb(0, 0, 136); box-sizing: border-box;">inline</span> Dtype Forward(<span class="hljs-keyword" style="color: rgb(0, 0, 136); box-sizing: border-box;">const</span> <span class="hljs-stl_container" style="box-sizing: border-box;"><span class="hljs-built_in" style="color: rgb(102, 0, 102); box-sizing: border-box;">vector</span><Blob<Dtype></span>*>&bottom, <span class="hljs-stl_container" style="box-sizing: border-box;"><span class="hljs-built_in" style="color: rgb(102, 0,102); box-sizing: border-box;">vector</span><Blob<Dtype></span>*>*top); <span class="hljs-keyword" style="color: rgb(0, 0, 136); box-sizing: border-box;">inline</span> <span class="hljs-keyword" style="color: rgb(0, 0, 136); box-sizing: border-box;">void</span> Backward(<span class="hljs-keyword"style="color: rgb(0, 0, 136); box-sizing: border-box;">const</span> <span class="hljs-stl_container" style="box-sizing: border-box;"><span class="hljs-built_in" style="color: rgb(102, 0, 102); box-sizing: border-box;">vector</span><Blob<Dtype></span>*>&top, <span class="hljs-keyword" style="color: rgb(0, 0, 136); box-sizing: border-box;">const</span> <span class="hljs-stl_container" style="box-sizing: border-box;"><span class="hljs-built_in" style="color: rgb(102, 0, 102); box-sizing: border-box;">vector</span><<span class="hljs-keyword" style="color: rgb(0, 0, 136); box-sizing: border-box;">bool</span>></span>& propagate_down, <span class="hljs-keyword" style="color: rgb(0, 0, 136); box-sizing: border-box;">const</span> <Blob<Dtype>*>* bottom);</code><ul class="pre-numbering" style="box-sizing: border-box; position: absolute; width: 50px; top: 0px; left: 0px; margin: 0px; padding: 6px 0px 40px; border-right-width: 1px; border-right-style: solid; border-right-color: rgb(221, 221, 221); list-style: none; text-align: right; background-color: rgb(238, 238, 238);"><li style="box-sizing: border-box; padding: 0px 5px;">1</li><li style="box-sizing: border-box; padding: 0px 5px;">2</li><li style="box-sizing: border-box; padding: 0px 5px;">3</li></ul><ul class="pre-numbering" style="box-sizing: border-box; position: absolute; width: 50px; top: 0px; left: 0px; margin: 0px; padding: 6px 0px 40px; border-right-width: 1px; border-right-style: solid; border-right-color: rgb(221, 221, 221); list-style: none; text-align: right; background-color: rgb(238, 238, 238);"><li style="box-sizing: border-box; padding: 0px 5px;">1</li><li style="box-sizing: border-box; padding: 0px 5px;">2</li><li style="box-sizing: border-box; padding: 0px 5px;">3</li></ul>
1.2. data_layers.hpp
data_layers.hpp这个头文件包含了这几个头文件:
<codeclass="hljs ruleslanguage has-numbering" style="display: block; padding: 0px; color: inherit; box-sizing: border-box; font-family: 'Source Code Pro', monospace;font-size:undefined; white-space: pre; border-radius: 0px; word-wrap: normal; background: transparent;"><span class="hljs-array" style="box-sizing: border-box;">#include </span><span class="hljs-string" style="color: rgb(0, 136, 0); box-sizing: border-box;">"boost/scoped_ptr.hpp"</span> <span class="hljs-array" style="box-sizing: border-box;">#include </span><span class="hljs-string" style="color: rgb(0, 136, 0); box-sizing: border-box;">"hdf5.h"</span> <span class="hljs-array" style="box-sizing: border-box;">#include </span><span class="hljs-string" style="color: rgb(0, 136, 0); box-sizing: border-box;">"leveldb/db.h"</span> <span class="hljs-array" style="box-sizing: border-box;">#include </span><span class="hljs-string" style="color: rgb(0, 136, 0); box-sizing: border-box;">"lmdb.h"</span><span class="hljs-array" style="box-sizing: border-box;">#include </span><span class="hljs-string" style="color: rgb(0, 136, 0); box-sizing: border-box;">"caffe/blob.hpp"</span> <span class="hljs-array" style="box-sizing: border-box;">#include </span><span class="hljs-string" style="color: rgb(0, 136, 0); box-sizing: border-box;">"caffe/common.hpp"</span> <span class="hljs-array" style="box-sizing: border-box;">#include </span><span class="hljs-string" style="color: rgb(0, 136, 0); box-sizing: border-box;">"caffe/filler.hpp"</span> <span class="hljs-array" style="box-sizing: border-box;">#include </span><span class="hljs-string" style="color: rgb(0, 136, 0); box-sizing: border-box;">"caffe/internal_thread.hpp"</span> <span class="hljs-array" style="box-sizing: border-box;">#include </span><span class="hljs-string" style="color: rgb(0, 136, 0); box-sizing: border-box;">"caffe/layer.hpp"</span> <span class="hljs-array" style="box-sizing: border-box;">#include </span><span class="hljs-string" style="color: rgb(0, 136, 0); box-sizing: border-box;">"caffe/proto/caffe.pb.h"</span></code><ulclass="pre-numbering" style="box-sizing: border-box; position: absolute; width: 50px; top: 0px; left: 0px; margin: 0px; padding: 6px 0px 40px; border-right-width: 1px; border-right-style: solid; border-right-color: rgb(221, 221, 221); list-style: none; text-align: right; background-color: rgb(238, 238, 238);"><li style="box-sizing: border-box; padding: 0px 5px;">1</li><li style="box-sizing: border-box; padding: 0px 5px;">2</li><li style="box-sizing: border-box; padding: 0px 5px;">3</li><li style="box-sizing: border-box; padding: 0px 5px;">4</li><li style="box-sizing: border-box; padding: 0px 5px;">5</li><li style="box-sizing: border-box; padding: 0px 5px;">6</li><li style="box-sizing: border-box; padding: 0px 5px;">7</li><li style="box-sizing: border-box; padding: 0px 5px;">8</li><li style="box-sizing: border-box; padding: 0px 5px;">9</li><li style="box-sizing: border-box; padding: 0px 5px;">10</li><listyle="box-sizing: border-box; padding: 0px 5px;">11</li></ul><ul class="pre-numbering" style="box-sizing: border-box; position: absolute; width: 50px; top: 0px; left: 0px; margin: 0px; padding: 6px 0px 40px; border-right-width: 1px; border-right-style: solid; border-right-color: rgb(221, 221, 221); list-style: none; text-align: right; background-color: rgb(238, 238, 238);"><li style="box-sizing: border-box; padding: 0px 5px;">1</li><li style="box-sizing: border-box; padding: 0px 5px;">2</li><li style="box-sizing: border-box; padding: 0px 5px;">3</li><li style="box-sizing: border-box; padding: 0px 5px;">4</li><li style="box-sizing: border-box; padding: 0px 5px;">5</li><li style="box-sizing: border-box; padding: 0px 5px;">6</li><li style="box-sizing: border-box; padding: 0px 5px;">7</li><li style="box-sizing: border-box; padding: 0px 5px;">8</li><li style="box-sizing: border-box; padding: 0px 5px;">9</li><li style="box-sizing: border-box; padding: 0px 5px;">10</li><li style="box-sizing: border-box; padding: 0px5px;">11</li></ul>
LevelDB是Google公司搞的一个高性能的key/value存储库,调用简单,数据是被Snappy压缩,据说效率很多,可以减少磁盘I/O,具体例子可以看看维基百科。
HDF(Hierarchical Data Format)是一种为存储和处理大容量科学数据而设计的文件格式及相应的库文件,当前最流行的版本是HDF5,其文件包含两种基本数据对象:
群组(group):类似文件夹,可以包含多个数据集或下级群组;
数据集(dataset):数据内容,可以是多维数组,也可以是更复杂的数据类型。
以上内容来自维基百科,关于使用可以参考HDF5 小试——高大上的多对象文件格式。
caffe/filler.hpp的作用是在网络初始化时,根据layer的定义进行初始参数的填充,下面的代码很直观,根据FillerParameter指定的类型进行对应的参数填充。
<codeclass="hljs fsharp has-numbering" style="display: block; padding: 0px; color: inherit; box-sizing: border-box; font-family: 'Source Code Pro', monospace;font-size:undefined; white-space: pre; border-radius: 0px; word-wrap: normal; background: transparent;"><span class="hljs-comment" style="color: rgb(136, 0, 0); box-sizing: border-box;">// A function to get a specific filler from the specification given in</span> <span class="hljs-comment" style="color: rgb(136, 0, 0); box-sizing: border-box;">// FillerParameter. Ideally this would be replaced by a factory pattern,</span> <span class="hljs-comment" style="color: rgb(136, 0, 0); box-sizing: border-box;">// but we will leave it this way for now.</span>template <typename Dtype> Filler<Dtype>* GetFiller(const FillerParameter& param) {const std::string& <span class="hljs-class" style="box-sizing: border-box;"><span class="hljs-keyword" style="color: rgb(0, 0, 136); box-sizing: border-box;">type</span> =</span> param.<span class="hljs-keyword" style="color: rgb(0, 0, 136); box-sizing: border-box;">type</span>();<span class="hljs-keyword" style="color: rgb(0, 0, 136); box-sizing: border-box;">if</span> (<span class="hljs-class"style="box-sizing: border-box;"><span class="hljs-keyword" style="color: rgb(0, 0, 136); box-sizing: border-box;">type</span> =</span>= <span class="hljs-string" style="color: rgb(0, 136, 0); box-sizing: border-box;">"constant"</span>) {<span class="hljs-keyword" style="color: rgb(0, 0, 136); box-sizing: border-box;">return</span> <span class="hljs-keyword" style="color: rgb(0, 0, 136); box-sizing: border-box;">new</span> ConstantFiller<Dtype>(param);} <span class="hljs-keyword" style="color: rgb(0, 0, 136); box-sizing: border-box;">else</span> <span class="hljs-keyword" style="color: rgb(0, 0, 136); box-sizing: border-box;">if</span> (<span class="hljs-class" style="box-sizing: border-box;"><span class="hljs-keyword" style="color: rgb(0, 0, 136); box-sizing: border-box;">type</span> =</span>= <span class="hljs-string" style="color: rgb(0, 136, 0); box-sizing: border-box;">"gaussian"</span>) {<span class="hljs-keyword" style="color: rgb(0, 0, 136); box-sizing: border-box;">return</span> <span class="hljs-keyword" style="color: rgb(0, 0, 136); box-sizing: border-box;">new</span> GaussianFiller<Dtype>(param);} <span class="hljs-keyword" style="color: rgb(0, 0, 136); box-sizing: border-box;">else</span> <span class="hljs-keyword" style="color: rgb(0, 0, 136); box-sizing: border-box;">if</span> (<span class="hljs-class" style="box-sizing: border-box;"><span class="hljs-keyword" style="color: rgb(0, 0, 136); box-sizing: border-box;">type</span> =</span>= <span class="hljs-string" style="color: rgb(0, 136, 0); box-sizing: border-box;">"positive_unitball"</span>) {<span class="hljs-keyword" style="color: rgb(0, 0, 136); box-sizing: border-box;">return</span> <span class="hljs-keyword" style="color: rgb(0, 0, 136); box-sizing: border-box;">new</span> PositiveUnitballFiller<Dtype>(param);} <span class="hljs-keyword" style="color: rgb(0, 0, 136); box-sizing: border-box;">else</span> <span class="hljs-keyword" style="color: rgb(0, 0, 136); box-sizing: border-box;">if</span> (<span class="hljs-class" style="box-sizing: border-box;"><span class="hljs-keyword" style="color: rgb(0, 0, 136); box-sizing: border-box;">type</span> =</span>= <span class="hljs-string" style="color: rgb(0, 136, 0); box-sizing: border-box;">"uniform"</span>) {<span class="hljs-keyword" style="color: rgb(0, 0, 136); box-sizing: border-box;">return</span> <span class="hljs-keyword" style="color: rgb(0, 0, 136); box-sizing: border-box;">new</span> UniformFiller<Dtype>(param);} <span class="hljs-keyword" style="color: rgb(0, 0, 136); box-sizing: border-box;">else</span> <span class="hljs-keyword" style="color: rgb(0, 0, 136); box-sizing: border-box;">if</span> (<span class="hljs-class" style="box-sizing: border-box;"><span class="hljs-keyword" style="color: rgb(0, 0, 136); box-sizing: border-box;">type</span> =</span>= <span class="hljs-string" style="color: rgb(0, 136, 0); box-sizing: border-box;">"xavier"</span>) {<span class="hljs-keyword" style="color: rgb(0, 0, 136); box-sizing: border-box;">return</span> <span class="hljs-keyword" style="color: rgb(0, 0, 136); box-sizing: border-box;">new</span> XavierFiller<Dtype>(param);} <span class="hljs-keyword" style="color: rgb(0, 0, 136); box-sizing: border-box;">else</span> {CHECK(<span class="hljs-keyword" style="color: rgb(0, 0, 136); box-sizing: border-box;">false</span>) << <span class="hljs-string" style="color: rgb(0, 136, 0); box-sizing: border-box;">"Unknown filler name: "</span> << param.<span class="hljs-keyword" style="color: rgb(0, 0, 136); box-sizing: border-box;">type</span>();}<span class="hljs-keyword" style="color: rgb(0, 0, 136); box-sizing: border-box;">return</span> (Filler<Dtype>*)(NULL); }</code><ul class="pre-numbering" style="box-sizing: border-box; position: absolute; width: 50px; top: 0px; left: 0px; margin: 0px; padding: 6px 0px 40px; border-right-width: 1px; border-right-style: solid; border-right-color: rgb(221, 221, 221); list-style: none; text-align: right; background-color: rgb(238, 238, 238);"><li style="box-sizing: border-box; padding: 0px 5px;">1</li><li style="box-sizing: border-box; padding: 0px 5px;">2</li><li style="box-sizing: border-box; padding: 0px 5px;">3</li><li style="box-sizing: border-box; padding: 0px 5px;">4</li><li style="box-sizing: border-box; padding: 0px 5px;">5</li><li style="box-sizing: border-box; padding: 0px 5px;">6</li><li style="box-sizing: border-box; padding: 0px 5px;">7</li><li style="box-sizing: border-box; padding: 0px 5px;">8</li><li style="box-sizing: border-box; padding: 0px 5px;">9</li><li style="box-sizing: border-box; padding: 0px 5px;">10</li><li style="box-sizing: border-box; padding: 0px5px;">11</li><li style="box-sizing: border-box; padding: 0px 5px;">12</li><li style="box-sizing: border-box; padding:0px 5px;">13</li><li style="box-sizing: border-box; padding: 0px 5px;">14</li><li style="box-sizing: border-box;padding: 0px 5px;">15</li><li style="box-sizing: border-box; padding: 0px 5px;">16</li><li style="box-sizing:border-box; padding: 0px 5px;">17</li><li style="box-sizing: border-box; padding: 0px 5px;">18</li><listyle="box-sizing: border-box; padding: 0px 5px;">19</li><li style="box-sizing: border-box; padding: 0px5px;">20</li><li style="box-sizing: border-box; padding: 0px 5px;">21</li><li style="box-sizing: border-box; padding:0px 5px;">22</li></ul><ul class="pre-numbering" style="box-sizing: border-box; position: absolute; width: 50px; top: 0px; left: 0px; margin: 0px; padding: 6px 0px 40px; border-right-width: 1px; border-right-style: solid; border-right-color: rgb(221, 221, 221); list-style: none; text-align: right; background-color: rgb(238, 238, 238);"><li style="box-sizing: border-box; padding: 0px 5px;">1</li><li style="box-sizing: border-box; padding: 0px 5px;">2</li><li style="box-sizing: border-box; padding: 0px 5px;">3</li><li style="box-sizing: border-box; padding: 0px 5px;">4</li><li style="box-sizing: border-box; padding: 0px 5px;">5</li><li style="box-sizing: border-box; padding: 0px 5px;">6</li><li style="box-sizing: border-box; padding: 0px 5px;">7</li><li style="box-sizing: border-box; padding: 0px 5px;">8</li><li style="box-sizing: border-box; padding: 0px 5px;">9</li><li style="box-sizing: border-box; padding: 0px 5px;">10</li><li style="box-sizing: border-box; padding: 0px5px;">11</li><li style="box-sizing: border-box; padding: 0px 5px;">12</li><li style="box-sizing: border-box; padding:0px 5px;">13</li><li style="box-sizing: border-box; padding: 0px 5px;">14</li><li style="box-sizing: border-box;padding: 0px 5px;">15</li><li style="box-sizing: border-box; padding: 0px 5px;">16</li><li style="box-sizing:border-box; padding: 0px 5px;">17</li><li style="box-sizing: border-box; padding: 0px 5px;">18</li><listyle="box-sizing: border-box; padding: 0px 5px;">19</li><li style="box-sizing: border-box; padding: 0px5px;">20</li><li style="box-sizing: border-box; padding: 0px 5px;">21</li><li style="box-sizing: border-box; padding:0px 5px;">22</li></ul>
internal_thread.hpp里面封装了pthread函数,继承的子类可以得到一个单独的线程,主要作用是在计算当前的一批数据时,在后台获取新一批的数据。
关于data_layer,基本要注意的我都在图片上标注了。
1.3. neuron_layers.hpp
1.4. vision_layers.hpp
1.5. loss_layers.hpp
1.6. common_layers.hpp
2. Detail
在这一Section中,我们深入到上一小节所讲的集中layer的细节中去。对于一些常用的layer,如卷积层,池化层(Pooling),还给出对应的proto代码。
2.1. 数据层(data_layers)
数据通过数据层进入Caffe,数据层在整个网络的底部。数据可以来自高效的数据库(LevelDB 或者 LMDB),直接来自内存。如果不追求高效性,可以以HDF5或者一般图像的格式从硬盘读取数据。
一些基本的操作,如:mean subtraction, scaling, random cropping, and mirroring均可以直接在数据层上进行指定。
2.1.1 Database
source: 包含数据的目录名称
batch_size: 一次处理的输入的数量
可选参数:
rand_skip: 在开始的时候从输入中跳过这个数值,这在异步随机梯度下降(SGD)的时候非常有用
backend [default LEVELDB]: 选择使用 LEVELDB 或者 LMDB
2.1.2 In-Memory
2.1.3 HDF5 Input
source: 需要读取的文件名
batch_size:一次处理的输入的数量
2.1.4 HDF5 Output
HDF5的作用和这节中的其他的层不一样,它是把输入的blobs写到硬盘
2.1.5 Images
source: text文件的名字,每一行给出一张图片的文件名和label
batch_size: 一个batch中图片的数量
可选参数:
rand_skip:在开始的时候从输入中跳过这个数值,这在异步随机梯度下降(SGD)的时候非常有用
shuffle [default false]
new_height, new_width: 把所有的图像resize到这个大小
2.1.6 Windows
2.1.7 Dummy
Dummy 层用于development 和debugging。具体参数DummyDataParameter。
2.2. 激励层(neuron_layers)
一般来说,激励层是element-wise的操作,输入和输出的大小相同,一般情况下就是一个非线性函数。
2.2.1 ReLU / Rectified-Linear and Leaky-ReLU
<codeclass="hljs css has-numbering" style="display: block; padding: 0px; color: inherit; box-sizing: border-box; font-family: 'Source Code Pro', monospace;font-size:undefined; white-space: pre; border-radius: 0px; word-wrap: normal; background: transparent;"><span class="hljs-tag" style="color: rgb(0, 0, 0); box-sizing: border-box;">layer</span> <span class="hljs-rules" style="box-sizing: border-box;">{<span class="hljs-rule" style="box-sizing: border-box;"><spanclass="hljs-attribute" style="box-sizing: border-box;">name</span>:<span class="hljs-value" style="box-sizing: border-box; color: rgb(0, 102, 102);"> <span class="hljs-string" style="color: rgb(0, 136, 0); box-sizing: border-box;">"relu1"</span>type: <span class="hljs-string" style="color: rgb(0, 136, 0); box-sizing: border-box;">"ReLU"</span>bottom: <span class="hljs-string" style="color: rgb(0, 136, 0); box-sizing: border-box;">"conv1"</span>top: <span class="hljs-string" style="color: rgb(0, 136, 0); box-sizing: border-box;">"conv1"</span> </span></span></span>}</code><ul class="pre-numbering" style="box-sizing: border-box; position: absolute;width: 50px; top: 0px; left: 0px; margin: 0px; padding: 6px 0px 40px; border-right-width: 1px; border-right-style: solid; border-right-color: rgb(221, 221, 221); list-style: none; text-align: right; background-color: rgb(238, 238, 238);"><li style="box-sizing: border-box; padding: 0px 5px;">1</li><li style="box-sizing: border-box; padding: 0px 5px;">2</li><li style="box-sizing: border-box; padding: 0px 5px;">3</li><li style="box-sizing: border-box; padding: 0px 5px;">4</li><li style="box-sizing: border-box; padding: 0px 5px;">5</li><li style="box-sizing: border-box; padding: 0px 5px;">6</li></ul><ul class="pre-numbering" style="box-sizing: border-box; position: absolute;width: 50px; top: 0px; left: 0px; margin: 0px; padding: 6px 0px 40px; border-right-width: 1px; border-right-style: solid; border-right-color: rgb(221, 221, 221); list-style: none; text-align: right; background-color: rgb(238, 238, 238);"><li style="box-sizing: border-box; padding: 0px 5px;">1</li><li style="box-sizing: border-box; padding: 0px 5px;">2</li><li style="box-sizing: border-box; padding: 0px 5px;">3</li><li style="box-sizing: border-box; padding: 0px 5px;">4</li><li style="box-sizing: border-box; padding: 0px 5px;">5</li><li style="box-sizing: border-box; padding: 0px 5px;">6</li></ul>
2.2.2 Sigmoid
<codeclass="hljs css has-numbering" style="display: block; padding: 0px; color: inherit; box-sizing: border-box; font-family: 'Source Code Pro', monospace;font-size:undefined; white-space: pre; border-radius: 0px; word-wrap: normal; background: transparent;"><span class="hljs-tag" style="color: rgb(0, 0, 0); box-sizing: border-box;">layer</span> <span class="hljs-rules" style="box-sizing: border-box;">{<span class="hljs-rule" style="box-sizing: border-box;"><spanclass="hljs-attribute" style="box-sizing: border-box;">name</span>:<span class="hljs-value" style="box-sizing: border-box; color: rgb(0, 102, 102);"> <span class="hljs-string" style="color: rgb(0, 136, 0); box-sizing: border-box;">"encode1neuron"</span>bottom: <span class="hljs-string" style="color: rgb(0, 136, 0); box-sizing: border-box;">"encode1"</span>top: <span class="hljs-string" style="color: rgb(0, 136, 0); box-sizing: border-box;">"encode1neuron"</span>type: <span class="hljs-string" style="color: rgb(0, 136, 0); box-sizing: border-box;">"Sigmoid"</span> </span></span></span>}</code><ul class="pre-numbering" style="box-sizing: border-box; position: absolute;width: 50px; top: 0px; left: 0px; margin: 0px; padding: 6px 0px 40px; border-right-width: 1px; border-right-style: solid; border-right-color: rgb(221, 221, 221); list-style: none; text-align: right; background-color: rgb(238, 238, 238);"><li style="box-sizing: border-box; padding: 0px 5px;">1</li><li style="box-sizing: border-box; padding: 0px 5px;">2</li><li style="box-sizing: border-box; padding: 0px 5px;">3</li><li style="box-sizing: border-box; padding: 0px 5px;">4</li><li style="box-sizing: border-box; padding: 0px 5px;">5</li><li style="box-sizing: border-box; padding: 0px 5px;">6</li></ul><ul class="pre-numbering" style="box-sizing: border-box; position: absolute;width: 50px; top: 0px; left: 0px; margin: 0px; padding: 6px 0px 40px; border-right-width: 1px; border-right-style: solid; border-right-color: rgb(221, 221, 221); list-style: none; text-align: right; background-color: rgb(238, 238, 238);"><li style="box-sizing: border-box; padding: 0px 5px;">1</li><li style="box-sizing: border-box; padding: 0px 5px;">2</li><li style="box-sizing: border-box; padding: 0px 5px;">3</li><li style="box-sizing: border-box; padding: 0px 5px;">4</li><li style="box-sizing: border-box; padding: 0px 5px;">5</li><li style="box-sizing: border-box; padding: 0px 5px;">6</li></ul>
Sigmoid层通过 sigmoid(x) 计算每一个输入x的输出,函数如下图。
2.2.3 TanH / Hyperbolic Tangent
<codeclass="hljs css has-numbering" style="display: block; padding: 0px; color: inherit; box-sizing: border-box; font-family: 'Source Code Pro', monospace;font-size:undefined; white-space: pre; border-radius: 0px; word-wrap: normal; background: transparent;"><span class="hljs-tag" style="color: rgb(0, 0, 0); box-sizing: border-box;">layer</span> <span class="hljs-rules" style="box-sizing: border-box;">{<span class="hljs-rule" style="box-sizing: border-box;"><spanclass="hljs-attribute" style="box-sizing: border-box;">name</span>:<span class="hljs-value" style="box-sizing: border-box; color: rgb(0, 102, 102);"> <span class="hljs-string" style="color: rgb(0, 136, 0); box-sizing: border-box;">"layer"</span>bottom: <span class="hljs-string" style="color: rgb(0, 136, 0); box-sizing: border-box;">"in"</span>top: <span class="hljs-string" style="color: rgb(0, 136, 0); box-sizing: border-box;">"out"</span>type: <span class="hljs-string" style="color: rgb(0, 136, 0); box-sizing: border-box;">"TanH"</span> </span></span></span>}</code><ul class="pre-numbering" style="box-sizing: border-box; position: absolute;width: 50px; top: 0px; left: 0px; margin: 0px; padding: 6px 0px 40px; border-right-width: 1px; border-right-style: solid; border-right-color: rgb(221, 221, 221); list-style: none; text-align: right; background-color: rgb(238, 238, 238);"><li style="box-sizing: border-box; padding: 0px 5px;">1</li><li style="box-sizing: border-box; padding: 0px 5px;">2</li><li style="box-sizing: border-box; padding: 0px 5px;">3</li><li style="box-sizing: border-box; padding: 0px 5px;">4</li><li style="box-sizing: border-box; padding: 0px 5px;">5</li><li style="box-sizing: border-box; padding: 0px 5px;">6</li></ul><ul class="pre-numbering" style="box-sizing: border-box; position: absolute;width: 50px; top: 0px; left: 0px; margin: 0px; padding: 6px 0px 40px; border-right-width: 1px; border-right-style: solid; border-right-color: rgb(221, 221, 221); list-style: none; text-align: right; background-color: rgb(238, 238, 238);"><li style="box-sizing: border-box; padding: 0px 5px;">1</li><li style="box-sizing: border-box; padding: 0px 5px;">2</li><li style="box-sizing: border-box; padding: 0px 5px;">3</li><li style="box-sizing: border-box; padding: 0px 5px;">4</li><li style="box-sizing: border-box; padding: 0px 5px;">5</li><li style="box-sizing: border-box; padding: 0px 5px;">6</li></ul>
TanH层通过 tanh(x) 计算每一个输入x的输出,函数如下图。请注意sigmoid函数和TanH函数在纵轴上的区别。sigmoid函数将实数映射到(0,1)。TanH将实数映射到(-1,1)。
2.2.4 Absolute Value
<codeclass="hljs css has-numbering" style="display: block; padding: 0px; color: inherit; box-sizing: border-box; font-family: 'Source Code Pro', monospace;font-size:undefined; white-space: pre; border-radius: 0px; word-wrap: normal; background: transparent;"><span class="hljs-tag" style="color: rgb(0, 0, 0); box-sizing: border-box;">layer</span> <span class="hljs-rules" style="box-sizing: border-box;">{<span class="hljs-rule" style="box-sizing: border-box;"><spanclass="hljs-attribute" style="box-sizing: border-box;">name</span>:<span class="hljs-value" style="box-sizing: border-box; color: rgb(0, 102, 102);"> <span class="hljs-string" style="color: rgb(0, 136, 0); box-sizing: border-box;">"layer"</span>bottom: <span class="hljs-string" style="color: rgb(0, 136, 0); box-sizing: border-box;">"in"</span>top: <span class="hljs-string" style="color: rgb(0, 136, 0); box-sizing: border-box;">"out"</span>type: <span class="hljs-string" style="color: rgb(0, 136, 0); box-sizing: border-box;">"AbsVal"</span> </span></span></span>}</code><ul class="pre-numbering" style="box-sizing: border-box; position: absolute;width: 50px; top: 0px; left: 0px; margin: 0px; padding: 6px 0px 40px; border-right-width: 1px; border-right-style: solid; border-right-color: rgb(221, 221, 221); list-style: none; text-align: right; background-color: rgb(238, 238, 238);"><li style="box-sizing: border-box; padding: 0px 5px;">1</li><li style="box-sizing: border-box; padding: 0px 5px;">2</li><li style="box-sizing: border-box; padding: 0px 5px;">3</li><li style="box-sizing: border-box; padding: 0px 5px;">4</li><li style="box-sizing: border-box; padding: 0px 5px;">5</li><li style="box-sizing: border-box; padding: 0px 5px;">6</li></ul><ul class="pre-numbering" style="box-sizing: border-box; position: absolute;width: 50px; top: 0px; left: 0px; margin: 0px; padding: 6px 0px 40px; border-right-width: 1px; border-right-style: solid; border-right-color: rgb(221, 221, 221); list-style: none; text-align: right; background-color: rgb(238, 238, 238);"><li style="box-sizing: border-box; padding: 0px 5px;">1</li><li style="box-sizing: border-box; padding: 0px 5px;">2</li><li style="box-sizing: border-box; padding: 0px 5px;">3</li><li style="box-sizing: border-box; padding: 0px 5px;">4</li><li style="box-sizing: border-box; padding: 0px 5px;">5</li><li style="box-sizing: border-box; padding: 0px 5px;">6</li></ul>
2.2.5 Power
<codeclass="hljs css has-numbering" style="display: block; padding: 0px; color: inherit; box-sizing: border-box; font-family: 'Source Code Pro', monospace;font-size:undefined; white-space: pre; border-radius: 0px; word-wrap: normal; background: transparent;"><span class="hljs-tag" style="color: rgb(0, 0, 0); box-sizing: border-box;">layer</span> <span class="hljs-rules" style="box-sizing: border-box;">{<span class="hljs-rule" style="box-sizing: border-box;"><spanclass="hljs-attribute" style="box-sizing: border-box;">name</span>:<span class="hljs-value" style="box-sizing: border-box; color: rgb(0, 102, 102);"> <span class="hljs-string" style="color: rgb(0, 136, 0); box-sizing: border-box;">"layer"</span>bottom: <span class="hljs-string" style="color: rgb(0, 136, 0); box-sizing: border-box;">"in"</span>top: <span class="hljs-string" style="color: rgb(0, 136, 0); box-sizing: border-box;">"out"</span>type: <span class="hljs-string" style="color: rgb(0, 136, 0); box-sizing: border-box;">"Power"</span>power_param {power: <span class="hljs-number" style="box-sizing: border-box;">1</span>scale: <span class="hljs-number" style="box-sizing: border-box;">1</span>shift: <span class="hljs-number" style="box-sizing: border-box;">0</span></span></span></span>} }</code><ul class="pre-numbering" style="box-sizing: border-box; position: absolute; width: 50px; top: 0px; left: 0px; margin: 0px; padding: 6px 0px 40px; border-right-width: 1px; border-right-style: solid; border-right-color: rgb(221, 221, 221); list-style: none; text-align: right; background-color: rgb(238, 238, 238);"><li style="box-sizing: border-box; padding: 0px 5px;">1</li><li style="box-sizing: border-box; padding: 0px 5px;">2</li><li style="box-sizing: border-box; padding: 0px 5px;">3</li><li style="box-sizing: border-box; padding: 0px 5px;">4</li><li style="box-sizing: border-box; padding: 0px 5px;">5</li><li style="box-sizing: border-box; padding: 0px 5px;">6</li><li style="box-sizing: border-box; padding: 0px 5px;">7</li><li style="box-sizing: border-box; padding: 0px 5px;">8</li><li style="box-sizing: border-box; padding: 0px 5px;">9</li><li style="box-sizing: border-box; padding: 0px 5px;">10</li><li style="box-sizing: border-box; padding: 0px5px;">11</li></ul><ul class="pre-numbering" style="box-sizing: border-box; position: absolute; width: 50px; top: 0px; left: 0px; margin: 0px; padding: 6px 0px 40px; border-right-width: 1px; border-right-style: solid; border-right-color: rgb(221, 221, 221); list-style: none; text-align: right; background-color: rgb(238, 238, 238);"><li style="box-sizing: border-box; padding: 0px 5px;">1</li><li style="box-sizing: border-box; padding: 0px 5px;">2</li><li style="box-sizing: border-box; padding: 0px 5px;">3</li><li style="box-sizing: border-box; padding: 0px 5px;">4</li><li style="box-sizing: border-box; padding: 0px 5px;">5</li><li style="box-sizing: border-box; padding: 0px 5px;">6</li><li style="box-sizing: border-box; padding: 0px 5px;">7</li><li style="box-sizing: border-box; padding: 0px 5px;">8</li><li style="box-sizing: border-box; padding: 0px 5px;">9</li><li style="box-sizing: border-box; padding: 0px 5px;">10</li><li style="box-sizing: border-box; padding: 0px5px;">11</li></ul>
power [default 1]
scale [default 1]
shift [default 0]
POWER层通过 (shift + scale * x) ^ power计算每一个输入x的输出。
2.2.6 BNLL
<codeclass="hljs css has-numbering" style="display: block; padding: 0px; color: inherit; box-sizing: border-box; font-family: 'Source Code Pro', monospace;font-size:undefined; white-space: pre; border-radius: 0px; word-wrap: normal; background: transparent;"><span class="hljs-tag" style="color: rgb(0, 0, 0); box-sizing: border-box;">layer</span> <span class="hljs-rules" style="box-sizing: border-box;">{<span class="hljs-rule" style="box-sizing: border-box;"><spanclass="hljs-attribute" style="box-sizing: border-box;">name</span>:<span class="hljs-value" style="box-sizing: border-box; color: rgb(0, 102, 102);"> <span class="hljs-string" style="color: rgb(0, 136, 0); box-sizing: border-box;">"layer"</span>bottom: <span class="hljs-string" style="color: rgb(0, 136, 0); box-sizing: border-box;">"in"</span>top: <span class="hljs-string" style="color: rgb(0, 136, 0); box-sizing: border-box;">"out"</span>type: BNLL </span></span></span>}</code><ul class="pre-numbering" style="box-sizing: border-box; position: absolute;width: 50px; top: 0px; left: 0px; margin: 0px; padding: 6px 0px 40px; border-right-width: 1px; border-right-style: solid; border-right-color: rgb(221, 221, 221); list-style: none; text-align: right; background-color: rgb(238, 238, 238);"><li style="box-sizing: border-box; padding: 0px 5px;">1</li><li style="box-sizing: border-box; padding: 0px 5px;">2</li><li style="box-sizing: border-box; padding: 0px 5px;">3</li><li style="box-sizing: border-box; padding: 0px 5px;">4</li><li style="box-sizing: border-box; padding: 0px 5px;">5</li><li style="box-sizing: border-box; padding: 0px 5px;">6</li></ul><ul class="pre-numbering" style="box-sizing: border-box; position: absolute;width: 50px; top: 0px; left: 0px; margin: 0px; padding: 6px 0px 40px; border-right-width: 1px; border-right-style: solid; border-right-color: rgb(221, 221, 221); list-style: none; text-align: right; background-color: rgb(238, 238, 238);"><li style="box-sizing: border-box; padding: 0px 5px;">1</li><li style="box-sizing: border-box; padding: 0px 5px;">2</li><li style="box-sizing: border-box; padding: 0px 5px;">3</li><li style="box-sizing: border-box; padding: 0px 5px;">4</li><li style="box-sizing: border-box; padding: 0px 5px;">5</li><li style="box-sizing: border-box; padding: 0px 5px;">6</li></ul>
BNLL (binomial normal log likelihood) 层通过 log(1 + exp(x)) 计算每一个输入x的输出。
2.3. 视觉层(vision_layers)
2.3.1 卷积层(Convolution)
<codeclass="hljs bash has-numbering" style="display: block; padding: 0px; color: inherit; box-sizing: border-box; font-family: 'Source Code Pro', monospace;font-size:undefined; white-space: pre; border-radius: 0px; word-wrap: normal; background: transparent;">layers { name: <span class="hljs-string" style="color: rgb(0, 136, 0); box-sizing: border-box;">"conv1"</span> <span class="hljs-built_in" style="color: rgb(102, 0, 102); box-sizing: border-box;">type</span>: CONVOLUTION bottom: <span class="hljs-string" style="color: rgb(0, 136, 0); box-sizing: border-box;">"data"</span> top: <span class="hljs-string" style="color: rgb(0, 136, 0); box-sizing: border-box;">"conv1"</span> blobs_lr: <span class="hljs-number" style="color: rgb(0, 102, 102); box-sizing: border-box;">1</span> <span class="hljs-comment" style="color: rgb(136, 0, 0); box-sizing: border-box;"># learning rate multiplier for the filters </span>blobs_lr: <span class="hljs-number" style="color: rgb(0, 102, 102); box-sizing: border-box;">2</span> <span class="hljs-comment" style="color: rgb(136, 0, 0); box-sizing: border-box;"># learning rate multiplier for the biases </span>weight_decay: <span class="hljs-number" style="color: rgb(0, 102,102); box-sizing: border-box;">1</span> <span class="hljs-comment" style="color: rgb(136, 0, 0); box-sizing: border-box;"># weight decay multiplier for the filters </span>weight_decay: <span class="hljs-number" style="color: rgb(0, 102,102); box-sizing: border-box;">0</span> <span class="hljs-comment" style="color: rgb(136, 0, 0); box-sizing: border-box;"># weight decay multiplier for the biases </span>convolution_param { num_output: <span class="hljs-number" style="color: rgb(0, 102, 102); box-sizing: border-box;">96</span> <span class="hljs-comment" style="color: rgb(136, 0, 0); box-sizing: border-box;"># learn 96 filters </span>kernel_size: <span class="hljs-number" style="color: rgb(0, 102, 102); box-sizing: border-box;">11</span> <span class="hljs-comment" style="color: rgb(136, 0, 0); box-sizing: border-box;"># each filter is 11x11 </span>stride: <span class="hljs-number" style="color: rgb(0, 102, 102); box-sizing: border-box;">4</span> <span class="hljs-comment" style="color: rgb(136, 0, 0); box-sizing: border-box;"># step 4 pixels between each filter application </span>weight_filler { <span class="hljs-built_in" style="color: rgb(102, 0, 102); box-sizing: border-box;">type</span>: <span class="hljs-string" style="color: rgb(0, 136, 0); box-sizing: border-box;">"gaussian"</span> <span class="hljs-comment" style="color: rgb(136, 0, 0); box-sizing: border-box;"># initialize the filters from a Gaussian </span>std: <span class="hljs-number" style="color: rgb(0, 102, 102); box-sizing: border-box;">0.01</span> <span class="hljs-comment" style="color: rgb(136, 0, 0); box-sizing: border-box;"># distribution with stdev 0.01 (default mean: 0) } </span>bias_filler { <span class="hljs-built_in" style="color: rgb(102, 0,102); box-sizing: border-box;">type</span>: <span class="hljs-string" style="color: rgb(0, 136, 0); box-sizing: border-box;">"constant"</span> <span class="hljs-comment" style="color: rgb(136, 0, 0); box-sizing: border-box;"># initialize the biases to zero (0) </span>value: <span class="hljs-number" style="color: rgb(0,102, 102); box-sizing: border-box;">0</span> } }} }</code><ul class="pre-numbering" style="box-sizing: border-box; position: absolute; width: 50px; top: 0px; left: 0px; margin: 0px; padding: 6px 0px 40px; border-right-width: 1px; border-right-style: solid; border-right-color: rgb(221, 221, 221); list-style: none; text-align: right; background-color: rgb(238, 238, 238);"><li style="box-sizing: border-box; padding: 0px 5px;">1</li><li style="box-sizing: border-box; padding: 0px 5px;">2</li><li style="box-sizing: border-box; padding: 0px 5px;">3</li><li style="box-sizing: border-box; padding: 0px 5px;">4</li><li style="box-sizing: border-box; padding: 0px 5px;">5</li><li style="box-sizing: border-box; padding: 0px 5px;">6</li><li style="box-sizing: border-box; padding: 0px 5px;">7</li><li style="box-sizing: border-box; padding: 0px 5px;">8</li><li style="box-sizing: border-box; padding: 0px 5px;">9</li><li style="box-sizing: border-box; padding: 0px 5px;">10</li><li style="box-sizing: border-box; padding: 0px5px;">11</li><li style="box-sizing: border-box; padding: 0px 5px;">12</li><li style="box-sizing: border-box; padding:0px 5px;">13</li><li style="box-sizing: border-box; padding: 0px 5px;">14</li><li style="box-sizing: border-box;padding: 0px 5px;">15</li><li style="box-sizing: border-box; padding: 0px 5px;">16</li><li style="box-sizing:border-box; padding: 0px 5px;">17</li><li style="box-sizing: border-box; padding: 0px 5px;">18</li><listyle="box-sizing: border-box; padding: 0px 5px;">19</li><li style="box-sizing: border-box; padding: 0px5px;">20</li><li style="box-sizing: border-box; padding: 0px 5px;">21</li><li style="box-sizing: border-box; padding:0px 5px;">22</li><li style="box-sizing: border-box; padding: 0px 5px;">23</li></ul><ul class="pre-numbering" style="box-sizing: border-box; position: absolute;width: 50px; top: 0px; left: 0px; margin: 0px; padding: 6px 0px 40px; border-right-width: 1px; border-right-style: solid; border-right-color: rgb(221, 221, 221); list-style: none; text-align: right; background-color: rgb(238, 238, 238);"><li style="box-sizing: border-box; padding: 0px 5px;">1</li><li style="box-sizing: border-box; padding: 0px 5px;">2</li><li style="box-sizing: border-box; padding: 0px 5px;">3</li><li style="box-sizing: border-box; padding: 0px 5px;">4</li><li style="box-sizing: border-box; padding: 0px 5px;">5</li><li style="box-sizing: border-box; padding: 0px 5px;">6</li><li style="box-sizing: border-box; padding: 0px 5px;">7</li><li style="box-sizing: border-box; padding: 0px 5px;">8</li><li style="box-sizing: border-box; padding: 0px 5px;">9</li><li style="box-sizing: border-box; padding: 0px 5px;">10</li><li style="box-sizing:border-box; padding: 0px 5px;">11</li><li style="box-sizing: border-box; padding: 0px 5px;">12</li><listyle="box-sizing: border-box; padding: 0px 5px;">13</li><li style="box-sizing: border-box; padding: 0px5px;">14</li><li style="box-sizing: border-box; padding: 0px 5px;">15</li><li style="box-sizing: border-box; padding:0px 5px;">16</li><li style="box-sizing: border-box; padding: 0px 5px;">17</li><li style="box-sizing: border-box;padding: 0px 5px;">18</li><li style="box-sizing: border-box; padding: 0px 5px;">19</li><li style="box-sizing:border-box; padding: 0px 5px;">20</li><li style="box-sizing: border-box; padding: 0px 5px;">21</li><listyle="box-sizing: border-box; padding: 0px 5px;">22</li><li style="box-sizing: border-box; padding: 0px5px;">23</li></ul>
blobs_lr: 学习率调整的参数,在上面的例子中设置权重学习率和运行中求解器给出的学习率一样,同时是偏置学习率为权重的两倍。
weight_decay:
num_output (c_o):过滤器的个数
kernel_size (or kernel_h and kernel_w):过滤器的大小(也就是所谓“核”的大小)。
建议参数:
- weight_filler [default type: ‘constant’ value: 0]:参数的初始化方法
可选参数:
bias_filler:偏置的初始化方法
bias_term [default true]:指定是否是否开启偏置项
pad (or pad_h and pad_w) [default 0]:指定在输入的每一边加上多少个像素
stride (or stride_h and stride_w) [default 1]:指定过滤器的步长
group (g) [default 1]: 如果g>1,那么将每个滤波器都限定只与某个输入的子集有关联。换句话说,将输入分为g组,同时将输出也分为g组。那么第i组输出只与第i组输入有关。
通过卷积后的大小变化:
输入:
- n×ci×hi×wi
输出:
- n×co×ho×wo
其中:ho=(hi+2×padh−kernelh)/strideh+1。wo通过同样的方法计算。
2.3.2 池化层(Pooling)
<codeclass="hljs bash has-numbering" style="display: block; padding: 0px; color: inherit; box-sizing: border-box; font-family: 'Source Code Pro', monospace;font-size:undefined; white-space: pre; border-radius: 0px; word-wrap: normal; background: transparent;">layers { name: <span class="hljs-string" style="color: rgb(0, 136, 0); box-sizing: border-box;">"pool1"</span> <span class="hljs-built_in" style="color: rgb(102, 0, 102); box-sizing: border-box;">type</span>: POOLING bottom: <span class="hljs-string" style="color: rgb(0, 136, 0); box-sizing: border-box;">"conv1"</span> top: <span class="hljs-string" style="color: rgb(0, 136, 0); box-sizing: border-box;">"pool1"</span> pooling_param { pool: MAX kernel_size: <span class="hljs-number" style="color: rgb(0, 102, 102); box-sizing: border-box;">3</span> <span class="hljs-comment" style="color: rgb(136, 0, 0); box-sizing: border-box;"># pool over a 3x3 region </span>stride: <span class="hljs-number" style="color: rgb(0, 102, 102); box-sizing: border-box;">2</span> <span class="hljs-comment" style="color: rgb(136, 0, 0); box-sizing: border-box;"># step two pixels (in the bottom blob) between pooling regions </span>} }</code><ul class="pre-numbering" style="box-sizing: border-box; position: absolute; width: 50px; top: 0px; left: 0px; margin: 0px; padding: 6px 0px 40px; border-right-width: 1px; border-right-style: solid; border-right-color: rgb(221, 221, 221); list-style: none; text-align: right; background-color: rgb(238, 238, 238);"><li style="box-sizing: border-box; padding: 0px 5px;">1</li><li style="box-sizing: border-box; padding: 0px 5px;">2</li><li style="box-sizing: border-box; padding: 0px 5px;">3</li><li style="box-sizing: border-box; padding: 0px 5px;">4</li><li style="box-sizing: border-box; padding: 0px 5px;">5</li><li style="box-sizing: border-box; padding: 0px 5px;">6</li><li style="box-sizing: border-box; padding: 0px 5px;">7</li><li style="box-sizing: border-box; padding: 0px 5px;">8</li><li style="box-sizing: border-box; padding: 0px 5px;">9</li><li style="box-sizing: border-box; padding: 0px 5px;">10</li><li style="box-sizing: border-box; padding: 0px5px;">11</li></ul><ul class="pre-numbering" style="box-sizing: border-box; position: absolute; width: 50px; top: 0px; left: 0px; margin: 0px; padding: 6px 0px 40px; border-right-width: 1px; border-right-style: solid; border-right-color: rgb(221, 221, 221); list-style: none; text-align: right; background-color: rgb(238, 238, 238);"><li style="box-sizing: border-box; padding: 0px 5px;">1</li><li style="box-sizing: border-box; padding: 0px 5px;">2</li><li style="box-sizing: border-box; padding: 0px 5px;">3</li><li style="box-sizing: border-box; padding: 0px 5px;">4</li><li style="box-sizing: border-box; padding: 0px 5px;">5</li><li style="box-sizing: border-box; padding: 0px 5px;">6</li><li style="box-sizing: border-box; padding: 0px 5px;">7</li><li style="box-sizing: border-box; padding: 0px 5px;">8</li><li style="box-sizing: border-box; padding: 0px 5px;">9</li><li style="box-sizing: border-box; padding: 0px 5px;">10</li><li style="box-sizing: border-box; padding: 0px5px;">11</li></ul>
pool [default MAX]:pooling的方法,目前有MAX, AVE, 和STOCHASTIC三种方法
pad (or pad_h and pad_w) [default 0]:指定在输入的每一遍加上多少个像素
stride (or stride_h and stride_w) [default 1]:指定过滤器的步长
通过池化后的大小变化:
输入:
- n×ci×hi×wi
输出:
- n×co×ho×wo
其中:ho=(hi+2×padh−kernelh)/strideh+1。wo通过同样的方法计算。
2.3.3 Local Response Normalization (LRN)
local_size [default 5]:对于cross channel LRN为需要求和的邻近channel的数量;对于within channel LRN为需要求和的空间区域的边长;
alpha [default 1]:scaling参数;
beta [default 5]:指数;
norm_region [default ACROSS_CHANNELS]: 选择LRN实现的方法:1. ACROSS_CHANNELS ;2. WITHIN_CHANNEL
LRN(Local Response Normalization)是对一个局部的输入区域进行的归一化。有两种不同的形式:1. ACCROSS_CHANNEL;2. WITHIN_CHANNEL。其实很好从字面上进行理解。第一种方法综合了不同的channel,而在一个channel里面只取1*1(所以size是localsize×1×1)。而在第二种方法中,不在channel方向上扩展,只在单一channel上进行空间扩展(所以size是1×localsize×localsize)。
计算公式:对每一个输入除以(1+(α/n)⋅∑ix2i)β
在这里,参数α是scaling参数,参数β是指数。而参数n对应local region的大小。
2.4. 损失层(Loss Layers)
2.4.1 Softmax
2.4.2 Sum-of-Squares / Euclidean
2.4.3 Hinge / Margin
<codeclass="hljs css has-numbering" style="display: block; padding: 0px; color: inherit; box-sizing: border-box; font-family: 'Source Code Pro', monospace;font-size:undefined; white-space: pre; border-radius: 0px; word-wrap: normal; background: transparent;"><span class="hljs-tag" style="color: rgb(0, 0, 0); box-sizing: border-box;">L1</span> <span class="hljs-tag" style="color: rgb(0, 0, 0); box-sizing: border-box;">Normlayers</span> <span class="hljs-rules" style="box-sizing: border-box;">{ <span class="hljs-rule" style="box-sizing: border-box;"><span class="hljs-attribute" style="box-sizing: border-box;">name</span>:<span class="hljs-value" style="box-sizing: border-box; color: rgb(0, 102, 102);"> <span class="hljs-string" style="color: rgb(0, 136, 0); box-sizing: border-box;">"loss"</span> type: HINGE_LOSS bottom: <span class="hljs-string" style="color: rgb(0, 136, 0); box-sizing: border-box;">"pred"</span> bottom: <span class="hljs-string" style="color: rgb(0, 136, 0); box-sizing: border-box;">"label"</span> </span></span></span>} <span class="hljs-tag" style="color: rgb(0, 0, 0); box-sizing: border-box;">L2</span> <span class="hljs-tag" style="color: rgb(0, 0, 0); box-sizing: border-box;">Normlayers</span> <span class="hljs-rules" style="box-sizing: border-box;">{ <span class="hljs-rule" style="box-sizing: border-box;"><span class="hljs-attribute" style="box-sizing: border-box;">name</span>:<span class="hljs-value" style="box-sizing: border-box; color: rgb(0, 102, 102);"> <span class="hljs-string" style="color: rgb(0, 136, 0); box-sizing: border-box;">"loss"</span> type: HINGE_LOSS bottom: <span class="hljs-string" style="color: rgb(0, 136, 0); box-sizing: border-box;">"pred"</span> bottom: <span class="hljs-string" style="color: rgb(0, 136, 0); box-sizing: border-box;">"label"</span> top: <span class="hljs-string" style="color: rgb(0, 136, 0); box-sizing: border-box;">"loss"</span> hinge_loss_param { norm: L2 </span></span></span>} }</code><ul class="pre-numbering" style="box-sizing: border-box; position: absolute; width: 50px; top: 0px; left: 0px; margin: 0px; padding: 6px 0px 40px; border-right-width: 1px; border-right-style: solid; border-right-color: rgb(221, 221, 221); list-style: none; text-align: right; background-color: rgb(238, 238, 238);"><li style="box-sizing: border-box; padding: 0px 5px;">1</li><li style="box-sizing: border-box; padding: 0px 5px;">2</li><li style="box-sizing: border-box; padding: 0px 5px;">3</li><li style="box-sizing: border-box; padding: 0px 5px;">4</li><li style="box-sizing: border-box; padding: 0px 5px;">5</li><li style="box-sizing: border-box; padding: 0px 5px;">6</li><li style="box-sizing: border-box; padding: 0px 5px;">7</li><li style="box-sizing: border-box; padding: 0px 5px;">8</li><li style="box-sizing: border-box; padding: 0px 5px;">9</li><li style="box-sizing: border-box; padding: 0px 5px;">10</li><li style="box-sizing: border-box; padding: 0px5px;">11</li><li style="box-sizing: border-box; padding: 0px 5px;">12</li><li style="box-sizing: border-box; padding:0px 5px;">13</li><li style="box-sizing: border-box; padding: 0px 5px;">14</li><li style="box-sizing: border-box;padding: 0px 5px;">15</li><li style="box-sizing: border-box; padding: 0px 5px;">16</li></ul><ul class="pre-numbering" style="box-sizing: border-box; position: absolute;width: 50px; top: 0px; left: 0px; margin: 0px; padding: 6px 0px 40px; border-right-width: 1px; border-right-style: solid; border-right-color: rgb(221, 221, 221); list-style: none; text-align: right; background-color: rgb(238, 238, 238);"><li style="box-sizing: border-box; padding: 0px 5px;">1</li><li style="box-sizing: border-box; padding: 0px 5px;">2</li><li style="box-sizing: border-box; padding: 0px 5px;">3</li><li style="box-sizing: border-box; padding: 0px 5px;">4</li><li style="box-sizing: border-box; padding: 0px 5px;">5</li><li style="box-sizing: border-box; padding: 0px 5px;">6</li><li style="box-sizing: border-box; padding: 0px 5px;">7</li><li style="box-sizing: border-box; padding: 0px 5px;">8</li><li style="box-sizing: border-box; padding: 0px 5px;">9</li><li style="box-sizing: border-box; padding: 0px 5px;">10</li><li style="box-sizing:border-box; padding: 0px 5px;">11</li><li style="box-sizing: border-box; padding: 0px 5px;">12</li><listyle="box-sizing: border-box; padding: 0px 5px;">13</li><li style="box-sizing: border-box; padding: 0px5px;">14</li><li style="box-sizing: border-box; padding: 0px 5px;">15</li><li style="box-sizing: border-box; padding:0px 5px;">16</li></ul>
2.4.4 Sigmoid Cross-Entropy
2.4.5 Infogain
2.4.6 Accuracy and Top-k
用来计算输出和目标的正确率,事实上这不是一个loss,而且没有backward这一步。
2.5. 一般层(Common Layers)
2.5.1 全连接层 Inner Product
<codeclass="hljs css has-numbering" style="display: block; padding: 0px; color: inherit; box-sizing: border-box; font-family: 'Source Code Pro', monospace;font-size:undefined; white-space: pre; border-radius: 0px; word-wrap: normal; background: transparent;"><span class="hljs-tag" style="color: rgb(0, 0, 0); box-sizing: border-box;">layer</span> <span class="hljs-rules" style="box-sizing: border-box;">{<span class="hljs-rule" style="box-sizing: border-box;"><spanclass="hljs-attribute" style="box-sizing: border-box;">name</span>:<span class="hljs-value" style="box-sizing: border-box; color: rgb(0, 102, 102);"> <span class="hljs-string" style="color: rgb(0, 136, 0); box-sizing: border-box;">"fc8"</span>type: <span class="hljs-string" style="color: rgb(0, 136, 0); box-sizing: border-box;">"InnerProduct"</span># learning rate and decay multipliers for the weightsparam { lr_mult: <span class="hljs-number" style="box-sizing: border-box;">1</span> decay_mult: <span class="hljs-number" style="box-sizing: border-box;">1</span> </span></span></span>}# <span class="hljs-tag" style="color: rgb(0, 0, 0); box-sizing: border-box;">learning</span> <span class="hljs-tag" style="color: rgb(0, 0, 0); box-sizing: border-box;">rate</span> <span class="hljs-tag" style="color: rgb(0, 0, 0); box-sizing: border-box;">and</span><span class="hljs-tag" style="color: rgb(0, 0, 0); box-sizing: border-box;">decay</span> <span class="hljs-tag" style="color: rgb(0, 0, 0); box-sizing: border-box;">multipliers</span> <span class="hljs-tag" style="color: rgb(0, 0, 0); box-sizing: border-box;">for</span><span class="hljs-tag" style="color: rgb(0, 0, 0); box-sizing: border-box;">the</span> <span class="hljs-tag" style="color:rgb(0, 0, 0); box-sizing: border-box;">biases</span><span class="hljs-tag" style="color: rgb(0, 0, 0); box-sizing: border-box;">param</span> <span class="hljs-rules" style="box-sizing: border-box;">{ <span class="hljs-rule" style="box-sizing: border-box;"><span class="hljs-attribute" style="box-sizing: border-box;">lr_mult</span>:<span class="hljs-value" style="box-sizing: border-box; color: rgb(0, 102, 102);"> <span class="hljs-number" style="box-sizing: border-box;">2</span> decay_mult: <span class="hljs-number" style="box-sizing: border-box;">0</span> </span></span></span>}<span class="hljs-tag" style="color: rgb(0, 0, 0); box-sizing: border-box;">inner_product_param</span> <span class="hljs-rules" style="box-sizing: border-box;">{<span class="hljs-rule" style="box-sizing: border-box;"><span class="hljs-attribute" style="box-sizing: border-box;">num_output</span>:<span class="hljs-value" style="box-sizing: border-box; color: rgb(0, 102, 102);"> <span class="hljs-number" style="box-sizing: border-box;">1000</span>weight_filler {type: <span class="hljs-string" style="color: rgb(0, 136, 0); box-sizing: border-box;">"gaussian"</span>std: <span class="hljs-number" style="box-sizing: border-box;">0.01</span></span></span></span>}<span class="hljs-tag" style="color: rgb(0, 0, 0); box-sizing: border-box;">bias_filler</span> <span class="hljs-rules" style="box-sizing: border-box;">{<span class="hljs-rule" style="box-sizing: border-box;"><span class="hljs-attribute" style="box-sizing: border-box;">type</span>:<span class="hljs-value" style="box-sizing: border-box; color: rgb(0, 102, 102);"> <span class="hljs-string" style="color: rgb(0, 136, 0); box-sizing: border-box;">"constant"</span>value: <span class="hljs-number" style="box-sizing: border-box;">0</span></span></span></span>}}<span class="hljs-tag" style="color: rgb(0, 0, 0); box-sizing: border-box;">bottom</span>: "<span class="hljs-tag" style="color: rgb(0, 0, 0); box-sizing: border-box;">fc7</span>"<span class="hljs-tag" style="color: rgb(0, 0, 0); box-sizing: border-box;">top</span>: "<span class="hljs-tag" style="color: rgb(0, 0, 0); box-sizing: border-box;">fc8</span>" }</code><ul class="pre-numbering" style="box-sizing: border-box; position: absolute; width: 50px; top: 0px; left: 0px; margin: 0px; padding: 6px 0px 40px; border-right-width: 1px; border-right-style: solid; border-right-color: rgb(221, 221, 221); list-style: none; text-align: right; background-color: rgb(238, 238, 238);"><li style="box-sizing: border-box; padding: 0px 5px;">1</li><li style="box-sizing: border-box; padding: 0px 5px;">2</li><li style="box-sizing: border-box; padding: 0px 5px;">3</li><li style="box-sizing: border-box; padding: 0px 5px;">4</li><li style="box-sizing: border-box; padding: 0px 5px;">5</li><li style="box-sizing: border-box; padding: 0px 5px;">6</li><li style="box-sizing: border-box; padding: 0px 5px;">7</li><li style="box-sizing: border-box; padding: 0px 5px;">8</li><li style="box-sizing: border-box; padding: 0px 5px;">9</li><li style="box-sizing: border-box; padding: 0px 5px;">10</li><li style="box-sizing: border-box; padding: 0px5px;">11</li><li style="box-sizing: border-box; padding: 0px 5px;">12</li><li style="box-sizing: border-box; padding:0px 5px;">13</li><li style="box-sizing: border-box; padding: 0px 5px;">14</li><li style="box-sizing: border-box;padding: 0px 5px;">15</li><li style="box-sizing: border-box; padding: 0px 5px;">16</li><li style="box-sizing:border-box; padding: 0px 5px;">17</li><li style="box-sizing: border-box; padding: 0px 5px;">18</li><listyle="box-sizing: border-box; padding: 0px 5px;">19</li><li style="box-sizing: border-box; padding: 0px5px;">20</li><li style="box-sizing: border-box; padding: 0px 5px;">21</li></ul><ul class="pre-numbering" style="box-sizing: border-box; position: absolute; width: 50px; top: 0px; left: 0px; margin: 0px; padding: 6px 0px 40px; border-right-width: 1px; border-right-style: solid; border-right-color: rgb(221, 221, 221); list-style: none; text-align: right; background-color: rgb(238, 238, 238);"><li style="box-sizing: border-box; padding: 0px 5px;">1</li><li style="box-sizing: border-box; padding: 0px 5px;">2</li><li style="box-sizing: border-box; padding: 0px 5px;">3</li><li style="box-sizing: border-box; padding: 0px 5px;">4</li><li style="box-sizing: border-box; padding: 0px 5px;">5</li><li style="box-sizing: border-box; padding: 0px 5px;">6</li><li style="box-sizing: border-box; padding: 0px 5px;">7</li><li style="box-sizing: border-box; padding: 0px 5px;">8</li><li style="box-sizing: border-box; padding: 0px 5px;">9</li><li style="box-sizing: border-box; padding: 0px 5px;">10</li><li style="box-sizing: border-box; padding: 0px5px;">11</li><li style="box-sizing: border-box; padding: 0px 5px;">12</li><li style="box-sizing: border-box; padding:0px 5px;">13</li><li style="box-sizing: border-box; padding: 0px 5px;">14</li><li style="box-sizing: border-box;padding: 0px 5px;">15</li><li style="box-sizing: border-box; padding: 0px 5px;">16</li><li style="box-sizing:border-box; padding: 0px 5px;">17</li><li style="box-sizing: border-box; padding: 0px 5px;">18</li><listyle="box-sizing: border-box; padding: 0px 5px;">19</li><li style="box-sizing: border-box; padding: 0px5px;">20</li><li style="box-sizing: border-box; padding: 0px 5px;">21</li></ul>
weight_filler [default type: ‘constant’ value: 0]:参数的初始化方法
bias_filler:偏置的初始化方法
bias_term [default true]:指定是否是否开启偏置项
通过全连接层后的大小变化:
输入:n×ci×hi×wi
输出:n×co×1×1
2.5.2 Splitting
Splitting层可以把一个输入blob分离成多个输出blobs。这个用在当需要把一个blob输入到多个输出层的时候。
2.5.3 Flattening
Flatten层是把一个输入的大小为n * c * h * w变成一个简单的向量,其大小为 n * (c*h*w) * 1 * 1。
2.5.4 Reshape
<codeclass="hljs vbscript has-numbering" style="display: block; padding: 0px; color: inherit; box-sizing: border-box; font-family: 'Source Code Pro', monospace;font-size:undefined; white-space: pre; border-radius: 0px; word-wrap: normal; background: transparent;"> layer {name: <span class="hljs-string" style="color: rgb(0, 136, 0); box-sizing: border-box;">"reshape"</span>type: <span class="hljs-string" style="color: rgb(0, 136, 0); box-sizing: border-box;">"Reshape"</span>bottom: <span class="hljs-string" style="color: rgb(0, 136, 0); box-sizing: border-box;">"input"</span>top: <span class="hljs-string" style="color: rgb(0, 136, 0); box-sizing: border-box;">"output"</span>reshape_param {shape {<span class="hljs-keyword" style="color: rgb(0, 0, 136); box-sizing: border-box;">dim</span>: <span class="hljs-number" style="color: rgb(0, 102, 102); box-sizing: border-box;">0</span> # copy the dimension from below<span class="hljs-keyword" style="color: rgb(0, 0, 136); box-sizing: border-box;">dim</span>: <span class="hljs-number" style="color: rgb(0, 102, 102); box-sizing: border-box;">2</span><span class="hljs-keyword" style="color: rgb(0, 0, 136); box-sizing: border-box;">dim</span>: <span class="hljs-number" style="color: rgb(0, 102, 102); box-sizing: border-box;">3</span><span class="hljs-keyword" style="color: rgb(0, 0, 136); box-sizing: border-box;">dim</span>: -<span class="hljs-number" style="color: rgb(0, 102, 102); box-sizing: border-box;">1</span> # infer it from the other dimensions}}}</code><ul class="pre-numbering" style="box-sizing: border-box; position: absolute; width: 50px; top: 0px; left: 0px; margin: 0px; padding: 6px 0px 40px; border-right-width: 1px; border-right-style: solid; border-right-color: rgb(221, 221, 221); list-style: none; text-align: right; background-color: rgb(238, 238, 238);"><li style="box-sizing: border-box; padding: 0px 5px;">1</li><li style="box-sizing: border-box; padding: 0px 5px;">2</li><li style="box-sizing: border-box; padding: 0px 5px;">3</li><li style="box-sizing: border-box; padding: 0px 5px;">4</li><li style="box-sizing: border-box; padding: 0px 5px;">5</li><li style="box-sizing: border-box; padding: 0px 5px;">6</li><li style="box-sizing: border-box; padding: 0px 5px;">7</li><li style="box-sizing: border-box; padding: 0px 5px;">8</li><li style="box-sizing: border-box; padding: 0px 5px;">9</li><li style="box-sizing: border-box; padding: 0px 5px;">10</li><li style="box-sizing: border-box; padding: 0px5px;">11</li><li style="box-sizing: border-box; padding: 0px 5px;">12</li><li style="box-sizing: border-box; padding:0px 5px;">13</li><li style="box-sizing: border-box; padding: 0px 5px;">14</li></ul><ul class="pre-numbering" style="box-sizing: border-box; position: absolute;width: 50px; top: 0px; left: 0px; margin: 0px; padding: 6px 0px 40px; border-right-width: 1px; border-right-style: solid; border-right-color: rgb(221, 221, 221); list-style: none; text-align: right; background-color: rgb(238, 238, 238);"><li style="box-sizing: border-box; padding: 0px 5px;">1</li><li style="box-sizing: border-box; padding: 0px 5px;">2</li><li style="box-sizing: border-box; padding: 0px 5px;">3</li><li style="box-sizing: border-box; padding: 0px 5px;">4</li><li style="box-sizing: border-box; padding: 0px 5px;">5</li><li style="box-sizing: border-box; padding: 0px 5px;">6</li><li style="box-sizing: border-box; padding: 0px 5px;">7</li><li style="box-sizing: border-box; padding: 0px 5px;">8</li><li style="box-sizing: border-box; padding: 0px 5px;">9</li><li style="box-sizing: border-box; padding: 0px 5px;">10</li><li style="box-sizing:border-box; padding: 0px 5px;">11</li><li style="box-sizing: border-box; padding: 0px 5px;">12</li><listyle="box-sizing: border-box; padding: 0px 5px;">13</li><li style="box-sizing: border-box; padding: 0px5px;">14</li></ul>
输出:同样的blob,但是它的维度已经被我们人为地改变,维度的数据由reshap_param定义。
Reshape层被用于改变输入的维度,而不改变输入的具体数据。就像Flatten层一样。只是维度被改变而已,这个过程不涉及数据的拷贝。
输出的维度由ReshapeParam proto控制。可以直接使用数字进行指定。设定输入的某一维到输出blob中去。此外,还有两个数字值得说一下:
0 直接从底层复制。例如,如果是底层是一个2在它的第一维,那么顶层在它的第一维也有一个2。
-1 从其他的数据里面推测这一维应该是多少。
2.5.5 Concatenation
<codeclass="hljs css has-numbering" style="display: block; padding: 0px; color: inherit; box-sizing: border-box; font-family: 'Source Code Pro', monospace;font-size:undefined; white-space: pre; border-radius: 0px; word-wrap: normal; background: transparent;"><span class="hljs-tag" style="color: rgb(0, 0, 0); box-sizing: border-box;">layer</span> <span class="hljs-rules" style="box-sizing: border-box;">{<span class="hljs-rule" style="box-sizing: border-box;"><spanclass="hljs-attribute" style="box-sizing: border-box;">name</span>:<span class="hljs-value" style="box-sizing: border-box; color: rgb(0, 102, 102);"> <span class="hljs-string" style="color: rgb(0, 136, 0); box-sizing: border-box;">"concat"</span>bottom: <span class="hljs-string" style="color: rgb(0, 136, 0); box-sizing: border-box;">"in1"</span>bottom: <span class="hljs-string" style="color: rgb(0, 136, 0); box-sizing: border-box;">"in2"</span>top: <span class="hljs-string" style="color: rgb(0, 136, 0); box-sizing: border-box;">"out"</span>type: <span class="hljs-string" style="color: rgb(0, 136, 0); box-sizing: border-box;">"Concat"</span>concat_param {axis: <span class="hljs-number" style="box-sizing: border-box;">1</span></span></span></span>} }</code><ul class="pre-numbering" style="box-sizing: border-box; position: absolute; width: 50px; top: 0px; left: 0px; margin: 0px; padding: 6px 0px 40px; border-right-width: 1px; border-right-style: solid; border-right-color: rgb(221, 221, 221); list-style: none; text-align: right; background-color: rgb(238, 238, 238);"><li style="box-sizing: border-box; padding: 0px 5px;">1</li><li style="box-sizing: border-box; padding: 0px 5px;">2</li><li style="box-sizing: border-box; padding: 0px 5px;">3</li><li style="box-sizing: border-box; padding: 0px 5px;">4</li><li style="box-sizing: border-box; padding: 0px 5px;">5</li><li style="box-sizing: border-box; padding: 0px 5px;">6</li><li style="box-sizing: border-box; padding: 0px 5px;">7</li><li style="box-sizing: border-box; padding: 0px 5px;">8</li><li style="box-sizing: border-box; padding: 0px 5px;">9</li><li style="box-sizing: border-box; padding: 0px 5px;">10</li></ul><ul class="pre-numbering" style="box-sizing: border-box; position: absolute; width: 50px; top: 0px; left: 0px; margin: 0px; padding: 6px 0px 40px; border-right-width: 1px; border-right-style: solid; border-right-color: rgb(221, 221, 221); list-style: none; text-align: right; background-color: rgb(238, 238, 238);"><li style="box-sizing: border-box; padding: 0px 5px;">1</li><li style="box-sizing: border-box; padding: 0px 5px;">2</li><li style="box-sizing: border-box; padding: 0px 5px;">3</li><li style="box-sizing: border-box; padding: 0px 5px;">4</li><li style="box-sizing: border-box; padding: 0px 5px;">5</li><li style="box-sizing: border-box; padding: 0px 5px;">6</li><li style="box-sizing: border-box; padding: 0px 5px;">7</li><li style="box-sizing: border-box; padding: 0px 5px;">8</li><li style="box-sizing: border-box; padding: 0px 5px;">9</li><li style="box-sizing: border-box; padding: 0px 5px;">10</li></ul>
如果axis = 0: (n1+n2+...+nK)×c1×h×w,需要保证所有输入的ci相同。
如果axis = 1: n1×(c1+c2+...+cK)×h×w,需要保证所有输入的n_i 相同。
通过Concatenation层,可以把多个的blobs链接成一个blob。
2.5.6 Slicing
<codeclass="hljs css has-numbering" style="display: block; padding: 0px; color: inherit; box-sizing: border-box; font-family: 'Source Code Pro', monospace;font-size:undefined; white-space: pre; border-radius: 0px; word-wrap: normal; background: transparent;"><span class="hljs-tag" style="color: rgb(0, 0, 0); box-sizing: border-box;">layer</span> <span class="hljs-rules" style="box-sizing: border-box;">{<span class="hljs-rule" style="box-sizing: border-box;"><spanclass="hljs-attribute" style="box-sizing: border-box;">name</span>:<span class="hljs-value" style="box-sizing: border-box; color: rgb(0, 102, 102);"> <span class="hljs-string" style="color: rgb(0, 136, 0); box-sizing: border-box;">"slicer_label"</span>type: <span class="hljs-string" style="color: rgb(0, 136, 0); box-sizing: border-box;">"Slice"</span>bottom: <span class="hljs-string" style="color: rgb(0, 136, 0); box-sizing: border-box;">"label"</span>## Example of label with a shape N x <span class="hljs-number" style="box-sizing: border-box;">3</span> x <span class="hljs-number" style="box-sizing: border-box;">1</span> x <span class="hljs-number" style="box-sizing: border-box;">1</span>top: <span class="hljs-string" style="color: rgb(0, 136, 0); box-sizing: border-box;">"label1"</span>top: <span class="hljs-string" style="color: rgb(0, 136, 0); box-sizing: border-box;">"label2"</span>top: <span class="hljs-string" style="color: rgb(0, 136, 0); box-sizing: border-box;">"label3"</span>slice_param {axis: <span class="hljs-number" style="box-sizing: border-box;">1</span>slice_point: <span class="hljs-number" style="box-sizing: border-box;">1</span>slice_point: <span class="hljs-number" style="box-sizing: border-box;">2</span></span></span></span>} }</code><ul class="pre-numbering" style="box-sizing: border-box; position: absolute; width: 50px; top: 0px; left: 0px; margin: 0px; padding: 6px 0px 40px; border-right-width: 1px; border-right-style: solid; border-right-color: rgb(221, 221, 221); list-style: none; text-align: right; background-color: rgb(238, 238, 238);"><li style="box-sizing: border-box; padding: 0px 5px;">1</li><li style="box-sizing: border-box; padding: 0px 5px;">2</li><li style="box-sizing: border-box; padding: 0px 5px;">3</li><li style="box-sizing: border-box; padding: 0px 5px;">4</li><li style="box-sizing: border-box; padding: 0px 5px;">5</li><li style="box-sizing: border-box; padding: 0px 5px;">6</li><li style="box-sizing: border-box; padding: 0px 5px;">7</li><li style="box-sizing: border-box; padding: 0px 5px;">8</li><li style="box-sizing: border-box; padding: 0px 5px;">9</li><li style="box-sizing: border-box; padding: 0px 5px;">10</li><li style="box-sizing: border-box; padding: 0px5px;">11</li><li style="box-sizing: border-box; padding: 0px 5px;">12</li><li style="box-sizing: border-box; padding:0px 5px;">13</li><li style="box-sizing: border-box; padding: 0px 5px;">14</li></ul><ul class="pre-numbering" style="box-sizing: border-box; position: absolute;width: 50px; top: 0px; left: 0px; margin: 0px; padding: 6px 0px 40px; border-right-width: 1px; border-right-style: solid; border-right-color: rgb(221, 221, 221); list-style: none; text-align: right; background-color: rgb(238, 238, 238);"><li style="box-sizing: border-box; padding: 0px 5px;">1</li><li style="box-sizing: border-box; padding: 0px 5px;">2</li><li style="box-sizing: border-box; padding: 0px 5px;">3</li><li style="box-sizing: border-box; padding: 0px 5px;">4</li><li style="box-sizing: border-box; padding: 0px 5px;">5</li><li style="box-sizing: border-box; padding: 0px 5px;">6</li><li style="box-sizing: border-box; padding: 0px 5px;">7</li><li style="box-sizing: border-box; padding: 0px 5px;">8</li><li style="box-sizing: border-box; padding: 0px 5px;">9</li><li style="box-sizing: border-box; padding: 0px 5px;">10</li><li style="box-sizing:border-box; padding: 0px 5px;">11</li><li style="box-sizing: border-box; padding: 0px 5px;">12</li><listyle="box-sizing: border-box; padding: 0px 5px;">13</li><li style="box-sizing: border-box; padding: 0px5px;">14</li></ul>
Slice层可以将输入层变成多个输出层。这些输出层沿一个给定的维度存在。axis指定了目标的轴,slice_point则指定了选择维度的序号。
2.5.7 Elementwise Operations
2.5.8 Argmax
2.5.9 Softmax
2.5.10 Mean-Variance Normalization
3. 后语
在这些抽象的基类头文件里,看起来挺累,好在各种搜索,也能学到一些技巧,如,巧用宏定义来简写C,C++代码,使用模板方法,将有大量重复接口和参数的类抽象为一个宏定义,达到简化代码的目的。
4. 参考资料
caffe中的layers相关推荐
- 【深度学习】caffe中那些layers
0. 引子 自从半年前开始进入418实验室研究深度学习,可以说caffe这一深度学习开源框架,一直伴随着我.经过了大半年的学习,虽然还有许多东西尚待完善,但起码对于caffe已经有了一个比较感性的认识 ...
- Caffe中对MNIST执行train操作执行流程解析
之前在 http://blog.csdn.net/fengbingchun/article/details/49849225 中简单介绍过使用Caffe train MNIST的文章,当时只是仿照ca ...
- caffe中各层的作用
caffe中各层的作用: 关于caffe中的solver: cafffe中的sover的方法都有: Stochastic Gradient Descent (type: "SGD" ...
- 怎么在Caffe中配置每一个层的结构
如何在Caffe中配置每一个层的结构 最近刚在电脑上装好Caffe,由于神经网络中有不同的层结构,不同类型的层又有不同的参数,所有就根据Caffe官网的说明文档做了一个简单的总结. 1. Vision ...
- caffe 中base_lr、weight_decay、lr_mult、decay_mult代表什么意思?
在机器学习或者模式识别中,会出现overfitting,而当网络逐渐overfitting时网络权值逐渐变大,因此,为了避免出现overfitting,会给误差函数添加一个惩罚项,常用的惩罚项是所有权 ...
- caffe中的loss函数简单介绍
本文转载自:http://www.cnblogs.com/jianyingzhou/p/4094095.html,感谢作者的热心分享! 进一步了解可以看这篇博文:http://www.cnblogs. ...
- Caffe中的损失函数
文章从自己的QQ(632846506)2年前的日志中移过来的.https://user.qzone.qq.com/632846506/infocenter. 机器学习的目的就是通过对训练样本输出与真实 ...
- caffe中的各种loss函数
转自:https://blog.csdn.net/u012177034/article/details/52144325 机器学习的目的就是通过对训练样本输出与真实值不一致的进行惩罚,得到损失Loss ...
- caffe中的多种loss函数
https://blog.csdn.net/u012177034/article/details/52144325 机器学习的目的就是通过对训练样本输出与真实值不一致的进行惩罚,得到损失Loss,然后 ...
- caffe中forward过程总结
caffe中最重要的两个部分就是forward和backward的过程,farward是根据输入数据正向预测输入属于哪一类:backward是根据输出的结果求得代价函数,然后根据代价函数反向求去其相对 ...
最新文章
- linux网络配置、测试及故障诊断方法和工具二
- 【Matlab 图像】灰度二值化处理
- 闲置硬盘自制nas私有云_闲置U盘不用扔,教你一招变云盘,随时随地备份数据、访问私有云...
- JavaScript 对象和包装类
- 级数求和(信息学奥赛一本通-T1087)
- sqlserver修改链接服务器,sqlserver怎么新建链接服务器
- System verilog随机系统函数$urandom_range使用方法
- 简单的动态JavaScript Ajax函数
- mysql 逗号 join_mysql – sql join表,其中1列有逗号
- 删除一个带有文件的文件夹
- django-celery beat报错 error pid
- python语法基础知识第四关第二题_【python基础语法】第4天作业练习题
- 从数组中取值,并且比较字符串的值
- 顺序表的基本操作(完整代码)
- 地震数据爬取——Scrapy爬虫框架应用
- 百分点大数据技术团队:互联网舆情系统的架构实践
- android逆向分析so,Android逆向——so反编译分析由浅入深(回帖奖励)
- C语言>>百钱百鸡问题。鸡翁一,值钱五;鸡母一,值钱三;鸡雏三,值钱一;百钱买百鸡,翁、母、雏各几何 ?
- python古诗默写_Python网络爬虫:爬取古诗文中的某个制定诗句来实现搜索
- 食用盐的12个美容方法