Stochastic Pooling简单理解
本文转自: http://www.cnblogs.com/tornadomeet/p/3432093.html
CNN中卷积完后有个步骤叫pooling, 在ICLR2013上,作者Zeiler提出了另一种pooling手段(最常见的就是mean-pooling和max-pooling),叫stochastic pooling,在他的文章还给出了效果稍差点的probability weighted pooling方法。
stochastic pooling方法非常简单,只需对feature map中的元素按照其概率值大小随机选择,即元素值大的被选中的概率也大。而不像max-pooling那样,永远只取那个最大值元素。
假设feature map中的pooling区域元素值如下:
3*3大小的,元素值和sum=0+1.1+2.5+0.9+2.0+1.0+0+1.5+1.0=10
方格中的元素同时除以sum后得到的矩阵元素为:
每个元素值表示对应位置处值的概率,现在只需要按照该概率来随机选一个,方法是:将其看作是9个变量的多项式分布,然后对该多项式分布采样即可,theano中有直接的multinomial()来函数完成。当然也可以自己用01均匀分布来采样,将单位长度1按照那9个概率值分成9个区间(概率越大,覆盖的区域越长,每个区间对应一个位置),然随机生成一个数后看它落在哪个区间。
比如如果随机采样后的矩阵为:
则这时候的poolng值为1.5
使用stochastic pooling时(即test过程),其推理过程也很简单,对矩阵区域求加权平均即可。比如对上面的例子求值过程为为:
0*0+1.1*0.11+2.5*0.25+0.9*0.09+2.0*0.2+1.0*0.1+0*0+1.5*0.15+1.0*0.1=1.625 说明此时对小矩形pooling后的结果为1.625.
在反向传播求导时,只需保留前向传播已经记录被选中节点的位置的值,其它值都为0,这和max-pooling的反向传播非常类似。
Stochastic pooling优点:
方法简单;
泛化能力更强;
可用于卷积层(文章中是与Dropout和DropConnect对比的,说是Dropout和DropConnect不太适合于卷积层. 不过个人感觉这没什么可比性,因为它们在网络中所处理的结构不同);
至于为什么stochastic pooling效果好,作者说该方法也是模型平均的一种,没怎么看懂。
关于Stochastic Pooling的前向传播过程和推理过程的代码可参考(没包括bp过程,所以代码中pooling选择的位置没有保存下来)
源码:pylearn2/stochastic_pool.py
""" An implementation of stochastic max-pooling, based onStochastic Pooling for Regularization of Deep Convolutional Neural Networks Matthew D. Zeiler, Rob Fergus, ICLR 2013 """__authors__ = "Mehdi Mirza" __copyright__ = "Copyright 2010-2012, Universite de Montreal" __credits__ = ["Mehdi Mirza", "Ian Goodfellow"] __license__ = "3-clause BSD" __maintainer__ = "Mehdi Mirza" __email__ = "mirzamom@iro"import numpy import theano from theano import tensor from theano.sandbox.rng_mrg import MRG_RandomStreams as RandomStreams from theano.gof.op import get_debug_valuesdef stochastic_max_pool_bc01(bc01, pool_shape, pool_stride, image_shape, rng = None):"""Stochastic max pooling for training as defined in:Stochastic Pooling for Regularization of Deep Convolutional Neural NetworksMatthew D. Zeiler, Rob Fergusbc01: minibatch in format (batch size, channels, rows, cols),IMPORTANT: All values should be poitiviepool_shape: shape of the pool region (rows, cols)pool_stride: strides between pooling regions (row stride, col stride)image_shape: avoid doing some of the arithmetic in theanorng: theano random stream"""r, c = image_shapepr, pc = pool_shapers, cs = pool_stridebatch = bc01.shape[0] #总共batch的个数channel = bc01.shape[1] #通道个数if rng is None:rng = RandomStreams(2022)# Compute index in pooled space of last needed pool# (needed = each input pixel must appear in at least one pool)def last_pool(im_shp, p_shp, p_strd):rval = int(numpy.ceil(float(im_shp - p_shp) / p_strd))assert p_strd * rval + p_shp >= im_shpassert p_strd * (rval - 1) + p_shp < im_shpreturn rval #表示pool过程中需要移动的次数return T.dot(x, self._W)# Compute starting row of the last poollast_pool_r = last_pool(image_shape[0] ,pool_shape[0], pool_stride[0]) * pool_stride[0] #最后一个pool的起始位置# Compute number of rows needed in image for all indexes to work outrequired_r = last_pool_r + pr #满足上面pool条件时所需要image的高度 last_pool_c = last_pool(image_shape[1] ,pool_shape[1], pool_stride[1]) * pool_stride[1]required_c = last_pool_c + pc# final result shaperes_r = int(numpy.floor(last_pool_r/rs)) + 1 #最后pool完成时图片的shaperes_c = int(numpy.floor(last_pool_c/cs)) + 1for bc01v in get_debug_values(bc01):assert not numpy.any(numpy.isinf(bc01v))assert bc01v.shape[2] == image_shape[0]assert bc01v.shape[3] == image_shape[1]# padding,如果不能整除移动,需要对原始图片进行扩充padded = tensor.alloc(0.0, batch, channel, required_r, required_c)name = bc01.nameif name is None:name = 'anon_bc01'bc01 = tensor.set_subtensor(padded[:,:, 0:r, 0:c], bc01)bc01.name = 'zero_padded_' + name# unravelingwindow = tensor.alloc(0.0, batch, channel, res_r, res_c, pr, pc)window.name = 'unravlled_winodows_' + namefor row_within_pool in xrange(pool_shape[0]):row_stop = last_pool_r + row_within_pool + 1for col_within_pool in xrange(pool_shape[1]):col_stop = last_pool_c + col_within_pool + 1win_cell = bc01[:,:,row_within_pool:row_stop:rs, col_within_pool:col_stop:cs]window = tensor.set_subtensor(window[:,:,:,:, row_within_pool, col_within_pool], win_cell) #windows中装的是所有的pooling数据块# find the normnorm = window.sum(axis = [4, 5]) #求和当分母用 norm = tensor.switch(tensor.eq(norm, 0.0), 1.0, norm) #如果norm为0,则将norm赋值为1norm = window / norm.dimshuffle(0, 1, 2, 3, 'x', 'x') #除以norm得到每个位置的概率# get probprob = rng.multinomial(pvals = norm.reshape((batch * channel * res_r * res_c, pr * pc)), dtype='float32') #multinomial()函数能够按照pvals产生多个多项式分布,元素值为0或1# selectres = (window * prob.reshape((batch, channel, res_r, res_c, pr, pc))).max(axis=5).max(axis=4) #window和后面的矩阵相乘是点乘,即对应元素相乘,numpy矩阵符号res.name = 'pooled_' + namereturn tensor.cast(res, theano.config.floatX)def weighted_max_pool_bc01(bc01, pool_shape, pool_stride, image_shape, rng = None):"""This implements test time probability weighted pooling defined in:Stochastic Pooling for Regularization of Deep Convolutional Neural NetworksMatthew D. Zeiler, Rob Fergusbc01: minibatch in format (batch size, channels, rows, cols),IMPORTANT: All values should be poitiviepool_shape: shape of the pool region (rows, cols)pool_stride: strides between pooling regions (row stride, col stride)image_shape: avoid doing some of the arithmetic in theano"""r, c = image_shapepr, pc = pool_shapers, cs = pool_stridebatch = bc01.shape[0]channel = bc01.shape[1]if rng is None: rng = RandomStreams(2022) # Compute index in pooled space of last needed pool # (needed = each input pixel must appear in at least one pool)def last_pool(im_shp, p_shp, p_strd):rval = int(numpy.ceil(float(im_shp - p_shp) / p_strd))assert p_strd * rval + p_shp >= im_shpassert p_strd * (rval - 1) + p_shp < im_shpreturn rval# Compute starting row of the last poollast_pool_r = last_pool(image_shape[0] ,pool_shape[0], pool_stride[0]) * pool_stride[0]# Compute number of rows needed in image for all indexes to work outrequired_r = last_pool_r + prlast_pool_c = last_pool(image_shape[1] ,pool_shape[1], pool_stride[1]) * pool_stride[1]required_c = last_pool_c + pc# final result shaperes_r = int(numpy.floor(last_pool_r/rs)) + 1res_c = int(numpy.floor(last_pool_c/cs)) + 1for bc01v in get_debug_values(bc01):assert not numpy.any(numpy.isinf(bc01v))assert bc01v.shape[2] == image_shape[0]assert bc01v.shape[3] == image_shape[1]# paddingpadded = tensor.alloc(0.0, batch, channel, required_r, required_c)name = bc01.nameif name is None:name = 'anon_bc01'bc01 = tensor.set_subtensor(padded[:,:, 0:r, 0:c], bc01)bc01.name = 'zero_padded_' + name# unravelingwindow = tensor.alloc(0.0, batch, channel, res_r, res_c, pr, pc)window.name = 'unravlled_winodows_' + namefor row_within_pool in xrange(pool_shape[0]):row_stop = last_pool_r + row_within_pool + 1for col_within_pool in xrange(pool_shape[1]):col_stop = last_pool_c + col_within_pool + 1win_cell = bc01[:,:,row_within_pool:row_stop:rs, col_within_pool:col_stop:cs]window = tensor.set_subtensor(window[:,:,:,:, row_within_pool, col_within_pool], win_cell)# find the normnorm = window.sum(axis = [4, 5])norm = tensor.switch(tensor.eq(norm, 0.0), 1.0, norm)norm = window / norm.dimshuffle(0, 1, 2, 3, 'x', 'x')# averageres = (window * norm).sum(axis=[4,5]) #前面的代码几乎和前向传播代码一样,这里只需加权求和即可res.name = 'pooled_' + namereturn res.reshape((batch, channel, res_r, res_c))
参考资料:
Stochastic Pooling for Regularization of Deep Convolutional Neural Networks. Matthew D. Zeiler, Rob Fergus.
pylearn2/stochastic_pool.py
Stochastic Pooling简单理解相关推荐
- 对CNN中pooling的理解
自己在看论文的过程中结合网上的一些资料,对pooling的一些理解汇总如下,以供参考: 1.pooling主要是在用于图像处理的卷积神经网络中,但随着深层神经网络的发展,pooling相关技术在其他领 ...
- 随机池化(Stochastic Pooling)
前言 CNN中卷积完后有个步骤叫pooling, 在ICLR2013上,作者Zeiler提出了另一种pooling手段(最常见的就是mean-pooling和max-pooling),叫stochas ...
- 人工智能:深度学习算法及应用——简单理解CNN卷积神经网络并python实现(带源码)
深度学习算法及应用 一. 实验目的 二. 实验要求 三. 实验的硬件.软件平台 四. 实验原理 1.1. 深度学习概述 1.2. 深度学习的常见结构 1.3. 卷积神经网络(CNN) **卷积** * ...
- Deep Reinforcement Learning: Pong from Pixels翻译和简单理解
原文链接: http://karpathy.github.io/2016/05/31/rl/ 文章目录 原文链接: 前言 Policy-Gradient结构流程图 Deep Reinforcement ...
- android 点击事件消费,Android View事件分发和消费源码简单理解
Android View事件分发和消费源码简单理解 前言: 开发过程中觉得View事件这块是特别烧脑的,看了好久,才自认为看明白.中间上网查了下singwhatiwanna粉丝的读书笔记,有种茅塞顿开 ...
- 【转载】Deep learning:十九(RBM简单理解)
Deep learning:十九(RBM简单理解) 这篇博客主要用来简单介绍下RBM网络,因为deep learning中的一个重要网络结构DBN就可以由RBM网络叠加而成,所以对RBM的理解有利于我 ...
- 学习:双机热备、集群、负载均衡、SQL故障转移群集简单理解(转)
双机热备.集群.负载均衡.SQL故障转移群集简单理解平常,大家常提到几个技术名词:双机热备.集群.负载均衡.SQL故障转移群集.这里,就我的理解,和大家简单探讨下,有不足或错误之处还请各位指出! 这些 ...
- 字符串匹配算法Java_如何简单理解字符串匹配算法?
这篇文章来说说如何简单理解KMP,BM算法.之前看过一些文章说,KMP算法很难理解. 可我并不觉得. 我反而觉得它容易理解.平时我们写java代码的时候, 判断一个字符串是否存在包含另一个字符串都是直 ...
- python yield的简单理解
2019独角兽企业重金招聘Python工程师标准>>> yield是个生成器,它可以使一个方法变成可迭代的方法,每次迭代返回yield后面的值 简单理解: >>>d ...
- linun——SElinux的简单理解
SElinux简单理解 一.什么是SElinux? SELinux 全称 Security Enhanced Linux (安全强化 Linux),是 MAC (Mandatory Access Co ...
最新文章
- saltstack 安装nginx
- P3455 [POI2007]ZAP-Queries
- 如何:创建公钥/私钥对
- BDTC 2017 | 中国大数据技术大会全日程和演讲者曝光
- 息中间件kafka与activeMQ、rabbitMQ、zeroMQ、rocketMQ的比较
- 李开复:21世纪最需要的7种人才
- 分享好用的Mac系统的FTP工具和SHELL工具
- 数字化项目建设管理难点分析与对策
- 愉快的在vscode上刷leetcode
- Wiki系列(一):Wiki系统选择
- html批量修改 快捷键,html怎么设置快捷键?
- 2007 合成孔径雷达成像算法与实现 各figure的matlab实现(第三章 - 图3.2)
- android L和android M的区别
- 2021年中国家具制造业经营现状分析:营业收入达8004.6亿元,利润总额达433.7亿元[图]
- 浅谈API开发安全之sign有效时间(三)
- python绘制曲面图_Python:有没有办法用Matplotlib绘制“部分”曲面图?
- 百度地图API调用问题-sn 的生成
- Python代码实现飞机大战
- 如今只见当年月,何曾再见当年人?
- CommandNotFoundError: Your shell has not been properly configured to use ‘conda activate‘. To initia