UFLDL教程: Exercise:Learning color features with Sparse Autoencoders
2024-04-08 20:52:59
Linear Decoders
Deep Learning and Unsupervised Feature Learning Tutorial Solutions
以三层的稀疏编码神经网络而言，在sparse autoencoder中的输出层满足下面的公式
从公式中可以看出，a3的输出值是f函数的输出，而在普通的sparse autoencoder中f函数一般为sigmoid函数，所以其输出值的范围为(0,1)，所以可以知道a3的输出值范围也在0到1之间。
另外我们知道，在稀疏模型中的输出层应该是尽量和输入层特征相同，也就是说a3=x1，这样就可以推导出x1也是在0和1之间，那就是要求我们对输入到网络中的数据要先变换到0和1之间，这一条件虽然在有些领域满足，比如前面实验中的MINIST数字识别。
但是有些领域，比如说使用了PCA Whitening后的数据，其范围却不一定在0和1之间。因此Linear Decoder方法就出现了。Linear Decoder是指在隐含层采用的激发函数是sigmoid函数，而在输出层的激发函数采用的是线性函数，比如说最特别的线性函数——等值函数。此时，也就是说输出层满足下面公式：
一个 S 型或 tanh 隐含层以及线性输出层构成的自编码器，我们称为线性解码器。
随着输出单元的激励函数的改变，这个输出单元梯度也相应变化。回顾之前每一个输出单元误差项定义为：
其中 y = x 是所期望的输出, 是自编码器的输出, 是激励函数.因为在输出层激励函数为 f(z) = z, 这样 f’(z) = 1，所以上述公式可以简化为
当然，若使用反向传播算法来计算隐含层的误差项时:
因为隐含层采用一个 S 型（或 tanh）的激励函数 f,在上述公式中， 依然是 S 型（或 tanh）函数的导数。
这样在用ＢＰ算法进行梯度的求解时，只需要更改误差的计算公式而已，改成如下公式
实验步骤
1.初始化参数，编写计算线性解码器代价函数及其梯度的函数sparseAutoencoderLinearCost.m，主要是在sparseAutoencoderCost.m的基础上稍微修改，然后再检查其梯度实现是否正确。
2.加载数据并原始数据进行ZCA Whitening的预处理。
3.学习特征，即用LBFG算法训练整个线性解码器网络，得到整个网络权值optTheta。
4.可视化第一层学习到的特征。
linearDecoderExercise.m
%% CS294A/CS294W Linear Decoder Exercise% Instructions
% ------------
%
% This file contains code that helps you get started on the
% linear decoder exericse. For this exercise, you will only need to modify
% the code in sparseAutoencoderLinearCost.m. You will not need to modify
% any code in this file.%%======================================================================
%% STEP 0: Initialization
% Here we initialize some parameters used for the exercise.imageChannels = 3; % number of channels (rgb, so 3)patchDim = 8; % patch dimension
numPatches = 100000; % number of patchesvisibleSize = patchDim * patchDim * imageChannels; % number of input units
outputSize = visibleSize; % number of output units
hiddenSize = 400; % number of hidden units sparsityParam = 0.035; % desired average activation of the hidden units.
lambda = 3e-3; % weight decay parameter
beta = 5; % weight of sparsity penalty term epsilon = 0.1; % epsilon for ZCA whitening%%======================================================================
%% STEP 1: Create and modify sparseAutoencoderLinearCost.m to use a linear decoder,
% and check gradients
% You should copy sparseAutoencoderCost.m from your earlier exercise
% and rename it to sparseAutoencoderLinearCost.m.
% Then you need to rename the function from sparseAutoencoderCost to
% sparseAutoencoderLinearCost, and modify it so that the sparse autoencoder
% uses a linear decoder instead. Once that is done, you should check
% your gradients to verify that they are correct.% NOTE: Modify sparseAutoencoderCost first!% To speed up gradient checking, we will use a reduced network and some
% dummy patchesdebugHiddenSize = 5;
debugvisibleSize = 8;
patches = rand([8 10]);%随机产生10个样本，每个样本为一个8维的列向量，元素值为0~1
theta = initializeParameters(debugHiddenSize, debugvisibleSize); [cost, grad] = sparseAutoencoderLinearCost(theta, debugvisibleSize, debugHiddenSize, ...lambda, sparsityParam, beta, ...patches);% Check gradients
numGrad = computeNumericalGradient( @(x) sparseAutoencoderLinearCost(x, debugvisibleSize, debugHiddenSize, ...lambda, sparsityParam, beta, ...patches), theta);% Use this to visually compare the gradients side by side
disp([numGrad grad]); diff = norm(numGrad-grad)/norm(numGrad+grad);
% Should be small. In our implementation, these values are usually less than 1e-9.
disp(diff); assert(diff < 1e-9, 'Difference too large. Check your gradient computation again');% NOTE: Once your gradients check out, you should run step 0 again to
% reinitialize the parameters
%}%%======================================================================
%% STEP 2: Learn features on small patches从pathes中学习特征
% In this step, you will use your sparse autoencoder (which now uses a
% linear decoder) to learn features on small patches sampled from related
% images.%% STEP 2a: Load patches 加载数据
% In this step, we load 100k patches sampled from the STL10 dataset and
% visualize them. Note that these patches have been scaled to [0,1]load stlSampledPatches.mat %里面自己定义了变量patches的值displayColorNetwork(patches(:, 1:100));%% STEP 2b: Apply preprocessing预处理
% In this sub-step, we preprocess the sampled patches, in particular,
% ZCA whitening them.
%
% In a later exercise on convolution and pooling, you will need to replicate
% exactly the preprocessing steps you apply to these patches before
% using the autoencoder to learn features on them. Hence, we will save the
% ZCA whitening and mean image matrices together with the learned features
% later on.% Subtract mean patch (hence zeroing the mean of the patches)
meanPatch = mean(patches, 2); %注意这里减掉的是每一维属性的均值
%为什么是对每行求平均，以前是对每列即每个样本求平均呀？因为以前是灰度图，现在是彩色图，如果现在对每列平均就是对三个通道求平均，这肯定不行
patches = bsxfun(@minus, patches, meanPatch);%每一维都均值化% Apply ZCA whitening
sigma = patches * patches' / numPatches;%协方差矩阵
[u, s, v] = svd(sigma);
ZCAWhite = u * diag(1 ./ sqrt(diag(s) + epsilon)) * u';%求出ZCAWhitening矩阵
patches = ZCAWhite * patches;displayColorNetwork(patches(:, 1:100));%% STEP 2c: Learn features
% You will now use your sparse autoencoder (with linear decoder) to learn
% features on the preprocessed patches. This should take around 45 minutes.theta = initializeParameters(hiddenSize, visibleSize);% Use minFunc to minimize the function
addpath minFunc/options = struct;
options.Method = 'lbfgs';
options.maxIter = 400;
options.display = 'on';[optTheta, cost] = minFunc( @(p) sparseAutoencoderLinearCost(p, ...visibleSize, hiddenSize, ...lambda, sparsityParam, ...beta, patches), ...theta, options);% Save the learned features and the preprocessing matrices for use in
% the later exercise on convolution and pooling
fprintf('Saving learned features and preprocessing matrices...\n');
save('STL10Features.mat', 'optTheta', 'ZCAWhite', 'meanPatch');
fprintf('Saved\n');%% STEP 2d: Visualize learned featuresW = reshape(optTheta(1:visibleSize * hiddenSize), hiddenSize, visibleSize);
b = optTheta(2*hiddenSize*visibleSize+1:2*hiddenSize*visibleSize+hiddenSize);
figure;
%这里为什么要用(W*ZCAWhite)'呢？首先，使用W*ZCAWhite是因为每个样本x输入网络，
%其输出等价于W*ZCAWhite*x；另外，由于W*ZCAWhite的每一行才是一个隐含节点的变换值
%而displayColorNetwork函数是把每一列显示一个小图像块的，所以需要对其转置。displayColorNetwork( (W*ZCAWhite)');
sparseAutoencoderLinearCost.m
function [cost,grad,features] = sparseAutoencoderLinearCost(theta, visibleSize, hiddenSize, ...lambda, sparsityParam, beta, data)
% -------------------- YOUR CODE HERE --------------------
% Instructions:
% Copy sparseAutoencoderCost in sparseAutoencoderCost.m from your
% earlier exercise onto this file, renaming the function to
% sparseAutoencoderLinearCost, and changing the autoencoder to use a
% linear decoder.
% -------------------- YOUR CODE HERE -------------------- %计算线性解码器代价函数及其梯度
% visibleSize:输入层神经单元节点数
% hiddenSize:隐藏层神经单元节点数
% lambda: 权重衰减系数
% sparsityParam: 稀疏性参数
% beta: 稀疏惩罚项的权重
% data: 训练集
% theta：参数向量，包含W1、W2、b1、b2W1 = reshape(theta(1:hiddenSize*visibleSize), hiddenSize, visibleSize);
W2 = reshape(theta(hiddenSize*visibleSize+1:2*hiddenSize*visibleSize), visibleSize, hiddenSize);
b1 = theta(2*hiddenSize*visibleSize+1:2*hiddenSize*visibleSize+hiddenSize);
b2 = theta(2*hiddenSize*visibleSize+hiddenSize+1:end);% Loss and gradient variables (your code needs to compute these values)
m = size(data, 2); % 样本数量%% ---------- YOUR CODE HERE --------------------------------------
% Instructions: Compute the loss for the Sparse Autoencoder and gradients
% W1grad, W2grad, b1grad, b2grad
%
% Hint: 1) data(:,i) is the i-th example
% 2) your computation of loss and gradients should match the size
% above for loss, W1grad, W2grad, b1grad, b2grad% z2 = W1 * x + b1
% a2 = f(z2)
% z3 = W2 * a2 + b2
% h_Wb = a3 = f(z3)z2 = W1 * data + repmat(b1, [1, m]);
a2 = sigmoid(z2);
z3 = W2 * a2 + repmat(b2, [1, m]);
a3 = z3;rhohats = mean(a2,2);
rho = sparsityParam;
KLsum = sum(rho * log(rho ./ rhohats) + (1-rho) * log((1-rho) ./ (1-rhohats)));squares = (a3 - data).^2;
squared_err_J = (1/2) * (1/m) * sum(squares(:)); %均方差项
weight_decay_J = (lambda/2) * (sum(W1(:).^2) + sum(W2(:).^2));%权重衰减项
sparsity_J = beta * KLsum; %惩罚项cost = squared_err_J + weight_decay_J + sparsity_J;%损失函数值% delta3 = -(data - a3) .* fprime(z3);
% but fprime(z3) = a3 * (1-a3)
delta3 = -(data - a3);
beta_term = beta * (- rho ./ rhohats + (1-rho) ./ (1-rhohats));
delta2 = ((W2' * delta3) + repmat(beta_term, [1,m]) ) .* a2 .* (1-a2);W2grad = (1/m) * delta3 * a2' + lambda * W2; % W2梯度
b2grad = (1/m) * sum(delta3, 2); % b2梯度
W1grad = (1/m) * delta2 * data' + lambda * W1; % W1梯度
b1grad = (1/m) * sum(delta2, 2); % b1梯度%-------------------------------------------------------------------
% Convert weights and bias gradients to a compressed form
% This step will concatenate and flatten all your gradients to a vector
% which can be used in the optimization method.
grad = [W1grad(:) ; W2grad(:) ; b1grad(:) ; b2grad(:)];%-------------------------------------------------------------------
% We are giving you the sigmoid function, you may find this function
% useful in your computation of the loss and the gradients.
function sigm = sigmoid(x)sigm = 1 ./ (1 + exp(-x));
endend
displayColorNetwork.m
function displayColorNetwork(A)% display receptive field(s) or basis vector(s) for image patches
%
% A the basis, with patches as column vectors% In case the midpoint is not set at 0, we shift it dynamically
if min(A(:)) >= 0 A = A - mean(A(:));%0均值化
endcols = round(sqrt(size(A, 2)));% 每行大图像中小图像块的个数channel_size = size(A,1) / 3;
dim = sqrt(channel_size);% 小图像块内每行或列像素点个数
dimp = dim+1;
rows = ceil(size(A,2)/cols);% 每列大图像中小图像块的个数
B = A(1:channel_size,:);% R通道像素值
C = A(channel_size+1:channel_size*2,:);% G通道像素值
D = A(2*channel_size+1:channel_size*3,:);% B通道像素值
B=B./(ones(size(B,1),1)*max(abs(B)));% 归一化
C=C./(ones(size(C,1),1)*max(abs(C)));
D=D./(ones(size(D,1),1)*max(abs(D)));
% Initialization of the image
I = ones(dim*rows+rows-1,dim*cols+cols-1,3);%Transfer features to this image matrix
for i=0:rows-1for j=0:cols-1if i*cols+j+1 > size(B, 2)breakend% This sets the patchI(i*dimp+1:i*dimp+dim,j*dimp+1:j*dimp+dim,1) = ...reshape(B(:,i*cols+j+1),[dim dim]);I(i*dimp+1:i*dimp+dim,j*dimp+1:j*dimp+dim,2) = ...reshape(C(:,i*cols+j+1),[dim dim]);I(i*dimp+1:i*dimp+dim,j*dimp+1:j*dimp+dim,3) = ...reshape(D(:,i*cols+j+1),[dim dim]);end
endI = I + 1;% 使I的范围从[-1,1]变为[0，2]
I = I / 2;% 使I的范围从[0，2]变为[0, 1]
imagesc(I);
axis equal% 等比坐标轴：设置屏幕高宽比，使得每个坐标轴的具有均匀的刻度间隔
axis off% 关闭所有的坐标轴标签、刻度、背景end参考文献
Exercise:Learning color features with Sparse Autoencoders
Deep learning：十七(Linear Decoders，Convolution和Pooling)
线性解码器
吴恩达 Andrew Ng 的公开课
UFLDL教程: Exercise:Learning color features with Sparse Autoencoders相关推荐
- 【DeepLearning】Exercise:Learning color features with Sparse Autoencoders
Exercise:Learning color features with Sparse Autoencoders 习题链接:Exercise:Learning color features with ...
- 深度学习笔记6：Learning color features with Sparse Autoencoders
线性解码器 动机 当采用稀疏自编码器的输出层采用sigmoid作为激励函数时,要求对输入进行缩放或限制,使其位于[0,1]范围中.但是有些输入值很难满足要求,如PCA白化处理的输入并不满足该范围要求. ...
- UFLDL教程: Exercise:Self-Taught Learning
自我学习 Deep Learning and Unsupervised Feature Learning Tutorial Solutions 1.先训练稀疏自编码器提取特征,再把特征和label给s ...
- UFLDL教程: Exercise: Implement deep networks for digit classification
Deep networks Deep Learning and Unsupervised Feature Learning Tutorial Solutions 深度网络的优势 比单层神经网络能学习到 ...
- Python数据处理 PCA/ZCA 白化(UFLDL教程:Exercise:PCA_in_2DPCA_and_Whitening)
Python数据处理 PCA/ZCA 白化 参考材料 PCA.白化 以及一份别人的课后作业答案 UFLDL教程答案(3):Exercise:PCA_in_2D&PCA_and_Whitenin ...
- UFLDL教程：数据预处理
数据预处理是深度学习中非常重要的一步!如果说原始数据的获得,是深度学习中最重要的一步,那么获得原始数据之后对它的预处理更是重要的一部分. 一般来说,算法的好坏一定程度上和数据是否归一化,是否白化有关. ...
- Deep Learning 9_深度学习UFLDL教程：linear decoder_exercise（斯坦福大学深度学习教程）...
前言 实验内容:Exercise:Learning color features with Sparse Autoencoders.即:利用线性解码器,从100000张8*8的RGB图像块中提取颜色特 ...
- 斯坦福大学UFLDL教程列表
UFLDL教程 说明:本教程将阐述无监督特征学习和深度学习的主要观点.通过学习,你也将实现多个功能学习/深度学习算法,能看到它们为你工作,并学习如何应用/适应这些想法到新问题上. 本教程假定机器学习的 ...
- UFLDL教程： Exercise: Sparse Autoencoder
自编码可以跟PCA 一样,给特征属性降维 一些matlab函数 bsxfun:C=bsxfun(fun,A,B)表达的是两个数组A和B间元素的二值操作,fun是函数句柄或者m文件,或者是内嵌的函数.在 ...
最新文章
- 神奇的requestAnimationFrame
- 分叉币众多，为什么说只有BCH能活下来？
- java中class_JAVA中的Class类
- php 函数分类,PHP Array 函数
- MongoDB非关系型数据库开发手册
- Jsoup获取所有链接
- 如何识别一个人是web前端程序员
- ShellCode初体验
- 03-19 分布式测试-Selenium Grid
- 无所不在的性能测试——《LoadRunner 没有告诉你的》之五
- 菜鸟谷歌浏览器打印组件技术分析
- windows11（win10大部分通用）系统C盘清理 | 深度优化
- 手把手教你Web数据可视化平台搭建
- 不懂英文能靠建站卖站达到月入上万？
- 2020最新整理：好用的免费/收费的第三方域名解析服务平台
- NB-IoT和eMTC对比
- 画E-R图·数据库笔记(四)
- Kerberos HA高可用配置
- 动态规划专题（三）——数位DP
- python下划线怎么输入_python中下划线的用法