BP网络反向传播

1.原理
2.Code
- bc.m
- fc.m
- bp_batch.m
- bp_online.m

本博客代码为本人选修《深度学习引论》课程所用，源代码与课件截图属于Zhang Yi教授。github地址
本篇仅为鄙人自己时隔一年整理的屑作，如有错误请见谅。

上一篇：【深度学习】深度学习门前徘徊——正向传播

1.原理

图1.1

图1.2

图1.3

好的，这大抵就是梯度下降了。训练模型的目的是为了降低误差，相当于研究自变量为www，因变量为JJJ的函数，如上图1.1（其中α\alphaα即学习率）：

w(k+1)=w(k)−αk⋅∂J∂w∣w(k)w(k+1)=w(k)-\alpha_k \cdot \left.\frac{\partial J}{\partial w}\right|_{w(k)}w(k+1)=w(k)−αk⋅∂w∂J∣∣∣∣w(k)

本篇同样研究XOR函数，在上一篇中是直接输入参数得到了以下表达式：

同样的先准备好data与label：

data   = [1 0 0 10 1 0 1]; % samples
labels = [1 1 0 0]; % labels
m = size(data, 2);

设定α\alphaα与epochsepochsepochs、初始化权值

alpha = 0.1;
epochs = 50000;
w1 = randn(2,3);
w2 = randn(1,3);

那么开始！先正向传播跑一遍：（函数内容后面有）

% forward calculation (invoke fc)a1 = data(:,i);[a2, z2] = fc(w1, [a1;1]);[a3, z3] = fc(w2, [a2;1]);% calculate cost functionJ(t) = power((a3 - labels(i)), 2)/2;

注意，我们每次用aia_iai与权值相乘后得到ai+1a_{i+1}ai+1，对ai+1a_{i+1}ai+1激活得到zi+1z_{i+1}zi+1。

下面准备反向传播：

图2.1	图2.2	图2.3
图2.4	图2.5	图2.6

看上图2.3，我们定义δil=∂J∂zil\delta^l_i= \frac{\partial J}{\partial z^l_i}δil=∂zil∂J。那么我们就发现：

那么我们就要算出每一层的δ\deltaδ，首先，如何计算最后一层的δ\deltaδ呢？（见图2.5）

那么向后传播中其他层的δ\deltaδ呢？（见图2.6）

上代码：

delta3 = (a3 - labels(i))*df(z3);
delta2 = bc(w2, z2, delta3);

其中function bc()为：

function delta = bc(w, z, delta_next)df = @(s) f(s) .* (1 - f(s));delta = df(z)*( w*delta_next );
end

好了，那么接下来更新权值就好：

% cumulate the total gradientsdw1 = dw1 + [a1;1] * delta2(1:2);dw2 = dw2 + [a2;1] * delta3;
% update weightsw1 = w1 - alpha * dw1';w2 = w2 - alpha * dw2';

接下来，考虑epochs与m的循环完善代码即可。

一页总结

2.Code

bc.m

function delta = bc(w, z, delta_next)% define the activation functionf = @(s) 1 ./ (1 + exp(-s));% define the derivative of activation functiondf = @(s) f(s) .* (1 - f(s));%%%%%%%%%%%%%%%%%%%%%%%%%%%%%% Your code BELOW%%%%%%%%%%%%%%%%%%%%%%%%%%%%%% backward computing (in either component or vector form)delta = df(z)*( w*delta_next );%%%%%%%%%%%%%%%%%%%%%%%%%%%%%% Your code ABOVE%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
end

fc.m


function [a_next, z_next] = fc(w, a)% define the activation functionf = @(s) 1 ./ (1 + exp(-s));%%%%%%%%%%%%%%%%%%%%%%%%%%%%%% Your code BELOW%%%%%%%%%%%%%%%%%%%%%%%%%%%%%% forward computing (in either component or vector form)z_next = w * a;a_next = f( z_next );%%%%%%%%%%%%%%%%%%%%%%%%%%%%%% Your code ABOVE%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
end

bp_batch.m


% clear the workspace
clear% define the activation function
f = @(s) 1 ./ (1 + exp(-s));
% define the derivative of activation function
df = @(s) f(s) .* (1 - f(s));% prepare the training data set
data   = [1 0 0 10 1 0 1]; % samples
labels = [1 1 0 0]; % labels
m = size(data, 2);% choose parameters, initialize the weights
alpha = 0.1;
epochs = 50000;
w1 = randn(2,3);
w2 = randn(1,3);
J = zeros(1,epochs);% loop until weights converge
for t = 1:epochs% reset the total gradientsdw1 = 0;dw2 = 0;%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
% Your code BELOW
%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
% for all samplesfor i = 1:m% forward calculation (invoke fc)a1 = data(:,i);[a2, z2] = fc(w1, [a1;1]);[a3, z3] = fc(w2, [a2;1]);% calculate cost functionJ(t) = power((a3 - labels(i)), 2)/2;% backwork calculation (invoke bc)delta3 = (a3 - labels(i))*df(z3);delta2 = bc(w2, z2, delta3);% cumulate the total gradientsdw1 = dw1 + [a1;1] * delta2(1:2);dw2 = dw2 + [a2;1] * delta3;        % --- ATTENTION --- % end for all samplesend% update weightsw1 = w1 - alpha * dw1';w2 = w2 - alpha * dw2';%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
% Your code ABOVE
%%%%%%%%%%%%%%%%%%%%%%%%%%%%%% end loopif mod(t,100) == 0fprintf('%i/%i epochs: J=%.4f\n', t, epochs, J(t));end
end% display the result
for i = 1:4a1 = data(:,i);[a2, z2] = fc(w1, [a1;1]);[a3, z3] = fc(w2, [a2;1]);fprintf('Sample [%i %i] (%i) is classified as %i.\n', data(1,i), data(2,i), labels(i), a3>0.5);
end

bp_online.m


% clear the workspace
clear% define the activation function
f = @(s) 1 ./ (1 + exp(-s));
% define the derivative of activation function
df = @(s) f(s) .* (1 - f(s));% prepare the training data set
data   = [1 0 0 10 1 0 1]; % samples
labels = [1 1 0 0]; % labels
m = size(data, 2);% choose parameters, initialize the weights
alpha = 0.1;
epochs = 50000;
w1 = randn(2,3);
w2 = randn(1,3);
J = zeros(1,epochs);% loop until weights converge
for t = 1:epochs%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
% Your code BELOW
%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
% for each samplesfor i = 1:m% forward calculation (invoke fc)a1 = data(:,i);[a2, z2] = fc(w1, [a1;1]);[a3, z3] = fc(w2, [a2;1]);% calculate cost functionJ(t) = power((a3 - labels(i)), 2)/2;% backwork calculation (invoke bc)delta3 = (a3 - labels(i))*df(z3);       % P29delta2 = bc(w2, z2, delta3);% calculate the gradientsdw1 = [a1;1] * delta2(1:2);dw2 = [a2;1] * delta3;% update weightsw1 = w1 - alpha * dw1';w2 = w2 - alpha * dw2';% end for each sampleend%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
% Your code ABOVE
%%%%%%%%%%%%%%%%%%%%%%%%%%%%%% end loopif mod(t,100) == 0fprintf('%i/%i epochs: J=%.4f\n', t, epochs, J(t));end
end% display the result
for i = 1:4a1 = data(:,i);[a2, z2] = fc(w1, [a1;1]);[a3, z3] = fc(w2, [a2;1]);fprintf('Sample [%i %i] (%i) is classified as %i.\n', data(1,i), data(2,i), labels(i), a3>0.5);
end

【深度学习】深度学习入门——BP网络反向传播相关推荐

神经网络训练中的Tricks之高效BP（反向传播算法）
神经网络训练中的Tricks之高效BP(反向传播算法) zouxy09@qq.com http://blog.csdn.net/zouxy09 Tricks!这是一个让人听了充满神秘和好奇的词.对于我 ...
转载：BP（反向传播算法）公式推导及例题解析
首发于深度学习与我的那些事写文章 BP(反向传播算法)公式推导及例题解析 WILL 深度学习搬砖者关注他 297 人赞同了该文章写在前面:最近赶上<模式识别>课程考试,就把B ...
深度学习之学习笔记（九）—— 误差反向传播法
误差反向传播法() 对神经网络模型而言,梯度下降法需要计算损失函数对参数的偏导数,如果用链式法则对每个参数逐一求偏导,这是一个非常艰巨的任务!这是因为: 模型参数非常多--现在的神经网络中经常会有上亿 ...
TensorFlow 2.0深度学习算法实战第七章反向传播算法
第七章反向传播算法 7.1 导数与梯度 7.2 导数常见性质 7.2.1 基本函数的导数 7.2.2 常用导数性质 7.2.3 导数求解实战 7.3 激活函数导数 7.3.1 Sigmoid 函数导 ...
深度学习与计算机视觉（四）反向传播及其直观理解
四.反向传播及其直观理解 4.1 引言问题描述和动机: 大家都知道,其实我们就是在给定的图像像素向量x和对应的函数f(x)f(x)f(x),然后我们希望能够计算fff在x上的梯度∇f(x)" ...
神经网络之BP（反向传播算法）的原理介绍
反向传播是人工神经网络中的一个重要算法,这个方法可以对网络中所有权重计算损失函数的梯度,然后这个梯度会反馈给最优化方法,用来更新权值以最小化损失函数. 反向传播算法的原理我们先直观的看一下反向传播的 ...
BP神经网络(反向传播算法原理、推导过程、计算步骤)
BP神经网络 1.反向传播算法的原理 2.反向传播算法参数学习的推导 3.反向传播算法参数更新案例 3.1 反向传播的具体计算步骤 3.1.1 计算输出层的误差 3.1.2 计算隐藏层误差 3.1.3 ...
人工神经网络——【BP】反向传播算法证明
第一步:前向传播 [注]此BP算法的证明仅限sigmoid激活函数情况.本博文讲道理是没错的,毕竟最后还利用代码还核对了一次理论证明结果. 关于更为严谨的BP证明,即严格通过上下标证明BP的博客请戳这 ...

【深度学习】深度学习入门——BP网络反向传播