多层感知机BP算法推导

前向计算

含有数据输入层，1个以上隐藏层，1个输出层。各层神经元之间全连接，同一层的神经元之间无连接。

在图中，z(l)=W(l)⋅a(l−1)+b(l)a(l)=f(l)(z(l))z^{(l)}=W^{(l)}\cdot a^{(l-1)}+b^{(l)}\\ a^{(l)}=f^{(l)}(z^{(l)})z(l)=W(l)⋅a(l−1)+b(l)a(l)=f(l)(z(l))
其中f(⋅)f(\cdot)f(⋅)是激励函数，aaa是该层的输出值
变量关系：
z1=g1(x,W1)z2=g2(z1,W2)⋯zl−1=gl−1(zl−2,Wl−1)zl=gl(zl−1,Wl)zl+1=gl+1(zl,Wl+1)⋯zL=gL(zL−1,WL)y=fL(zL)J(W,y)z^{1}=g_{1}(x,W^{1})\\ z^{2}=g_{2}(z^{1},W^{2})\\ \cdots\\ z^{l-1}=g_{l-1}(z^{l-2},W^{l-1})\\ z^{l}=g_{l}(z^{l-1},W^{l})\\ z^{l+1}=g_{l+1}(z^{l},W^{l+1})\\ \cdots\\ z^{L}=g_{L}(z^{L-1},W^{L})\\ y=f_{L}(z^{L})\\ J(W,y)z1=g1(x,W1)z2=g2(z1,W2)⋯zl−1=gl−1(zl−2,Wl−1)zl=gl(zl−1,Wl)zl+1=gl+1(zl,Wl+1)⋯zL=gL(zL−1,WL)y=fL(zL)J(W,y)
变量依赖：
J(W,y)J(W,y)J(W,y)与xxx的依赖关系：J(W,y)=J(W,f(gL(...g2(g1(x,W1),W2)...,WL))J(W,y)=J(W,f(g_{L}(...g_{2}(g_{1}(x,W^{1}),W^{2})...,W^{L}))J(W,y)=J(W,f(gL(...g2(g1(x,W1),W2)...,WL))
J(W,y)J(W,y)J(W,y)与z1z^{1}z1的依赖关系：J(W,y)=J(W,f(gL(...g2(z1,W2)...,WL))J(W,y)=J(W,f(g_{L}(...g_{2}(z^{1},W^{2})...,W^{L}))J(W,y)=J(W,f(gL(...g2(z1,W2)...,WL))
J(W,y)J(W,y)J(W,y)与z2z^{2}z2的依赖关系：J(W,y)=J(W,f(gL(..g3(z2,W3)...,WL))J(W,y)=J(W,f(g_{L}(..g_{3}(z^{2},W^{3})...,W^{L}))J(W,y)=J(W,f(gL(..g3(z2,W3)...,WL))
… …
J(W,y)J(W,y)J(W,y)与zlz^{l}zl的依赖关系：J(W,y)=J(W,f(gL(..gl+1(zl,Wl+1)...,WL))J(W,y)=J(W,f(g_{L}(..g_{l+1}(z^{l},W^{l+1})...,W^{L}))J(W,y)=J(W,f(gL(..gl+1(zl,Wl+1)...,WL))

反向传播

目标是最小化损失函数，通过梯度下降：
W(l)=W(l)−α∂J(W,b)∂W(l)=W(l)−α∂1N∑i=1NJ(W,b;x(i),y(i))∂W(l)b(l)=b(l)−α∂J(W,b)∂b(l)=b(l)−α∂1N∑i=1NJ(W,b;x(i),y(i))∂b(l)W^{(l)}=W^{(l)}-\alpha \frac{\partial J(W,\bm{b})}{\partial W^{(l)}} =W^{(l)}-\alpha \frac{\partial \frac{1}{N}\sum_{i=1}^{N}J(W,\bm{b};\bm{x}^{(i)},y^{(i)})}{\partial W^{(l)}}\\ \bm{b}^{(l)}=\bm{b}^{(l)}-\alpha \frac{\partial J(W,\bm{b})}{\partial \bm{b}^{(l)}} =\bm{b}^{(l)}-\alpha \frac{\partial \frac{1}{N}\sum_{i=1}^{N}J(W,\bm{b};\bm{x}^{(i)},y^{(i)})}{\partial \bm{b}^{(l)}}W(l)=W(l)−α∂W(l)∂J(W,b)=W(l)−α∂W(l)∂N1∑i=1NJ(W,b;x(i),y(i))b(l)=b(l)−α∂b(l)∂J(W,b)=b(l)−α∂b(l)∂N1∑i=1NJ(W,b;x(i),y(i))
局部梯度迭代：

第lll层zlz^{l}zl的梯度为δ(l)\delta^{(l)}δ(l):
δ(l)=∂J(W,b;x,y)∂z(l)=∂z(l+1)∂z(l)⋅∂J(W,b;x,y)∂z(l+1)=∂a(l)∂z(l)⋅∂z(l+1)∂a(l)⋅∂J(W,b;x,y)∂z(l+1)=∂a(l)∂z(l)⋅∂z(l+1)∂a(l)⋅δ(l+1)\delta^{(l)}=\frac{\partial J(W,b;x,y)}{\partial z^{(l)}}=\frac{\partial z^{(l+1)}}{\partial z^{(l)}}\cdot \frac{\partial J(W,b;x,y)}{\partial z^{(l+1)}}\\ =\frac{\partial a^{(l)}}{\partial z^{(l)}}\cdot \frac{\partial z^{(l+1)}}{\partial a^{(l)}}\cdot \frac{\partial J(W,b;x,y)}{\partial z^{(l+1)}}\\ =\frac{\partial a^{(l)}}{\partial z^{(l)}}\cdot \frac{\partial z^{(l+1)}}{\partial a^{(l)}}\cdot \delta^{(l+1)}δ(l)=∂z(l)∂J(W,b;x,y)=∂z(l)∂z(l+1)⋅∂z(l+1)∂J(W,b;x,y)=∂z(l)∂a(l)⋅∂a(l)∂z(l+1)⋅∂z(l+1)∂J(W,b;x,y)=∂z(l)∂a(l)⋅∂a(l)∂z(l+1)⋅δ(l+1)
上述的形式是矩阵优化的形式，下面求具体的某一个连接参数的优化迭代式：
第l+1l+1l+1层的梯度δ(l+1)\delta^{(l+1)}δ(l+1)已知，求此时的lll层的梯度δ(l)\delta^{(l)}δ(l)
对于第jjj个神经元输出值zj(l+1)=∑iai(l)wij(l+1)=∑ifi(l)(zi(l))wij(l+1)z_{j}^{(l+1)}=\sum_{i}a_{i}^{(l)}w_{ij}^{(l+1)}=\sum_{i}f_{i}^{(l)}(z_{i}^{(l)})w_{ij}^{(l+1)}zj(l+1)=∑iai(l)wij(l+1)=∑ifi(l)(zi(l))wij(l+1)，

由上式可得到：
∂zj(l+1)∂zi(l)=∂ail∂zi(l)⋅∂zj(l+1)∂ai(l)=fi′(l)(zi(l))wij(l+1)\frac{\partial z_{j}^{(l+1)}}{\partial z_{i}^{(l)}}=\frac{\partial a_{i}^{l}}{\partial z_{i}^{(l)}}\cdot \frac{\partial z_{j}^{(l+1)}}{\partial a_{i}^{(l)}}=f_{i}^{'(l)}(z_{i}^{(l)})w_{ij}^{(l+1)}∂zi(l)∂zj(l+1)=∂zi(l)∂ail⋅∂ai(l)∂zj(l+1)=fi′(l)(zi(l))wij(l+1)
第lll层第iii个输出值zi(l)z_{i}^{(l)}zi(l)的梯度为：
δi(l)=∂L∂zi(l)=∑j∂zj(l+1)∂zi(l)∂L∂zj(l+1)=∑j∂zj(l+1)∂zi(l)δj(l+1)=∑jfi′(l)(zi(l))wij(l+1)δj(l+1)=fi′(l)(zi(l))∑jwij(l+1)δj(l+1)\delta_{i}^{(l)}=\frac{\partial L}{\partial z_{i}^{(l)}}=\sum_{j}\frac{\partial z_{j}^{(l+1)}}{\partial z_{i}^{(l)}}\frac{\partial L}{\partial z_{j}^{(l+1)}}=\sum_{j}\frac{\partial z_{j}^{(l+1)}}{\partial z_{i}^{(l)}}\delta_{j}^{(l+1)}\\ =\sum_{j}f_{i}^{'(l)}(z_{i}^{(l)})w_{ij}^{(l+1)}\delta_{j}^{(l+1)}=f_{i}^{'(l)}(z_{i}^{(l)})\sum_{j}w_{ij}^{(l+1)}\delta_{j}^{(l+1)}δi(l)=∂zi(l)∂L=j∑∂zi(l)∂zj(l+1)∂zj(l+1)∂L=j∑∂zi(l)∂zj(l+1)δj(l+1)=j∑fi′(l)(zi(l))wij(l+1)δj(l+1)=fi′(l)(zi(l))j∑wij(l+1)δj(l+1)
最后一层输出层的梯度为：δo(L)=∂L∂zo(L)=∂aoL∂zo(L)∂L∂aoL=fo′(L)(z0(L))∂L∂aoL\delta_{o}^{(L)}=\frac{\partial L}{\partial z_{o}^{(L)}}=\frac{\partial a_{o}^{L}}{\partial z_{o}^{(L)}}\frac{\partial L}{\partial a_{o}^{L}}=f_{o}^{'(L)}(z_{0}^{(L)})\frac{\partial L}{\partial a_{o}^{L}}δo(L)=∂zo(L)∂L=∂zo(L)∂aoL∂aoL∂L=fo′(L)(z0(L))∂aoL∂L
梯度更新沿着网络反向计算：

求解zi(l)z_{i}^{(l)}zi(l)对应的权重{wki(l)}k=1K(K\{w_{ki}^{(l)}\}_{k=1}^{K}(K{wki(l)}k=1K(K表示l−1l-1l−1层的神经元个数)和偏置bi(l)b_{i}^{(l)}bi(l)的梯度:
∂J∂wki(l)=∂zi(l)∂wki(l)∂J∂zi(l)=ak(l−1)δi(l)∂J∂bi(l)=∂zi(l)∂bi(l)∂J∂zi(l)=δi(l)\frac{\partial J}{\partial w_{ki}^{(l)}}=\frac{\partial z_{i}^{(l)}}{\partial w_{ki}^{(l)}}\frac{\partial J}{\partial z_{i}^{(l)}}=a_{k}^{(l-1)}\delta_{i}^{(l)}\\ \ \\ \frac{\partial J}{\partial b_{i}^{(l)}}=\frac{\partial z_{i}^{(l)}}{\partial b_{i}^{(l)}}\frac{\partial J}{\partial z_{i}^{(l)}}=\delta_{i}^{(l)}∂wki(l)∂J=∂wki(l)∂zi(l)∂zi(l)∂J=ak(l−1)δi(l) ∂bi(l)∂J=∂bi(l)∂zi(l)∂zi(l)∂J=δi(l)
可总结出BP算法的一般步骤。

MLP的BP算法的步骤

（1）前向计算，并记录zi(l)z_{i}^{(l)}zi(l)
（2）反向计算zi(l)z_{i}^{(l)}zi(l)的梯度δi(l)\delta_{i}^{(l)}δi(l):
先计算输出层：δoL=fo′(L)(z0(L))∂L∂aoL\delta_{o}^{L}=f_{o}^{'(L)}(z_{0}^{(L)})\frac{\partial L}{\partial a_{o}^{L}}δoL=fo′(L)(z0(L))∂aoL∂L
从后向前依次计算：
δi(l)=fi′(l)(zi(l))∑jwij(l+1)δj(l+1)\delta_{i}^{(l)}=f_{i}^{'(l)}(z_{i}^{(l)})\sum_{j}w_{ij}^{(l+1)}\delta_{j}^{(l+1)}δi(l)=fi′(l)(zi(l))j∑wij(l+1)δj(l+1)
（3）计算权重和偏置参数的梯度：
∂J∂wki(l)=∂zi(l)∂wki(l)∂J∂zi(l)=ak(l−1)δi(l)∂J∂bi(l)=∂zi(l)∂bi(l)∂J∂zi(l)=δi(l)\frac{\partial J}{\partial w_{ki}^{(l)}}=\frac{\partial z_{i}^{(l)}}{\partial w_{ki}^{(l)}}\frac{\partial J}{\partial z_{i}^{(l)}}=a_{k}^{(l-1)}\delta_{i}^{(l)}\\ \ \\ \frac{\partial J}{\partial b_{i}^{(l)}}=\frac{\partial z_{i}^{(l)}}{\partial b_{i}^{(l)}}\frac{\partial J}{\partial z_{i}^{(l)}}=\delta_{i}^{(l)}∂wki(l)∂J=∂wki(l)∂zi(l)∂zi(l)∂J=ak(l−1)δi(l) ∂bi(l)∂J=∂bi(l)∂zi(l)∂zi(l)∂J=δi(l)