02.改善深层神经网络：超参数调试、正则化以及优化 W2.优化算法（作业：优化方法）

文章目录

1. 梯度下降
2. mini-Batch 梯度下降
3. 动量
4. Adam
5. 不同优化算法下的模型
- 5.1 Mini-batch梯度下降
- 5.2 带动量的Mini-batch梯度下降
- 5.3 带Adam的Mini-batch梯度下降
- 5.4 对比总结

测试题：参考博文

笔记：02.改善深层神经网络：超参数调试、正则化以及优化 W2.优化算法

导入一些包

import numpy as np
import matplotlib.pyplot as plt
import scipy.io
import math
import sklearn
import sklearn.datasetsfrom opt_utils import load_params_and_grads, initialize_parameters, forward_propagation, backward_propagation
from opt_utils import compute_cost, predict, predict_dec, plot_decision_boundary, load_dataset
from testCases import *%matplotlib inline
plt.rcParams['figure.figsize'] = (7.0, 4.0) # set default size of plots
plt.rcParams['image.interpolation'] = 'nearest'
plt.rcParams['image.cmap'] = 'gray'

1. 梯度下降

（Batch）Gradient Descent 梯度下降对每一层都进行：

W[l]=W[l]−α∗dW[l]W^{[l]} =W^{[l]}-\alpha *d W^{[l]}W[l]=W[l]−α∗dW[l]

b[l]=b[l]−α∗db[l]b^{[l]} =b^{[l]}-\alpha *d b^{[l]}b[l]=b[l]−α∗db[l]

lll 是层号，α\alphaα 是学习率

# GRADED FUNCTION: update_parameters_with_gddef update_parameters_with_gd(parameters, grads, learning_rate):"""Update parameters using one step of gradient descentArguments:parameters -- python dictionary containing your parameters to be updated:parameters['W' + str(l)] = Wlparameters['b' + str(l)] = blgrads -- python dictionary containing your gradients to update each parameters:grads['dW' + str(l)] = dWlgrads['db' + str(l)] = dbllearning_rate -- the learning rate, scalar.Returns:parameters -- python dictionary containing your updated parameters """L = len(parameters) // 2 # number of layers in the neural networks# Update rule for each parameterfor l in range(L):### START CODE HERE ### (approx. 2 lines)parameters["W" + str(l+1)] = parameters['W'+str(l+1)] - learning_rate * grads['dW'+str(l+1)]parameters["b" + str(l+1)] = parameters['b'+str(l+1)] - learning_rate * grads['db'+str(l+1)]### END CODE HERE ###return parameters

Stochastic Gradient Descent 随机梯度下降

每次只用1个样本来更新梯度，当训练集很大的时候，SGD 很快
其寻优过程有震荡

代码差异：

(Batch) Gradient Descent:

X = data_input
Y = labels
parameters = initialize_parameters(layers_dims)
for i in range(0, num_iterations):# Forward propagationa, caches = forward_propagation(X, parameters)# Compute cost.cost = compute_cost(a, Y)# Backward propagation.grads = backward_propagation(a, caches, parameters)# Update parameters.parameters = update_parameters(parameters, grads)

Stochastic Gradient Descent:

X = data_input
Y = labels
parameters = initialize_parameters(layers_dims)
for i in range(0, num_iterations):for j in range(0, m):# Forward propagationa, caches = forward_propagation(X[:,j], parameters)# Compute costcost = compute_cost(a, Y[:,j])# Backward propagationgrads = backward_propagation(a, caches, parameters)# Update parameters.parameters = update_parameters(parameters, grads)

3者的差别在于，一次梯度更新时，用到的样本数量不同

调好参数的 mini-batch 梯度下降，通常优于梯度下降或随机梯度下降（特别是当训练集很大时）

2. mini-Batch 梯度下降

如何从训练集 (X,Y)(X,Y)(X,Y) 建立 mini-batches

步骤1：随机打乱数据，X 和 Y 是同步进行的，保持对应关系

步骤2：切分数据集（每个子集大小为 mini_batch_size，最后一个可能不够，没关系）

# GRADED FUNCTION: random_mini_batchesdef random_mini_batches(X, Y, mini_batch_size = 64, seed = 0):"""Creates a list of random minibatches from (X, Y)Arguments:X -- input data, of shape (input size, number of examples)Y -- true "label" vector (1 for blue dot / 0 for red dot), of shape (1, number of examples)mini_batch_size -- size of the mini-batches, integerReturns:mini_batches -- list of synchronous (mini_batch_X, mini_batch_Y)"""np.random.seed(seed)            # To make your "random" minibatches the same as oursm = X.shape[1]                  # number of training examplesmini_batches = []# Step 1: Shuffle (X, Y)permutation = list(np.random.permutation(m))shuffled_X = X[:, permutation]shuffled_Y = Y[:, permutation].reshape((1,m))# Step 2: Partition (shuffled_X, shuffled_Y). Minus the end case.num_complete_minibatches = math.floor(m/mini_batch_size) # number of mini batches of size mini_batch_size in your partitionningfor k in range(0, num_complete_minibatches):### START CODE HERE ### (approx. 2 lines)mini_batch_X = X[:, k*mini_batch_size : (k+1)*mini_batch_size]mini_batch_Y = Y[:, k*mini_batch_size : (k+1)*mini_batch_size]### END CODE HERE ###mini_batch = (mini_batch_X, mini_batch_Y)mini_batches.append(mini_batch)# Handling the end case (last mini-batch < mini_batch_size)if m % mini_batch_size != 0:### START CODE HERE ### (approx. 2 lines)mini_batch_X = X[:, num_complete_minibatches*mini_batch_size : ]mini_batch_Y = Y[:, num_complete_minibatches*mini_batch_size : ]### END CODE HERE ###mini_batch = (mini_batch_X, mini_batch_Y)mini_batches.append(mini_batch)return mini_batches

3. 动量

带动量的梯度下降可以降低 mini-batch 梯度下降时的震荡

原因：Momentum 考虑过去的梯度对当前的梯度进行平滑，梯度不会剧烈变化

初始化梯度的初速度为 0

# GRADED FUNCTION: initialize_velocitydef initialize_velocity(parameters):"""Initializes the velocity as a python dictionary with:- keys: "dW1", "db1", ..., "dWL", "dbL" - values: numpy arrays of zeros of the same shape as the corresponding gradients/parameters.Arguments:parameters -- python dictionary containing your parameters.parameters['W' + str(l)] = Wlparameters['b' + str(l)] = blReturns:v -- python dictionary containing the current velocity.v['dW' + str(l)] = velocity of dWlv['db' + str(l)] = velocity of dbl"""L = len(parameters) // 2 # number of layers in the neural networksv = {}# Initialize velocityfor l in range(L):### START CODE HERE ### (approx. 2 lines)v["dW" + str(l+1)] = np.zeros(parameters['W'+str(l+1)].shape)v["db" + str(l+1)] = np.zeros(parameters['b'+str(l+1)].shape)### END CODE HERE ###return v

对每一层，更新动量
{vdW[l]=βvdW[l]+(1−β)dW[l]W[l]=W[l]−αvdW[l]\begin{cases} v_{dW^{[l]}} = \beta v_{dW^{[l]}} + (1 - \beta) dW^{[l]} \\ W^{[l]} = W^{[l]} - \alpha v_{dW^{[l]}} \end{cases}{vdW[l]=βvdW[l]+(1−β)dW[l]W[l]=W[l]−αvdW[l]

{vdb[l]=βvdb[l]+(1−β)db[l]b[l]=b[l]−αvdb[l]\begin{cases} v_{db^{[l]}} = \beta v_{db^{[l]}} + (1 - \beta) db^{[l]} \\ b^{[l]} = b^{[l]} - \alpha v_{db^{[l]}} \end{cases}{vdb[l]=βvdb[l]+(1−β)db[l]b[l]=b[l]−αvdb[l]

# GRADED FUNCTION: update_parameters_with_momentumdef update_parameters_with_momentum(parameters, grads, v, beta, learning_rate):"""Update parameters using MomentumArguments:parameters -- python dictionary containing your parameters:parameters['W' + str(l)] = Wlparameters['b' + str(l)] = blgrads -- python dictionary containing your gradients for each parameters:grads['dW' + str(l)] = dWlgrads['db' + str(l)] = dblv -- python dictionary containing the current velocity:v['dW' + str(l)] = ...v['db' + str(l)] = ...beta -- the momentum hyperparameter, scalarlearning_rate -- the learning rate, scalarReturns:parameters -- python dictionary containing your updated parameters v -- python dictionary containing your updated velocities"""L = len(parameters) // 2 # number of layers in the neural networks# Momentum update for each parameterfor l in range(L):### START CODE HERE ### (approx. 4 lines)# compute velocitiesv["dW" + str(l+1)] = beta* v["dW" + str(l+1)] + (1-beta)*grads['dW' + str(l+1)]v["db" + str(l+1)] = beta* v["db" + str(l+1)] + (1-beta)*grads['db' + str(l+1)]# update parametersparameters["W" + str(l+1)] = parameters["W" + str(l+1)] - learning_rate*v["dW" + str(l+1)]parameters["b" + str(l+1)] = parameters["b" + str(l+1)] - learning_rate*v["db" + str(l+1)]### END CODE HERE ###return parameters, v

注意：

速度 v 初始化为 0，算法需要几次迭代后才能把 v 加上来，然后开始采用大的步长
β=0\beta = 0β=0 就是不带动量的标准梯度下降

如何选择 β\betaβ：

β\betaβ 越大，考虑的过去的梯度越多，梯度输出也更光滑，太大也不行，过度光滑
经常取值为 0.8 - 0.999 之间，如果不确定，0.9 是个合理的默认值
参数验证选取，看其如何影响损失函数

4. Adam

参看笔记

对每一层：
{vdW[l]=β1vdW[l]+(1−β1)∂J∂W[l]vdW[l]corrected=vdW[l]1−(β1)tsdW[l]=β2sdW[l]+(1−β2)(∂J∂W[l])2sdW[l]corrected=sdW[l]1−(β2)tW[l]=W[l]−αvdW[l]correctedsdW[l]corrected+ε\begin{cases} v_{dW^{[l]}} = \beta_1 v_{dW^{[l]}} + (1 - \beta_1) \frac{\partial \mathcal{J} }{ \partial W^{[l]} } \\ v^{corrected}_{dW^{[l]}} = \frac{v_{dW^{[l]}}}{1 - (\beta_1)^t} \\ s_{dW^{[l]}} = \beta_2 s_{dW^{[l]}} + (1 - \beta_2) (\frac{\partial \mathcal{J} }{\partial W^{[l]} })^2 \\ s^{corrected}_{dW^{[l]}} = \frac{s_{dW^{[l]}}}{1 - (\beta_2)^t} \\ W^{[l]} = W^{[l]} - \alpha \frac{v^{corrected}_{dW^{[l]}}}{\sqrt{s^{corrected}_{dW^{[l]}}} + \varepsilon} \end{cases}⎩⎪⎪⎪⎪⎪⎪⎪⎪⎨⎪⎪⎪⎪⎪⎪⎪⎪⎧vdW[l]=β1vdW[l]+(1−β1)∂W[l]∂JvdW[l]corrected=1−(β1)tvdW[l]sdW[l]=β2sdW[l]+(1−β2)(∂W[l]∂J)2sdW[l]corrected=1−(β2)tsdW[l]W[l]=W[l]−αsdW[l]corrected+εvdW[l]corrected

初始化为 0

# GRADED FUNCTION: initialize_adamdef initialize_adam(parameters) :"""Initializes v and s as two python dictionaries with:- keys: "dW1", "db1", ..., "dWL", "dbL" - values: numpy arrays of zeros of the same shape as the corresponding gradients/parameters.Arguments:parameters -- python dictionary containing your parameters.parameters["W" + str(l)] = Wlparameters["b" + str(l)] = blReturns: v -- python dictionary that will contain the exponentially weighted average of the gradient.v["dW" + str(l)] = ...v["db" + str(l)] = ...s -- python dictionary that will contain the exponentially weighted average of the squared gradient.s["dW" + str(l)] = ...s["db" + str(l)] = ..."""L = len(parameters) // 2 # number of layers in the neural networksv = {}s = {}# Initialize v, s. Input: "parameters". Outputs: "v, s".for l in range(L):### START CODE HERE ### (approx. 4 lines)v["dW" + str(l+1)] = np.zeros(parameters["W" + str(l+1)].shape)v["db" + str(l+1)] = np.zeros(parameters["b" + str(l+1)].shape)s["dW" + str(l+1)] = np.zeros(parameters["W" + str(l+1)].shape)s["db" + str(l+1)] = np.zeros(parameters["b" + str(l+1)].shape)### END CODE HERE ###return v, s

迭代更新

# GRADED FUNCTION: update_parameters_with_adamdef update_parameters_with_adam(parameters, grads, v, s, t, learning_rate = 0.01,beta1 = 0.9, beta2 = 0.999,  epsilon = 1e-8):"""Update parameters using AdamArguments:parameters -- python dictionary containing your parameters:parameters['W' + str(l)] = Wlparameters['b' + str(l)] = blgrads -- python dictionary containing your gradients for each parameters:grads['dW' + str(l)] = dWlgrads['db' + str(l)] = dblv -- Adam variable, moving average of the first gradient, python dictionarys -- Adam variable, moving average of the squared gradient, python dictionarylearning_rate -- the learning rate, scalar.beta1 -- Exponential decay hyperparameter for the first moment estimates beta2 -- Exponential decay hyperparameter for the second moment estimates epsilon -- hyperparameter preventing division by zero in Adam updatesReturns:parameters -- python dictionary containing your updated parameters v -- Adam variable, moving average of the first gradient, python dictionarys -- Adam variable, moving average of the squared gradient, python dictionary"""L = len(parameters) // 2                 # number of layers in the neural networksv_corrected = {}                         # Initializing first moment estimate, python dictionarys_corrected = {}                         # Initializing second moment estimate, python dictionary# Perform Adam update on all parametersfor l in range(L):# Moving average of the gradients. Inputs: "v, grads, beta1". Output: "v".### START CODE HERE ### (approx. 2 lines)v["dW" + str(l+1)] = beta1*v["dW" + str(l+1)] + (1-beta1)*grads['dW' + str(l+1)]v["db" + str(l+1)] = beta1*v["db" + str(l+1)] + (1-beta1)*grads['db' + str(l+1)]### END CODE HERE #### Compute bias-corrected first moment estimate. Inputs: "v, beta1, t". Output: "v_corrected".### START CODE HERE ### (approx. 2 lines)v_corrected["dW" + str(l+1)] = v["dW" + str(l+1)]/(1-np.power(beta1,t))v_corrected["db" + str(l+1)] = v["db" + str(l+1)]/(1-np.power(beta1,t))### END CODE HERE #### Moving average of the squared gradients. Inputs: "s, grads, beta2". Output: "s".### START CODE HERE ### (approx. 2 lines)s["dW" + str(l+1)] = beta2*s["dW" + str(l+1)] + (1-beta2)*grads['dW' + str(l+1)]**2s["db" + str(l+1)] = beta2*s["db" + str(l+1)] + (1-beta2)*grads['db' + str(l+1)]**2### END CODE HERE #### Compute bias-corrected second raw moment estimate. Inputs: "s, beta2, t". Output: "s_corrected".### START CODE HERE ### (approx. 2 lines)s_corrected["dW" + str(l+1)] = s["dW" + str(l+1)]/(1-np.power(beta2,t))s_corrected["db" + str(l+1)] = s["db" + str(l+1)]/(1-np.power(beta2,t))### END CODE HERE #### Update parameters. Inputs: "parameters, learning_rate, v_corrected, s_corrected, epsilon". Output: "parameters".### START CODE HERE ### (approx. 2 lines)parameters["W" + str(l+1)] = parameters["W" + str(l+1)] - learning_rate*v_corrected["dW" + str(l+1)]/(np.sqrt(s_corrected["dW" + str(l+1)])+epsilon)parameters["b" + str(l+1)] = parameters["b" + str(l+1)] - learning_rate*v_corrected["db" + str(l+1)]/(np.sqrt(s_corrected["db" + str(l+1)])+epsilon)### END CODE HERE ###return parameters, v, s

5. 不同优化算法下的模型

数据集：使用以下数据集进行测试

3层神经网络模型：

Mini-batch Gradient Descent:：
使用函数 update_parameters_with_gd()
Mini-batch Momentum:：
使用函数 initialize_velocity() 和 update_parameters_with_momentum()
Mini-batch Adam：
使用函数 initialize_adam() 和 update_parameters_with_adam()

def model(X, Y, layers_dims, optimizer, learning_rate = 0.0007, mini_batch_size = 64, beta = 0.9,beta1 = 0.9, beta2 = 0.999,  epsilon = 1e-8, num_epochs = 10000, print_cost = True):"""3-layer neural network model which can be run in different optimizer modes.Arguments:X -- input data, of shape (2, number of examples)Y -- true "label" vector (1 for blue dot / 0 for red dot), of shape (1, number of examples)layers_dims -- python list, containing the size of each layerlearning_rate -- the learning rate, scalar.mini_batch_size -- the size of a mini batchbeta -- Momentum hyperparameterbeta1 -- Exponential decay hyperparameter for the past gradients estimates beta2 -- Exponential decay hyperparameter for the past squared gradients estimates epsilon -- hyperparameter preventing division by zero in Adam updatesnum_epochs -- number of epochsprint_cost -- True to print the cost every 1000 epochsReturns:parameters -- python dictionary containing your updated parameters """L = len(layers_dims)             # number of layers in the neural networkscosts = []                       # to keep track of the costt = 0                            # initializing the counter required for Adam updateseed = 10                        # For grading purposes, so that your "random" minibatches are the same as ours# Initialize parametersparameters = initialize_parameters(layers_dims)# Initialize the optimizerif optimizer == "gd":pass # no initialization required for gradient descentelif optimizer == "momentum":v = initialize_velocity(parameters)elif optimizer == "adam":v, s = initialize_adam(parameters)# Optimization loopfor i in range(num_epochs):# Define the random minibatches. We increment the seed to reshuffle differently the dataset after each epochseed = seed + 1minibatches = random_mini_batches(X, Y, mini_batch_size, seed)for minibatch in minibatches:# Select a minibatch(minibatch_X, minibatch_Y) = minibatch# Forward propagationa3, caches = forward_propagation(minibatch_X, parameters)# Compute costcost = compute_cost(a3, minibatch_Y)# Backward propagationgrads = backward_propagation(minibatch_X, minibatch_Y, caches)# Update parametersif optimizer == "gd":parameters = update_parameters_with_gd(parameters, grads, learning_rate)elif optimizer == "momentum":parameters, v = update_parameters_with_momentum(parameters, grads, v, beta, learning_rate)elif optimizer == "adam":t = t + 1 # Adam counterparameters, v, s = update_parameters_with_adam(parameters, grads, v, s,t, learning_rate, beta1, beta2,  epsilon)# Print the cost every 1000 epochif print_cost and i % 1000 == 0:print ("Cost after epoch %i: %f" %(i, cost))if print_cost and i % 100 == 0:costs.append(cost)# plot the costplt.plot(costs)plt.ylabel('cost')plt.xlabel('epochs (per 100)')plt.title("Learning rate = " + str(learning_rate))plt.show()return parameters

5.1 Mini-batch梯度下降

# train 3-layer model
layers_dims = [train_X.shape[0], 5, 2, 1]
parameters = model(train_X, train_Y, layers_dims, optimizer = "gd")# Predict
predictions = predict(train_X, train_Y, parameters)# Plot decision boundary
plt.title("Model with Gradient Descent optimization")
axes = plt.gca()
axes.set_xlim([-1.5,2.5])
axes.set_ylim([-1,1.5])
plot_decision_boundary(lambda x: predict_dec(parameters, x.T), train_X, train_Y)

表现：

Accuracy: 0.79 （Mini-batch梯度下降）

5.2 带动量的Mini-batch梯度下降

# train 3-layer model
layers_dims = [train_X.shape[0], 5, 2, 1]
parameters = model(train_X, train_Y, layers_dims, beta = 0.9, optimizer = "momentum")# Predict
predictions = predict(train_X, train_Y, parameters)# Plot decision boundary
plt.title("Model with Momentum optimization")
axes = plt.gca()
axes.set_xlim([-1.5,2.5])
axes.set_ylim([-1,1.5])
plot_decision_boundary(lambda x: predict_dec(parameters, x.T), train_X, train_Y)

表现：

Accuracy: 0.79（带动量的Mini-batch梯度下降）

本例子由于太过简单，所以动量的优势没有体现出来，在大的数据集上会较不带动量的模型更好

5.3 带Adam的Mini-batch梯度下降

# train 3-layer model
layers_dims = [train_X.shape[0], 5, 2, 1]
parameters = model(train_X, train_Y, layers_dims, optimizer = "adam")# Predict
predictions = predict(train_X, train_Y, parameters)# Plot decision boundary
plt.title("Model with Adam optimization")
axes = plt.gca()
axes.set_xlim([-1.5,2.5])
axes.set_ylim([-1,1.5])
plot_decision_boundary(lambda x: predict_dec(parameters, x.T), train_X, train_Y)

表现：

Accuracy: 0.9366666666666666（带Adam的Mini-batch梯度下降）