吴恩达深度学习2.1练习_Improving Deep Neural Networks_Gradient Checking
转载自吴恩达老师深度学习练习notebook
Gradient Checking
Welcome to the final assignment for this week! In this assignment you will learn to implement and use gradient checking.
You are part of a team working to make mobile payments available globally, and are asked to build a deep learning model to detect fraud–whenever someone makes a payment, you want to see if the payment might be fraudulent, such as if the user’s account has been taken over by a hacker.
But backpropagation is quite challenging to implement, and sometimes has bugs. Because this is a mission-critical application, your company’s CEO wants to be really certain that your implementation of backpropagation is correct. Your CEO says, “Give me a proof that your backpropagation is actually working!” To give this reassurance, you are going to use “gradient checking”.
Let’s do it!
# Packages
import numpy as np
from testCases import *
from gc_utils import sigmoid, relu, dictionary_to_vector, vector_to_dictionary, gradients_to_vector
1) How does gradient checking work?
Backpropagation computes the gradients ∂J∂θ\frac{\partial J}{\partial \theta}∂θ∂J, where θ\thetaθ denotes the parameters of the model. JJJ is computed using forward propagation and your loss function.
Because forward propagation is relatively easy to implement, you’re confident you got that right, and so you’re almost 100% sure that you’re computing the cost JJJ correctly. Thus, you can use your code for computing JJJ to verify the code for computing ∂J∂θ\frac{\partial J}{\partial \theta}∂θ∂J.
Let’s look back at the definition of a derivative (or gradient):
(1)∂J∂θ=limε→0J(θ+ε)−J(θ−ε)2ε\frac{\partial J}{\partial \theta} = \lim_{\varepsilon \to 0} \frac{J(\theta + \varepsilon) - J(\theta - \varepsilon)}{2 \varepsilon} \tag{1}∂θ∂J=ε→0lim2εJ(θ+ε)−J(θ−ε)(1)
If you’re not familiar with the “limε→0\displaystyle \lim_{\varepsilon \to 0}ε→0lim” notation, it’s just a way of saying “when ε\varepsilonε is really really small.”
We know the following:
- ∂J∂θ\frac{\partial J}{\partial \theta}∂θ∂J is what you want to make sure you’re computing correctly.
- You can compute J(θ+ε)J(\theta + \varepsilon)J(θ+ε) and J(θ−ε)J(\theta - \varepsilon)J(θ−ε) (in the case that θ\thetaθ is a real number), since you’re confident your implementation for JJJ is correct.
Lets use equation (1) and a small value for ε\varepsilonε to convince your CEO that your code for computing ∂J∂θ\frac{\partial J}{\partial \theta}∂θ∂J is correct!
2) 1-dimensional gradient checking
Consider a 1D linear function J(θ)=θxJ(\theta) = \theta xJ(θ)=θx. The model contains only a single real-valued parameter θ\thetaθ, and takes xxx as input.
You will implement code to compute J(.)J(.)J(.) and its derivative ∂J∂θ\frac{\partial J}{\partial \theta}∂θ∂J. You will then use gradient checking to make sure your derivative computation for JJJ is correct.
Figure 1 : 1D linear model
The diagram above shows the key computation steps: First start with xxx, then evaluate the function J(x)J(x)J(x) (“forward propagation”). Then compute the derivative ∂J∂θ\frac{\partial J}{\partial \theta}∂θ∂J (“backward propagation”).
Exercise: implement “forward propagation” and “backward propagation” for this simple function. I.e., compute both J(.)J(.)J(.) (“forward propagation”) and its derivative with respect to θ\thetaθ (“backward propagation”), in two separate functions.
# GRADED FUNCTION: forward_propagationdef forward_propagation(x, theta):"""Implement the linear forward propagation (compute J) presented in Figure 1 (J(theta) = theta * x)Arguments:x -- a real-valued inputtheta -- our parameter, a real number as wellReturns:J -- the value of function J, computed using the formula J(theta) = theta * x"""### START CODE HERE ### (approx. 1 line)J = np.dot(theta, x)### END CODE HERE ###return J
x, theta = 2, 4
J = forward_propagation(x, theta)
print ("J = " + str(J))
J = 8
Expected Output:
Exercise: Now, implement the backward propagation step (derivative computation) of Figure 1. That is, compute the derivative of J(θ)=θxJ(\theta) = \theta xJ(θ)=θx with respect to θ\thetaθ. To save you from doing the calculus, you should get dtheta=∂J∂θ=xdtheta = \frac { \partial J }{ \partial \theta} = xdtheta=∂θ∂J=x.
# GRADED FUNCTION: backward_propagationdef backward_propagation(x, theta):"""Computes the derivative of J with respect to theta (see Figure 1).Arguments:x -- a real-valued inputtheta -- our parameter, a real number as wellReturns:dtheta -- the gradient of the cost with respect to theta"""### START CODE HERE ### (approx. 1 line)dtheta = x### END CODE HERE ###return dtheta
x, theta = 2, 4
dtheta = backward_propagation(x, theta)
print ("dtheta = " + str(dtheta))
dtheta = 2
Expected Output:
| dtheta | 2 |
Exercise: To show that the backward_propagation() function is correctly computing the gradient ∂J∂θ\frac{\partial J}{\partial \theta}∂θ∂J, let’s implement gradient checking.
Instructions:
- First compute “gradapprox” using the formula above (1) and a small value of ε\varepsilonε. Here are the Steps to follow:
- θ+=θ+ε\theta^{+} = \theta + \varepsilonθ+=θ+ε
- θ−=θ−ε\theta^{-} = \theta - \varepsilonθ−=θ−ε
- J+=J(θ+)J^{+} = J(\theta^{+})J+=J(θ+)
- J−=J(θ−)J^{-} = J(\theta^{-})J−=J(θ−)
- gradapprox=J+−J−2εgradapprox = \frac{J^{+} - J^{-}}{2 \varepsilon}gradapprox=2εJ+−J−
- Then compute the gradient using backward propagation, and store the result in a variable “grad”
- Finally, compute the relative difference between “gradapprox” and the “grad” using the following formula:
(2)difference=∣∣grad−gradapprox∣∣2∣∣grad∣∣2+∣∣gradapprox∣∣2difference = \frac {\mid\mid grad - gradapprox \mid\mid_2}{\mid\mid grad \mid\mid_2 + \mid\mid gradapprox \mid\mid_2} \tag{2}difference=∣∣grad∣∣2+∣∣gradapprox∣∣2∣∣grad−gradapprox∣∣2(2)
You will need 3 Steps to compute this formula:- 1’. compute the numerator using np.linalg.norm(…)
- 2’. compute the denominator. You will need to call np.linalg.norm(…) twice.
- 3’. divide them.
- If this difference is small (say less than 10−710^{-7}10−7), you can be quite confident that you have computed your gradient correctly. Otherwise, there may be a mistake in the gradient computation.
# GRADED FUNCTION: gradient_checkdef gradient_check(x, theta, epsilon = 1e-7):"""Implement the backward propagation presented in Figure 1.Arguments:x -- a real-valued inputtheta -- our parameter, a real number as wellepsilon -- tiny shift to the input to compute approximated gradient with formula(1)Returns:difference -- difference (2) between the approximated gradient and the backward propagation gradient"""# Compute gradapprox using left side of formula (1). epsilon is small enough, you don't need to worry about the limit.### START CODE HERE ### (approx. 5 lines)thetaplus = theta + epsilon # Step 1thetaminus = theta - epsilon # Step 2J_plus = np.dot(thetaplus, x) # Step 3J_minus = np.dot(thetaminus, x) # Step 4gradapprox = (J_plus - J_minus) / 2 / epsilon # Step 5### END CODE HERE #### Check if gradapprox is close enough to the output of backward_propagation()### START CODE HERE ### (approx. 1 line)grad = backward_propagation(x, theta)### END CODE HERE ###### START CODE HERE ### (approx. 1 line)numerator = np.linalg.norm(grad - gradapprox) # Step 1'denominator = np.linalg.norm(grad) + np.linalg.norm(gradapprox) # Step 2'difference = numerator / denominator # Step 3'### END CODE HERE ###if difference < 1e-7:print ("The gradient is correct!")else:print ("The gradient is wrong!")return difference
x, theta = 2, 4
difference = gradient_check(x, theta)
print("difference = " + str(difference))
The gradient is correct!
difference = 2.919335883291695e-10
Expected Output:
The gradient is correct!
| difference | 2.9193358103083e-10 |
Congrats, the difference is smaller than the 10−710^{-7}10−7 threshold. So you can have high confidence that you’ve correctly computed the gradient in backward_propagation().
Now, in the more general case, your cost function JJJ has more than a single 1D input. When you are training a neural network, θ\thetaθ actually consists of multiple matrices W[l]W^{[l]}W[l] and biases b[l]b^{[l]}b[l]! It is important to know how to do a gradient check with higher-dimensional inputs. Let’s do it!
3) N-dimensional gradient checking
The following figure describes the forward and backward propagation of your fraud detection model.
Figure 2 : deep neural network
*LINEAR -> RELU -> LINEAR -> RELU -> LINEAR -> SIGMOID*
Let’s look at your implementations for forward propagation and backward propagation.
def forward_propagation_n(X, Y, parameters):"""Implements the forward propagation (and computes the cost) presented in Figure 3.Arguments:X -- training set for m examplesY -- labels for m examples parameters -- python dictionary containing your parameters "W1", "b1", "W2", "b2", "W3", "b3":W1 -- weight matrix of shape (5, 4)b1 -- bias vector of shape (5, 1)W2 -- weight matrix of shape (3, 5)b2 -- bias vector of shape (3, 1)W3 -- weight matrix of shape (1, 3)b3 -- bias vector of shape (1, 1)Returns:cost -- the cost function (logistic cost for one example)"""# retrieve parametersm = X.shape[1]W1 = parameters["W1"]b1 = parameters["b1"]W2 = parameters["W2"]b2 = parameters["b2"]W3 = parameters["W3"]b3 = parameters["b3"]# LINEAR -> RELU -> LINEAR -> RELU -> LINEAR -> SIGMOIDZ1 = np.dot(W1, X) + b1A1 = relu(Z1)Z2 = np.dot(W2, A1) + b2A2 = relu(Z2)Z3 = np.dot(W3, A2) + b3A3 = sigmoid(Z3)# Costlogprobs = np.multiply(-np.log(A3),Y) + np.multiply(-np.log(1 - A3), 1 - Y)cost = 1./m * np.sum(logprobs)cache = (Z1, A1, W1, b1, Z2, A2, W2, b2, Z3, A3, W3, b3)return cost, cache
Now, run backward propagation.
def backward_propagation_n(X, Y, cache):"""Implement the backward propagation presented in figure 2.Arguments:X -- input datapoint, of shape (input size, 1)Y -- true "label"cache -- cache output from forward_propagation_n()Returns:gradients -- A dictionary with the gradients of the cost with respect to each parameter, activation and pre-activation variables."""m = X.shape[1](Z1, A1, W1, b1, Z2, A2, W2, b2, Z3, A3, W3, b3) = cachedZ3 = A3 - YdW3 = 1./m * np.dot(dZ3, A2.T)db3 = 1./m * np.sum(dZ3, axis=1, keepdims = True)dA2 = np.dot(W3.T, dZ3)dZ2 = np.multiply(dA2, np.int64(A2 > 0))dW2 = 1./m * np.dot(dZ2, A1.T)db2 = 1./m * np.sum(dZ2, axis=1, keepdims = True)dA1 = np.dot(W2.T, dZ2)dZ1 = np.multiply(dA1, np.int64(A1 > 0))dW1 = 1./m * np.dot(dZ1, X.T)db1 = 1./m * np.sum(dZ1, axis=1, keepdims = True)gradients = {"dZ3": dZ3, "dW3": dW3, "db3": db3,"dA2": dA2, "dZ2": dZ2, "dW2": dW2, "db2": db2,"dA1": dA1, "dZ1": dZ1, "dW1": dW1, "db1": db1}return gradients
You obtained some results on the fraud detection test set but you are not 100% sure of your model. Nobody’s perfect! Let’s implement gradient checking to verify if your gradients are correct.
How does gradient checking work?.
As in 1) and 2), you want to compare “gradapprox” to the gradient computed by backpropagation. The formula is still:
(1)∂J∂θ=limε→0J(θ+ε)−J(θ−ε)2ε\frac{\partial J}{\partial \theta} = \lim_{\varepsilon \to 0} \frac{J(\theta + \varepsilon) - J(\theta - \varepsilon)}{2 \varepsilon} \tag{1}∂θ∂J=ε→0lim2εJ(θ+ε)−J(θ−ε)(1)
However, θ\thetaθ is not a scalar anymore. It is a dictionary called “parameters”. We implemented a function “dictionary_to_vector()” for you. It converts the “parameters” dictionary into a vector called “values”, obtained by reshaping all parameters (W1, b1, W2, b2, W3, b3) into vectors and concatenating them.
The inverse function is “vector_to_dictionary” which outputs back the “parameters” dictionary.
Figure 2 : dictionary_to_vector() and vector_to_dictionary()
You will need these functions in gradient_check_n()
We have also converted the “gradients” dictionary into a vector “grad” using gradients_to_vector(). You don’t need to worry about that.
Exercise: Implement gradient_check_n().
Instructions: Here is pseudo-code that will help you implement the gradient check.
For each i in num_parameters:
- To compute
J_plus[i]:- Set θ+\theta^{+}θ+ to
np.copy(parameters_values) - Set θi+\theta^{+}_iθi+ to θi++ε\theta^{+}_i + \varepsilonθi++ε
- Calculate Ji+J^{+}_iJi+ using to
forward_propagation_n(x, y, vector_to_dictionary(θ+\theta^{+}θ+)).
- Set θ+\theta^{+}θ+ to
- To compute
J_minus[i]: do the same thing with θ−\theta^{-}θ− - Compute gradapprox[i]=Ji+−Ji−2εgradapprox[i] = \frac{J^{+}_i - J^{-}_i}{2 \varepsilon}gradapprox[i]=2εJi+−Ji−
Thus, you get a vector gradapprox, where gradapprox[i] is an approximation of the gradient with respect to parameter_values[i]. You can now compare this gradapprox vector to the gradients vector from backpropagation. Just like for the 1D case (Steps 1’, 2’, 3’), compute:
(3)difference=∥grad−gradapprox∥2∥grad∥2+∥gradapprox∥2difference = \frac {\| grad - gradapprox \|_2}{\| grad \|_2 + \| gradapprox \|_2 } \tag{3}difference=∥grad∥2+∥gradapprox∥2∥grad−gradapprox∥2(3)
# GRADED FUNCTION: gradient_check_ndef gradient_check_n(parameters, gradients, X, Y, epsilon = 1e-7):"""Checks if backward_propagation_n computes correctly the gradient of the cost output by forward_propagation_nArguments:parameters -- python dictionary containing your parameters "W1", "b1", "W2", "b2", "W3", "b3":grad -- output of backward_propagation_n, contains gradients of the cost with respect to the parameters. x -- input datapoint, of shape (input size, 1)y -- true "label"epsilon -- tiny shift to the input to compute approximated gradient with formula(1)Returns:difference -- difference (2) between the approximated gradient and the backward propagation gradient"""# Set-up variablesparameters_values, _ = dictionary_to_vector(parameters)grad = gradients_to_vector(gradients)num_parameters = parameters_values.shape[0]J_plus = np.zeros((num_parameters, 1))J_minus = np.zeros((num_parameters, 1))gradapprox = np.zeros((num_parameters, 1))# Compute gradapproxfor i in range(num_parameters):# Compute J_plus[i]. Inputs: "parameters_values, epsilon". Output = "J_plus[i]".# "_" is used because the function you have to outputs two parameters but we only care about the first one### START CODE HERE ### (approx. 3 lines)thetaplus = np.copy(parameters_values) # Step 1thetaplus[i][0] = thetaplus[i][0] + epsilon # Step 2J_plus[i], _ = forward_propagation_n(X, Y, vector_to_dictionary(thetaplus)) # Step 3### END CODE HERE #### Compute J_minus[i]. Inputs: "parameters_values, epsilon". Output = "J_minus[i]".### START CODE HERE ### (approx. 3 lines)thetaminus = np.copy(parameters_values) # Step 1thetaminus[i][0] = thetaminus[i][0] - epsilon # Step 2 J_minus[i], _ = forward_propagation_n(X, Y, vector_to_dictionary(thetaminus)) # Step 3### END CODE HERE #### Compute gradapprox[i]### START CODE HERE ### (approx. 1 line)gradapprox[i] = (J_plus[i] - J_minus[i]) / 2 / epsilon### END CODE HERE #### Compare gradapprox to backward propagation gradients by computing difference.### START CODE HERE ### (approx. 1 line)numerator = np.linalg.norm(grad - gradapprox) # Step 1'denominator = np.linalg.norm(grad) + np.linalg.norm(gradapprox) # Step 2'difference = numerator / denominator # Step 3'### END CODE HERE ###if difference > 1e-7:print ("\033[93m" + "There is a mistake in the backward propagation! difference = " + str(difference) + "\033[0m")else:print ("\033[92m" + "Your backward propagation works perfectly fine! difference = " + str(difference) + "\033[0m")return difference
X, Y, parameters = gradient_check_n_test_case()cost, cache = forward_propagation_n(X, Y, parameters)
gradients = backward_propagation_n(X, Y, cache)
difference = gradient_check_n(parameters, gradients, X, Y)
[93mThere is a mistake in the backward propagation! difference = 1.1885552035482147e-07[0m
Expected output:
| *There is a mistake in the backward propagation! | difference = 0.285093156781 |
It seems that there were errors in the backward_propagation_n code we gave you! Good that you’ve implemented the gradient check. Go back to backward_propagation and try to find/correct the errors (Hint: check dW2 and db1). Rerun the gradient check when you think you’ve fixed it. Remember you’ll need to re-execute the cell defining backward_propagation_n() if you modify the code.
Can you get gradient check to declare your derivative computation correct? Even though this part of the assignment isn’t graded, we strongly urge you to try to find the bug and re-run gradient check until you’re convinced backprop is now correctly implemented.
Note
- Gradient Checking is slow! Approximating the gradient with ∂J∂θ≈J(θ+ε)−J(θ−ε)2ε\frac{\partial J}{\partial \theta} \approx \frac{J(\theta + \varepsilon) - J(\theta - \varepsilon)}{2 \varepsilon}∂θ∂J≈2εJ(θ+ε)−J(θ−ε) is computationally costly. For this reason, we don’t run gradient checking at every iteration during training. Just a few times to check if the gradient is correct.
- Gradient Checking, at least as we’ve presented it, doesn’t work with dropout. You would usually run the gradient check algorithm without dropout to make sure your backprop is correct, then add dropout.
Congrats, you can be confident that your deep learning model for fraud detection is working correctly! You can even use this to convince your CEO. ?
What you should remember from this notebook:
- Gradient checking verifies closeness between the gradients from backpropagation and the numerical approximation of the gradient (computed using forward propagation).
- Gradient checking is slow, so we don’t run it in every iteration of training. You would usually run it only to make sure your code is correct, then turn it off and use backprop for the actual learning process.
吴恩达深度学习2.1练习_Improving Deep Neural Networks_Gradient Checking相关推荐
- 吴恩达深度学习2.3练习_Improving Deep Neural Networks_Tensorflow
转载自吴恩达老师深度学习练习notebook TensorFlow Tutorial Welcome to this week's programming assignment. Until now, ...
- 吴恩达深度学习2.1练习_Improving Deep Neural Networks_Regularization
转载自吴恩达老师深度学习练习notebook Regularization Welcome to the second assignment of this week. Deep Learning m ...
- 吴恩达深度学习2.1练习_Improving Deep Neural Networks_initialization
转载自吴恩达老师深度学习练习notebook Initialization Welcome to the first assignment of "Improving Deep Neural ...
- 吴恩达深度学习2.2练习_Improving Deep Neural Networks_Optimization
版权声明:本文为博主原创文章,未经博主允许不得转载. https://blog.csdn.net/weixin_42432468 学习心得: 1.每周的视频课程看一到两遍 2.做笔记 3.做每周的作业 ...
- 吴恩达深度学习2.1练习_Improving Deep Neural Networks(Initialization_Regularization_Gradientchecking)
版权声明:本文为博主原创文章,未经博主允许不得转载. https://blog.csdn.net/weixin_42432468 学习心得: 1.每周的视频课程看一到两遍 2.做笔记 3.做每周的作业 ...
- 吴恩达深度学习2.3笔记_Improving Deep Neural Networks_超参数调试 和 Batch Norm
版权声明:本文为博主原创文章,未经博主允许不得转载. https://blog.csdn.net/weixin_42432468 学习心得: 1.每周的视频课程看一到两遍 2.做笔记 3.做每周的作业 ...
- 吴恩达深度学习2.2笔记_Improving Deep Neural Networks_优化算法
版权声明:本文为博主原创文章,未经博主允许不得转载. https://blog.csdn.net/weixin_42432468 学习心得: 1.每周的视频课程看一到两遍 2.做笔记 3.做每周的作业 ...
- 吴恩达深度学习2.1笔记_Improving Deep Neural Networks_深度学习的实践层面
版权声明:本文为博主原创文章,未经博主允许不得转载. https://blog.csdn.net/weixin_42432468 学习心得: 1.每周的视频课程看一到两遍 2.做笔记 3.做每周的作业 ...
- 吴恩达深度学习笔记——卷积神经网络（Convolutional Neural Networks）
深度学习笔记导航 前言 传送门 卷积神经网络(Convolutional Neural Networks) 卷积神经网络基础(Foundations of Convolutional Neural N ...
最新文章
- python使用退格键时出现^H解决方法
- ajax提交表单，导致浏览器卡死
- View.Post () 的身世大揭秘
- 如何让body背景图自适应浏览器窗口大小
- 企业即时通讯一个全新行业的诞生
- 解决ffmpeg中的时间戳同步问题
- 关于二进制兼容(二进制边界)的解释
- scala中的协变和逆变
- WPF-14：绑定中数据模型必须为public问题
- 乐高ev3搭建图_你所不知道的乐高EV3发展史
- 基于微信小程序的医疗监督反馈小程序的设计与实现-计算机毕业设计源码+LW文档
- 连续优化、离散优化、组合优化、整数优化和凸优化
- html特殊符号怎么输入法,教您特殊符号怎么打出来
- photoshop---压缩图片大小/给人物换衣服
- 互联网“香饽饽”？数据挖掘或成热门行业
- Android 系统各版本新特性总结
- Real-Time Rendering 翻译 3.图形处理单元
- 电脑剪辑视频用什么工具？好用的视频剪辑工具推荐
- 如何解决win11“无法枚举容器中的对象，访问被拒绝”、“右键新建只有文件夹，没有其他选项”的问题。
- 区块链职业技术技能标准明确了人才培养的框架