人工智能实战_第三次作业

第三次作业：使用minibatch的方式进行梯度下降

一、简要概述

项目	内容
课程	人工智能实战2019
作业要求	作业要求
我在这个课程的目标是	了解人工智能理论，提升coding能力
这个作业在哪个具体方面帮助我实现目标	了解单层神经工作原理，掌握几种梯度下降法的优缺点，自己实现简单算法

二、单层神经网络原理以及权值更新依据

单变量随机梯度下降 SDG(Stochastic Grident Descent)
正向计算过程：
\[Z^{n \times 1}=W^{n \times f} \cdot X^{f \times 1} + B^{n \times 1}\]
\[A^{n \times 1}=a(Z)\]
反向计算过程：
\[ \Delta Z^{n \times 1} = J'(W,B) = A^{n \times 1} - Y^{1 \times 1}\]
\[ W^{n \times f} = W^{n \times f} - \eta \cdot (\Delta Z^{n \times 1} \cdot X_T^{1 \times f})\]
\[ B^{n \times 1} = B^{n \times 1} - \eta \cdot \Delta Z^{n \times 1}\]
- 其中：
  \[f=特征值数，m=样本数，n=神经元数，\eta=步长 \\ A=预测值，Y=标签值，X=输入值，X_T=X的转置\]

三、作业实现步骤

模拟PDF上的步骤，仿写出相应的单层神经网络模型算法。
采用SGD梯度下降算法更新每次的权值
统计每一个Epoch权值更新完之后的代价值。
更改batch_size的值为5，10，15，观察在不同的batch_size下函数代价函数的收敛速度与效果。

四、代码

#-*-coding:utf-8-*-
import numpy as np
import matplotlib.pyplot as plt
from pathlib import Pathx_data_name = "TemperatureControlXData.dat" #read the data from x
y_data_name = "TemperatureControlYData.dat" # 两个文件名保存着数据文件的路径#下面这个类保存着每个线性模型的参数class CData(object):def __init__(self, loss, w, b, epoch, iteration):self.loss = lossself.w = wself.b = bself.epoch = epochself.iteration = iteration# 下面这个函数将X，Y数据从文件中读出，并且通过reshape函数将其规整为一行的结果def ReadData():Xfile = Path(x_data_name)Yfile = Path(y_data_name)if Xfile.exists() & Yfile.exists():X = np.load(Xfile)Y = np.load(Yfile)return X.reshape(1, -1), Y.reshape(1, -1) #将这两种数据全部转化为一个列else:return None, None
# 前向传播过程 z = w*x + b
# 得到相应的z
def Fowrad(w,b,x):z = np.dot(w,x)+breturn z;# 反向求导过程，求出w和b的值应该变化的量
def BackPropagation(x,y,z):m = np.shape(x)[1] # 代表样本的数量# delta z = z - ydeltaZ = z - y# delta b = sum(delta z)deltaB = deltaZ.sum(axis=1, keepdims=True) / m# delta w = (delta z) * (x')deltaW = np.dot(deltaZ, x.T) / mreturn deltaW,deltaB# 每次反向求导后跟新w和b的值
def UpdateWeights(w, b, deltaW, deltaB, eta):w = w - eta * deltaWb = b - eta * deltaBreturn w, b# 示例中给出的初始化w和b的方法，根据输入和输出的个数确定不同
# 大小的矩阵
def SetParam(num_input,num_output, flag):if flag == 0:# zeroW = np.zeros((num_output, num_input))elif flag == 1:# normalizeW = np.random.normal(size=(num_output, num_input))elif flag == 2:# xavierW = np.random.uniform(-np.sqrt(6 / (num_input + num_output)),np.sqrt(6 / (num_input + num_output)),size=(num_output, num_input))B = np.zeros((num_output, 1))return W, B#   这个函数计算在每一次迭代更新完权值之后的误差
def GetLoss(w,b,x,y):m = x.shape[1]z = np.dot(w, x) + bLOSS = (z - y) ** 2loss = LOSS.sum() / m / 2return loss
# 在每次迭代的时候从x，y中取出合适大小的batch来进行训练
def GetBatchSamples(X, Y, batch_size, iteration):num_feature = X.shape[0]start = iteration * batch_sizeend = start + batch_sizebatch_x = X[0:num_feature, start:end].reshape(num_feature, batch_size)batch_y = Y[0, start:end].reshape(1, batch_size)return batch_x, batch_ydef shuffle_batch(X, y, batch_size):rnd_idx = np.random.permutation(len(X))n_batches = len(X)//batch_sizefor batch_idx in np.array_split(rnd_idx, n_batches):X_batch, y_batch = X[batch_idx], y[batch_idx]yield X_batch, y_batchif __name__ == '__main__':# 首先设置学习率 eta, batch_size, epoch的个数eta = 0.1size = {5,10,15}epochMax = 50#   x,y就是我们的训练数据x,y = ReadData()print np.shape(x),np.shape(y)plt.figure()for batch_size in size:loss = []#   iterMax是我们每个epoch迭代的次数,epochMax是我们循环的次数iterMax = np.shape(x)[1] / batch_sizew, b = SetParam(1, 1, 2)for epoch in range(epochMax):for x_batch, y_batch in shuffle_batch(X, y, batch_size):# 获得每一个iter的batch数据# 获得前向传播后的z值z_batch = Fowrad(w,b,x_batch)# 反向传播deltaW,deltaB = BackPropagation(x_batch,y_batch,z_batch)# 更新w和b的值w,b = UpdateWeights(w,b,deltaW,deltaB,eta)# 每个epoch后获取新的参数得出的lossc = GetLoss(w,b,x,y)print("Epoch = %d , w = %f,b = %f ,loss = %f"%(epoch,w,b,c))loss.append(c)axisX = np.arange(5,epochMax,1)plt.plot(axisX,loss)plt.legend(['batch_size : 5','batch_size : 10','batch_size : 15'],loc ='upper right')plt.show()plt.figure()min = np.min(x,1)max = np.max(x,1)lineX = np.arange(min,max,0.01)lineX = lineX.reshape(1,-1)lineY = lineX * w + bprint np.shape(lineY)plt.scatter(x,y,s=1)plt.plot(lineX[0,:],lineY[0,:],'r')plt.show()

五、实验结果

以上是不同batch_size下的代价函数随着epoch的增加而变化的情况可以看当batchsize取10的时候下降最快
当我们改变其学习率为0.01,0.1,0.2,0.5时我们看其变化
- eta = 0.01时
- eta = 0.1时
- eta = 0.2时
- eta = 0.5时
我们看到随着学习率的变大，SGD的抖动也在逐渐增加。
这是在最后取得的权值下，生成的拟合曲线。

代价函数变化:

batch_size = 5

Epoch = 0 , w = 1.263874,b = 3.099159 ,loss = 0.061878
Epoch = 1 , w = 1.454535,b = 3.274156 ,loss = 0.017243
Epoch = 2 , w = 1.529773,b = 3.253623 ,loss = 0.014271
Epoch = 3 , w = 1.589247,b = 3.223812 ,loss = 0.012070
Epoch = 4 , w = 1.640776,b = 3.196955 ,loss = 0.010385
Epoch = 5 , w = 1.685762,b = 3.173438 ,loss = 0.009097
Epoch = 6 , w = 1.725059,b = 3.152890 ,loss = 0.008113
Epoch = 7 , w = 1.759388,b = 3.134939 ,loss = 0.007362
Epoch = 8 , w = 1.789379,b = 3.119256 ,loss = 0.006788
Epoch = 9 , w = 1.815578,b = 3.105557 ,loss = 0.006349
Epoch = 10 , w = 1.838466,b = 3.093589 ,loss = 0.006014
Epoch = 11 , w = 1.858460,b = 3.083133 ,loss = 0.005758
Epoch = 12 , w = 1.875927,b = 3.074000 ,loss = 0.005562
Epoch = 13 , w = 1.891186,b = 3.066021 ,loss = 0.005412
Epoch = 14 , w = 1.904517,b = 3.059050 ,loss = 0.005297
Epoch = 15 , w = 1.916162,b = 3.052961 ,loss = 0.005210
Epoch = 16 , w = 1.926335,b = 3.047641 ,loss = 0.005142
Epoch = 17 , w = 1.935222,b = 3.042994 ,loss = 0.005091
Epoch = 18 , w = 1.942986,b = 3.038934 ,loss = 0.005051
Epoch = 19 , w = 1.949769,b = 3.035387 ,loss = 0.005021
Epoch = 20 , w = 1.955694,b = 3.032289 ,loss = 0.004998
Epoch = 21 , w = 1.960870,b = 3.029582 ,loss = 0.004980
Epoch = 22 , w = 1.965392,b = 3.027218 ,loss = 0.004966
Epoch = 23 , w = 1.969342,b = 3.025152 ,loss = 0.004956
Epoch = 24 , w = 1.972793,b = 3.023348 ,loss = 0.004948
Epoch = 25 , w = 1.975808,b = 3.021771 ,loss = 0.004941
Epoch = 26 , w = 1.978442,b = 3.020394 ,loss = 0.004937
Epoch = 27 , w = 1.980742,b = 3.019191 ,loss = 0.004933
Epoch = 28 , w = 1.982752,b = 3.018140 ,loss = 0.004930
Epoch = 29 , w = 1.984508,b = 3.017222 ,loss = 0.004928
Epoch = 30 , w = 1.986042,b = 3.016420 ,loss = 0.004926
Epoch = 31 , w = 1.987382,b = 3.015719 ,loss = 0.004925
Epoch = 32 , w = 1.988553,b = 3.015107 ,loss = 0.004924
Epoch = 33 , w = 1.989575,b = 3.014572 ,loss = 0.004923
Epoch = 34 , w = 1.990469,b = 3.014105 ,loss = 0.004922
Epoch = 35 , w = 1.991249,b = 3.013697 ,loss = 0.004922
Epoch = 36 , w = 1.991931,b = 3.013340 ,loss = 0.004921
Epoch = 37 , w = 1.992527,b = 3.013029 ,loss = 0.004921
Epoch = 38 , w = 1.993047,b = 3.012757 ,loss = 0.004921
Epoch = 39 , w = 1.993502,b = 3.012519 ,loss = 0.004920
Epoch = 40 , w = 1.993899,b = 3.012311 ,loss = 0.004920
Epoch = 41 , w = 1.994246,b = 3.012130 ,loss = 0.004920
Epoch = 42 , w = 1.994549,b = 3.011972 ,loss = 0.004920
Epoch = 43 , w = 1.994813,b = 3.011833 ,loss = 0.004920
Epoch = 44 , w = 1.995045,b = 3.011712 ,loss = 0.004920
Epoch = 45 , w = 1.995247,b = 3.011607 ,loss = 0.004920
Epoch = 46 , w = 1.995423,b = 3.011514 ,loss = 0.004920
Epoch = 47 , w = 1.995577,b = 3.011434 ,loss = 0.004920
Epoch = 48 , w = 1.995712,b = 3.011363 ,loss = 0.004920
Epoch = 49 , w = 1.995830,b = 3.011302 ,loss = 0.004920

batch_size = 10

Epoch = 0 , w = 2.627484,b = 2.662620 ,loss = 0.022283
Epoch = 1 , w = 2.482123,b = 2.755555 ,loss = 0.014914
Epoch = 2 , w = 2.366523,b = 2.817388 ,loss = 0.010696
Epoch = 3 , w = 2.278350,b = 2.864496 ,loss = 0.008255
Epoch = 4 , w = 2.211115,b = 2.900418 ,loss = 0.006844
Epoch = 5 , w = 2.159845,b = 2.927810 ,loss = 0.006031
Epoch = 6 , w = 2.120750,b = 2.948697 ,loss = 0.005563
Epoch = 7 , w = 2.090938,b = 2.964625 ,loss = 0.005295
Epoch = 8 , w = 2.068205,b = 2.976770 ,loss = 0.005142
Epoch = 9 , w = 2.050870,b = 2.986031 ,loss = 0.005055
Epoch = 10 , w = 2.037652,b = 2.993094 ,loss = 0.005006
Epoch = 11 , w = 2.027572,b = 2.998479 ,loss = 0.004979
Epoch = 12 , w = 2.019886,b = 3.002585 ,loss = 0.004965
Epoch = 13 , w = 2.014025,b = 3.005717 ,loss = 0.004957
Epoch = 14 , w = 2.009556,b = 3.008104 ,loss = 0.004953
Epoch = 15 , w = 2.006148,b = 3.009925 ,loss = 0.004951
Epoch = 16 , w = 2.003549,b = 3.011314 ,loss = 0.004951
Epoch = 17 , w = 2.001568,b = 3.012372 ,loss = 0.004950
Epoch = 18 , w = 2.000056,b = 3.013180 ,loss = 0.004951
Epoch = 19 , w = 1.998904,b = 3.013795 ,loss = 0.004951
Epoch = 20 , w = 1.998025,b = 3.014265 ,loss = 0.004951
Epoch = 21 , w = 1.997355,b = 3.014623 ,loss = 0.004951
Epoch = 22 , w = 1.996845,b = 3.014896 ,loss = 0.004951
Epoch = 23 , w = 1.996455,b = 3.015104 ,loss = 0.004952
Epoch = 24 , w = 1.996158,b = 3.015263 ,loss = 0.004952
Epoch = 25 , w = 1.995931,b = 3.015384 ,loss = 0.004952
Epoch = 26 , w = 1.995759,b = 3.015476 ,loss = 0.004952
Epoch = 27 , w = 1.995627,b = 3.015546 ,loss = 0.004952
Epoch = 28 , w = 1.995526,b = 3.015600 ,loss = 0.004952
Epoch = 29 , w = 1.995450,b = 3.015641 ,loss = 0.004952
Epoch = 30 , w = 1.995391,b = 3.015672 ,loss = 0.004952
Epoch = 31 , w = 1.995347,b = 3.015696 ,loss = 0.004952
Epoch = 32 , w = 1.995313,b = 3.015714 ,loss = 0.004952
Epoch = 33 , w = 1.995287,b = 3.015728 ,loss = 0.004952
Epoch = 34 , w = 1.995267,b = 3.015738 ,loss = 0.004952
Epoch = 35 , w = 1.995252,b = 3.015746 ,loss = 0.004952
Epoch = 36 , w = 1.995241,b = 3.015753 ,loss = 0.004952
Epoch = 37 , w = 1.995232,b = 3.015757 ,loss = 0.004952
Epoch = 38 , w = 1.995225,b = 3.015761 ,loss = 0.004952
Epoch = 39 , w = 1.995220,b = 3.015764 ,loss = 0.004952
Epoch = 40 , w = 1.995216,b = 3.015766 ,loss = 0.004952
Epoch = 41 , w = 1.995213,b = 3.015767 ,loss = 0.004952
Epoch = 42 , w = 1.995211,b = 3.015768 ,loss = 0.004952
Epoch = 43 , w = 1.995209,b = 3.015769 ,loss = 0.004952
Epoch = 44 , w = 1.995208,b = 3.015770 ,loss = 0.004952
Epoch = 45 , w = 1.995207,b = 3.015771 ,loss = 0.004952
Epoch = 46 , w = 1.995206,b = 3.015771 ,loss = 0.004952
Epoch = 47 , w = 1.995206,b = 3.015771 ,loss = 0.004952
Epoch = 48 , w = 1.995205,b = 3.015772 ,loss = 0.004952
Epoch = 49 , w = 1.995205,b = 3.015772 ,loss = 0.004952

batch_size = 15

Epoch = 0 , w = 0.359754,b = 3.013313 ,loss = 0.427990
Epoch = 1 , w = 0.753299,b = 3.498736 ,loss = 0.075995
Epoch = 2 , w = 0.903888,b = 3.542616 ,loss = 0.055048
Epoch = 3 , w = 1.004769,b = 3.512235 ,loss = 0.046501
Epoch = 4 , w = 1.090577,b = 3.472006 ,loss = 0.039700
Epoch = 5 , w = 1.167951,b = 3.433004 ,loss = 0.034029
Epoch = 6 , w = 1.238552,b = 3.396928 ,loss = 0.029284
Epoch = 7 , w = 1.303122,b = 3.363848 ,loss = 0.025311
Epoch = 8 , w = 1.362202,b = 3.333566 ,loss = 0.021986
Epoch = 9 , w = 1.416263,b = 3.305853 ,loss = 0.019201
Epoch = 10 , w = 1.465732,b = 3.280493 ,loss = 0.016871
Epoch = 11 , w = 1.511001,b = 3.257288 ,loss = 0.014919
Epoch = 12 , w = 1.552425,b = 3.236052 ,loss = 0.013286
Epoch = 13 , w = 1.590330,b = 3.216621 ,loss = 0.011919
Epoch = 14 , w = 1.625017,b = 3.198839 ,loss = 0.010774
Epoch = 15 , w = 1.656758,b = 3.182568 ,loss = 0.009816
Epoch = 16 , w = 1.685803,b = 3.167679 ,loss = 0.009014
Epoch = 17 , w = 1.712381,b = 3.154054 ,loss = 0.008343
Epoch = 18 , w = 1.736702,b = 3.141586 ,loss = 0.007782
Epoch = 19 , w = 1.758958,b = 3.130177 ,loss = 0.007312
Epoch = 20 , w = 1.779323,b = 3.119737 ,loss = 0.006918
Epoch = 21 , w = 1.797959,b = 3.110184 ,loss = 0.006589
Epoch = 22 , w = 1.815012,b = 3.101442 ,loss = 0.006314
Epoch = 23 , w = 1.830617,b = 3.093442 ,loss = 0.006083
Epoch = 24 , w = 1.844897,b = 3.086122 ,loss = 0.005890
Epoch = 25 , w = 1.857964,b = 3.079423 ,loss = 0.005729
Epoch = 26 , w = 1.869921,b = 3.073294 ,loss = 0.005594
Epoch = 27 , w = 1.880863,b = 3.067685 ,loss = 0.005481
Epoch = 28 , w = 1.890875,b = 3.062552 ,loss = 0.005387
Epoch = 29 , w = 1.900037,b = 3.057855 ,loss = 0.005308
Epoch = 30 , w = 1.908421,b = 3.053557 ,loss = 0.005242
Epoch = 31 , w = 1.916093,b = 3.049624 ,loss = 0.005186
Epoch = 32 , w = 1.923113,b = 3.046026 ,loss = 0.005140
Epoch = 33 , w = 1.929538,b = 3.042732 ,loss = 0.005102
Epoch = 34 , w = 1.935416,b = 3.039719 ,loss = 0.005070
Epoch = 35 , w = 1.940796,b = 3.036961 ,loss = 0.005043
Epoch = 36 , w = 1.945718,b = 3.034438 ,loss = 0.005020
Epoch = 37 , w = 1.950222,b = 3.032129 ,loss = 0.005001
Epoch = 38 , w = 1.954344,b = 3.030016 ,loss = 0.004986
Epoch = 39 , w = 1.958116,b = 3.028082 ,loss = 0.004973
Epoch = 40 , w = 1.961568,b = 3.026313 ,loss = 0.004962
Epoch = 41 , w = 1.964726,b = 3.024694 ,loss = 0.004953
Epoch = 42 , w = 1.967616,b = 3.023212 ,loss = 0.004945
Epoch = 43 , w = 1.970261,b = 3.021856 ,loss = 0.004939
Epoch = 44 , w = 1.972681,b = 3.020616 ,loss = 0.004933
Epoch = 45 , w = 1.974895,b = 3.019480 ,loss = 0.004929
Epoch = 46 , w = 1.976922,b = 3.018442 ,loss = 0.004925
Epoch = 47 , w = 1.978776,b = 3.017491 ,loss = 0.004922
Epoch = 48 , w = 1. 980473,b = 3.016621 ,loss = 0.004920
Epoch = 49 , w = 1.982026,b = 3.015825 ,loss = 0.004918

六、实验结论

SGD的迭代效果会和batch_size有关，batch_size过大或者过小都会导致权值的更新速度变慢。此处，batch_size = 10就是一个比较合适的大小。SGD在随着epoch增加的时候会发生上下波动，这也正是其随机性的体现。

七、问题解答

问题2：为什么是椭圆而不是圆？如何把这个图变成一个圆？
- 答：因为损失函数的表达式是\(f(w,b) = \Sigma(w*x+b-y)^2\)=\(w^2*x^2+b^2+y^2+C*w+D*b+K\),因此w前面有一个x的系数，这样的话这个函数就明显是一个椭圆形的函数。如果我们想要画成一个圆，只需要把损失函数\(f(w,b)\)改成\(f(w,b) = (w*y-w*x-b)^2\),这样的画画出来的损失函数关于\(w和b\)的图像将会是一个圆形。但是此时的损失函数的意义没有原来的那么明确。
问题3:为什么中心是个椭圆区域而不是一个点？
- 答：我的理解是
  - 1、我们的数据精度不够，导致原本不同的点由于精度的原因认为其相等。
  - 2、由于SGD只能得到局部最优解，而不能得到全局最优解，因此局部最优解可能有很多个，因此得到是一个区域。
  - 3、实际上我们的w和b也不是连续的而是一堆离散的点，因此会存在很多个近似相等的最优解。

转载于:https://www.cnblogs.com/somnus-chen/p/10574360.html

人工智能实战_第三次作业_陈泽寅相关推荐

人工智能实战_第三次作业_杨佳宁
使用Minibatch的方式进行梯度下降作业要求项目内容这个作业属于哪个课程班级博客这个作业的要求在哪里作业要求我在这个课程的目标是对于人工智能有一定的了解这个作业在哪个具体方面帮 ...
人工智能实战课第三次作业-尹正超
作业三:使用minibatch的方式进行梯度下降项目内容这个作业属于哪个课程人工智能实战这个作业的要求在哪里第三次作业-使用minibatch的方式进行梯度下降我在这个课程的目标是学习 ...
人工智能实战第三次作业_尹正超
作业三:使用minibatch的方式进行梯度下降项目内容这个作业属于哪个课程人工智能实战这个作业的要求在哪里第三次作业-使用minibatch的方式进行梯度下降我在这个课程的目标是学习 ...
人工智能实战_第八次作业_手写数学式子识别_廖盈嘉
第8次作业:手写数学式子识别项目内容这个作业属于哪个课程人工智能实战2019 这个作业的要求在哪里人工智能实战第八次作业我在这个课程的目标是学会.理解和应用神经网络知识来完成一个app ...
人工智能实战_第五次作业_杨佳宁
训练逻辑与门和逻辑或门作业要求项目内容这个作业属于哪个课程班级博客这个作业的要求在哪里作业要求我在这个课程的目标是对于人工智能有一定的了解这个作业在哪个具体方面帮助我实现目标了解 ...
操作系统_第三章处理器管理_批处理作业的调度算法
在批处理操作系统控制下的作业称为批处理作业. 在操作系统中,我们把磁盘上用来存放作业信息的专用区域称为输入井,把在输入井中等待处理的作业称为后备作业. 批处理操作系统应怎样从输入井中选取作业装入主存储 ...
人工智能实战2019第七次作业(黄金点) 16721088 焦宇恒
项目内容这个作业属于哪个课程人工智能实战2019 这个作业的要求在哪里黄金点游戏我在本次作业的目标体验黄金点游戏,构思AI策略黄金点游戏游戏介绍 N个同学(N通常大于10),每人写 ...
人工智能实战2019 第五次作业焦宇恒 16721088
标题内容这个作业属于哪个课程人工智能实战2019 这个作业的要求在哪里逻辑与非门这个作业在哪个具体方面帮助我实现目标神经网络二分类法逻辑与门训练样本 X1 X2 Y 0 0 0 0 1 ...
人工智能实战2019第六次作业焦宇恒
标题内容这个作业属于哪个课程人工智能实战2019 这个作业的要求在哪里练习调整超参这个作业在哪个具体方面帮助我实现目标体验超参对学习结果的影响实验结果本次实验,假设各个变量相互独立使用 ...

人工智能实战_第三次作业_陈泽寅