ML Notes: Week 1 - Univariate Linear Regression

1. The Basic Theory

Hypothesis: hθ(x)=θ0+θ1xh_\theta(x)=\theta_0+\theta_1xhθ(x)=θ0+θ1x

Parameters: θ0,θ1\theta_0, \theta_1θ0,θ1

Cost Function: J(θ)=12m∑i=1m(hθ(x(i))−y(i))2J(\theta)=\frac{1}{2m}\sum\limits_{i=1}^m(h_\theta(x^{(i)})-y^{(i)})^2J(θ)=2m1i=1∑m(hθ(x(i))−y(i))2

Objective Function: min⁡θ0,θ1J(θ)\min\limits_{\theta_0,\theta_1}J(\theta)θ0,θ1minJ(θ)

2. Gradient descent of linear regression

2.1 What is the gradient descent ?

We adopt the Gradient Descent Method to find the optimum of θ0,θ1\theta_0,\theta_1θ0,θ1

θj=θj−α∂∂θjJ(θ0,θ1)=θj−αm∑i=1m(hθ(x(i))−y(i))∂hθ(x)∂θj\theta_j=\theta_j-\alpha \frac{\partial}{\partial\theta_j}J(\theta_0,\theta_1) =\theta_j-\frac{\alpha}{m}\sum\limits_{i=1}^m(h_\theta(x^{(i)})-y^{(i)})\frac{\partial h_\theta(x)}{\partial\theta_j}θj=θj−α∂θj∂J(θ0,θ1)=θj−mαi=1∑m(hθ(x(i))−y(i))∂θj∂hθ(x)

so, when j=0j=0j=0, θ0=θ0−αm∑i=1m(hθ(x(i))−y(i))\theta_0=\theta_0-\frac{\alpha}{m}\sum\limits_{i=1}^m(h_\theta(x^{(i)})-y^{(i)})θ0=θ0−mαi=1∑m(hθ(x(i))−y(i)).

j=1j=1j=1, θ1=θ1−αm∑i=1m[(hθ(x(i))−y(i))∗x(i)]\theta_1=\theta_1-\frac{\alpha}{m}\sum\limits_{i=1}^m[(h_\theta(x^{(i)})-y^{(i)})*x^{(i)}]θ1=θ1−mαi=1∑m[(hθ(x(i))−y(i))∗x(i)].

We’d better to update the parameters simultaneously.

2.2 How to implement it？

STEP 1: Initialize the θ0,θ1\theta_0,\theta_1θ0,θ1

STEP 2: Calculate the hθ(x)h_\theta(x)hθ(x)

STEP 3: Come up with the new θ0,θ1\theta_0,\theta_1θ0,θ1 under the condition of the hθ(x)h_\theta(x)hθ(x) in STEP 2

STEP 4: Loop from STEP 2 - STEP 3

When to stop? We can set the times of iteration.

2.3 Matlab code for gradient descent

%% ================== Data Generation ===================
x = normrnd(0, 0.6, 100, 1);
noise = normrnd(0, 0.03, 100, 1);
y = 0.43 + 0.22 * x + noise;%% ================== Linear Regression ===================
m = length(y);
X = [ones(m,1),x];
% initialize the theta
theta = [0;0];theta_length = length(theta);
itera = 5000;
alpha = 0.01;
costFunc = zeros(m,1);
theta_itera = zeros(m,2);for j = 1:iteratheta_itera(j,:) = theta';  % record all the theta values during the processhypothesis = X * theta;costFunc(j) = (1/2*m) * sum(((hypothesis - y) .^ 2)); % cost function% using the gradienet descent algorithm to find the optimumfor i = 1:theta_lengththeta(i) = theta(i) - (alpha/m) * ((hypothesis - y)'* X(:,i));  end
end%% ================== Visualization ===================
%% plot the dataset and the fit line
figure;
for j = 1:10:iteraplot(x,y,'bx');hold onh2 = plot(X(:,2),X*theta_itera(j,:)','r');pause(0.5);delete(h2);
end%% plot the iterate curve
figure;
plot3(theta_itera(:,1),theta_itera(:,2),costFunc,'r');
xlabel('\theta_0'); ylabel('\theta_1');
hold on%% plot the costFunction
% Grid over which we will calculate J
theta0_vals = linspace(-1, 1, 100);
theta1_vals = linspace(-1, 1, 100);
plot_costFunc = zeros(length(theta0_vals), length(theta1_vals));
% Fill
for i = 1:length(theta0_vals)for j = 1:length(theta1_vals)t = [theta0_vals(i); theta1_vals(j)];plot_hypothesis = X * t;plot_costFunc(i,j) = (1/2*m) * sum(((plot_hypothesis - y) .^ 2));end
end% Surface plot
%   In this case, the vertices of the surface patches are the triples
%   (x(j), y(i), Z(i,j)). Note that x corresponds to the columns of Z
%   and y corresponds to the rows.
plot_costFunc = plot_costFunc';
surf(theta0_vals, theta1_vals,plot_costFunc)
xlabel('\theta_0'); ylabel('\theta_1');%% Contour map of the costFunction
figure;
contour(theta0_vals, theta1_vals,  plot_costFunc, logspace(-2, 3, 20))
xlabel('\theta_0'); ylabel('\theta_1');
hold on;
plot(theta_itera(:,1), theta_itera(:,2), 'r');