回归篇 : LinearRegression

梯度下降
最小二乘

1. 梯度下降

从mmm样本数据(x,y)(x, y)(x,y)中去学习到一组权值向量,尽可能的拟合yyy
f(x)=x1w1+x2w2+....xnwn+b=y^f(x) = x_1w_1 + x_2w_2 + .... x_nw_n+b = \hat{y}f(x)=x1w1+x2w2+....xnwn+b=y^
目标函数
J(w)=12m∑i=1m(f(xi)−yi)2J(w) = \frac{1}{2m}\sum^{m}_{i=1}(f(x^i) - y^i)^2J(w)=2m1i=1∑m(f(xi)−yi)2
当J(w)J(w)J(w)最小时,即www为最优,梯度反向下,即函数J(w)J(w)J(w)的下降方向
dw=(y−y^)x,db=y−y^d_w = (y-\hat{y})x, d_b = y-\hat{y}dw=(y−y^)x,db=y−y^
更新梯度
w=w−dw\pmb w = \pmb w - d_wwww=www−dw

2. 最小二乘

方程组(xn+11=b1=1x^1_{n+1}= b^1 = 1xn+11=b1=1)
x11w1+x21w2+x31w3+....xn1wn+xn+11wn+1=f(x1)x^1_1w_1 + x^1_2w_2 + x^1_3w_3 + .... x^1_nw_n + x^1_{n+1}w_{n+1} = f(x^1)x11w1+x21w2+x31w3+....xn1wn+xn+11wn+1=f(x1)
x12w1+x22w2+x32w3+....xn2wn+xn+12wn+1=f(x2)x^2_1w_1 + x^2_2w_2 + x^2_3w_3 + .... x^2_nw_n + x^2_{n+1}w_{n+1} = f(x^2)x12w1+x22w2+x32w3+....xn2wn+xn+12wn+1=f(x2)
x13w1+x23w2+x33w3+....xn3wn+xn+13wn+1=f(x3)x^3_1w_1 + x^3_2w_2 + x^3_3w_3 + .... x^3_nw_n + x^3_{n+1}w_{n+1} = f(x^3)x13w1+x23w2+x33w3+....xn3wn+xn+13wn+1=f(x3)
.............................................................................................................................................
x1mw1+x2mw2+x3mw3+....xnmwn+xn+1mwn+1=f(xm)x^m_1w_1 + x^m_2w_2 + x^m_3w_3 + .... x^m_nw_n + x^m_{n+1}w_{n+1} = f(x^m)x1mw1+x2mw2+x3mw3+....xnmwn+xn+1mwn+1=f(xm)
矩阵乘法
[x11x21x31....xn1b1x12x22x32....xn2b2x13x23x33....xn3b3:::....::x1mx2mx3m....xnmbm][w1w2w3:wn+1]=[f(x1)f(x2)f(x3):f(xm)]\left[\begin{matrix}x^1_1 & x^1_2 & x^1_3 &....& x^1_n & b^1 \\ x^2_1 & x^2_2 & x^2_3 &....& x^2_n & b^2 \\ x^3_1 & x^3_2 & x^3_3 &....& x^3_n & b^3 \\ : & : & : & ....&: &:\\ x^m_1 & x^m_2 & x^m_3 &....& x^m_n & b^m\end{matrix}\right] \left[\begin{matrix}w_1 \\ w_2 \\ w_3 \\ : \\ w_{n+1} \end{matrix}\right] = \left[\begin{matrix}f(x^1) \\ f(x^2) \\ f(x^3) \\ : \\ f(x^m) \end{matrix}\right]⎣⎢⎢⎢⎢⎡x11x12x13:x1mx21x22x23:x2mx31x32x33:x3m....................xn1xn2xn3:xnmb1b2b3:bm⎦⎥⎥⎥⎥⎤⎣⎢⎢⎢⎢⎡w1w2w3:wn+1⎦⎥⎥⎥⎥⎤=⎣⎢⎢⎢⎢⎡f(x1)f(x2)f(x3):f(xm)⎦⎥⎥⎥⎥⎤
简化 : Xw=y^X \pmb w = \hat{y}Xwww=y^
X=[x11x21x31....xn1b1x12x22x32....xn2b2x13x23x33....xn3b3:::....::x1mx2mx3m....xnmbm],w=[w1w2w3:wn+1],y^=[f(x1)f(x2)f(x3):f(xm)]X = \left[\begin{matrix}x^1_1 & x^1_2 & x^1_3 &....& x^1_n & b^1 \\ x^2_1 & x^2_2 & x^2_3 &....& x^2_n & b^2 \\ x^3_1 & x^3_2 & x^3_3 &....& x^3_n & b^3 \\ : & : & : & ....&: &:\\ x^m_1 & x^m_2 & x^m_3 &....& x^m_n & b^m\end{matrix}\right], \pmb w = \left[\begin{matrix}w_1 \\ w_2 \\ w_3 \\ : \\ w_{n+1} \end{matrix}\right], \hat{y} = \left[\begin{matrix}f(x^1) \\ f(x^2) \\ f(x^3) \\ : \\ f(x^m) \end{matrix}\right]X=⎣⎢⎢⎢⎢⎡x11x12x13:x1mx21x22x23:x2mx31x32x33:x3m....................xn1xn2xn3:xnmb1b2b3:bm⎦⎥⎥⎥⎥⎤,www=⎣⎢⎢⎢⎢⎡w1w2w3:wn+1⎦⎥⎥⎥⎥⎤,y^=⎣⎢⎢⎢⎢⎡f(x1)f(x2)f(x3):f(xm)⎦⎥⎥⎥⎥⎤

对回归任务来说,要使的误差最小,即使f(x)≈yf(x) \approx yf(x)≈y

即(残差平方和最小):
J(w)=∣∣Xw−y∣∣2J(\pmb w) = || X\pmb w - \pmb y||^2J(www)=∣∣Xwww−yyy∣∣2

当w=w^\pmb w = \hat{w}www=w^时,J(w)J(w)J(w)最小,Xw^≈yX\hat{w} \approx \pmb yXw^≈yyy:
w^=argmin(J(w))\hat{w} = argmin(J(w))w^=argmin(J(w))

XTXw^=XTyX^TX\hat{w} = X^T\pmb yXTXw^=XTyyy

如果矩阵XTXX^TXXTX可逆则计算其逆矩阵,不可逆求其广义逆矩阵(伪逆)
w^=(XTX)−1XTy\hat{w} = (X^TX)^{-1}X^T\pmb yw^=(XTX)−1XTyyy

Linear Regression

import math
import warnings
import numpy as np
import seaborn as sns
import matplotlib.pyplot as plt
%matplotlib inline
warnings.filterwarnings('ignore')

class LinearRegression():"""gradient_descent: [True|False],True : 梯度下降,False : 最小二乘regularization : 正则化方法regularization.grad : 正则项"""def __init__(self, n_iterations=100, learning_rate=0.001, gradient_descent=True):self.n_iterations = n_iterationsself.learning_rate = learning_rateself.gradient_descent = gradient_descentself.regularization = lambda x: 0self.regularization.grad = lambda x: 0def initialize_weights(self, n_features):""" 随机初始化权重 """limit = 1 / math.sqrt(n_features)self.w = np.random.uniform(-limit, limit, (n_features, ))#self.w = np.ones(n_features)def fit(self, X, y):# 最小二乘if not self.gradient_descent:# 偏置1X = np.insert(X, 0, 1, axis=1)# 求逆U, S, V = np.linalg.svd(X.T.dot(X))S = np.diag(S)X_sq_reg_inv = V.dot(np.linalg.pinv(S)).dot(U.T)self.w = X_sq_reg_inv.dot(X.T).dot(y)else:# 增加偏置X = np.insert(X, 0, 1, axis=1)self.training_errors = []    # 训练过程中的误差self.weights = []            # 训练过程中的权重self.grads = []          # 训练过程中的梯度self.initialize_weights(n_features=X.shape[1])self.weights.append(self.w.copy())# 梯度下降for i in range(self.n_iterations):y_pred = X.dot(self.w)# 损失(MSE+正则项)mse = np.mean(0.5 * (y - y_pred)**2 + self.regularization(self.w))self.training_errors.append(mse)# 梯度grad_w = -(y - y_pred).dot(X) + self.regularization.grad(self.w)self.grads.append(grad_w)# 更新权值self.w -= self.learning_rate * grad_wself.weights.append(self.w.copy())def predict(self, X):X = np.insert(X, 0, 1, axis=1)y_pred = X.dot(self.w)return y_pred

Load dataset

sklearn中的数据集合

datasets.load_boston([return_X_y])          # Load and return the boston house-prices dataset (regression).
datasets.load_breast_cancer([return_X_y])     #Load and return the breast cancer wisconsin dataset (classification).
datasets.load_diabetes([return_X_y])        #Load and return the diabetes dataset (regression).
datasets.load_digits([n_class, return_X_y])    #Load and return the digits dataset (classification).
datasets.load_files(container_path[, …])     #Load text files with categories as subfolder names.
datasets.load_iris([return_X_y])           #Load and return the iris dataset (classification).
datasets.load_linnerud([return_X_y])        #Load and return the linnerud dataset (multivariate regression).
datasets.load_sample_image(image_name)       #Load the numpy array of a single sample image
datasets.load_sample_images()             #Load sample images for image manipulation.
datasets.load_wine([return_X_y])           #Load and return the wine dataset (classification).

from sklearn.datasets import load_diabetes
from sklearn.preprocessing import Normalizer
from sklearn.metrics import mean_squared_error
data = load_diabetes()

数据的简单可视化

plt.figure(figsize=(14, 6))
sns.heatmap(data.data)
plt.title('data visualization',fontsize=18)
_=plt.xticks(np.arange(10)+0.5, data.feature_names,rotation=30,fontsize=18)

训练模型

梯度下降
最小二乘

梯度下降

# 数据标准化
normalizer = Normalizer().fit(data.data)
norm_data = normalizer.transform(data.data)

plt.figure(figsize=(14, 5))
sns.heatmap(norm_data)
_=plt.xticks(np.arange(10)+0.5, data.feature_names,rotation=45)

标准化后的数据

lr_1 = LinearRegression(gradient_descent=True, n_iterations=200)lr_1.fit(norm_data, data.target)

权重的变化

迭代过程中梯度的变化

训练误差曲线

training error : 1464.4834476392352

最小二乘

lr_2 = LinearRegression(gradient_descent=False)lr_2.fit(norm_data, data.target)

training error

print('train error :',mean_squared_error(lr_2.predict(norm_data), data.target))

train error : 10796.47034363991

NumpyML : Linear Regression相关推荐

【cs229-Lecture2】Linear Regression with One Variable (Week 1)(含测试数据和源码)
从Ⅱ到Ⅳ都在讲的是线性回归,其中第Ⅱ章讲得是简单线性回归(simple linear regression, SLR)(单变量),第Ⅲ章讲的是线代基础,第Ⅳ章讲的是多元回归(大于一个自变量). 本文的 ...
Linear Regression（一）——
Linear Regression(一)-- 机器学习回归定义回归的定义在平面上存在这些点我希望能用一条直线尽可能经过它们. 于是我们画了下面的一条直线这样的过程就叫做回归. 这个过程中我们 ...
R语言使用lm函数拟合多元线性回归模型、假定预测变量没有交互作用（Multiple linear regression）
R语言使用lm函数拟合多元线性回归模型.假定预测变量没有交互作用(Multiple linear regression) 目录
Python使用sklearn和statsmodels构建多元线性回归模型（Multiple Linear Regression）并解读
Python使用sklearn和statsmodels构建多元线性回归模型(Multiple Linear Regression)并解读 #仿真数据集(预测股票指数) 这里的目标是根据两个宏观经济变量 ...
R语言可视化包ggplot2绘制线性回归模型曲线实战（ Linear Regression Line）
R语言可视化包ggplot2绘制线性回归模型曲线实战( Linear Regression Line) 目录 R语言可视化包ggplot2绘制线性回归模型曲线实战( Linear Regression ...
R语言基于线性回归（Linear Regression）进行特征筛选（feature selection）
R语言基于线性回归(Linear Regression)进行特征筛选(feature selection) 对一个学习任务来说,给定属性集,有些属性很有用,另一些则可能没什么用.这里的属性即称为&qu ...
局部加权线性回归(Local Weighted Linear Regression)+局部加权回归+局部线性回归
局部加权线性回归(Local Weighted Linear Regression)+局部加权回归+局部线性回归 locally weighted scatterplot smoothing,LOWE ...
python中的linearregression_【python+机器学习（2）】python实现Linear Regression
欢迎关注哈希大数据微信公众号[哈希大数据] python实现多元线性回归算法(lr) 想必大家在很早之前就接触过函数的概念,即寻找自变量和因变量之间的对应关系,一元一次.多元一次.一元二次等等,表示的 ...
Linear regression with one variable算法实例讲解（绘制图像，cost_Function ,Gradient Desent, 拟合曲线, 轮廓图绘制）_矩阵操作...
%测试数据 'ex1data1.txt', 第一列为 population of City in 10,000s, 第二列为 Profit in $10,000s 1 6.1101,17.592 2 ...
机器学习之线性回归（Linear Regression）
线性学习中最基础的回归之一,本文从线性回归的数学假设,公式推导,模型算法以及实际代码运行几方面对这一回归进行全面的剖析~ 一:线性回归的数学假设 1.假设输入的X和Y是线性关系,预测的y与X通过线性方 ...

NumpyML : Linear Regression