1、基于概率的机器学习算法

机器学习算法可以分为基于概率、基于距离、基于树和基于神经网络四类。基于概率的机器学习算法本质上是计算每个样本属于对应类别的概率，然后利用极大似然估计法对模型进行训练。基于概率的机器学习算法的损失函数为负的log似然函数。
基于概率的机器学习算法包括朴素贝叶斯算法、Logistic Regression算法、Softmax Regression算法和Factorization Machine算法等。

2、逻辑回归算法原理

2.1、分离超平面

Logistic Regression算法是二分类线性分类算法，分离超平面采用线性函数：
Wx+b=0Wx + b = 0Wx+b=0
xxx是样本特征矩阵，特征数为mmm，其中W（1∗m）W（1*m）W（1∗m）是权重矩阵。通过分类超平面可以将数据分成正负两个类别，类别为正的样本标签标记为1，类别为负的样本标签标记为0。

2.2、阈值函数

通过阈值函数可以将样本到分离超平面的距离映射到不同的类别，Logistic Regression算法中阈值函数采用Sigmoid函数：
f(x)=11+e−xf(x) = \frac{1}{{1 + {e^{ - x}}}}f(x)=1+e−x1
sigmoid函数的图像如下：

对于样本xxx，其到分离超平面的几何距离DDD为：
D=Wx+bD = Wx + bD=Wx+b

2.3、样本概率

假设样本xxx为正类别，则其概率为：
p(y=1∣x,W,b)=σ(Wx+b)=11+e−(Wx+b)p\left( {y = 1\left| {x,W,b} \right.} \right) = \sigma \left( {Wx + b} \right) = \frac{1}{{1 + {e^{ - \left( {Wx + b} \right)}}}}p(y=1∣x,W,b)=σ(Wx+b)=1+e−(Wx+b)1
负类别样本的概率：
p(y=0∣x,W,b)=1−p(y=1∣x,W,b)=e−(Wx+b)1+e−(Wx+b)p\left( {y = 0\left| {x,W,b} \right.} \right) = 1 - p\left( {y = 1\left| {x,W,b} \right.} \right) = \frac{{{e^{ - \left( {Wx + b} \right)}}}}{{1 + {e^{ - \left( {Wx + b} \right)}}}}p(y=0∣x,W,b)=1−p(y=1∣x,W,b)=1+e−(Wx+b)e−(Wx+b)

将两种类别合并，属于类别yyy的概率为：
p(y∣x,W,b)=σ(Wx+b)y(1−σ(Wx+b))1−yp\left( {y\left| {x,W,b} \right.} \right) = \sigma {\left( {Wx + b} \right)^y}{\left( {1 - \sigma \left( {Wx + b} \right)} \right)^{1 - y}}p(y∣x,W,b)=σ(Wx+b)y(1−σ(Wx+b))1−y

2.4、损失函数

设训练数据集有nnn个训练样本{(x1,y1),(x2,y2),⋯ ,(xn,yn)}\left\{ {\left( {{x_1},{y_1}} \right),\left( {{x_2},{y_2}} \right), \cdots ,\left( {{x_n},{y_n}} \right)} \right\}{(x1,y1),(x2,y2),⋯,(xn,yn)}，其似然函数为：

LW,b=∏i=1n[σ(Wxi+b)yi(1−σ(Wxi+b))1−yi]{L_{W,b}} = \prod\limits_{i = 1}^n {\left[ {\sigma {{\left( {W{x_i} + b} \right)}^{{y_i}}}{{\left( {1 - \sigma \left( {W{x_i} + b} \right)} \right)}^{1 - {y_i}}}} \right]}LW,b=i=1∏n[σ(Wxi+b)yi(1−σ(Wxi+b))1−yi]

Logistic Regression算法的损失函数为负的log似然函数：

lW,b=−1n∑i=1n[yilog⁡(σ(Wxi+b))+(1−yi)log⁡(1−σ(Wxi+b))]{l_{W,b}} = - \frac{1}{n}\sum\limits_{i = 1}^n {\left[ {{y_i}\log \left( {\sigma \left( {W{x_i} + b} \right)} \right) + \left( {1 - {y_i}} \right)\log \left( {1 - \sigma \left( {W{x_i} + b} \right)} \right)} \right]}lW,b=−n1i=1∑n[yilog(σ(Wxi+b))+(1−yi)log(1−σ(Wxi+b))]

模型训练是为了求取最优的权值矩阵WWW和偏置bbb，将模型训练问题转化为最小化损失函数：
min⁡W,blW,b\mathop {\min }\limits_{W,b} {l_{W,b}}W,bminlW,b

3、基于梯度下降法的模型训练

本博文中，最小化损失函数的求解采用梯度下降法。
第一步：初始化权重矩阵W=W0W=W_0W=W0和偏置b=b0b=b_0b=b0。

第二步：重复如下过程：
计算参数的梯度下降方向：

∂Wi=−∂lW,b∂W∣Wi\partial {W_i} = - \frac{{\partial {l_{W,b}}}}{{\partial W}}\left| {_{{W_i}}} \right.∂Wi=−∂W∂lW,b∣Wi

∂bi=−∂lW,b∂b∣bi\partial {b_i} = - \frac{{\partial {l_{W,b}}}}{{\partial b}}\left| {_{{b_i}}} \right.∂bi=−∂b∂lW,b∣bi

选择步长α\alphaα
更新参数：
Wi+1=Wi+α⋅∂Wi{W_{i + 1}} = {W_i} + \alpha \cdot \partial {W_i}Wi+1=Wi+α⋅∂Wi
bi+1=bi+α⋅∂bi{b_{i + 1}} = {b_i} + \alpha \cdot \partial {b_i}bi+1=bi+α⋅∂bi

第三步：判断是否达到终止条件。
假设xijx_i^jxij是样本xix_ixi的第jjj个特征分量，wjw_jwj为权重矩阵WWW的第jjj个分量，取w0=b{w_0} = bw0=b，则权重矩阵中第jjj个分量的梯度方向为：

∂wj=−1n∑i=1n(yi−σ(Wxi+b))xij\partial {w_j} = - \frac{1}{n}\sum\limits_{i = 1}^n {\left( {{y_i} - \sigma \left( {W{x_i} + b} \right)} \right)} x_i^j∂wj=−n1i=1∑n(yi−σ(Wxi+b))xij

4、java实现

完整代码和数据样本地址：https://github.com/shiluqiang/Logistic_Regression_java
首先：导入样本特征和标签。

import java.io.*;public class LoadData {//导入样本特征public static double[][] Loadfeature(String filename) throws IOException{File f = new File(filename);FileInputStream fip = new FileInputStream(f);// 构建FileInputStream对象InputStreamReader reader = new InputStreamReader(fip,"UTF-8");// 构建InputStreamReader对象StringBuffer sb = new StringBuffer();while(reader.ready()) {sb.append((char) reader.read());}reader.close();fip.close();//将读入的数据流转换为字符串String sb1 = sb.toString();//按行将字符串分割,计算二维数组行数String [] a = sb1.split("\n");int n = a.length;System.out.println("二维数组行数为:" + n);//计算二维数组列数String [] a0 = a[0].split("\t");int m = a0.length;System.out.println("二维数组列数为:" + m);double [][] feature = new double[n][m];for (int i = 0; i < n; i ++) {String [] tmp = a[i].split("\t");for(int j = 0; j < m; j ++) {if (j == m-1) {feature[i][j] = (double) 1;}else {feature[i][j] = Double.parseDouble(tmp[j]);}            }           }return feature;        }//导入样本标签public static double[] LoadLabel(String filename) throws IOException{File f = new File(filename);FileInputStream fip = new FileInputStream(f);// 构建FileInputStream对象InputStreamReader reader = new InputStreamReader(fip,"UTF-8");// 构建InputStreamReader对象,编码与写入相同StringBuffer sb = new StringBuffer();while(reader.ready()) {sb.append((char) reader.read());}reader.close();fip.close();//将读入的数据流转换为字符串String sb1 = sb.toString();//按行将字符串分割,计算二维数组行数String [] a = sb1.split("\n");int n = a.length;System.out.println("二维数组行数为:" + n);//计算二维数组列数String [] a0 = a[0].split("\t");int m = a0.length;System.out.println("二维数组列数为:" + m);double [] Label = new double[n];for (int i = 0; i < n; i ++) {String [] tmp = a[i].split("\t");Label[i] = Double.parseDouble(tmp[m-1]);            }return Label;      }
}

然后，利用梯度下降算法优化Logistic Regression模型。

public class LRtrainGradientDescent {int paraNum; //权重参数的个数double rate; //学习率int samNum; //样本个数double [][] feature; //样本特征矩阵double [] Label;//样本标签int maxCycle; //最大迭代次数public LRtrainGradientDescent(double [][] feature, double [] Label, int paraNum,double rate, int samNum,int maxCycle) {this.feature = feature;this.Label = Label;this.maxCycle = maxCycle;this.paraNum = paraNum;this.rate = rate;this.samNum = samNum;        }// 权值矩阵初始化public double [] ParaInitialize(int paraNum) {double [] W = new double[paraNum];for (int i = 0; i < paraNum; i ++) {W[i] =  1.0;}return W;       }//计算每次迭代后的预测误差public double [] PreVal(int samNum,int paraNum, double [][] feature,double [] W) {double [] Preval = new double[samNum];for (int i = 0; i< samNum; i ++) {double tmp = 0;for(int j = 0; j < paraNum; j ++) {tmp += feature[i][j] * W[j];}Preval[i] = Sigmoid.sigmoid(tmp);}return Preval;}//计算误差率public double error_rate(int samNum, double [] Label, double [] Preval) {double sum_err = 0.0;for(int i = 0; i < samNum; i ++) {sum_err += Math.pow(Label[i] - Preval[i], 2);         }return sum_err;}//LR模型训练public double[] Updata(double [][] feature, double[] Label, int maxCycle, double rate) {// 先计算样本个数和特征个数int samNum = feature.length;int paraNum = feature[0].length;//初始化权重矩阵double [] W = ParaInitialize(paraNum);// 循环迭代优化权重矩阵for (int i = 0; i < maxCycle; i ++) {// 每次迭代后，样本预测值double [] Preval = PreVal(samNum,paraNum,feature,W);double sum_err = error_rate(samNum,Label,Preval);if (i % 10 == 0) {System.out.println("第" + i + "次迭代的预测误差为:" + sum_err);}//预测值与标签的误差double [] err = new double[samNum];for(int j = 0; j < samNum; j ++) {err[j] = Label[j] - Preval[j];}// 计算权重矩阵的梯度方向double [] Delt_W = new double[paraNum];for (int n = 0 ; n < paraNum; n ++) {double tmp = 0;for(int m = 0; m < samNum; m ++) {tmp += feature[m][n] * err[m];}Delt_W[n] = tmp / samNum;} for(int m = 0; m < paraNum; m ++) {W[m] = W[m] + rate * Delt_W[m];}}return W;}
}

Sigmoid函数

public class Sigmoid {public static double sigmoid(double x) {double i = 1.0;double y = i / (i + Math.exp(-x));return y;}
}

Logistic Regression模型参数和测试结果存储。

import java.io.*;
public class SaveModel {public static void savemodel(String filename, double [] W) throws IOException{File f = new File(filename);// 构建FileOutputStream对象FileOutputStream fip = new FileOutputStream(f);// 构建OutputStreamWriter对象OutputStreamWriter writer = new OutputStreamWriter(fip,"UTF-8");//计算模型矩阵的元素个数int n = W.length;StringBuffer sb = new StringBuffer();for (int i = 0; i < n-1; i ++) {sb.append(String.valueOf(W[i]));sb.append("\t");}sb.append(String.valueOf(W[n-1]));String sb1 = sb.toString();writer.write(sb1);writer.close();fip.close();}public static void saveresults(String filename, double [] pre_results) throws IOException{File f = new File(filename);// 构建FileOutputStream对象FileOutputStream fip = new FileOutputStream(f);// 构建OutputStreamWriter对象OutputStreamWriter writer = new OutputStreamWriter(fip,"UTF-8");//计算预测结果的个数int n = pre_results.length;StringBuffer sb = new StringBuffer();for (int i = 0; i < n-1; i ++) {sb.append(String.valueOf(pre_results[i]));sb.append("\n");}sb.append(String.valueOf(pre_results[n-1]));String sb1 = sb.toString();writer.write(sb1);writer.close();fip.close();}
}

主类。

import java.io.*;
public class LRMain {public static void main(String[] args) throws IOException{// filename String filename = "data.txt";// 导入样本特征和标签double [][] feature = LoadData.Loadfeature(filename);double [] Label = LoadData.LoadLabel(filename); // 参数设置int samNum = feature.length;int paraNum = feature[0].length;double rate = 0.01;int maxCycle = 1000;// LR模型训练LRtrainGradientDescent LR = new LRtrainGradientDescent(feature,Label,paraNum,rate,samNum,maxCycle);double [] W = LR.Updata(feature, Label, maxCycle, rate);//保存模型String model_path = "wrights.txt";SaveModel.savemodel(model_path, W);//模型测试double [] pre_results = LRTest.lrtest(paraNum, samNum, feature, W);//保存测试结果String results_path = "pre_results.txt";SaveModel.saveresults(results_path, pre_results);}
}

【机器学习】Logistic Regression逻辑回归原理与java实现相关推荐

【李宏毅机器学习】Logistic Regression 逻辑回归（p11）学习笔记
李宏毅机器学习学习笔记汇总课程链接文章目录 Logistic Regression Step 1: Function Set Step 2: Goodness of a Function Step ...
Logistic Regression逻辑回归的损失函数与梯度下降训练
有一篇博文提到logistic regression的简单理解(Logistic Regression逻辑回归的简单解释).逻辑回归实际上是odds取对数后的反函数,其函数形式也称为sigmoid f ...
Logistic Regression 逻辑回归数学原理、python代码实现、实际应用
说在前面第一次写博客,主要目的是再梳理一下学到东西的逻辑,如果可以帮助到其他在学习的人就更好啦.本篇主要参考的:<机器学习>西瓜书.博主文章:文章链接.以及知乎.百度等大神们的解惑文章 ...
Logistic Regression逻辑回归的简单解释
Logistic Regression也叫Logit Regression,在机器学习中属于参数估计的模型.逻辑回归与普通线性回归(Linear Regression)有很大的关系.在应用上,它们有所 ...
Tensorflow【实战Google深度学习框架】—Logistic regression逻辑回归模型实例讲解
文章目录 1.前言 2.程序详细讲解环境设定数据读取准备好placeholder,开好容器来装数据准备好参数/权重拿到每个类别的score 计算多分类softmax的loss functio ...
Logistic Regression逻辑回归
参考自: http://blog.sina.com.cn/s/blog_74cf26810100ypzf.html http://blog.sina.com.cn/s/blog_64ecfc2f010 ...
Logistic Regression(逻辑回归) +python3.6(pycharm)实现
数学基础知识略过,可自行查询探究. 遇到的bugs: 1.AttributeError: module 'scipy' has no attribute '__version__' 解决办法:inst ...
机器学习入门04-线性回归原理与java实现多元线性回归
线性回归原理: 线性回归公式:y = b + w*x,w表示权重b表示偏置. 在实际实现中可以将公式写作:y = w[0] * x[0] + w[1] * x[1],x[0]=1,这样就可以很方便的进 ...
机器学习-非线性回归( Unlinear Regression) -逻辑回归（Logistic Regression）算法
学习彭亮<深度学习基础介绍:机器学习>课程概率定义概率(Probability): 对一件事情发生的可能性的衡量范围 0 <= P <= 1 计算方法根据个人置信根 ...

【机器学习】Logistic Regression逻辑回归原理与java实现