Spark中组件Mllib的学习27之逻辑回归-多元逻辑回归，较大数据集，带预测准确度计算

更多代码请见：https://github.com/xubo245/SparkLearning
Spark中组件Mllib的学习之逻辑回归篇
1解释
但预测较多数据集，需要去计算准确度

2.代码：

/*** @author xubo*         ref:Spark MlLib机器学习实战*         more code:https://github.com/xubo245/SparkLearning*         more blog:http://blog.csdn.net/xubo245*/
package org.apache.spark.mllib.learning.regressionimport org.apache.spark.mllib.classification.LogisticRegressionWithSGD
import org.apache.spark.mllib.evaluation.MulticlassMetrics
import org.apache.spark.mllib.regression.LabeledPoint
import org.apache.spark.mllib.util.MLUtils
import org.apache.spark.{SparkConf, SparkContext}/*** Created by xubo on 2016/5/23.* 多元逻辑回归,带验证*/
object LogisticRegression3Learning {def main(args: Array[String]) {val conf = new SparkConf().setMaster("local[4]").setAppName(this.getClass().getSimpleName().filter(!_.equals('$')))val sc = new SparkContext(conf)val data = MLUtils.loadLibSVMFile(sc, "file/data/mllib/input/regression/sample_libsvm_data.txt") //读取数据文件val splits = data.randomSplit(Array(0.6, 0.4), seed = 11L) //对数据集切分val parsedData = splits(0) //分割训练数据val parseTtest = splits(1) //分割测试数据val model = LogisticRegressionWithSGD.train(parsedData, 50) //训练模型println(model.weights) //打印θ值println("model.weights.size:" + model.weights.size) //打印θ数量val predictionAndLabels = parseTtest.map {//计算测试值case LabeledPoint(label, features) => //计算测试值val prediction = model.predict(features) //计算测试值(prediction, label) //存储测试和预测值}val metrics = new MulticlassMetrics(predictionAndLabels) //创建验证类val precision = metrics.precision //计算验证值println("data:" + data.count())println("parsedData:" + parsedData.count())println("parseTtest:" + parseTtest.count())println("Precision = " + precision) //打印验证值predictionAndLabels.take(10).foreach(println)sc.stop}
}

数据请见【3】

3.结果：

[0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,-0.4375,-1.9296875,0.26179428118654746,3.0555843001036025,3.956521016335986,4.299210393492351,0.8225802776670239,0.5234375,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,1.6031886937633835,4.77184399437807,2.489657736197254,-0.29118810889564173,1.84375,4.235675955010707,1.1769835598176726,-3.7047185056219294,-4.02175582125693,-7.542190343589054,-6.633736893650203,-1.0921125522771242,11.25260678716201,1.7762432743011,-0.6171875,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,1.5466055633952638,4.545511472905593,4.17943440746599,3.234121907465991,-1.101411437626905,-5.395106196318806,-1.3486748093031236,-8.04846850562193,-21.833636404670226,-26.340069558943227,-20.72913918349567,-6.776928228735921,-10.443710946436127,-0.8509002191452768,-4.605160592753207,-4.03125,-1.109375,-0.6640625,-0.109375,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,3.3006826048069655,3.382559407465991,0.8903719074659908,-1.9065030925340092,-9.119375680979342,-9.23669789928171,-15.54846850562193,-20.30628100562193,-19.74853923693182,-18.320277125928367,-16.08914880046484,-11.132220113435054,-16.88888593632102,-13.489860141109073,-16.165984023126715,-7.086473373361595,-3.9508895738313474,-1.953125,-1.140625,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,-0.7782439350714825,-0.7773471349678798,-2.265878092534009,-6.967146920243104,-11.384984336908616,-20.54470035889218,-23.026983716572694,-20.12800355795577,-5.8774845289734055,-8.868526284103204,-11.504061076868366,-26.616593931366943,-19.45246271914528,-21.16340021914528,-21.07746271914528,-6.5384002191452755,-2.355998877566452,-2.25,-1.953125,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,-0.1796875,-3.53125,-5.944106805214448,-7.33619059253401,-11.567543183534765,-23.610383313342695,-27.293936842776702,-17.03895243618413,-4.675236883299926,11.237650630594029,6.449887142702322,-1.5044209462714406,-12.443030870150169,-16.32746271914528,-25.225900219145284,-28.116525219145284,-19.737647698120988,-6.432994672891841,-3.5546875,-1.953125,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,-1.5078125,-5.7109375,-9.029584348159403,-14.67810076431383,-17.726429764762642,-18.01793852197583,-26.224336597465598,-8.584882291834965,9.68157181876785,16.164195470070425,18.039361220317293,13.918001418680438,5.764308537096623,-3.999337719145277,-25.671212719145284,-39.77170347242209,-34.19363593037352,-21.328125,-6.0703125,-1.9921875,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,-2.078125,-8.4765625,-13.819753643457837,-14.785942248852573,-19.635706075069162,-26.002176377629976,-24.956552126028495,-9.095098375851723,13.079114164488029,29.84531051269869,33.92217372031729,23.41883970349368,18.42421408780125,1.1412872808547232,-21.340875276157192,-47.24916171518676,-38.3046875,-26.8359375,-10.6484375,-2.765625,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,-4.9140625,-12.343425233427979,-9.994403485648718,-12.00126286075676,-21.58600936591399,-21.526636951378265,-27.20725167385357,-10.29318917385357,22.525377816277064,43.94938515648298,48.0808351098884,34.42121999551059,24.628456800885576,3.1016659243125595,-23.508895738313477,-41.2421875,-37.7890625,-31.1484375,-15.09375,-4.7421875,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,-8.12332343037351,-7.73121588760918,-4.539295174125693,-19.641355920795963,-32.063265395566816,-31.45888425270881,-21.898347956478467,-7.359238860285283,32.72275296800366,56.62341651394423,60.165528252319255,46.23999316089327,22.707692235445972,-4.340898259337763,-23.296875,-39.3046875,-37.390625,-34.2734375,-20.3515625,-7.5078125,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,-5.028991590650778,-7.186837719145277,-22.192866206133665,-37.620836282071245,-42.03881077786638,-33.60829304035597,-21.33898302300573,-3.241385960121552,50.05977294581118,64.1064511008563,65.25668077685344,46.58247310700115,19.248986212459045,-13.53793329532539,-22.640625,-37.7265625,-38.8515625,-34.4375,-22.234375,-9.875,0.0,0.0,0.0,0.0,0.0,0.0,0.0,-0.1484375,-4.2510717467231895,-20.301020457705086,-35.40205857593378,-43.1328125,-38.328125,-25.796875,-16.0234375,8.63208310634022,57.06457938088267,66.10225204100462,67.15834906814594,38.49519519391323,14.760683541292167,-12.4765625,-25.2421875,-36.734375,-36.6875,-31.0234375,-23.0390625,-9.8125,0.0,0.0,0.0,0.0,0.0,0.0,0.0,-0.265625,-16.03125,-31.53125,-39.1875,-46.125,-36.1953125,-21.796875,-6.2265625,23.759898650818428,59.12480270961393,73.11658429273758,60.25909547478095,32.47309810700115,4.50803519157861,-10.845129871165135,-27.71875,-36.875,-33.3671875,-27.875,-22.0390625,-8.3828125,0.0,0.0,0.0,0.0,0.0,0.0,0.0,-0.4140625,-19.6953125,-33.3046875,-43.796875,-44.8046875,-32.078125,-17.671875,2.4296875,29.5593197586919,55.482781494378074,72.83283092988812,51.00100354650988,22.750099224983508,3.026258355767928,-13.839521896215203,-31.02929856748394,-37.06272997852752,-34.3515625,-21.7578125,-18.859375,-8.171875,0.0,0.0,0.0,0.0,0.0,0.0,0.0,-1.640625,-20.7421875,-33.578125,-46.234375,-43.6171875,-31.625,-11.8984375,12.640625,29.9499447586919,54.389869779191315,62.02138284915817,44.45412854650988,19.333816018149744,-3.2379912924656056,-19.900109921776355,-30.029091702962905,-35.691600155418186,-29.5859375,-19.609375,-16.7265625,-6.234375,0.0,0.0,0.0,0.0,0.0,0.0,0.0,-1.5859375,-18.734375,-32.03125,-47.15625,-45.2109375,-32.109375,-4.288673267751275,19.997823136254897,33.86267139290246,57.02352410489334,55.09050201636554,37.11188586959847,11.15952931397337,-14.478409911767342,-26.447571628501258,-31.080972886159802,-24.82862711730755,-20.36425546985404,-18.6015625,-12.84375,-3.625,0.0,0.0,0.0,0.0,0.0,0.0,0.0,-0.859375,-13.0234375,-29.6640625,-45.32489465152724,-50.005269702102765,-32.787368060889996,-3.2826949954408393,18.18372167890559,32.31072891356959,51.13479079683827,51.08706664849493,28.628069053171927,1.1427713356495692,-18.504881894662258,-22.688604733473678,-21.158327300657415,-14.308393674875742,-18.441326985126373,-15.7421875,-8.765625,-1.0859375,0.0,0.0,0.0,0.0,0.0,0.0,0.0,-0.2265625,-7.3203125,-23.093085747521062,-34.79680738417467,-44.58199858131957,-32.25387358131957,-13.897639364550447,3.861299914080677,8.542740588737214,21.059299034017343,21.23538993647642,-0.09923012859382341,-13.614412728434514,-18.07738249495379,-9.899488504497675,-19.777557035737598,-16.039351316924147,-13.57092488982853,-11.875,-3.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,-2.3515625,-2.7250988472913926,-14.353679070561082,-29.966441684584918,-29.62883910654871,-21.749758236997273,-10.638976980474904,-14.47932433220015,-5.1571812247672035,-5.5712437247672035,-11.919570181311165,-18.383743724767207,-18.40873278065874,-17.873137097935928,-16.84400309253401,-12.29712809253401,-9.112039482105114,-5.0078125,-0.3046875,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,-0.640625,5.25279018781412,3.156154835989414,-7.560968340725907,-14.404145002168885,-17.532752786118134,-10.938431224767204,-15.508743724767205,-17.200455484481278,-15.204801962497713,-14.181554982948022,-19.211799382354275,-14.786989537131406,-11.92212809253401,-7.75025309253401,-5.68775309253401,-1.4948519821051138,-1.078125,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,-0.8872718607470209,1.1233693967346479,2.002583803981441,3.7897247808547228,-4.749337719145277,-11.382150219145277,-22.731209356295828,-23.845089683403984,-18.33924467289184,-12.729624719391614,-12.772878643457837,-9.469114696829891,-5.372953307258785,-2.414315592534009,1.3591219074659908,2.512960517894886,-0.078125,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,-0.3125,-1.8671875,-1.8984375,0.75,1.046875,-6.8984375,-12.296875,-14.7890625,-10.9140625,-6.625,-6.2109375,-5.125,-2.8515625,-0.8515625]
model.weights.size:692
data:100
parsedData:64
parseTtest:36
Precision = 1.0
(0.0,0.0)
(1.0,1.0)
(0.0,0.0)
(0.0,0.0)
(1.0,1.0)
(1.0,1.0)
(1.0,1.0)
(0.0,0.0)
(1.0,1.0)
(1.0,1.0)

准确度100%

参考
【1】http://spark.apache.org/docs/1.5.2/mllib-guide.html
【2】http://spark.apache.org/docs/1.5.2/programming-guide.html
【3】https://github.com/xubo245/SparkLearning

Spark中组件Mllib的学习27之逻辑回归-多元逻辑回归，较大数据集，带预测准确度计算相关推荐

Spark中组件Mllib的学习16之分布式行矩阵的四种形式
来源:http://blog.csdn.net/xubo245/article/details/51483995 更多代码请见:https://github.com/xubo245/SparkLear ...
Spark中组件Mllib的学习15之创建分布式矩阵
更多代码请见:https://github.com/xubo245/SparkLearning Spark中组件Mllib的学习之基础概念篇 1解释创建分布式矩阵 2.代码: /*** @autho ...
Spark中组件Mllib的学习40之梯度提升树（GBT）用于回归
更多代码请见:https://github.com/xubo245/SparkLearning Spark中组件Mllib的学习之分类篇 1解释 GBRT(Gradient Boost Regr ...
Spark中组件Mllib的学习19之分层抽样
更多代码请见:https://github.com/xubo245/SparkLearning Spark中组件Mllib的学习之基础概念篇 1解释分层抽样的概念就不讲了,具体的操作: RDD有个操 ...
Spark中组件Mllib的学习41之保序回归（Isotonic regression）
更多代码请见:https://github.com/xubo245/SparkLearning Spark中组件Mllib的学习之分类篇 1解释问题描述:给定一个无序数字序列,要求不改变每个元素的位 ...
Spark中组件Mllib的学习11之使用ALS对movieLens中一百万条（1M）数据集进行训练，并对输入的新用户数据进行电影推荐
更多代码请见:https://github.com/xubo245/SparkLearning 1解释 spark-1.5.2 数据集:http://grouplens.org/datasets/mo ...
Spark中组件Mllib的学习1之Kmeans错误解决
更多代码请见:https://github.com/xubo245/SparkLearning 解决办法:(中间比较多,为了方便看到,放在最开始) txt文件格式不对,用WPS转存的是UTF-16,s ...
Spark四大组件包括Spark Streaming、Spark SQL、Spark MLlib和Spark GraphX。
Spark四大组件包括Spark Streaming.Spark SQL.Spark MLlib和Spark GraphX.它们的主要应用场景是: Spark Streaming: Spark Str ...
华为云DAYU使用Spark组件开发的学习使用心得
自己学习的心得,如有错误欢迎指正- 简单认识首先华为DAYU平台中有两套Spark组件,一个是DLI Spark另一个是MRS Spark. DLI是数据湖探索服务,是完全兼容Apache Spar ...
中国Spark技术峰会（上）：Spark与生态圈中组件结合实战
5月13日-15日,由全球最大中文IT社区CSDN主办的"2016中国云计算技术大会"(Cloud Computing Technology Conference 2016,简称C ...

Spark中组件Mllib的学习27之逻辑回归-多元逻辑回归，较大数据集，带预测准确度计算

Spark中组件Mllib的学习27之逻辑回归-多元逻辑回归，较大数据集，带预测准确度计算相关推荐

最新文章

热门文章