http://blog.csdn.net/u011239443/article/details/51858825

数据集处理

import org.apache.spark.mllib.linalg._
import org.apache.spark.mllib.regression._
val rawData = sc.textFile("covtype.data")
val data = rawData.map{ line =>val values = line.split(",").map( _.toDouble)//init返回除最后一个值外的所有值val featureVector = Vectors.dense(values.init)//决策树要求label从0开始val label = values.last -1LabeledPoint( label,featureVector)
}val Array(trainData,cvData,testData) = data.randomSplit( Array(0.8,0.1,0.1))
trainData.cache()
cvData.cache() //交叉检验集
testData.cache()

模型训练

import org.apache.spark.mllib.tree._
import org.apache.spark.mllib.tree.model._
import org.apache.spark.rdd._def getMetrics(model: DecisionTreeModel,dta: RDD[ LabeledPoint ]):MulticlassMetrics = {val predictionsAndLabels = data.map( example =>( model.predict( example.features), example.label))new MulticlassMetrics( predictionsAndLabels)}val model = DecisionTree.trainClassifier(trainData,7,Map[Int,Int](),"gini",4,100)

决策树有训练分类模型的函数trainClassifier和回归模型的函数trainRegressor，这里我们使用trainClassifier。
我们来看看trainClassifier都需要什么参数：

scala> DecisionTree.trainClassifier
<console>:42: error: ambiguous reference to overloaded definition,
both method trainClassifier in object DecisionTree of type (input: org.apache.spark.api.java.JavaRDD[org.apache.spark.mllib.regression.LabeledPoint], numClasses: Int, categoricalFeaturesInfo: java.util.Map[Integer,Integer], impurity: String, maxDepth: Int, maxBins: Int)org.apache.spark.mllib.tree.model.DecisionTreeModel
and  method trainClassifier in object DecisionTree of type (input: org.apache.spark.rdd.RDD[org.apache.spark.mllib.regression.LabeledPoint], numClasses: Int, categoricalFeaturesInfo: Map[Int,Int], impurity: String, maxDepth: Int, maxBins: Int)org.apache.spark.mllib.tree.model.DecisionTreeModel
match expected type ?DecisionTree.trainClassifier^

input:数据的LabeledPoint
numClasses：类别数量
categoricalFeaturesInfo：
看下Saprk官网的介绍:

categoricalFeaturesInfo: Specifies which features are categorical and how many categorical values each of those features can take. This is given as a map from feature indices to feature arity (number of categories). Any features not in this map are treated as continuous.

E.g., Map(0 -> 2, 4 -> 10) specifies that feature 0 is binary (taking values 0 or 1) and that feature 4 has 10 categories (values {0, 1, …, 9}). Note that feature indices are 0-based: features 0 and 4 are the 1st and 5th elements of an instance’s feature vector.

Note that you do not have to specify categoricalFeaturesInfo. The algorithm will still run and may get reasonable results. However, performance should be better if categorical features are properly designated.

impurity：不纯度的类型，有基尼不纯度——“gini”，熵——“entropy”
maxDepth：对层数进行限制，避免过拟合
maxBins：决策规则集，可以理解成是决策树的孩子节点的数量

性能评估

import org.apache.spark.mllib.evaluation._
val metrics = getMetrics(model,cvData)
metrics.confusionMatrix
/*
res6: org.apache.spark.mllib.linalg.Matrix =
156710.0  51350.0   203.0    0.0  0.0    0.0  3577.0
68735.0   207253.0  6883.0   0.0  42.0   0.0  388.0
0.0       5872.0    29882.0  0.0  0.0    0.0  0.0
0.0       0.0       2747.0   0.0  0.0    0.0  0.0
105.0     8702.0    557.0    0.0  129.0  0.0  0.0
0.0       4475.0    12892.0  0.0  0.0    0.0  0.0
11290.0   239.0     55.0     0.0  0.0    0.0  8926.0
*/

因为一共有7种类别，所以生成的是7*7的矩阵，aij 表示实际类别是i，而被预测类别是j的次数。

metrics.precision
//res7: Double = 0.6934452300468837

决策树调优

val evaluations =for (impurity <- Array("gini", "entropy");depth    <- Array(1, 20);bins     <- Array(10, 300))yield {val model = DecisionTree.trainClassifier(trainData, 7, Map[Int,Int](), impurity, depth, bins)val accuracy = getMetrics(model, cvData).precision((impurity, depth, bins), accuracy)}
evaluations.sortBy(_._2).reverse.foreach( println)
/*
((entropy,20,300),0.9380098861985638)
((gini,20,300),0.9319721451536285)
((entropy,20,10),0.9273681094366382)
((gini,20,10),0.9195954644654499)
((gini,1,10),0.633916339077334)
((gini,1,300),0.6335772755123819)
((entropy,1,300),0.48759922342395684)
((entropy,1,10),0.48759922342395684)
*/

scala语法：

  for (impurity <- Array("gini", "entropy");depth    <- Array(1, 20);bins     <- Array(10, 300))yield {}

相当于关于impurity，depth，bins的三层循环。

关于categoricalFeaturesInfo

关于categoricalFeaturesInfo这个参数，我们前面直接不设定取值个数:

Map[Int,Int]()

但是，我们可以参阅下covtype.info关于数据集的描述：

……

Hillshade_9am quantitative 0 to 255 index Hillshade index at 9am, summer solstice

Hillshade_Noon quantitative 0 to 255 index Hillshade index at noon, summer soltice

Hillshade_3pm quantitative 0 to 255 index Hillshade index at 3pm, summer solstice

Wilderness_Area (4 binary columns) qualitative 0 (absence) or 1 (presence) Wilderness area designation

Soil_Type (40 binary columns) qualitative 0 (absence) or 1 (presence) Soil Type designation

……

Wilderness Areas: 1 – Rawah Wilderness Area
2 – Neota Wilderness Area
3 – Comanche Peak Wilderness Area
4 – Cache la Poudre Wilderness Area

Soil Types: 1 to 40 : based on the USFS Ecological
Landtype Units (ELUs) for this study area

可知：
- 三个Hillshade都有256种取值
- Wilderness Areas 有4中类别，Soil Types 有40种。数据集中是以二元特征的形式，有4列，如取值为3，那么第三列为1，其它列都为0

重新处理数据集

  def unencodeOneHot(rawData: RDD[String]): RDD[LabeledPoint] = {rawData.map { line =>val values = line.split(',').map(_.toDouble)/*我们可以从covtype.info中得知：wilderness是从第10行开始的，slice(10, 14) 截取 10 到 13 行indexOf(1.0)  返回值为1的位置编号*/val wilderness = values.slice(10, 14).indexOf(1.0).toDoubleval soil = values.slice(14, 54).indexOf(1.0).toDoubleval featureVector = Vectors.dense(values.slice(0, 10) :+ wilderness :+ soil)val label = values.last - 1LabeledPoint(label, featureVector)}}val data = unencodeOneHot(rawData)val Array(trainData, cvData, testData) = data.randomSplit(Array(0.8, 0.1, 0.1))trainData.cache()cvData.cache()testData.cache()

重新评估性能

这里进行参数设置时发现这样的错误：

java.lang.IllegalArgumentException: requirement failed: DecisionTree requires maxBins (= 40) to be at least as large as the number of values in each categorical feature, but categorical feature 6 has 256 values. Considering remove this and other categorical features with a large number of values, or add more training examples.

所以：bins数量必须大于等于Max（各个feature的values数量）

val evaluations =for (impurity <- Array("gini", "entropy");depth    <- Array(10, 20,30);bins     <- Array(256, 300))yield{val model = DecisionTree.trainClassifier(trainData,7,Map(6 -> 256,7 -> 256,8 -> 256,10 -> 4,11 -> 40),impurity, depth, bins)val accurary = getMetrics(model, cvData).precision(( impurity,depth,bins), accurary)}evaluations.sortBy(_._2).reverse.foreach( println)
/*
((gini,30,300),0.6327390828416625)
((gini,20,300),0.6319645721602997)
((gini,10,256),0.6190078690285227)
((gini,30,256),0.6165724632193483)
((gini,20,256),0.6149373851142489)
((gini,10,300),0.596522963381135)
((entropy,30,256),0.5868863293701335)
((entropy,20,256),0.5792754710746078)
((entropy,30,300),0.570642258679683)
((entropy,10,256),0.5678006650465051)
((entropy,20,300),0.5645890274211204)
((entropy,10,300),0.5548353562404907)*/

可以看到，结果反而比之前差了很多。这说明这些特征的类别取值有倾斜。

随机森林

随机森林可以理解将数据集合分成n个子集，然后在每个子集上建立决策树，最后结果是n棵决策树的平均值。
我们看一下所需要的参数：

scala> RandomForest.trainClassifier
<console>:42: error: ambiguous reference to overloaded definition,
both method trainClassifier in object RandomForest of type (input: org.apache.spark.api.java.JavaRDD[org.apache.spark.mllib.regression.LabeledPoint], numClasses: Int, categoricalFeaturesInfo: java.util.Map[Integer,Integer], numTrees: Int, featureSubsetStrategy: String, impurity: String, maxDepth: Int, maxBins: Int, seed: Int)org.apache.spark.mllib.tree.model.RandomForestModel
and  method trainClassifier in object RandomForest of type (input: org.apache.spark.rdd.RDD[org.apache.spark.mllib.regression.LabeledPoint], numClasses: Int, categoricalFeaturesInfo: Map[Int,Int], numTrees: Int, featureSubsetStrategy: String, impurity: String, maxDepth: Int, maxBins: Int, seed: Int)org.apache.spark.mllib.tree.model.RandomForestModel
match expected type ?RandomForest.trainClassifier^

这里新增的参数有：

numTrees：树的数量
featureSubsetStrategy：我们看下spark文档:

featureSubsetStrategy: Number of features to use as candidates for splitting at each tree node. The number is specified as a fraction or function of the total number of features. Decreasing this number will speed up training, but can sometimes impact performance if too low.

我们可以将featureSubsetStrategy设置为auto，让算法自己来决定。

val forest = RandomForest.trainClassifier(
trainData, 7, Map[Int,Int](), 20, "auto", "entropy", 30, 300)
val predictionsAndLabels = data.map(example =>(forest.predict(example.features), example.label)
val mul = new  MulticlassMetrics(predictionsAndLabels)
mul.precision
//res59: Double = 0.8690027056239802

【Spark Mllib】决策树，随机森林——预测森林植被类型相关推荐

python spark2.0_Python+Spark2.0+hadoop学习笔记——Python Spark MLlib决策树二分类
决策树是在数据分类问题中的一种常用且经典的机器学习方法,在本例里使用决策树模型来分析StumbleUpon数据集,来预测网页是暂时的(ephemeral)还是长青的(evergreen),并且调教参数 ...
基于spark mllib 随机森林分类代码记录后续完善
scala+spark+randomForests 代码实现分为读取hive数据.随机森林建模训练.数据预测随机森林建模训练的代码主类(实现流程) package com.inspur.mr.Ins ...
机器学习之路：python 集成分类器随机森林分类RandomForestClassifier 梯度提升决策树分类GradientBoostingClassifier 预测泰坦尼克号幸存者...
python3 学习使用随机森林分类器梯度提升决策树分类的api,并将他们和单一决策树预测结果做出对比附上我的git,欢迎大家来参考我其他分类器的代码: https://github.com/l ...
python在Scikit-learn中用决策树和随机森林预测NBA获胜者
在本文中,我们将以Scikit-learn的决策树和随机森林预测NBA获胜者.美国国家篮球协会(NBA)是北美主要的男子职业篮球联赛,被广泛认为是首屈一指的男子职业篮球联赛在世界上.它有30个团队(美 ...
python在Scikit-learn中用决策树和随机森林预测NBA获胜者 1
最近我们被要求撰写关于预测NBA的研究报告,包括一些图形和统计输出.我们将以Scikit-learn的决策树和随机森林预测NBA获胜者.美国国家篮球协会(NBA)是北美主要的男子职业篮球联赛,被广泛认 ...
决策树随机森林 xgboost_一文看懂随机森林-RandomForest（附4个构造步骤+4种实现方式评测+10个优缺点）...
随机森林是一种由决策树构成的集成算法,他在很多情况下都能有不错的表现.本文将介绍随机森林的基本概念.4 个构造步骤.4 种方式的对比评测.10 个优缺点和 4 个应用方向. 什么是随机森林? 随机森林 ...
RandomForest：随机森林预测生物标记biomarker——分类
随机森林简介 R randomForest包安装与加载分类Classification 分类结果主坐轴分析随机选取2/3预测,1/3验证无监督分类分层抽样 Reference 猜你喜欢写在 ...
在spark上构造随机森林模型过程的一点理解
这篇文章仅仅是为了帮助自己理解在分布式环境下是如何进行随机森林模型构建的,文章中记录的内容可能不太准确,仅仅是大致上的一个理解. 1.特征切分点统计不管是连续取值型特征还是离散取值型特征,分裂树结点 ...
决策树随机森林 xgboost_从决策树到随机森林理论篇从人话到鬼话：看不懂来找我麻烦...
从决策树产生的一些列的问题,过度到随机森林: 全文大概要阅读10分钟: 随机森林算法范畴监督学习分类算法,基模型是由决策树组成决策树决策树模型与学习特征选择决策树生成决策树剪枝 CART算 ...
Python使用随机森林预测泰坦尼克号生存
tags: 随机森林 kaggle 数据挖掘 categories: 数据挖掘 mathjax: true 文章目录前言: 1 数据预处理 1.1 读入数据 1.2 训练集与数据集 1.2.1 查看 ...

【Spark Mllib】决策树，随机森林——预测森林植被类型