Spark中组件Mllib的学习19之分层抽样
更多代码请见:https://github.com/xubo245/SparkLearning
Spark中组件Mllib的学习之基础概念篇
1解释
分层抽样的概念就不讲了,具体的操作:
RDD有个操作可以直接进行抽样:sampleByKey和sample等,这里主要介绍这两个
(1)将字符串长度为2划分为层1和层2,对层1和层2按不同的概率进行抽样
数据
aa
bb
cc
dd
ee
aaa
bbb
ccc
ddd
eee
比如:
val fractions: Map[Int, Double] = (List((1, 0.2), (2, 0.8))).toMap //设定抽样格式
sampleByKey(withReplacement = false, fractions, 0)
fractions表示在层1抽0.2,在层2中抽0.8
withReplacement false表示不重复抽样
0表示随机的seed
源码:
/*** Return a subset of this RDD sampled by key (via stratified sampling).** Create a sample of this RDD using variable sampling rates for different keys as specified by* `fractions`, a key to sampling rate map, via simple random sampling with one pass over the* RDD, to produce a sample of size that's approximately equal to the sum of* math.ceil(numItems * samplingRate) over all key values.** @param withReplacement whether to sample with or without replacement* @param fractions map of specific keys to sampling rates* @param seed seed for the random number generator* @return RDD containing the sampled subset*/def sampleByKey(withReplacement: Boolean,fractions: Map[K, Double],seed: Long = Utils.random.nextLong): RDD[(K, V)] = self.withScope {require(fractions.values.forall(v => v >= 0.0), "Negative sampling rates.")val samplingFunc = if (withReplacement) {StratifiedSamplingUtils.getPoissonSamplingFunction(self, fractions, false, seed)} else {StratifiedSamplingUtils.getBernoulliSamplingFunction(self, fractions, false, seed)}self.mapPartitionsWithIndex(samplingFunc, preservesPartitioning = true)}
(2)和(3)类似,请看代码
2.代码:
/*** @author xubo* ref:Spark MlLib机器学习实战* more code:https://github.com/xubo245/SparkLearning* more blog:http://blog.csdn.net/xubo245*/
package org.apache.spark.mllib.learning.basicimport org.apache.spark.{SparkConf, SparkContext}/*** Created by xubo on 2016/5/23.*/
object StratifiedSamplingLearning {def main(args: Array[String]) {val conf = new SparkConf().setMaster("local[4]").setAppName(this.getClass().getSimpleName().filter(!_.equals('$')))val sc = new SparkContext(conf)println("First:")val data = sc.textFile("file/data/mllib/input/basic/StratifiedSampling.txt") //读取数.map(row => {//开始处理if (row.length == 3) //判断字符数(row, 1) //建立对应mapelse (row, 2) //建立对应map}).map(each => (each._2, each._1))data.foreach(println)println("sampleByKey:")val fractions: Map[Int, Double] = (List((1, 0.2), (2, 0.8))).toMap //设定抽样格式val approxSample = data.sampleByKey(withReplacement = false, fractions, 0) //计算抽样样本approxSample.foreach(println)println("Second:")//http://homepage.cs.latrobe.edu.au/zhe/ZhenHeSparkRDDAPIExamples.html#sampleByKeyval randRDD = sc.parallelize(List((7, "cat"), (6, "mouse"), (7, "cup"), (6, "book"), (7, "tv"), (6, "screen"), (7, "heater")))val sampleMap = List((7, 0.4), (6, 0.8)).toMapval sample2 = randRDD.sampleByKey(false, sampleMap, 42).collectsample2.foreach(println)println("Third:")//http://bbs.csdn.net/topics/390953396val a = sc.parallelize(1 to 20, 3)val b = a.sample(true, 0.8, 0)val c = a.sample(false, 0.8, 0)println("RDD a : " + a.collect().mkString(" , "))println("RDD b : " + b.collect().mkString(" , "))println("RDD c : " + c.collect().mkString(" , "))sc.stop}
}
3.结果:
First:
2016-05-23 22:37:34 WARN :139 - Your hostname, xubo-PC resolves to a loopback/non-reachable address: fe80:0:0:0:482:722f:5976:ce1f%20, but we couldn't find any external IP address!
(2,aa)
(1,bbb)
(2,bb)
(1,ccc)
(2,cc)
(1,ddd)
(2,dd)
(1,eee)
(2,ee)
(1,aaa)
sampleByKey:
(2,aa)
(2,bb)
(2,cc)
(2,ee)
Second:
(7,cat)
(6,mouse)
(6,book)
(6,screen)
(7,heater)
Third:
RDD a : 1 , 2 , 3 , 4 , 5 , 6 , 7 , 8 , 9 , 10 , 11 , 12 , 13 , 14 , 15 , 16 , 17 , 18 , 19 , 20
RDD b : 2 , 4 , 5 , 6 , 10 , 14 , 19 , 20
RDD c : 1 , 2 , 4 , 5 , 8 , 10 , 12 , 13 , 14 , 15 , 16 , 17 , 18 , 19 , 20
参考
【1】http://spark.apache.org/docs/1.5.2/mllib-guide.html
【2】http://spark.apache.org/docs/1.5.2/programming-guide.html
【3】https://github.com/xubo245/SparkLearning
Spark中组件Mllib的学习19之分层抽样相关推荐
- Spark中组件Mllib的学习16之分布式行矩阵的四种形式
来源:http://blog.csdn.net/xubo245/article/details/51483995 更多代码请见:https://github.com/xubo245/SparkLear ...
- Spark中组件Mllib的学习15之创建分布式矩阵
更多代码请见:https://github.com/xubo245/SparkLearning Spark中组件Mllib的学习之基础概念篇 1解释 创建分布式矩阵 2.代码: /*** @autho ...
- Spark中组件Mllib的学习40之梯度提升树(GBT)用于回归
更多代码请见:https://github.com/xubo245/SparkLearning Spark中组件Mllib的学习之分类篇 1解释 GBRT(Gradient Boost Regr ...
- Spark中组件Mllib的学习41之保序回归(Isotonic regression)
更多代码请见:https://github.com/xubo245/SparkLearning Spark中组件Mllib的学习之分类篇 1解释 问题描述:给定一个无序数字序列,要求不改变每个元素的位 ...
- Spark中组件Mllib的学习11之使用ALS对movieLens中一百万条(1M)数据集进行训练,并对输入的新用户数据进行电影推荐
更多代码请见:https://github.com/xubo245/SparkLearning 1解释 spark-1.5.2 数据集:http://grouplens.org/datasets/mo ...
- Spark中组件Mllib的学习1之Kmeans错误解决
更多代码请见:https://github.com/xubo245/SparkLearning 解决办法:(中间比较多,为了方便看到,放在最开始) txt文件格式不对,用WPS转存的是UTF-16,s ...
- Spark四大组件包括Spark Streaming、Spark SQL、Spark MLlib和Spark GraphX。
Spark四大组件包括Spark Streaming.Spark SQL.Spark MLlib和Spark GraphX.它们的主要应用场景是: Spark Streaming: Spark Str ...
- 中国Spark技术峰会(上):Spark与生态圈中组件结合实战
5月13日-15日,由全球最大中文IT社区CSDN主办的"2016中国云计算技术大会"(Cloud Computing Technology Conference 2016,简称C ...
- jsx 调用php,JavaScript_JavaScript的React框架中的JSX语法学习入门教程,什么是JSX?
在用React写组件的 - phpStudy...
JavaScript的React框架中的JSX语法学习入门教程 什么是JSX? 在用React写组件的时候,通常会用到JSX语法,粗看上去,像是在Javascript代码里直接写起了XML标签,实质上 ...
最新文章
- BorderDet(论文解读)
- 代码恒久远,GitHub 永流传!
- 爬虫,如何防止被ban之策略大集合
- CornerNet 测试:
- 选择排序之——堆排序(c/c++)
- 第四届蓝桥杯省赛javaB组试题解析
- 面试刷题7:int和Integer有什么区别?
- jquery关于checkbox在ie下的一个bug
- java选择安装路径的功能怎么实现_水槽怎么选择,从安装方式,材质功能,江水平给你一次性说清楚...
- Long Shadows Generate是一款在线使用纯CSS3实现长阴影的效果,一款强大的扁平化长投影制造器。...
- 目标识别:如何从人脸图片中扣出眼图,实时人脸人眼检测和识别
- C# async await
- Mutect2​案例 / 有或没有正常样本配对的肿瘤体细胞变异检测结果对比
- ASP.NET 安全认证(转自:寒羽枫cityhunter172)
- splash : mouse_click()方法
- 目标检测论文解读复现之十九:基于YOLOv5网络模型的人员口罩佩戴实时检测
- QQ群排名,引流营销做推广的好方法
- 区块链钱包开发(Android篇)
- html游戏让目标人物移动,用仿ActionScript的语法来编写html5——第三篇,鼠标事件与游戏人物移动...
- 史上最长最全!围绕故障管理谈SRE体系建设
热门文章
- keras中EarlyStopping(早停止)的用法和原理详解
- LXC、LXD、Docker的区别与联系
- jupyter内核无法连接,出现error,代码无法运行解决办法
- 修改注册表出现“无法编辑UncheckedValue:写该值的新内容时出错”的解决方法
- 中国科学技术大学计算机专业排名,2019中国科学技术大学专业排名
- antd 每次打开modal 初始化数据
- SpringMVC项目升级SpringBoot项目参考
- 041 :魔法方法:构造和析构
- 【解析几何】解析几何(Analytic geometry)
- matlab在解析几何教学中的应用,Matlab在解析几何教学中的应用