spark count统计元素个数
太简单了，直接上代码，不解析
public static void myCount(){
SparkConf conf=new SparkConf()
.setMaster("local")
.setAppName("myCount");
JavaSparkContext sc=new JavaSparkContext(conf);
List<Integer> list=Arrays.asList(1,2,3,4,4);
JavaRDD<Integer> listRdd=sc.parallelize(list, 2);
long counts=listRdd.count();
System.out.println("count:"+counts);
sc.close();
}
结果：
count:5
我始终觉得打印出来的日志很重要：
Using Spark's default log4j profile: org/apache/spark/log4j-defaults.properties
16/05/03 22:54:42 INFO SparkContext: Running Spark version 1.6.1
16/05/03 22:55:02 WARN NativeCodeLoader: Unable to load native-hadoop library for your platform... using builtin-java classes where applicable
16/05/03 22:55:02 INFO SecurityManager: Changing view acls to: admin
16/05/03 22:55:02 INFO SecurityManager: Changing modify acls to: admin
16/05/03 22:55:02 INFO SecurityManager: SecurityManager: authentication disabled; ui acls disabled; users with view permissions: Set(admin); users with modify permissions: Set(admin)
16/05/03 22:55:04 INFO Utils: Successfully started service 'sparkDriver' on port 55095.
16/05/03 22:55:05 INFO Slf4jLogger: Slf4jLogger started
16/05/03 22:55:05 INFO Remoting: Starting remoting
16/05/03 22:55:05 INFO Remoting: Remoting started; listening on addresses :[akka.tcp://sparkDriverActorSystem@192.168.213.1:55108]
16/05/03 22:55:06 INFO Utils: Successfully started service 'sparkDriverActorSystem' on port 55108.
16/05/03 22:55:06 INFO SparkEnv: Registering MapOutputTracker
16/05/03 22:55:06 INFO SparkEnv: Registering BlockManagerMaster
16/05/03 22:55:06 INFO DiskBlockManager: Created local directory at C:\Users\admin\AppData\Local\Temp\blockmgr-3aaa5046-0d05-4c75-8734-02b0121f3a1e
16/05/03 22:55:06 INFO MemoryStore: MemoryStore started with capacity 2.4 GB
16/05/03 22:55:06 INFO SparkEnv: Registering OutputCommitCoordinator
16/05/03 22:55:07 INFO Utils: Successfully started service 'SparkUI' on port 4040.
16/05/03 22:55:07 INFO SparkUI: Started SparkUI at http://192.168.213.1:4040
16/05/03 22:55:07 INFO Executor: Starting executor ID driver on host localhost
16/05/03 22:55:07 INFO Utils: Successfully started service 'org.apache.spark.network.netty.NettyBlockTransferService' on port 55116.
16/05/03 22:55:07 INFO NettyBlockTransferService: Server created on 55116
16/05/03 22:55:07 INFO BlockManagerMaster: Trying to register BlockManager
16/05/03 22:55:07 INFO BlockManagerMasterEndpoint: Registering block manager localhost:55116 with 2.4 GB RAM, BlockManagerId(driver, localhost, 55116)
16/05/03 22:55:07 INFO BlockManagerMaster: Registered BlockManager
16/05/03 22:55:09 INFO SparkContext: Starting job: count at ActionOperation.java:84
16/05/03 22:55:09 INFO DAGScheduler: Got job 0 (count at ActionOperation.java:84) with 2 output partitions
16/05/03 22:55:09 INFO DAGScheduler: Final stage: ResultStage 0 (count at ActionOperation.java:84)
16/05/03 22:55:09 INFO DAGScheduler: Parents of final stage: List()
16/05/03 22:55:09 INFO DAGScheduler: Missing parents: List()
16/05/03 22:55:09 INFO DAGScheduler: Submitting ResultStage 0 (ParallelCollectionRDD[0] at parallelize at ActionOperation.java:83), which has no missing parents
16/05/03 22:55:10 INFO MemoryStore: Block broadcast_0 stored as values in memory (estimated size 1320.0 B, free 1320.0 B)
16/05/03 22:55:10 INFO MemoryStore: Block broadcast_0_piece0 stored as bytes in memory (estimated size 927.0 B, free 2.2 KB)
16/05/03 22:55:10 INFO BlockManagerInfo: Added broadcast_0_piece0 in memory on localhost:55116 (size: 927.0 B, free: 2.4 GB)
16/05/03 22:55:10 INFO SparkContext: Created broadcast 0 from broadcast at DAGScheduler.scala:1006
16/05/03 22:55:10 INFO DAGScheduler: Submitting 2 missing tasks from ResultStage 0 (ParallelCollectionRDD[0] at parallelize at ActionOperation.java:83)
16/05/03 22:55:10 INFO TaskSchedulerImpl: Adding task set 0.0 with 2 tasks
16/05/03 22:55:10 INFO TaskSetManager: Starting task 0.0 in stage 0.0 (TID 0, localhost, partition 0,PROCESS_LOCAL, 2140 bytes)
16/05/03 22:55:10 INFO Executor: Running task 0.0 in stage 0.0 (TID 0)
16/05/03 22:55:10 INFO Executor: Finished task 0.0 in stage 0.0 (TID 0). 953 bytes result sent to driver
16/05/03 22:55:10 INFO TaskSetManager: Starting task 1.0 in stage 0.0 (TID 1, localhost, partition 1,PROCESS_LOCAL, 2145 bytes)
16/05/03 22:55:10 INFO Executor: Running task 1.0 in stage 0.0 (TID 1)
16/05/03 22:55:10 INFO TaskSetManager: Finished task 0.0 in stage 0.0 (TID 0) in 257 ms on localhost (1/2)
16/05/03 22:55:10 INFO Executor: Finished task 1.0 in stage 0.0 (TID 1). 953 bytes result sent to driver
16/05/03 22:55:10 INFO TaskSetManager: Finished task 1.0 in stage 0.0 (TID 1) in 55 ms on localhost (2/2)
16/05/03 22:55:10 INFO DAGScheduler: ResultStage 0 (count at ActionOperation.java:84) finished in 0.352 s
16/05/03 22:55:10 INFO TaskSchedulerImpl: Removed TaskSet 0.0, whose tasks have all completed, from pool
16/05/03 22:55:10 INFO DAGScheduler: Job 0 finished: count at ActionOperation.java:84, took 1.136850 s
count:5
16/05/03 22:55:10 INFO SparkUI: Stopped Spark web UI at http://192.168.213.1:4040
16/05/03 22:55:10 INFO MapOutputTrackerMasterEndpoint: MapOutputTrackerMasterEndpoint stopped!
16/05/03 22:55:10 INFO MemoryStore: MemoryStore cleared
16/05/03 22:55:10 INFO BlockManager: BlockManager stopped
16/05/03 22:55:10 INFO BlockManagerMaster: BlockManagerMaster stopped
16/05/03 22:55:10 INFO OutputCommitCoordinator$OutputCommitCoordinatorEndpoint: OutputCommitCoordinator stopped!
16/05/03 22:55:10 INFO SparkContext: Successfully stopped SparkContext
16/05/03 22:55:10 INFO ShutdownHookManager: Shutdown hook called
16/05/03 22:55:10 INFO ShutdownHookManager: Deleting directory C:\Users\admin\AppData\Local\Temp\spark-221cf0db-fb4a-4577-8785-ac392b53425e
16/05/03 22:55:10 INFO RemoteActorRefProvider$RemotingTerminator: Shutting down remote daemon.
spark count统计元素个数相关推荐
- python统计元素个数_python怎么统计列表中元素的个数
python统计列表中元素的个数的方法:可以通过count()方法来实现.该方法可以统计字符串中某个字符出现的次数,并返回子字符串在字符串中出现的次数.具体用法如:[count=List.count( ...
- 统计多维数组php_PHP多维数组中统计元素个数
Array ( [0] => Array ( [0] => Array ( [0] => Array ( [id] => 12 [name] => '1' ) [1] = ...
- mysql统计唯一个数_统计数组元素的个数和唯一性的函数
有些函数可以用来确定数组中的值总数及唯一值的个数.使用函数count()对元素个数进行统计,sizeof()函数时count()的别名,他们的功能是一样的. ①函数count() 函数count()的 ...
- php 统计数组个数,php统计数组元素的个数和唯一性
大家好,今天给大家分享的是php统计数组元素的个数和唯一性,希望大家喜欢. 我们在学习php数组的时候,如何来统计数组元素的个数和唯一性呢? 那么下面我们来说下 1,函数count() 统计数组元素个 ...
- php数组的元素个数,php怎么统计数组元素的个数
这篇文章主要介绍了php统计数组元素个数的方法的相关资料,需要的朋友可以参考下 count():对数组中的元素个数进行统计; sizeof():和count()具有同样的用途,这两个函数都可以返回数组 ...
- winform 统计大量数据重复的元素个数_面试系列：十个海量数据处理方法大总结...
本文将简单总结下一些处理海量数据问题的常见方法.当然这些方法可能并不能完全覆盖所有的问题,但是这样的一些方法也基本可以处理绝大多数遇到的问题.下面的一些问题基本直接来源于公司的面试笔试题目,方法不一定 ...
- java map 元素个数_Java 小模块之--统计字符串中元素个数
Java 小模块之--统计字符串中元素个数 曾经看过我Stream或者Guava类库等文章的小伙伴应该很明白我这篇博文的意义所在了 一是给读者提供综合的博文入口 二是自己也总结一下思路 ps: 之前没 ...
- 统计多维数组php_php统计多维数组元素个数的方法介绍（附代码）
详细内容 本篇文章给大家带来的内容是关于php统计多维数组元素个数的方法介绍(附代码),有一定的参考价值,有需要的朋友可以参考一下,希望对你有所帮助. 一般情况下,使用count可以直接统计数组的元素 ...
- 汇编中的length（返回利用dup定义的数组中的元素个数，即重复操作符dup前的count值）
LENGTH是属于析值操作符之一,也称为数值回送操作符,原因是这些操作符把一些特征或存储器地址的一部分作为数据返回.length的用法:length 变量名 .作用是返回利用dup定义的数组中的元素个 ...
最新文章
- Codeforces Global Round 13 E. Fib-tree
- hihocoder1147 时空阵(bfs树+DP)
- Python使用wordnet工具计算词集与词条基本用法（三）
- 软件测试用例质量不高?我教你如何编写高质量的测试用例!
- 盛情难却：北京，QECon来了
- python中怎样使用re模块_python如何导入re模块
- 文档大小超出上传限制怎么办_一键PDF转Word、PPT、图片等文档，这才是办公族必备的效率神器！...
- 相机姿态估计（五）--DLS
- js调用摄像头拍照上传图片
- 【原创工具 | NetSM】开源跨平台命令行网速监测（纯 Python 开发）
- 设计房屋租赁管理系统--PostgreSQL--数据库原理及应用
- 【最短路】 Johnson 算法
- 斯坦福大学CS224N-深度学习与自然语言处理：课程1-笔记
- WinPcap vs Npcap
- 目标检测-ImageAI从安装到使用详解
- 怎么用python输出百分比_Python 输出百分比
- 分享一个通过网络链接PDF转JPG的公用方法
- 利用echarts做图表统计
- 初步接触houdini---零零散散
- 使用JS实现当当购物车结算页面