spark Error initializing SparkContext System memory 466092032 must be at least 471859200.
ERROR SparkContext: Error initializing SparkContext.
Java.lang.IllegalArgumentException: System memory 259522560 must be at least 4.718592E8. Please use a larger heap size.
在Eclipse里开发Spark项目,尝试直接在spark里运行程序的时候,遇到下面这个报错:
ERROR SparkContext: Error initializing SparkContext.
java.lang.IllegalArgumentException: System memory 468189184 must be at least 4.718592E8. Please use a larger heap size.
很明显,这是JVM申请的memory不够导致无法启动SparkContext。但是该怎么设呢?
通过查看spark源码,发现源码是这么写的:
- /**
- * Return the total amount of memory shared between execution and storage, in bytes.
- */
- private def getMaxMemory(conf: SparkConf): Long = {
- val systemMemory = conf.getLong("spark.testing.memory", Runtime.getRuntime.maxMemory)
- val reservedMemory = conf.getLong("spark.testing.reservedMemory",
- if (conf.contains("spark.testing")) 0 else RESERVED_SYSTEM_MEMORY_BYTES)
- val minSystemMemory = reservedMemory * 1.5
- if (systemMemory < minSystemMemory) {
- throw new IllegalArgumentException(s"System memory $systemMemory must " +
- s"be at least $minSystemMemory. Please use a larger heap size.")
- }
- val usableMemory = systemMemory - reservedMemory
- val memoryFraction = conf.getDouble("spark.memory.fraction", 0.75)
- (usableMemory * memoryFraction).toLong
- }
所以,这里主要是val systemMemory = conf.getLong("spark.testing.memory", Runtime.getRuntime.maxMemory)。
conf.getLong()的定义和解释是
[java] view plain copy
- getLong(key: String, defaultValue: Long): Long
- Get a parameter as a long, falling back to a default if not set
所以,我们应该在conf里设置一下spark.testing.memory.
通过尝试,发现可以有2个地方可以设置
1. 自己的源代码处,可以在conf之后加上:
val conf = new SparkConf().setAppName("word count")
conf.set("spark.testing.memory", "2147480000")//后面的值大于512m即可
2. 可以在Eclipse的Run Configuration处,有一栏是Arguments,下面有VMarguments,在下面添加下面一行(值也是只要大于512m即可)
-Dspark.testing.memory=1073741824
其他的参数,也可以动态地在这里设置,比如-Dspark.master=spark://hostname:7077
再运行就不会报这个错误了。
spark Error initializing SparkContext System memory 466092032 must be at least 471859200.相关推荐
- Spark-submit:System memory 466092032 must be at least 471859200
在运行Standalone-client模式时遇到如下错误: Spark Executor Command: "/usr/local/jdk1.8.0_181/bin/java" ...
- ERROR SparkContext: Error initializing SparkContext. java.lang.IllegalArgumentException: System memo
ERROR SparkContext: Error initializing SparkContext. java.lang.IllegalArgumentException: System memo ...
- ERROR SparkContext: Error initializing SparkContext. org.apache.spark.SparkException: Could not pars
ERROR SparkContext: Error initializing SparkContext. org.apache.spark.SparkException: Could not pars ...
- System memory 466092032 must be at least
Error initializing SparkContext. dependencies | java.lang.IllegalArgumentException: System memory 46 ...
- Spark解决 System memory 259522560 must be at least 471859200
本地运行spark出现问题: 22/04/26 20:11:42 ERROR SparkContext: Error initializing SparkContext. java.lang.Ille ...
- SparkContext: Error initializing SparkContext解决方法
SparkContext: Error initializing SparkContext解决方法 Spark在配置高可用集群中,出现报错 ERROR SparkContext: Error init ...
- Spark本地测试异常之 System memory 259522560 must be at least 471859200.
解决Spark本地测试异常之 System memory 259522560 must be at least 471859200 一.异常如下 二.抛出异常原因 三.解决办法 一.异常如下 java ...
- 【Spark】java.lang.IllegalArgumentException: System memory 259522560 must be at least 471859200.
报错 java.lang.IllegalArgumentException: System memory 259522560 must be at least 471859200. Please in ...
- System memory 249364480 must be at least 471859200
我是在eclipse run遇到这个问题的,所以解决办法是来到这里加上: 表示运行的时候虚拟空间最小为128,最大为512 在linux上运行jar 如果用控制的虚拟机内存,句子要这么写: java ...
最新文章
- 谷歌像素递归超分辨率研究:怎么消灭低分辨率图像马赛克?
- (android硬件应用实战)摄像头拍照实现和总结
- Android复习03(Get请求访问网络[详解]、获取文章列表)
- boost::histogram::indexed用法的测试程序
- linux搜索指定文件夹里文件是否存在,Linux文件操作之文件查询与搜索命令详解...
- 在MVC项目中使用Ninject
- 【C/C 】浅谈C/C 中函数指针与回调函数
- MyEclipse 10, 2013, 2014 破解、注册码
- 如何用鲲鹏弹性云服务器部署《Hadoop伪分布式》
- 充分发挥计算机在教学中的辅助作,充分发挥计算机在教学中的辅助作用
- [ Typescript 手册] JavaScript `Date` 在 Typescript 中的接口
- ArcEngine9.1结合VS2005开发技巧2则
- C++通信录管理系统
- Splunk lookup 实践
- 二阶偏导相等的一个充分条件
- 文献综述在哪儿能找到?
- 十大畅销经济学类书籍
- 天池大赛——天猫用户复购预测
- ZOJ 3960 What Kind of Friends Are You?(读题+思维)
- PostgreSQL修炼之道之PostgreSQL的核心架构(十一)