启动hive --service metastore

启动 dfs yarn

[root@bigdatastorm bin]# ./spark-sql --master yarn --deploy-mode client --driver-memory 512m --executor-memory 512m --total-executor-cores 1

spark-sql>select count(*)  ;

Log

=======================================================

SLF4J: Class path contains multiple SLF4J bindings.

SLF4J: Found binding in [jar:file:/opt/hadoop-2.5.1/nm-local-dir/usercache/root/filecache/11/spark-assembly-1.6.0-hadoop2.6.0.jar!/org/slf4j/impl/StaticLoggerBinder.class]

SLF4J: Found binding in [jar:file:/usr/local/hadoop/share/hadoop/common/lib/slf4j-log4j12-1.7.5.jar!/org/slf4j/impl/StaticLoggerBinder.class]

SLF4J: See http://www.slf4j.org/codes.html#multiple_bindings for an explanation.

SLF4J: Actual binding is of type [org.slf4j.impl.Log4jLoggerFactory]

16/09/05 21:59:45 INFO yarn.ApplicationMaster: Registered signal handlers for [TERM, HUP, INT]

16/09/05 21:59:50 INFO yarn.ApplicationMaster: ApplicationAttemptId: appattempt_1473082245027_0003_000001

16/09/05 21:59:53 INFO spark.SecurityManager: Changing view acls to: root

16/09/05 21:59:53 INFO spark.SecurityManager: Changing modify acls to: root

16/09/05 21:59:53 INFO spark.SecurityManager: SecurityManager: authentication disabled; ui acls disabled; users with view permissions: Set(root); users with modify permissions: Set(root)

16/09/05 21:59:55 INFO yarn.ApplicationMaster: Waiting for Spark driver to be reachable.

16/09/05 21:59:55 INFO yarn.ApplicationMaster: Driver now available: 192.168.184.188:45475

16/09/05 21:59:57 INFO yarn.ApplicationMaster$AMEndpoint: Add WebUI Filter. AddWebUIFilter(org.apache.hadoop.yarn.server.webproxy.amfilter.AmIpFilter,Map(PROXY_HOSTS -> bigdatastorm, PROXY_URI_BASES -> http://bigdatastorm:8088/proxy/application_1473082245027_0003),/proxy/application_1473082245027_0003)

16/09/05 21:59:57 INFO yarn.YarnRMClient: Registering the ApplicationMaster

16/09/05 21:59:58 INFO yarn.YarnAllocator: Will request 1 executor containers, each with 1 cores and 896 MB memory including 384 MB overhead

16/09/05 21:59:58 INFO yarn.YarnAllocator: Container request (host: Any, capability: <896 vcores:1>)896>

16/09/05 21:59:58 INFO yarn.ApplicationMaster: Started progress reporter thread with (heartbeat : 3000, initial allocation : 200) intervals

16/09/05 21:59:58 INFO impl.AMRMClientImpl: Received new token for : bigdatastorm:59055

16/09/05 21:59:58 INFO yarn.YarnAllocator: Launching container container_1473082245027_0003_01_000002 for on host bigdatastorm

16/09/05 21:59:58 INFO yarn.YarnAllocator: Launching ExecutorRunnable. driverUrl: spark://CoarseGrainedScheduler@192.168.184.188:45475, executorHostname: bigdatastorm

16/09/05 21:59:58 INFO yarn.YarnAllocator: Received 1 containers from YARN, launching executors on 1 of them.

16/09/05 21:59:58 INFO yarn.ExecutorRunnable: Starting Executor Container

16/09/05 21:59:58 INFO impl.ContainerManagementProtocolProxy: yarn.client.max-cached-nodemanagers-proxies : 0

16/09/05 21:59:58 INFO yarn.ExecutorRunnable: Setting up ContainerLaunchContext

16/09/05 21:59:58 INFO yarn.ExecutorRunnable: Preparing Local resources

16/09/05 21:59:59 INFO yarn.ExecutorRunnable: Prepared Local resources Map(__spark__.jar -> resource { scheme: "hdfs" host: "mycluster" port: -1 file: "/user/root/.sparkStaging/application_1473082245027_0003/spark-assembly-1.6.0-hadoop2.6.0.jar" } size: 187548272 timestamp: 1473083954792 type: FILE visibility: PRIVATE)

16/09/05 21:59:59 INFO yarn.ExecutorRunnable:

===============================================================================

YARN executor launch context:

env:

CLASSPATH -> {{PWD}}{{PWD}}/__spark__.jar$HADOOP_CONF_DIR$HADOOP_COMMON_HOME/share/hadoop/common/*$HADOOP_COMMON_HOME/share/hadoop/common/lib/*$HADOOP_HDFS_HOME/share/hadoop/hdfs/*$HADOOP_HDFS_HOME/share/hadoop/hdfs/lib/*$HADOOP_YARN_HOME/share/hadoop/yarn/*$HADOOP_YARN_HOME/share/hadoop/yarn/lib/*$HADOOP_MAPRED_HOME/share/hadoop/mapreduce/*$HADOOP_MAPRED_HOME/share/hadoop/mapreduce/lib/*

SPARK_LOG_URL_STDERR -> http://bigdatastorm:8042/node/containerlogs/container_1473082245027_0003_01_000002/root/stderr?start=-4096

SPARK_YARN_STAGING_DIR -> .sparkStaging/application_1473082245027_0003

SPARK_YARN_CACHE_FILES_FILE_SIZES -> 187548272

SPARK_USER -> root

SPARK_YARN_CACHE_FILES_VISIBILITIES -> PRIVATE

SPARK_YARN_MODE -> true

SPARK_YARN_CACHE_FILES_TIME_STAMPS -> 1473083954792

SPARK_LOG_URL_STDOUT -> http://bigdatastorm:8042/node/containerlogs/container_1473082245027_0003_01_000002/root/stdout?start=-4096

SPARK_YARN_CACHE_FILES -> hdfs://mycluster/user/root/.sparkStaging/application_1473082245027_0003/spark-assembly-1.6.0-hadoop2.6.0.jar#__spark__.jar

command:

{{JAVA_HOME}}/bin/java -server -XX:OnOutOfMemoryError='kill %p' -Xms512m -Xmx512m -Djava.io.tmpdir={{PWD}}/tmp '-Dspark.driver.port=45475' -Dspark.yarn.app.container.log.dir= -XX:MaxPermSize=256m org.apache.spark.executor.CoarseGrainedExecutorBackend --driver-url spark://CoarseGrainedScheduler@192.168.184.188:45475 --executor-id 1 --hostname bigdatastorm --cores 1 --app-id application_1473082245027_0003 --user-class-path file:$PWD/__app__.jar 1> /stdout 2> /stderr

===============================================================================

16/09/05 21:59:59 INFO impl.ContainerManagementProtocolProxy: Opening proxy : bigdatastorm:59055

16/09/05 22:09:45 INFO yarn.YarnAllocator: Completed container container_1473082245027_0003_01_000002 on host: bigdatastorm (state: COMPLETE, exit status: 50)

16/09/05 22:09:45 WARN yarn.YarnAllocator: Container marked as failed: container_1473082245027_0003_01_000002 on host: bigdatastorm. Exit status: 50. Diagnostics: Exception from container-launch: ExitCodeException exitCode=50:

ExitCodeException exitCode=50:

at org.apache.hadoop.util.Shell.runCommand(Shell.java:538)

at org.apache.hadoop.util.Shell.run(Shell.java:455)

at org.apache.hadoop.util.Shell$ShellCommandExecutor.execute(Shell.java:702)

at org.apache.hadoop.yarn.server.nodemanager.DefaultContainerExecutor.launchContainer(DefaultContainerExecutor.java:195)

at org.apache.hadoop.yarn.server.nodemanager.containermanager.launcher.ContainerLaunch.call(ContainerLaunch.java:300)

at org.apache.hadoop.yarn.server.nodemanager.containermanager.launcher.ContainerLaunch.call(ContainerLaunch.java:81)

at java.util.concurrent.FutureTask.run(FutureTask.java:262)

at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)

at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)

at java.lang.Thread.run(Thread.java:745)

Container exited with a non-zero exit code 50

16/09/05 22:09:48 INFO yarn.YarnAllocator: Will request 1 executor containers, each with 1 cores and 896 MB memory including 384 MB overhead

16/09/05 22:09:48 INFO yarn.YarnAllocator: Container request (host: Any, capability: <896 vcores:1>)896>

16/09/05 22:09:49 INFO yarn.YarnAllocator: Launching container container_1473082245027_0003_01_000003 for on host bigdatastorm

16/09/05 22:09:49 INFO yarn.YarnAllocator: Launching ExecutorRunnable. driverUrl: spark://CoarseGrainedScheduler@192.168.184.188:45475, executorHostname: bigdatastorm

16/09/05 22:09:49 INFO yarn.YarnAllocator: Received 1 containers from YARN, launching executors on 1 of them.

16/09/05 22:09:49 INFO yarn.ExecutorRunnable: Starting Executor Container

16/09/05 22:09:49 INFO impl.ContainerManagementProtocolProxy: yarn.client.max-cached-nodemanagers-proxies : 0

16/09/05 22:09:49 INFO yarn.ExecutorRunnable: Setting up ContainerLaunchContext

16/09/05 22:09:49 INFO yarn.ExecutorRunnable: Preparing Local resources

16/09/05 22:09:49 INFO yarn.ExecutorRunnable: Prepared Local resources Map(__spark__.jar -> resource { scheme: "hdfs" host: "mycluster" port: -1 file: "/user/root/.sparkStaging/application_1473082245027_0003/spark-assembly-1.6.0-hadoop2.6.0.jar" } size: 187548272 timestamp: 1473083954792 type: FILE visibility: PRIVATE)

16/09/05 22:09:49 INFO yarn.ExecutorRunnable:

===============================================================================

YARN executor launch context:

env:

CLASSPATH -> {{PWD}}{{PWD}}/__spark__.jar$HADOOP_CONF_DIR$HADOOP_COMMON_HOME/share/hadoop/common/*$HADOOP_COMMON_HOME/share/hadoop/common/lib/*$HADOOP_HDFS_HOME/share/hadoop/hdfs/*$HADOOP_HDFS_HOME/share/hadoop/hdfs/lib/*$HADOOP_YARN_HOME/share/hadoop/yarn/*$HADOOP_YARN_HOME/share/hadoop/yarn/lib/*$HADOOP_MAPRED_HOME/share/hadoop/mapreduce/*$HADOOP_MAPRED_HOME/share/hadoop/mapreduce/lib/*

SPARK_LOG_URL_STDERR -> http://bigdatastorm:8042/node/containerlogs/container_1473082245027_0003_01_000003/root/stderr?start=-4096

SPARK_YARN_STAGING_DIR -> .sparkStaging/application_1473082245027_0003

SPARK_YARN_CACHE_FILES_FILE_SIZES -> 187548272

SPARK_USER -> root

SPARK_YARN_CACHE_FILES_VISIBILITIES -> PRIVATE

SPARK_YARN_MODE -> true

SPARK_YARN_CACHE_FILES_TIME_STAMPS -> 1473083954792

SPARK_LOG_URL_STDOUT -> http://bigdatastorm:8042/node/containerlogs/container_1473082245027_0003_01_000003/root/stdout?start=-4096

SPARK_YARN_CACHE_FILES -> hdfs://mycluster/user/root/.sparkStaging/application_1473082245027_0003/spark-assembly-1.6.0-hadoop2.6.0.jar#__spark__.jar

command:

{{JAVA_HOME}}/bin/java -server -XX:OnOutOfMemoryError='kill %p' -Xms512m -Xmx512m -Djava.io.tmpdir={{PWD}}/tmp '-Dspark.driver.port=45475' -Dspark.yarn.app.container.log.dir= -XX:MaxPermSize=256m org.apache.spark.executor.CoarseGrainedExecutorBackend --driver-url spark://CoarseGrainedScheduler@192.168.184.188:45475 --executor-id 2 --hostname bigdatastorm --cores 1 --app-id application_1473082245027_0003 --user-class-path file:$PWD/__app__.jar 1> /stdout 2> /stderr

===============================================================================

16/09/05 22:09:49 INFO impl.ContainerManagementProtocolProxy: Opening proxy : bigdatastorm:59055

16/09/05 22:12:14 INFO yarn.YarnAllocator: Completed container container_1473082245027_0003_01_000003 on host: bigdatastorm (state: COMPLETE, exit status: 1)

16/09/05 22:12:14 WARN yarn.YarnAllocator: Container marked as failed: container_1473082245027_0003_01_000003 on host: bigdatastorm. Exit status: 1. Diagnostics: Exception from container-launch: ExitCodeException exitCode=1:

ExitCodeException exitCode=1:

at org.apache.hadoop.util.Shell.runCommand(Shell.java:538)

at org.apache.hadoop.util.Shell.run(Shell.java:455)

at org.apache.hadoop.util.Shell$ShellCommandExecutor.execute(Shell.java:702)

at org.apache.hadoop.yarn.server.nodemanager.DefaultContainerExecutor.launchContainer(DefaultContainerExecutor.java:195)

at org.apache.hadoop.yarn.server.nodemanager.containermanager.launcher.ContainerLaunch.call(ContainerLaunch.java:300)

at org.apache.hadoop.yarn.server.nodemanager.containermanager.launcher.ContainerLaunch.call(ContainerLaunch.java:81)

at java.util.concurrent.FutureTask.run(FutureTask.java:262)

at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)

at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)

at java.lang.Thread.run(Thread.java:745)

Container exited with a non-zero exit code 1

16/09/05 22:12:17 INFO yarn.YarnAllocator: Will request 1 executor containers, each with 1 cores and 896 MB memory including 384 MB overhead

16/09/05 22:12:17 INFO yarn.YarnAllocator: Container request (host: Any, capability: <896 vcores:1>)896>

16/09/05 22:12:18 INFO impl.AMRMClientImpl: Received new token for : bigdatahadoop:39892

16/09/05 22:12:18 INFO yarn.YarnAllocator: Launching container container_1473082245027_0003_01_000004 for on host bigdatahadoop

16/09/05 22:12:18 INFO yarn.YarnAllocator: Launching ExecutorRunnable. driverUrl: spark://CoarseGrainedScheduler@192.168.184.188:45475, executorHostname: bigdatahadoop

16/09/05 22:12:18 INFO yarn.YarnAllocator: Received 1 containers from YARN, launching executors on 1 of them.

16/09/05 22:12:18 INFO yarn.ExecutorRunnable: Starting Executor Container

16/09/05 22:12:18 INFO impl.ContainerManagementProtocolProxy: yarn.client.max-cached-nodemanagers-proxies : 0

16/09/05 22:12:18 INFO yarn.ExecutorRunnable: Setting up ContainerLaunchContext

16/09/05 22:12:18 INFO yarn.ExecutorRunnable: Preparing Local resources

16/09/05 22:12:18 INFO yarn.ExecutorRunnable: Prepared Local resources Map(__spark__.jar -> resource { scheme: "hdfs" host: "mycluster" port: -1 file: "/user/root/.sparkStaging/application_1473082245027_0003/spark-assembly-1.6.0-hadoop2.6.0.jar" } size: 187548272 timestamp: 1473083954792 type: FILE visibility: PRIVATE)

16/09/05 22:12:18 INFO yarn.ExecutorRunnable:

===============================================================================

YARN executor launch context:

env:

CLASSPATH -> {{PWD}}{{PWD}}/__spark__.jar$HADOOP_CONF_DIR$HADOOP_COMMON_HOME/share/hadoop/common/*$HADOOP_COMMON_HOME/share/hadoop/common/lib/*$HADOOP_HDFS_HOME/share/hadoop/hdfs/*$HADOOP_HDFS_HOME/share/hadoop/hdfs/lib/*$HADOOP_YARN_HOME/share/hadoop/yarn/*$HADOOP_YARN_HOME/share/hadoop/yarn/lib/*$HADOOP_MAPRED_HOME/share/hadoop/mapreduce/*$HADOOP_MAPRED_HOME/share/hadoop/mapreduce/lib/*

SPARK_LOG_URL_STDERR -> http://bigdatahadoop:8042/node/containerlogs/container_1473082245027_0003_01_000004/root/stderr?start=-4096

SPARK_YARN_STAGING_DIR -> .sparkStaging/application_1473082245027_0003

SPARK_YARN_CACHE_FILES_FILE_SIZES -> 187548272

SPARK_USER -> root

SPARK_YARN_CACHE_FILES_VISIBILITIES -> PRIVATE

SPARK_YARN_MODE -> true

SPARK_YARN_CACHE_FILES_TIME_STAMPS -> 1473083954792

SPARK_LOG_URL_STDOUT -> http://bigdatahadoop:8042/node/containerlogs/container_1473082245027_0003_01_000004/root/stdout?start=-4096

SPARK_YARN_CACHE_FILES -> hdfs://mycluster/user/root/.sparkStaging/application_1473082245027_0003/spark-assembly-1.6.0-hadoop2.6.0.jar#__spark__.jar

command:

{{JAVA_HOME}}/bin/java -server -XX:OnOutOfMemoryError='kill %p' -Xms512m -Xmx512m -Djava.io.tmpdir={{PWD}}/tmp '-Dspark.driver.port=45475' -Dspark.yarn.app.container.log.dir= -XX:MaxPermSize=256m org.apache.spark.executor.CoarseGrainedExecutorBackend --driver-url spark://CoarseGrainedScheduler@192.168.184.188:45475 --executor-id 3 --hostname bigdatahadoop --cores 1 --app-id application_1473082245027_0003 --user-class-path file:$PWD/__app__.jar 1> /stdout 2> /stderr

===============================================================================

16/09/05 22:12:18 INFO impl.ContainerManagementProtocolProxy: Opening proxy : bigdatahadoop:39892

16/09/05 22:14:36 INFO yarn.YarnAllocator: Completed container container_1473082245027_0003_01_000004 on host: bigdatahadoop (state: COMPLETE, exit status: 1)

16/09/05 22:14:36 WARN yarn.YarnAllocator: Container marked as failed: container_1473082245027_0003_01_000004 on host: bigdatahadoop. Exit status: 1. Diagnostics: Exception from container-launch: ExitCodeException exitCode=1:

ExitCodeException exitCode=1:

at org.apache.hadoop.util.Shell.runCommand(Shell.java:538)

at org.apache.hadoop.util.Shell.run(Shell.java:455)

at org.apache.hadoop.util.Shell$ShellCommandExecutor.execute(Shell.java:702)

at org.apache.hadoop.yarn.server.nodemanager.DefaultContainerExecutor.launchContainer(DefaultContainerExecutor.java:195)

at org.apache.hadoop.yarn.server.nodemanager.containermanager.launcher.ContainerLaunch.call(ContainerLaunch.java:300)

at org.apache.hadoop.yarn.server.nodemanager.containermanager.launcher.ContainerLaunch.call(ContainerLaunch.java:81)

at java.util.concurrent.FutureTask.run(FutureTask.java:262)

at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)

at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)

at java.lang.Thread.run(Thread.java:745)

Container exited with a non-zero exit code 1

16/09/05 22:14:39 INFO yarn.ApplicationMaster: Final app status: FAILED, exitCode: 11, (reason: Max number of executor failures (3) reached)

16/09/05 22:14:42 INFO util.ShutdownHookManager: Shutdown hook called

select报错 spark_spark-sql master on yarn 模式运行 select count(*) 报错日志相关推荐

  1. jupyter-notebook 以yarn模式运行出现的问题及解决

    jupyter-notebook 以yarn模式运行出现的问题及解决 原创小白programmer 最后发布于2018-11-21 10:53:01 阅读数 519  收藏 展开 jupyter-no ...

  2. linux切换root报错,Linux用户切换到root后运行图形程序报错（*GLib-GIO-CRITICAL **）

    用su切换到root用户后,运行某些带图形的程序,会报如下错误: (ImageProc_qt:3158): GLib-GIO-CRITICAL **: g_dbus_connection_regist ...

  3. flinkx 部署，on yarn模式运行

    目录 1.flinkx部署 参考官方安装文档,但是会有一些坑 2.FlinkX版本需要与Flink版本保持一致,最好小版本也保持一致 下载最新编译包 下载后,lib里面缺少flinkx-launche ...

  4. vscode 终端运行yarn 报错 “因为在此系统上禁止运行脚本”

    问题: 在vscode终端中使用命令yarn 结果运行不起来报错 后面了解到是 PowerShell  的问题 解决办法: 1.首先以管理员的身份打开powershell 2.输入 get-Execu ...

  5. 【错误记录】PyCharm 运行 Python 程序报错 ( UnicodeDecodeError: ‘ascii‘ codec can‘t decode byte 0xe5 in positio )

    文章目录 一.报错信息 二.解决方案 一.报错信息 PyCharm 运行 Python 程序报错 : D:\002_Project\011_Python\APK\venv\Scripts\python ...

  6. Spark内核（上）——附：两种Yarn模式源码解析

    文章目录 一.Spark内核概述 1.1 Spark核心组件回顾 1.1.1 Driver 1.1.2 Executor 1.2 Spark通用运行流程概述 二.Spark通信架构概述 2.1 Spa ...

  7. 报错型sql注入原理分析

    0x00:前言 关于sql注入,经久不衰,现在的网站一般对sql注入的防护也相对加强了,2016年的***测试报告中,出现最多的是xss(跨站脚本***)和明文传输等,但是对sql注入的利用方式,也相 ...

  8. 基于GET报错的sql注入,sqli-lab 1~4

    根据注入类型可将sql注入分为两类:数字型和字符型 例如: 数字型: sleect * from table where if =用户输入id 字符型:select * from table wher ...

  9. SQL Server 合并复制遇到identity range check报错的解决

    最近帮一个客户搭建跨洋的合并复制,由于数据库非常大,跨洋网络条件不稳定,因此只能通过备份初始化,在初始化完成后向海外订阅端插入数据时发现报出如下错误: Msg 548, Level 16, State ...

最新文章

  1. rollback 最后判断成功_面试自我介绍随便说说就行？注意3忌2套路帮你提升50％成功率...
  2. GIT SSH连接遇到到的问题
  3. 袋鼯麻麻——智能购物平台
  4. java注解如何设置自增长_java如何自定义注解（一）
  5. php socket(服务端与客户端)demo
  6. hdu 1078 记忆化搜索
  7. SAP CRM WebClient UI的on_new_focus应该怎么理解
  8. uva 12442 . Forwarding Emails
  9. 正则不等于一个字符串_乳饮料不等于酸奶，记住一个关键词，花最少的钱买到真正的好酸奶...
  10. 《剑指Offer》 调整数组顺序使奇数位于偶数前面
  11. 如何实现DataGridView刷新数据？
  12. BIND配置文件详解（二）
  13. 接口测试之发包工具介绍
  14. linux下opendir的使用
  15. 峰值信噪比PSNR~均方差MSE matlab实现
  16. c语言怎么写最小公倍数的函数,C语言 · 最小公倍数
  17. 「补课」进行时：设计模式(2)——通过一个超级汽车工厂来了解工厂模式
  18. C语言图片变黑白,用c语言把bmp格式的彩色图片转换成黑白的
  19. 达内终端端mysql命令_如何从Windows命令行启动MySQL
  20. JVM基础 -＞ GC分代: MinorGC MajorGC FullGC Mixed GC

热门文章

  1. C++中派生类隐式调用与显式调用基类的构造函数
  2. tensorflow实现回归
  3. Inspeckage，安卓动态分析工具
  4. Hack.Chat 在浏览器里快速建立简单、随用即丢线上聊天室，无须下载安装软体
  5. oracle中 使用不了,Oracle 中不使用NOT IN 和 NOT EXISTS的另一种方法
  6. 如何传入比较器_typescript专题(五) 装饰器
  7. opencv python 直方图反向投影_python OpenCV学习笔记直方图反向投影的实现
  8. mysql按字段同步_MySQL同步(一) 基础知识
  9. 如何做推荐系统 java_Java程序员的日常——SpringMVC+Mybatis开发流程、推荐系统
  10. 双机热备_双机热备软件哪个好？双机热备软件推荐