运行 Giraph 提示 too many counters
运行 Giraph 提示 too many counters
在加入 -ca mapreduce.job.counters.limit=1000
后,仍然运行失败
16/10/20 08:56:08 INFO job.GiraphJob: Waiting for resources... Job will start only when it gets all 2 mappers
16/10/20 08:56:38 INFO job.HaltApplicationUtils$DefaultHaltInstructionsWriter: writeHaltInstructions: To halt after next superstep execute: 'bin/halt-application --zkServer s3:22181 --zkNode /_hadoopBsp/job_1476868823433_0017/_haltComputation'
16/10/20 08:56:38 INFO mapreduce.Job: Running job: job_1476868823433_0017
16/10/20 08:56:39 INFO mapreduce.Job: Job job_1476868823433_0017 running in uber mode : false
16/10/20 08:56:39 INFO mapreduce.Job: map 50% reduce 0%
16/10/20 08:56:47 INFO mapreduce.Job: map 100% reduce 0%
16/10/20 08:56:47 INFO mapreduce.Job: Job job_1476868823433_0017 failed with state FAILED due to: Task failed task_1476868823433_0017_m_000000
Job failed as tasks failed. failedMaps:1 failedReduces:016/10/20 08:56:47 INFO mapreduce.Job: Counters: 34File System CountersFILE: Number of bytes read=0FILE: Number of bytes written=97529FILE: Number of read operations=0FILE: Number of large read operations=0FILE: Number of write operations=0HDFS: Number of bytes read=76HDFS: Number of bytes written=0HDFS: Number of read operations=8HDFS: Number of large read operations=0HDFS: Number of write operations=4Job Counters Failed map tasks=1Launched map tasks=2Other local map tasks=2Total time spent by all maps in occupied slots (ms)=33269Total time spent by all reduces in occupied slots (ms)=0Total time spent by all map tasks (ms)=33269Total vcore-seconds taken by all map tasks=33269Total megabyte-seconds taken by all map tasks=34067456Map-Reduce FrameworkMap input records=1Map output records=0Input split bytes=44Spilled Records=0Failed Shuffles=0Merged Map outputs=0GC time elapsed (ms)=130CPU time spent (ms)=7280Physical memory (bytes) snapshot=186077184Virtual memory (bytes) snapshot=823398400Total committed heap usage (bytes)=200802304Zookeeper base path/_hadoopBsp/job_1476868823433_0017=0Zookeeper halt node/_hadoopBsp/job_1476868823433_0017/_haltComputation=0Zookeeper server:ports3:22181=0File Input Format Counters Bytes Read=0File Output Format Counters Bytes Written=0
Log 日志显示:
2016-10-20 08:56:38,569 INFO [org.apache.giraph.master.MasterThread] org.apache.giraph.master.MasterThread: masterThread: Coordination of superstep 199 took 0.016 seconds ended with state ALL_SUPERSTEPS_DONE and is now on superstep 200
2016-10-20 08:56:38,573 INFO [org.apache.giraph.master.MasterThread] org.apache.giraph.master.BspServiceMaster: setJobState: {"_stateKey":"FINISHED","_applicationAttemptKey":-1,"_superstepKey":-1} on superstep 200
2016-10-20 08:56:38,574 INFO [org.apache.giraph.master.MasterThread] org.apache.giraph.master.BspServiceMaster: setJobState: {"_stateKey":"FINISHED","_applicationAttemptKey":-1,"_superstepKey":-1}
2016-10-20 08:56:38,574 INFO [ProcessThread(sid:0 cport:-1):] org.apache.zookeeper.server.PrepRequestProcessor: Got user-level KeeperException when processing sessionid:0x157df96ea710000 type:create cxid:0x236f zxid:0x143b txntype:-1 reqpath:n/a Error Path:/_hadoopBsp/job_1476868823433_0017/_cleanedUpDir Error:KeeperErrorCode = NoNode for /_hadoopBsp/job_1476868823433_0017/_cleanedUpDir
2016-10-20 08:56:38,574 INFO [ProcessThread(sid:0 cport:-1):] org.apache.zookeeper.server.PrepRequestProcessor: Got user-level KeeperException when processing sessionid:0x157df96ea710001 type:create cxid:0xd8f zxid:0x143c txntype:-1 reqpath:n/a Error Path:/_hadoopBsp/job_1476868823433_0017/_masterJobState Error:KeeperErrorCode = NodeExists for /_hadoopBsp/job_1476868823433_0017/_masterJobState
2016-10-20 08:56:38,575 INFO [org.apache.giraph.master.MasterThread] org.apache.giraph.master.BspServiceMaster: cleanup: Notifying master its okay to cleanup with /_hadoopBsp/job_1476868823433_0017/_cleanedUpDir/0_master
2016-10-20 08:56:38,575 INFO [ProcessThread(sid:0 cport:-1):] org.apache.zookeeper.server.PrepRequestProcessor: Got user-level KeeperException when processing sessionid:0x157df96ea710000 type:create cxid:0x2375 zxid:0x143f txntype:-1 reqpath:n/a Error Path:/_hadoopBsp/job_1476868823433_0017/_cleanedUpDir Error:KeeperErrorCode = NodeExists for /_hadoopBsp/job_1476868823433_0017/_cleanedUpDir
2016-10-20 08:56:38,575 INFO [org.apache.giraph.master.MasterThread] org.apache.giraph.master.BspServiceMaster: cleanUpZooKeeper: Node /_hadoopBsp/job_1476868823433_0017/_cleanedUpDir already exists, no need to create.
2016-10-20 08:56:38,576 INFO [org.apache.giraph.master.MasterThread] org.apache.giraph.master.BspServiceMaster: cleanUpZooKeeper: Got 1 of 2 desired children from /_hadoopBsp/job_1476868823433_0017/_cleanedUpDir
2016-10-20 08:56:38,576 INFO [org.apache.giraph.master.MasterThread] org.apache.giraph.master.BspServiceMaster: cleanedUpZooKeeper: Waiting for the children of /_hadoopBsp/job_1476868823433_0017/_cleanedUpDir to change since only got 1 nodes.
2016-10-20 08:56:40,710 INFO [communication thread] org.apache.hadoop.mapred.Task: Communication exception: org.apache.hadoop.ipc.RemoteException(org.apache.hadoop.ipc.RpcServerException): IPC server unable to read call parameters: Too many counters: 121 max=120at org.apache.hadoop.ipc.Client.call(Client.java:1411)at org.apache.hadoop.ipc.Client.call(Client.java:1364)at org.apache.hadoop.ipc.WritableRpcEngine$Invoker.invoke(WritableRpcEngine.java:231)at com.sun.proxy.$Proxy7.statusUpdate(Unknown Source)at org.apache.hadoop.mapred.Task$TaskReporter.run(Task.java:737)at java.lang.Thread.run(Thread.java:745)2016-10-20 08:56:40,879 INFO [main-EventThread] org.apache.giraph.bsp.BspService: process: cleanedUpChildrenChanged signaled
2016-10-20 08:56:40,880 INFO [org.apache.giraph.master.MasterThread] org.apache.giraph.master.BspServiceMaster: cleanUpZooKeeper: Got 2 of 2 desired children from /_hadoopBsp/job_1476868823433_0017/_cleanedUpDir
2016-10-20 08:56:40,880 INFO [ProcessThread(sid:0 cport:-1):] org.apache.zookeeper.server.PrepRequestProcessor: Processed session termination for sessionid: 0x157df96ea710001
2016-10-20 08:56:40,882 INFO [NIOServerCxn.Factory:0.0.0.0/0.0.0.0:22181] org.apache.zookeeper.server.NIOServerCnxn: Closed socket connection for client /219.223.239.57:49390 which had sessionid 0x157df96ea710001
2016-10-20 08:56:40,888 INFO [org.apache.giraph.master.MasterThread] org.apache.giraph.master.BspServiceMaster: cleanup: Removed HDFS checkpoint directory (_bsp/_checkpoints//job_1476868823433_0017) with return = false since the job Giraph: cost.Test succeeded
2016-10-20 08:56:40,888 INFO [org.apache.giraph.master.MasterThread] org.apache.giraph.comm.netty.NettyClient: stop: Halting netty client
2016-10-20 08:56:40,890 INFO [netty-client-worker-0] org.apache.giraph.comm.netty.NettyClient: stop: reached wait threshold, 1 connections closed, releasing resources now.
2016-10-20 08:56:43,095 INFO [org.apache.giraph.master.MasterThread] org.apache.giraph.comm.netty.NettyClient: stop: Netty client halted
2016-10-20 08:56:43,095 INFO [org.apache.giraph.master.MasterThread] org.apache.giraph.comm.netty.NettyServer: stop: Halting netty server
2016-10-20 08:56:43,106 INFO [org.apache.giraph.master.MasterThread] org.apache.giraph.comm.netty.NettyServer: stop: Start releasing resources
2016-10-20 08:56:43,780 INFO [communication thread] org.apache.hadoop.mapred.Task: Communication exception: org.apache.hadoop.ipc.RemoteException(org.apache.hadoop.ipc.RpcServerException): IPC server unable to read call parameters: Too many counters: 121 max=120at org.apache.hadoop.ipc.Client.call(Client.java:1411)at org.apache.hadoop.ipc.Client.call(Client.java:1364)at org.apache.hadoop.ipc.WritableRpcEngine$Invoker.invoke(WritableRpcEngine.java:231)at com.sun.proxy.$Proxy7.statusUpdate(Unknown Source)at org.apache.hadoop.mapred.Task$TaskReporter.run(Task.java:737)at java.lang.Thread.run(Thread.java:745)2016-10-20 08:56:43,793 INFO [communication thread] org.apache.hadoop.mapred.Task: Process Thread Dump: Communication exception
46 active threads
Thread 56 (netty-server-worker-15):State: RUNNABLEBlocked count: 0Waited count: 1Stack:sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:269)sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:79)sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:87)sun.nio.ch.SelectorImpl.select(SelectorImpl.java:98)io.netty.channel.nio.NioEventLoop.select(NioEventLoop.java:596)io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:306)io.netty.util.concurrent.SingleThreadEventExecutor$2.run(SingleThreadEventExecutor.java:101)java.lang.Thread.run(Thread.java:745)
Thread 55 (netty-server-worker-14):State: RUNNABLEBlocked count: 0Waited count: 1
还是 Counters 的问题
网上查好像是说在命令行设置 mapreduce.job.counters.limit
属性没有作用
于是设置 $HADOOP_HOME/conf/mapred-site.xml 中的属性
<property><name>mapreduce.job.counters.limit</name><value>20000</value><description>Limit on the number of counters allowed per job. The default value is 200.</description>
</property>
一切正常~
转载于:https://www.cnblogs.com/chyf/p/6055952.html
运行 Giraph 提示 too many counters相关推荐
- 电脑显示没有被指定在上运行_win10系统运行程序提示“dll没有被指定在windows上运行”的办法...
有关win10系统运行程序提示"dll没有被指定在windows上运行"的操作方法想必大家有所耳闻.但是能够对win10系统运行程序提示"dll没有被指定在windows ...
- eclipse中运行tomcat提示端口被占的4种解决方案
eclipse中运行tomcat提示端口被占的4种解决方案 参考文章: (1)eclipse中运行tomcat提示端口被占的4种解决方案 (2)https://www.cnblogs.com/wyhl ...
- cx_oracle windows缺少100 dll,Windows10无法运行QQ提示缺少MSVCR100.dll的解决方案
QQ是一款广受用户喜爱的即时通讯软件,我们在工作和生活中都经常会使用到它.不过,一些用户反馈在win10 32位系统中双击打开QQ后,会出现缺少MSVCR100.dll的提示,且QQ无法运行,那是怎么 ...
- 无法启动此程序因为计算机中丢失msvcr110,无法启动此程序因为计算机中丢失msvcr110,教您无法运行程序提示计算机中丢失...
用户在一次运行某程序时,运到"无法启动此程序,因为计算机中丢失msvcr.dll.尝试重新安装该程序以解决此问题."的提示,当时很无语,因为系统是刚刚安装好的,怎么处理呢?下面,小 ...
- tomcat7 https 拒绝连接_Ubuntu上运行Docker提示权限拒绝,如何处理?
请关注本头条号,每天坚持更新原创干货技术文章. 如需学习视频,请在微信搜索公众号"智传网优"直接开始自助视频学习 1. 前言 如果您在Ubuntu上运行Docker,但被提示Per ...
- win10运行C语言的程序,win10系统运行软件提示应用程序发生异常0xc0000409的具体教程...
有关win10系统运行软件提示应用程序发生异常0xc0000409的操作方法想必大家有所耳闻.但是能够对win10系统运行软件提示应用程序发生异常0xc0000409进行实际操作的人却不多.其实解决w ...
- 计算机蓝屏代码0xc0000020,Win10运行程序提示“损坏的映像 错误0xc0000020”怎么解决?...
Win10运行程序提示"损坏的映像 错误0xc0000020"怎么解决?很多用户在Win10系统下运行软件时,会出现损坏的映像提示框,内容显示为"C:\xx\xxx.dl ...
- Android之运行PopupWindow提示Unable to add window -- token null is not valid; is your activity running?
1.问题 在使用PopupWindow的时候,我们构建好了直接放在Activity的onCreate函数里面直接运行,提示这个错误 Unable to add window -- token null ...
- python安装运行时提示不是内部或外部命令怎么办_如何解决cmd运行python提示不是内部命令...
python安装完成后,直接运行python.exe能够正常执行python程序.但是进入到cmd命令窗口(同时按下win+r组合键后输入cmd进入),输入python命令提示"不是内部或外 ...
最新文章
- 如何成为一名专家级的开发人员
- 《易学Python》——1.4 Linux
- jquery ajax json table 添加行,使用jQuery从AJAX响应构建表行(json)
- tp5 引入 没有命名空间的类库的方法(以微信支付SDK为例)
- java程序设计教程实验报告_java程序设计课程--实验报告-实验13.doc
- 每日一题:leetcode82. 删除排序链表中的重复元素 II
- 快速入门系列之 Rust 语言 GitChat连接
- 2017.8.15 阿狸的打字机 失败总结
- mysql设置user权限允许远程_mysql 用户及权限管理 允许远程连接
- 双指针 -- 验证回文串
- 【JOURNAL】《题虾壶》
- 2021年10月程序员薪资出炉,北京最高工资达45000元!
- ZOJ 4093 浙大校赛D题 Robot Cleaner II(思维+构造题)
- 检查android sdk是否安装成功,Android判断手机是否安装微信
- 网络知识汇总(1)-朗文和牛津英语词典网址
- xiuno4.0 火车头发布模块一Web登陆版
- 小白必备!Rust 编程语言入门教程
- Rabbitmq消息中心_消息中心总体方案
- atlas mysql 分表,Atlas实现Mysql读写分离
- 2.文件---open