Hadoop3.0.3-SYNTH运行

hadoop的迭代实在太快，导致出现很多bug。

在运行SYNTH模式时。发现两个问题。

1.官方给出的SYNTH的json脚本：

{"description" : "tiny jobs workload",    //description of the meaning of this collection of workloads"num_nodes" : 10,  //total nodes in the simulated cluster"nodes_per_rack" : 4, //number of nodes in each simulated rack"num_jobs" : 10, // total number of jobs being simulated"rand_seed" : 2, //the random seed used for deterministic randomized runs// a list of “workloads”, each of which has job classes, and temporal properties"workloads" : [{"workload_name" : "tiny-test", // name of the workload"workload_weight": 0.5,  // used for weighted random selection of which workload to sample from"queue_name" : "sls_queue_1", //queue the job will be submitted to//different classes of jobs for this workload"job_classes" : [{"class_name" : "class_1", //name of the class"class_weight" : 1.0, //used for weighted random selection of class within workload//nextr group controls average and standard deviation of a LogNormal distribution that//determines the number of mappers and reducers for thejob."mtasks_avg" : 5,"mtasks_stddev" : 1,"rtasks_avg" : 5,"rtasks_stddev" : 1,//averge and stdev input param of LogNormal distribution controlling job duration"dur_avg" : 60,"dur_stddev" : 5,//averge and stdev input param of LogNormal distribution controlling mappers and reducers durations"mtime_avg" : 10,"mtime_stddev" : 2,"rtime_avg" : 20,"rtime_stddev" : 4,//averge and stdev input param of LogNormal distribution controlling memory and cores for map and reduce"map_max_memory_avg" : 1024,"map_max_memory_stddev" : 0.001,"reduce_max_memory_avg" : 2048,"reduce_max_memory_stddev" : 0.001,"map_max_vcores_avg" : 1,"map_max_vcores_stddev" : 0.001,"reduce_max_vcores_avg" : 2,"reduce_max_vcores_stddev" : 0.001,//probability of running this job with a reservation"chance_of_reservation" : 0.5,//input parameters of LogNormal distribution that determines the deadline slack (as a multiplier of job duration)"deadline_factor_avg" : 10.0,"deadline_factor_stddev" : 0.001,}],// for each workload determines with what probability each time bucket is picked to choose the job starttime.// In the example below the jobs have twice as much chance to start in the first minute than in the second minute// of simulation, and then zero chance thereafter."time_distribution" : [{ "time" : 1, "weight" : 66 },{ "time" : 60, "weight" : 33 },{ "time" : 120, "jobs" : 0 }]}]
}

首先json文件中不能有任何注释，因此要删除这些注释才能运行。其次在

"deadline_factor_stddev" : 0.001,

这一行最后的逗号不能有，否则不符合json文件的格式，运行报错。

这里给出一个在线查看json文件是否合格的网站：

https://jsonlint.com/

2.在运行过程中

$HADOOP_HOME/share/hadoop/tools/sls/bin/slsrun.sh --tracetype=SYNTH --tracelocation=/home/c/sls/output2/SYNTH.json --output-dir=/home/c/sls/output1 --print-simulation

报错如下：

java.lang.IllegalArgumentException: Null userat org.apache.hadoop.security.UserGroupInformation.createRemoteUser(UserGroupInformation.java:1225)at org.apache.hadoop.security.UserGroupInformation.createRemoteUser(UserGroupInformation.java:1212)at org.apache.hadoop.yarn.sls.appmaster.AMSimulator.submitReservationWhenSpecified(AMSimulator.java:177)at org.apache.hadoop.yarn.sls.appmaster.AMSimulator.firstStep(AMSimulator.java:154)at org.apache.hadoop.yarn.sls.scheduler.TaskRunner$Task.run(TaskRunner.java:88)at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)at java.lang.Thread.run(Thread.java:748)
java.lang.IllegalArgumentException: Null user

报错信息是NULL user传入了。

在最开始传入该参数时：

  private void startAMFromSynthGenerator() throws YarnException, IOException {Configuration localConf = new Configuration();localConf.set("fs.defaultFS", "file:///");long baselineTimeMS = 0;// if we use the nodeFile this could have been not initialized yet.if (stjp == null) {stjp = new SynthTraceJobProducer(getConf(), new Path(inputTraces[0]));}SynthJob job = null;// we use stjp, a reference to the job producer instantiated during node// creationwhile ((job = (SynthJob) stjp.getNextJob()) != null) {// only support MapReduce currentlyString user = job.getUser();

getUser()返回后没有判断是不是为NULL。导致错误。

而对于从SLS和rumen输入的函数，得到user时是做了判断的：

  private void createAMForJob(Map jsonJob) throws YarnException {long jobStartTime = Long.parseLong(jsonJob.get(SLSConfiguration.JOB_START_MS).toString());long jobFinishTime = 0;if (jsonJob.containsKey(SLSConfiguration.JOB_END_MS)) {jobFinishTime = Long.parseLong(jsonJob.get(SLSConfiguration.JOB_END_MS).toString());}String user = (String) jsonJob.get(SLSConfiguration.JOB_USER);if (user == null) {user = "default";}

所以我在想是不是hadoop官方对SYNTH的支持不是很完善。

Hadoop3.0.3-SYNTH运行相关推荐

Hadoop3.0基础平台搭建(三节点),以及案例运行并使用python生成词云
一.设备配置列表,软件包二.主机名映射三.生成密钥四.将密钥发送到各个节点五.注意事项及验证六.关闭防火墙和SeLinux安全模块(所有主机均需操作七.解压JDK.hadoop包,设置Ja ...
hadoop3.0 分布式搭建/安装
原文:http://blog.csdn.net/u010499619/article/details/52886872 最近打算尝试在服务器上安装hadoop3.0,由于服务器上本来搭建好hadoop ...
配置开发者本地hadoop-3.0.0环境
1 配置开发者本地hadoop环境参考网址:https://blog.csdn.net/songhaifengshuaige/article/details/79575308 1.1.下载hadoo ...
centos7 hadoop3.0.1安装
安装JDK rpm -qa | grep java 找出OPENJDK yum -y remove java 上一步复制的jdk文件名卸载openjdk rpm -ivh jdk文件名安装jdk ...
mac下编译hadoop3.0.*版本的native lib
编译好的hadoop3.0.2的native 见https://download.csdn.net/download/qq_14811559/10413344 本人的hadoop版本为3.0.0,用3 ...
大数据学习系列：Hadoop3.0苦命学习（一）
传送门: 大数据学习系列:Hadoop3.0苦命学习(一) 大数据学习系列:Hadoop3.0苦命学习(二) 大数据学习系列:Hadoop3.0苦命学习(三) 大数据学习系列:Hadoop3.0苦命学 ...
大数据学习系列：Hadoop3.0苦命学习（七）
传送门: 大数据学习系列:Hadoop3.0苦命学习(一) 大数据学习系列:Hadoop3.0苦命学习(二) 大数据学习系列:Hadoop3.0苦命学习(三) 大数据学习系列:Hadoop3.0苦命学 ...
【Debug跟踪Hadoop3.0.0源码之MapReduce Job提交流程】第三节 Job提交前的初始化
[Debug跟踪Hadoop3.0.0源码之MapReduce Job提交流程]第三节 Job提交前的初始化回顾 Job提交前的初始化后记跳转回顾上一节中我们对 jobSubmitter(提 ...
hadoop下载地址与配置hadoop3.0集群
hadoop版本下载访问地址:http://mirrors.hust.edu.cn/apache/hadoop/common 搭建hadoop集群访问地址:https://blog.csdn.net/ ...
体验Hadoop3.0生态圈-CDH6.1时代的来临
体验Hadoop3.0生态圈-CDH6.1时代的来临作者:尹正杰版权声明:原创作品,谢绝转载!否则将追究法律责任. 我在公司使用的是CDH5.15.1这个发行版本,具体的部署文档之前也有给大家分享 ...

Hadoop3.0.3-SYNTH运行

Hadoop3.0.3-SYNTH运行相关推荐

最新文章

热门文章