Spark学习-事件日志EventLog
2024-04-26 00:12:20
事件日志EventLog
- 背景
- 系统结构
- Spark Event
- 介绍
- 事件类型
- 事件内容
- 事件日志
- 事件分析
背景
最近工作需要使用Flink对Spark的应用事件日志进行处理，帮助发现Spark应用中常见异常问题，为用户提供方便快捷的常见问题排查，在此作一下简单的记录和学习。
系统结构
Spark开启Thrift Server进行Spark SQL查询，Spark的Jobhistory服务对应用的事件进行记录和保存，利用FileBeat对历史事件日志传送到Kafka Topic中，最后使用Flink对Topic中的事件日志进行处理，并输出到ElasticSearch作结果展示。
Spark Event
介绍
Spark Event是Spark为Spark App定义的运行事件，用于展示一个Spark App完整的生命周期，Spark History Server服务可将App的事件日志存储到Hdfs中(即Event Log，JSON文件格式),一般来说一个Spark Job对应一个事件日志，通过分析这些事件日志，我们可以发现一些应用中的错误和待提升的点，从而进一步去优化应用性能。
事件类型
Spark事件类型可分为如下类型，具体可参考Spark源码SparkListener.class。
Spark的运行过程正是由上图中的多种不同事件组合而成，事件归纳一下主要分为Spark App运行事件、资源变更事件、环境变更事件和其他事件，图中相同颜色代表属于同一过程的不同状态的事件。
事件内容
| 事件名称 | 事件描述 | 事件内容 |
|---|---|---|
| SparkListenerApplicationStart | App开始事件 |
App Name:App名称 App ID:App的主键 Timestamp: 事件触发时间 User:提交App的用户 |
| SparkListenerApplicationEnd | App结束事件 | Timestamp:事件触发时间 |
| SparkListenerJobStart | Job开始事件 |
Job ID:Job的主键 Submission Time:Job提交时间 Stage Infos: Job中所有stage的详细信息 Stage IDs:Job中所有stage的ID列表 Properties:Job的配置信息 |
| SparkListenerJobEnd | Job结束事件 |
Job ID:Job的主键 Completion Time:Job完成时间 Job Result: Job的执行结果信息 |
| SparkListenerStageSubmitted | Stage提交事件 |
Stage Info:当前提交Stage的详细信息 Properties:Stage的配置信息 |
| SparkListenerStageCompleted | Stage完成事件 | Stage Info:当前完成Stage的详细信息 |
| SparkListenerTaskStart | Task启动事件 |
Stage ID:任务所在Stage的主键 Stage Attempt ID:上次尝试提交Stage的主键 Task Info:Task任务的详细信息 |
| SparkListenerTaskGettingResult | Task下载任务结果事件 | Task Info:Task执行的详细信息 |
| SparkListenerTaskEnd | Task任务结束事件 |
Stage ID:任务所在Stage的主键 Stage Attempt ID:上次尝试提交Stage的主键 Task Type:Task的类型 Task End Reason:Task结束的原因 Task Info:Task任务的详细信息 Task Executor Metrics:执行Task的Executor的JVM指标信息 Task Metrics:Task运行的指标信息 |
| SparkListenerEnvironmentUpdate | 环境更新事件 |
JVM Information:Java和Scala的版本信息 Spark Properties:App的Spark参数配置信息 Hadoop Properties:App的Yarn参数配置信息 System Properties:提交App的系统环境配置信息 Classpath Entries:环境变量信息 |
| SparkListenerBlockManagerAdded | 块管理器添加事件 |
Block Manager ID:块管理器的主键信息 Maximum Memory:最大内存值 Timestamp:事件触发时间 Maximum Onheap Memory:最大堆内内存值 Maximum Offheap Memory:最大堆外内存值 |
| SparkListenerBlockManagerRemoved | 块管理器移除事件 |
Block Manager ID:块管理器的主键信息 Timestamp:事件触发时间 |
| SparkListenerExecutorMetricsUpdate | Task指标更新事件 |
Executor ID:Executor的主键 Metrics Updated:Task更新的指标值 |
| SparkListenerExecutorAdded | Executor注册事件 |
Executor ID:Executor的主键 Executor Info:Executor的详细信息 Timestamp:事件触发时间 |
| SparkListenerExecutorRemoved | Executor移除事件 |
Executor ID:Executor的主键 Timestamp:事件触发时间 Removed Reason:Executor移除原因 |
| SparkListenerBlockUpdated | block更新事件 | Block Updated Info:Block更新的详细信息 |
| SparkListenerUnpersistRDD | 取消 RDD缓存事件 | RDD ID:RDD的主键 |
| SparkListenerExecutorBlacklisted | Executor黑名单事件 |
time:事件触发时间 executorId:Executor的主键 taskFailures:task失败次数 |
| SparkListenerExecutorBlacklistedForStage | 针对Stage的Executor黑名单事件 |
time:事件触发时间 executorId:Executor的主键 taskFailures:task失败次数 stageId:Stage的主键 stageAttemptId:上次尝试提交Stage的主键 |
| SparkListenerNodeBlacklisted | Node黑名单事件 |
time:事件触发时间 hostId:节点的主键 executorFailures:executor的失败次数 |
| SparkListenerNodeBlacklistedForStage | 针对Stage的Node黑名单事件 |
time:事件触发时间 executorId:Executor的主键 executorFailures:executor的失败次数 stageId:Stage的主键 stageAttemptId:上次尝试提交Stage的主键 |
| SparkListenerExecutorUnblacklisted | Executor黑名单移除事件 |
time:事件触发时间 executorId:Executor的主键 |
| SparkListenerNodeUnblacklisted | Node黑名单移除事件 |
time:事件触发时间 hostId:节点的主键 |
表格事件内容列中，红色字段是会在事件日志中输出的JSON字段，蓝色字段是不会在事件日志中输出的后台字段。
事件日志
Spark Pi运行的事件日志如下所示：
{"Event":"SparkListenerLogStart","Spark Version":"3.0.1"}
{"Event":"SparkListenerBlockManagerAdded","Block Manager ID":{"Executor ID":"driver","Host":"master","Port":43711},"Maximum Memory":97832140,"Timestamp":1603121632911,"Maximum Onheap Memory":97832140,"Maximum Offheap Memory":0}
{"Event":"SparkListenerEnvironmentUpdate","JVM Information":{"Java Home":"/home/java/jre","Java Version":"1.8.0_162 (Oracle Corporation)","Scala Version":"version 2.12.10"},"Spark Properties":{"spark.serializer":"org.apache.spark.serializer.KryoSerializer","spark.driver.host":"master","spark.eventLog.enabled":"true","spark.driver.port":"46515","spark.jars":"file:/home/spark-3.0.1/examples/jars/spark-examples_2.12-3.0.1.jar","spark.app.name":"Spark Pi","spark.scheduler.mode":"FIFO","spark.driver.memory":"512m","spark.submit.pyFiles":"","spark.executor.id":"driver","spark.submit.deployMode":"client","spark.master":"spark://master:7077","spark.executor.memory":"512m","spark.eventLog.dir":"hdfs://master:9000/spark_event","spark.executor.cores":"1","spark.app.id":"app-20201019233352-0002"},"Hadoop Properties":{"hadoop.service.shutdown.timeout":"30s","yarn.resourcemanager.amlauncher.thread-count":"50","yarn.sharedcache.enabled":"false","fs.s3a.connection.maximum":"15","yarn.nodemanager.numa-awareness.numactl.cmd":"/usr/bin/numactl","fs.s3a.impl":"org.apache.hadoop.fs.s3a.S3AFileSystem","yarn.app.mapreduce.am.scheduler.heartbeat.interval-ms":"1000","yarn.timeline-service.timeline-client.number-of-async-entities-to-merge":"10","hadoop.proxyuser.root.hosts":"*","hadoop.security.kms.client.timeout":"60","hadoop.http.authentication.kerberos.principal":"HTTP/_HOST@LOCALHOST","mapreduce.jobhistory.loadedjob.tasks.max":"-1","mapreduce.framework.name":"yarn","yarn.sharedcache.uploader.server.thread-count":"50","yarn.nodemanager.linux-container-executor.nonsecure-mode.user-pattern":"^[_.A-Za-z0-9][-@_.A-Za-z0-9]{0,255}?[$]?$","tfile.fs.output.buffer.size":"262144","yarn.app.mapreduce.am.job.task.listener.thread-count":"30","hadoop.security.groups.cache.background.reload.threads":"3","yarn.resourcemanager.webapp.cross-origin.enabled":"false","fs.AbstractFileSystem.ftp.impl":"org.apache.hadoop.fs.ftp.FtpFs","hadoop.registry.secure":"false","hadoop.shell.safely.delete.limit.num.files":"100","dfs.bytes-per-checksum":"512","mapreduce.job.acl-view-job":" ","fs.s3a.s3guard.ddb.background.sleep":"25ms","fs.s3a.retry.limit":"${fs.s3a.attempts.maximum}","mapreduce.jobhistory.loadedjobs.cache.size":"5","fs.s3a.s3guard.ddb.table.create":"false","yarn.nodemanager.amrmproxy.enabled":"false","yarn.timeline-service.entity-group-fs-store.with-user-dir":"false","mapreduce.input.fileinputformat.split.minsize":"0","yarn.resourcemanager.container.liveness-monitor.interval-ms":"600000","yarn.resourcemanager.client.thread-count":"50","io.seqfile.compress.blocksize":"1000000","yarn.sharedcache.checksum.algo.impl":"org.apache.hadoop.yarn.sharedcache.ChecksumSHA256Impl","yarn.nodemanager.amrmproxy.interceptor-class.pipeline":"org.apache.hadoop.yarn.server.nodemanager.amrmproxy.DefaultRequestInterceptor","dfs.datanode.data.dir":"/data/hadoop/hdfs/data","dfs.replication":"3","yarn.timeline-service.entity-group-fs-store.leveldb-cache-read-cache-size":"10485760","mapreduce.reduce.shuffle.fetch.retry.interval-ms":"1000","mapreduce.task.profile.maps":"0-2","yarn.scheduler.include-port-in-node-name":"false","yarn.nodemanager.admin-env":"MALLOC_ARENA_MAX=$MALLOC_ARENA_MAX","yarn.resourcemanager.node-removal-untracked.timeout-ms":"60000","mapreduce.am.max-attempts":"2","hadoop.security.kms.client.failover.sleep.base.millis":"100","mapreduce.jobhistory.webapp.https.address":"0.0.0.0:19890","yarn.node-labels.fs-store.impl.class":"org.apache.hadoop.yarn.nodelabels.FileSystemNodeLabelsStore","yarn.nodemanager.collector-service.address":"${yarn.nodemanager.hostname}:8048","fs.trash.checkpoint.interval":"0","mapreduce.job.map.output.collector.class":"org.apache.hadoop.mapred.MapTask$MapOutputBuffer","yarn.resourcemanager.node-ip-cache.expiry-interval-secs":"-1","hadoop.http.authentication.signature.secret.file":"*********(redacted)","hadoop.jetty.logs.serve.aliases":"true","yarn.resourcemanager.placement-constraints.handler":"disabled","yarn.timeline-service.handler-thread-count":"10","yarn.resourcemanager.max-completed-applications":"1000","yarn.resourcemanager.system-metrics-publisher.enabled":"false","yarn.resourcemanager.placement-constraints.algorithm.class":"org.apache.hadoop.yarn.server.resourcemanager.scheduler.constraint.algorithm.DefaultPlacementAlgorithm","yarn.sharedcache.webapp.address":"0.0.0.0:8788","yarn.resourcemanager.delegation.token.renew-interval":"*********(redacted)","yarn.sharedcache.nm.uploader.replication.factor":"10","hadoop.security.groups.negative-cache.secs":"30","yarn.app.mapreduce.task.container.log.backups":"0","mapreduce.reduce.skip.proc-count.auto-incr":"true","hadoop.security.group.mapping.ldap.posix.attr.gid.name":"gidNumber","rpc.engine.org.apache.hadoop.hdfs.protocolPB.ClientNamenodeProtocolPB":"org.apache.hadoop.ipc.ProtobufRpcEngine","ipc.client.fallback-to-simple-auth-allowed":"false","yarn.nodemanager.resource.memory.enforced":"true","hadoop.proxyuser.root.groups":"*","yarn.client.failover-proxy-provider":"org.apache.hadoop.yarn.client.ConfiguredRMFailoverProxyProvider","yarn.timeline-service.http-authentication.simple.anonymous.allowed":"true","ha.health-monitor.check-interval.ms":"1000","yarn.acl.reservation-enable":"false","yarn.resourcemanager.store.class":"org.apache.hadoop.yarn.server.resourcemanager.recovery.FileSystemRMStateStore","yarn.app.mapreduce.am.hard-kill-timeout-ms":"10000","fs.s3a.etag.checksum.enabled":"false","yarn.nodemanager.container-metrics.enable":"true","yarn.timeline-service.client.fd-clean-interval-secs":"60","yarn.resourcemanager.nodemanagers.heartbeat-interval-ms":"1000","hadoop.common.configuration.version":"3.0.0","fs.s3a.s3guard.ddb.table.capacity.read":"500","yarn.nodemanager.remote-app-log-dir-suffix":"logs","yarn.nodemanager.windows-container.cpu-limit.enabled":"false","yarn.nodemanager.runtime.linux.docker.privileged-containers.allowed":"false","file.blocksize":"67108864","hadoop.registry.zk.retry.ceiling.ms":"60000","yarn.scheduler.configuration.leveldb-store.path":"${hadoop.tmp.dir}/yarn/system/confstore","yarn.sharedcache.store.in-memory.initial-delay-mins":"10","mapreduce.jobhistory.principal":"jhs/_HOST@REALM.TLD","mapreduce.map.skip.proc-count.auto-incr":"true","fs.s3a.committer.name":"file","mapreduce.task.profile.reduces":"0-2","hadoop.zk.num-retries":"1000","yarn.webapp.xfs-filter.enabled":"true","seq.io.sort.mb":"100","yarn.scheduler.configuration.max.version":"100","yarn.timeline-service.webapp.https.address":"${yarn.timeline-service.hostname}:8190","yarn.resourcemanager.scheduler.address":"${yarn.resourcemanager.hostname}:8030","yarn.node-labels.enabled":"false","yarn.resourcemanager.webapp.ui-actions.enabled":"true","mapreduce.task.timeout":"600000","yarn.sharedcache.client-server.thread-count":"50","hadoop.security.groups.shell.command.timeout":"0s","hadoop.security.crypto.cipher.suite":"AES/CTR/NoPadding","yarn.nodemanager.elastic-memory-control.oom-handler":"org.apache.hadoop.yarn.server.nodemanager.containermanager.linux.resources.DefaultOOMHandler","yarn.resourcemanager.connect.max-wait.ms":"900000","fs.defaultFS":"hdfs://master:9000","yarn.minicluster.use-rpc":"false","fs.har.impl.disable.cache":"true","yarn.webapp.ui2.enable":"false","io.compression.codec.bzip2.library":"system-native","yarn.nodemanager.distributed-scheduling.enabled":"false","mapreduce.shuffle.connection-keep-alive.timeout":"5","yarn.resourcemanager.webapp.https.address":"${yarn.resourcemanager.hostname}:8090","mapreduce.jobhistory.address":"0.0.0.0:10020","yarn.resourcemanager.nm-tokens.master-key-rolling-interval-secs":"*********(redacted)","yarn.is.minicluster":"false","yarn.nodemanager.address":"${yarn.nodemanager.hostname}:0","fs.abfss.impl":"org.apache.hadoop.fs.azurebfs.SecureAzureBlobFileSystem","fs.AbstractFileSystem.s3a.impl":"org.apache.hadoop.fs.s3a.S3A","mapreduce.task.combine.progress.records":"10000","yarn.resourcemanager.epoch.range":"0","yarn.resourcemanager.am.max-attempts":"2","yarn.nodemanager.linux-container-executor.cgroups.hierarchy":"/hadoop-yarn","fs.AbstractFileSystem.wasbs.impl":"org.apache.hadoop.fs.azure.Wasbs","yarn.timeline-service.entity-group-fs-store.cache-store-class":"org.apache.hadoop.yarn.server.timeline.MemoryTimelineStore","fs.ftp.transfer.mode":"BLOCK_TRANSFER_MODE","ipc.server.log.slow.rpc":"false","yarn.resourcemanager.node-labels.provider.fetch-interval-ms":"1800000","yarn.router.webapp.https.address":"0.0.0.0:8091","yarn.nodemanager.webapp.cross-origin.enabled":"false","fs.wasb.impl":"org.apache.hadoop.fs.azure.NativeAzureFileSystem","yarn.resourcemanager.auto-update.containers":"false","yarn.app.mapreduce.am.job.committer.cancel-timeout":"60000","yarn.scheduler.configuration.zk-store.parent-path":"/confstore","yarn.nodemanager.default-container-executor.log-dirs.permissions":"710","yarn.app.attempt.diagnostics.limit.kc":"64","ftp.bytes-per-checksum":"512","yarn.nodemanager.resource.memory-mb":"-1","fs.AbstractFileSystem.abfs.impl":"org.apache.hadoop.fs.azurebfs.Abfs","yarn.timeline-service.writer.flush-interval-seconds":"60","fs.s3a.fast.upload.active.blocks":"4","hadoop.security.credential.clear-text-fallback":"true","yarn.nodemanager.collector-service.thread-count":"5","fs.azure.secure.mode":"false","mapreduce.jobhistory.joblist.cache.size":"20000","fs.ftp.host":"0.0.0.0","yarn.resourcemanager.fs.state-store.num-retries":"0","yarn.resourcemanager.nodemanager-connect-retries":"10","yarn.nodemanager.log-aggregation.num-log-files-per-app":"30","hadoop.security.kms.client.encrypted.key.cache.low-watermark":"0.3f","fs.s3a.committer.magic.enabled":"false","yarn.timeline-service.client.max-retries":"30","dfs.ha.fencing.ssh.connect-timeout":"30000","yarn.log-aggregation-enable":"false","yarn.system-metrics-publisher.enabled":"false","mapreduce.reduce.markreset.buffer.percent":"0.0","fs.AbstractFileSystem.viewfs.impl":"org.apache.hadoop.fs.viewfs.ViewFs","mapreduce.task.io.sort.factor":"10","yarn.nodemanager.amrmproxy.client.thread-count":"25","ha.failover-controller.new-active.rpc-timeout.ms":"60000","yarn.nodemanager.container-localizer.java.opts":"-Xmx256m","mapreduce.jobhistory.datestring.cache.size":"200000","mapreduce.job.acl-modify-job":" ","yarn.nodemanager.windows-container.memory-limit.enabled":"false","yarn.timeline-service.webapp.address":"${yarn.timeline-service.hostname}:8188","yarn.app.mapreduce.am.job.committer.commit-window":"10000","yarn.nodemanager.container-manager.thread-count":"20","yarn.minicluster.fixed.ports":"false","hadoop.tags.system":"YARN,HDFS,NAMENODE,DATANODE,REQUIRED,SECURITY,KERBEROS,PERFORMANCE,CLIENT\n ,SERVER,DEBUG,DEPRECATED,COMMON,OPTIONAL","yarn.cluster.max-application-priority":"0","yarn.timeline-service.ttl-enable":"true","mapreduce.jobhistory.recovery.store.fs.uri":"${hadoop.tmp.dir}/mapred/history/recoverystore","hadoop.caller.context.signature.max.size":"40","yarn.client.load.resource-types.from-server":"false","ha.zookeeper.session-timeout.ms":"10000","tfile.io.chunk.size":"1048576","fs.s3a.s3guard.ddb.table.capacity.write":"100","mapreduce.job.speculative.slowtaskthreshold":"1.0","io.serializations":"org.apache.hadoop.io.serializer.WritableSerialization, org.apache.hadoop.io.serializer.avro.AvroSpecificSerialization, org.apache.hadoop.io.serializer.avro.AvroReflectSerialization","hadoop.security.kms.client.failover.sleep.max.millis":"2000","hadoop.security.group.mapping.ldap.directory.search.timeout":"10000","yarn.scheduler.configuration.store.max-logs":"1000","yarn.nodemanager.node-attributes.provider.fetch-interval-ms":"600000","fs.swift.impl":"org.apache.hadoop.fs.swift.snative.SwiftNativeFileSystem","yarn.nodemanager.local-cache.max-files-per-directory":"8192","hadoop.http.cross-origin.enabled":"false","hadoop.zk.acl":"world:anyone:rwcda","mapreduce.map.sort.spill.percent":"0.80","yarn.timeline-service.entity-group-fs-store.scan-interval-seconds":"60","yarn.node-attribute.fs-store.impl.class":"org.apache.hadoop.yarn.server.resourcemanager.nodelabels.FileSystemNodeAttributeStore","fs.s3a.retry.interval":"500ms","yarn.timeline-service.client.best-effort":"false","yarn.resourcemanager.webapp.delegation-token-auth-filter.enabled":"*********(redacted)","hadoop.security.group.mapping.ldap.posix.attr.uid.name":"uidNumber","fs.AbstractFileSystem.swebhdfs.impl":"org.apache.hadoop.fs.SWebHdfs","yarn.nodemanager.elastic-memory-control.timeout-sec":"5","mapreduce.ifile.readahead":"true","yarn.timeline-service.leveldb-timeline-store.ttl-interval-ms":"300000","yarn.timeline-service.reader.webapp.address":"${yarn.timeline-service.webapp.address}","yarn.resourcemanager.placement-constraints.algorithm.pool-size":"1","yarn.timeline-service.hbase.coprocessor.jar.hdfs.location":"/hbase/coprocessor/hadoop-yarn-server-timelineservice.jar","hadoop.security.kms.client.encrypted.key.cache.num.refill.threads":"2","yarn.resourcemanager.scheduler.class":"org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.CapacityScheduler","yarn.app.mapreduce.am.command-opts":"-Xmx1024m","mapreduce.cluster.local.dir":"${hadoop.tmp.dir}/mapred/local","io.mapfile.bloom.error.rate":"0.005","fs.client.resolve.topology.enabled":"false","yarn.nodemanager.runtime.linux.allowed-runtimes":"default","yarn.sharedcache.store.class":"org.apache.hadoop.yarn.server.sharedcachemanager.store.InMemorySCMStore","ha.failover-controller.graceful-fence.rpc-timeout.ms":"5000","ftp.replication":"3","hadoop.security.uid.cache.secs":"14400","mapreduce.job.maxtaskfailures.per.tracker":"3","fs.s3a.metadatastore.impl":"org.apache.hadoop.fs.s3a.s3guard.NullMetadataStore","io.skip.checksum.errors":"false","yarn.app.mapreduce.client-am.ipc.max-retries-on-timeouts":"3","yarn.timeline-service.webapp.xfs-filter.xframe-options":"SAMEORIGIN","fs.s3a.connection.timeout":"200000","mapreduce.job.max.split.locations":"15","yarn.resourcemanager.nm-container-queuing.max-queue-length":"15","hadoop.registry.zk.session.timeout.ms":"60000","yarn.federation.cache-ttl.secs":"300","mapreduce.jvm.system-properties-to-log":"os.name,os.version,java.home,java.runtime.version,java.vendor,java.version,java.vm.name,java.class.path,java.io.tmpdir,user.dir,user.name","yarn.resourcemanager.opportunistic-container-allocation.nodes-used":"10","yarn.timeline-service.entity-group-fs-store.active-dir":"/tmp/entity-file-history/active","mapreduce.shuffle.transfer.buffer.size":"131072","yarn.timeline-service.client.retry-interval-ms":"1000","yarn.http.policy":"HTTP_ONLY","fs.s3a.socket.send.buffer":"8192","fs.AbstractFileSystem.abfss.impl":"org.apache.hadoop.fs.azurebfs.Abfss","yarn.sharedcache.uploader.server.address":"0.0.0.0:8046","yarn.resourcemanager.delegation-token.max-conf-size-bytes":"*********(redacted)","hadoop.http.authentication.token.validity":"*********(redacted)","mapreduce.shuffle.max.connections":"0","yarn.minicluster.yarn.nodemanager.resource.memory-mb":"4096","mapreduce.job.emit-timeline-data":"false","yarn.nodemanager.resource.system-reserved-memory-mb":"-1","hadoop.kerberos.min.seconds.before.relogin":"60","mapreduce.jobhistory.move.thread-count":"3","yarn.resourcemanager.admin.client.thread-count":"1","yarn.dispatcher.drain-events.timeout":"300000","fs.s3a.buffer.dir":"${hadoop.tmp.dir}/s3a","hadoop.ssl.enabled.protocols":"TLSv1,SSLv2Hello,TLSv1.1,TLSv1.2","mapreduce.jobhistory.admin.address":"0.0.0.0:10033","yarn.log-aggregation-status.time-out.ms":"600000","fs.s3a.assumed.role.sts.endpoint.region":"us-west-1","mapreduce.shuffle.port":"13562","yarn.resourcemanager.max-log-aggregation-diagnostics-in-memory":"10","yarn.nodemanager.health-checker.interval-ms":"600000","yarn.router.clientrm.interceptor-class.pipeline":"org.apache.hadoop.yarn.server.router.clientrm.DefaultClientRequestInterceptor","yarn.resourcemanager.zk-appid-node.split-index":"0","ftp.blocksize":"67108864","yarn.nodemanager.runtime.linux.sandbox-mode.local-dirs.permissions":"read","yarn.router.rmadmin.interceptor-class.pipeline":"org.apache.hadoop.yarn.server.router.rmadmin.DefaultRMAdminRequestInterceptor","yarn.nodemanager.log-container-debug-info.enabled":"true","yarn.client.max-cached-nodemanagers-proxies":"0","yarn.nodemanager.linux-container-executor.cgroups.delete-delay-ms":"20","yarn.nodemanager.delete.debug-delay-sec":"0","yarn.nodemanager.pmem-check-enabled":"true","yarn.nodemanager.disk-health-checker.max-disk-utilization-per-disk-percentage":"90.0","mapreduce.app-submission.cross-platform":"false","yarn.resourcemanager.work-preserving-recovery.scheduling-wait-ms":"10000","yarn.nodemanager.container-retry-minimum-interval-ms":"1000","hadoop.security.groups.cache.secs":"300","yarn.federation.enabled":"false","fs.azure.local.sas.key.mode":"false","ipc.maximum.data.length":"67108864","mapreduce.shuffle.max.threads":"0","yarn.router.pipeline.cache-max-size":"25","yarn.resourcemanager.nm-container-queuing.load-comparator":"QUEUE_LENGTH","hadoop.security.authorization":"false","mapreduce.job.complete.cancel.delegation.tokens":"*********(redacted)","fs.s3a.paging.maximum":"5000","nfs.exports.allowed.hosts":"* rw","yarn.nodemanager.amrmproxy.ha.enable":"false","mapreduce.jobhistory.http.policy":"HTTP_ONLY","yarn.sharedcache.store.in-memory.check-period-mins":"720","hadoop.security.group.mapping.ldap.ssl":"false","yarn.client.application-client-protocol.poll-interval-ms":"200","yarn.scheduler.configuration.leveldb-store.compaction-interval-secs":"86400","yarn.timeline-service.writer.class":"org.apache.hadoop.yarn.server.timelineservice.storage.HBaseTimelineWriterImpl","ha.zookeeper.parent-znode":"/hadoop-ha","yarn.nodemanager.log-aggregation.policy.class":"org.apache.hadoop.yarn.server.nodemanager.containermanager.logaggregation.AllContainerLogAggregationPolicy","mapreduce.reduce.shuffle.merge.percent":"0.66","hadoop.security.group.mapping.ldap.search.filter.group":"(objectClass=group)","yarn.resourcemanager.placement-constraints.scheduler.pool-size":"1","yarn.nodemanager.resourcemanager.minimum.version":"NONE","mapreduce.job.speculative.speculative-cap-running-tasks":"0.1","yarn.admin.acl":"*","yarn.nodemanager.recovery.supervised":"false","yarn.sharedcache.admin.thread-count":"1","yarn.resourcemanager.ha.automatic-failover.enabled":"true","mapreduce.reduce.skip.maxgroups":"0","mapreduce.reduce.shuffle.connect.timeout":"180000","yarn.resourcemanager.address":"${yarn.resourcemanager.hostname}:8032","ipc.client.ping":"true","mapreduce.task.local-fs.write-limit.bytes":"-1","fs.adl.oauth2.access.token.provider.type":"*********(redacted)","mapreduce.shuffle.ssl.file.buffer.size":"65536","yarn.resourcemanager.ha.automatic-failover.embedded":"true","yarn.nodemanager.resource-plugins.gpu.docker-plugin":"nvidia-docker-v1","hadoop.ssl.enabled":"false","fs.s3a.multipart.purge":"false","yarn.scheduler.configuration.store.class":"file","yarn.resourcemanager.nm-container-queuing.queue-limit-stdev":"1.0f","mapreduce.job.end-notification.max.attempts":"5","mapreduce.output.fileoutputformat.compress.codec":"org.apache.hadoop.io.compress.DefaultCodec","yarn.nodemanager.container-monitor.procfs-tree.smaps-based-rss.enabled":"false","ipc.client.bind.wildcard.addr":"false","yarn.resourcemanager.webapp.rest-csrf.enabled":"false","ha.health-monitor.connect-retry-interval.ms":"1000","yarn.nodemanager.keytab":"/etc/krb5.keytab","mapreduce.jobhistory.keytab":"/etc/security/keytab/jhs.service.keytab","fs.s3a.threads.max":"10","mapreduce.reduce.shuffle.input.buffer.percent":"0.70","yarn.nodemanager.runtime.linux.docker.allowed-container-networks":"host,none,bridge","yarn.nodemanager.node-labels.resync-interval-ms":"120000","hadoop.tmp.dir":"/home/hadoop-3.2.1/tmp/","mapreduce.job.maps":"2","mapreduce.jobhistory.webapp.rest-csrf.custom-header":"X-XSRF-Header","mapreduce.job.end-notification.max.retry.interval":"5000","yarn.log-aggregation.retain-check-interval-seconds":"-1","yarn.resourcemanager.resource-tracker.client.thread-count":"50","yarn.rm.system-metrics-publisher.emit-container-events":"false","yarn.timeline-service.leveldb-timeline-store.start-time-read-cache-size":"10000","yarn.resourcemanager.ha.automatic-failover.zk-base-path":"/yarn-leader-election","io.seqfile.local.dir":"${hadoop.tmp.dir}/io/local","fs.s3a.s3guard.ddb.throttle.retry.interval":"100ms","fs.AbstractFileSystem.wasb.impl":"org.apache.hadoop.fs.azure.Wasb","mapreduce.client.submit.file.replication":"10","mapreduce.jobhistory.minicluster.fixed.ports":"false","fs.s3a.multipart.threshold":"2147483647","yarn.resourcemanager.webapp.xfs-filter.xframe-options":"SAMEORIGIN","mapreduce.jobhistory.done-dir":"${yarn.app.mapreduce.am.staging-dir}/history/done","dfs.namenode.name.dir":"/data/hadoop/hdfs/name","ipc.client.idlethreshold":"4000","yarn.nodemanager.linux-container-executor.cgroups.strict-resource-usage":"false","mapreduce.reduce.input.buffer.percent":"0.0","yarn.nodemanager.runtime.linux.docker.userremapping-gid-threshold":"1","yarn.nodemanager.webapp.rest-csrf.enabled":"false","fs.ftp.host.port":"21","ipc.ping.interval":"60000","yarn.resourcemanager.history-writer.multi-threaded-dispatcher.pool-size":"10","yarn.resourcemanager.admin.address":"${yarn.resourcemanager.hostname}:8033","file.client-write-packet-size":"65536","ipc.client.kill.max":"10","mapreduce.reduce.speculative":"true","hadoop.security.key.default.bitlength":"128","mapreduce.job.reducer.unconditional-preempt.delay.sec":"300","yarn.nodemanager.disk-health-checker.interval-ms":"120000","yarn.nodemanager.log.deletion-threads-count":"4","yarn.webapp.filter-entity-list-by-user":"false","ipc.client.connection.maxidletime":"10000","mapreduce.task.io.sort.mb":"100","yarn.nodemanager.localizer.client.thread-count":"5","io.erasurecode.codec.rs.rawcoders":"rs_native,rs_java","io.erasurecode.codec.rs-legacy.rawcoders":"rs-legacy_java","yarn.sharedcache.admin.address":"0.0.0.0:8047","yarn.resourcemanager.placement-constraints.algorithm.iterator":"SERIAL","yarn.nodemanager.localizer.cache.cleanup.interval-ms":"600000","hadoop.security.crypto.codec.classes.aes.ctr.nopadding":"org.apache.hadoop.crypto.OpensslAesCtrCryptoCodec, org.apache.hadoop.crypto.JceAesCtrCryptoCodec","mapreduce.job.cache.limit.max-resources-mb":"0","fs.s3a.connection.ssl.enabled":"true","yarn.nodemanager.process-kill-wait.ms":"5000","mapreduce.job.hdfs-servers":"${fs.defaultFS}","hadoop.workaround.non.threadsafe.getpwuid":"true","fs.df.interval":"60000","fs.s3a.multiobjectdelete.enable":"true","yarn.sharedcache.cleaner.resource-sleep-ms":"0","yarn.nodemanager.disk-health-checker.min-healthy-disks":"0.25","hadoop.shell.missing.defaultFs.warning":"false","io.file.buffer.size":"65536","hadoop.security.group.mapping.ldap.search.attr.member":"member","hadoop.security.random.device.file.path":"/dev/urandom","hadoop.security.sensitive-config-keys":"*********(redacted)","fs.s3a.s3guard.ddb.max.retries":"9","hadoop.rpc.socket.factory.class.default":"org.apache.hadoop.net.StandardSocketFactory","dfs.permissions.enabled":"false","yarn.intermediate-data-encryption.enable":"false","yarn.resourcemanager.connect.retry-interval.ms":"30000","yarn.nodemanager.container.stderr.pattern":"{*stderr*,*STDERR*}","yarn.scheduler.minimum-allocation-mb":"1024","yarn.app.mapreduce.am.staging-dir":"/tmp/hadoop-yarn/staging","mapreduce.reduce.shuffle.read.timeout":"180000","hadoop.http.cross-origin.max-age":"1800","io.erasurecode.codec.xor.rawcoders":"xor_native,xor_java","fs.s3a.connection.establish.timeout":"5000","mapreduce.job.running.map.limit":"0","yarn.minicluster.control-resource-monitoring":"false","hadoop.ssl.require.client.cert":"false","hadoop.kerberos.kinit.command":"kinit","yarn.federation.state-store.class":"org.apache.hadoop.yarn.server.federation.store.impl.MemoryFederationStateStore","mapreduce.reduce.log.level":"INFO","hadoop.security.dns.log-slow-lookups.threshold.ms":"1000","mapreduce.job.ubertask.enable":"false","adl.http.timeout":"-1","yarn.resourcemanager.placement-constraints.retry-attempts":"3","hadoop.caller.context.enabled":"false","yarn.nodemanager.vmem-pmem-ratio":"2.1","hadoop.rpc.protection":"authentication","ha.health-monitor.rpc-timeout.ms":"45000","yarn.nodemanager.remote-app-log-dir":"/tmp/logs","hadoop.zk.timeout-ms":"10000","fs.s3a.s3guard.cli.prune.age":"86400000","yarn.nodemanager.resource.pcores-vcores-multiplier":"1.0","yarn.nodemanager.runtime.linux.sandbox-mode":"disabled","yarn.app.mapreduce.am.containerlauncher.threadpool-initial-size":"10","fs.s3a.committer.threads":"8","hadoop.zk.retry-interval-ms":"1000","hadoop.security.crypto.buffer.size":"8192","yarn.nodemanager.node-labels.provider.fetch-interval-ms":"600000","mapreduce.jobhistory.recovery.store.leveldb.path":"${hadoop.tmp.dir}/mapred/history/recoverystore","yarn.client.failover-retries-on-socket-timeouts":"0","yarn.nodemanager.resource.memory.enabled":"false","fs.azure.authorization.caching.enable":"true","hadoop.security.instrumentation.requires.admin":"false","yarn.nodemanager.delete.thread-count":"4","mapreduce.job.finish-when-all-reducers-done":"true","hadoop.registry.jaas.context":"Client","yarn.timeline-service.leveldb-timeline-store.path":"${hadoop.tmp.dir}/yarn/timeline","io.map.index.interval":"128","yarn.resourcemanager.nm-container-queuing.max-queue-wait-time-ms":"100","fs.abfs.impl":"org.apache.hadoop.fs.azurebfs.AzureBlobFileSystem","mapreduce.job.counters.max":"120","mapreduce.jobhistory.webapp.rest-csrf.enabled":"false","yarn.timeline-service.store-class":"org.apache.hadoop.yarn.server.timeline.LeveldbTimelineStore","mapreduce.jobhistory.move.interval-ms":"180000","yarn.nodemanager.localizer.fetch.thread-count":"4","yarn.resourcemanager.scheduler.client.thread-count":"50","hadoop.ssl.hostname.verifier":"DEFAULT","yarn.timeline-service.leveldb-state-store.path":"${hadoop.tmp.dir}/yarn/timeline","mapreduce.job.classloader":"false","mapreduce.task.profile.map.params":"${mapreduce.task.profile.params}","ipc.client.connect.timeout":"20000","hadoop.security.auth_to_local.mechanism":"hadoop","yarn.timeline-service.app-collector.linger-period.ms":"60000","yarn.nm.liveness-monitor.expiry-interval-ms":"600000","yarn.resourcemanager.reservation-system.planfollower.time-step":"1000","yarn.nodemanager.runtime.linux.docker.enable-userremapping.allowed":"true","yarn.webapp.api-service.enable":"false","yarn.nodemanager.recovery.enabled":"false","mapreduce.job.end-notification.retry.interval":"1000","fs.du.interval":"600000","fs.ftp.impl":"org.apache.hadoop.fs.ftp.FTPFileSystem","yarn.nodemanager.container.stderr.tail.bytes":"4096","hadoop.security.group.mapping.ldap.read.timeout.ms":"60000","hadoop.security.groups.cache.warn.after.ms":"5000","file.bytes-per-checksum":"512","mapreduce.outputcommitter.factory.scheme.s3a":"org.apache.hadoop.fs.s3a.commit.S3ACommitterFactory","hadoop.security.groups.cache.background.reload":"false","yarn.nodemanager.container-monitor.enabled":"true","yarn.nodemanager.elastic-memory-control.enabled":"false","net.topology.script.number.args":"100","mapreduce.task.merge.progress.records":"10000","yarn.nodemanager.localizer.address":"${yarn.nodemanager.hostname}:8040","yarn.timeline-service.keytab":"/etc/krb5.keytab","mapreduce.reduce.shuffle.fetch.retry.timeout-ms":"30000","yarn.resourcemanager.rm.container-allocation.expiry-interval-ms":"600000","mapreduce.fileoutputcommitter.algorithm.version":"2","yarn.resourcemanager.work-preserving-recovery.enabled":"true","mapreduce.map.skip.maxrecords":"0","yarn.sharedcache.root-dir":"/sharedcache","fs.s3a.retry.throttle.limit":"${fs.s3a.attempts.maximum}","hadoop.http.authentication.type":"simple","mapreduce.job.cache.limit.max-resources":"0","mapreduce.task.userlog.limit.kb":"0","yarn.resourcemanager.scheduler.monitor.enable":"false","ipc.client.connect.max.retries":"10","hadoop.registry.zk.retry.times":"5","dfs.namenode.http-address":"master:50070","yarn.nodemanager.resource-monitor.interval-ms":"3000","yarn.nodemanager.resource-plugins.gpu.allowed-gpu-devices":"auto","mapreduce.job.sharedcache.mode":"disabled","yarn.nodemanager.webapp.rest-csrf.custom-header":"X-XSRF-Header","mapreduce.shuffle.listen.queue.size":"128","yarn.scheduler.configuration.mutation.acl-policy.class":"org.apache.hadoop.yarn.server.resourcemanager.scheduler.DefaultConfigurationMutationACLPolicy","mapreduce.map.cpu.vcores":"1","yarn.log-aggregation.file-formats":"TFile","yarn.timeline-service.client.fd-retain-secs":"300","hadoop.user.group.static.mapping.overrides":"dr.who=;","fs.azure.sas.expiry.period":"90d","mapreduce.jobhistory.recovery.store.class":"org.apache.hadoop.mapreduce.v2.hs.HistoryServerFileSystemStateStoreService","yarn.resourcemanager.fail-fast":"${yarn.fail-fast}","yarn.resourcemanager.proxy-user-privileges.enabled":"false","yarn.router.webapp.interceptor-class.pipeline":"org.apache.hadoop.yarn.server.router.webapp.DefaultRequestInterceptorREST","yarn.nodemanager.resource.memory.cgroups.soft-limit-percentage":"90.0","mapreduce.job.reducer.preempt.delay.sec":"0","hadoop.util.hash.type":"murmur","yarn.nodemanager.disk-validator":"basic","yarn.app.mapreduce.client.job.max-retries":"3","mapreduce.reduce.shuffle.retry-delay.max.ms":"60000","hadoop.security.group.mapping.ldap.connection.timeout.ms":"60000","mapreduce.task.profile.params":"-agentlib:hprof=cpu=samples,heap=sites,force=n,thread=y,verbose=n,file=%s","yarn.app.mapreduce.shuffle.log.backups":"0","yarn.nodemanager.container-diagnostics-maximum-size":"10000","hadoop.registry.zk.retry.interval.ms":"1000","yarn.nodemanager.linux-container-executor.cgroups.delete-timeout-ms":"1000","fs.AbstractFileSystem.file.impl":"org.apache.hadoop.fs.local.LocalFs","yarn.nodemanager.log-aggregation.roll-monitoring-interval-seconds":"-1","mapreduce.jobhistory.cleaner.interval-ms":"86400000","hadoop.registry.zk.quorum":"localhost:2181","mapreduce.output.fileoutputformat.compress":"false","yarn.resourcemanager.am-rm-tokens.master-key-rolling-interval-secs":"*********(redacted)","fs.s3a.assumed.role.session.duration":"30m","hadoop.security.group.mapping.ldap.conversion.rule":"none","hadoop.ssl.server.conf":"ssl-server.xml","fs.s3a.retry.throttle.interval":"1000ms","seq.io.sort.factor":"100","yarn.sharedcache.cleaner.initial-delay-mins":"10","mapreduce.client.completion.pollinterval":"5000","hadoop.ssl.keystores.factory.class":"org.apache.hadoop.security.ssl.FileBasedKeyStoresFactory","yarn.app.mapreduce.am.resource.cpu-vcores":"1","yarn.timeline-service.enabled":"false","yarn.nodemanager.runtime.linux.docker.capabilities":"CHOWN,DAC_OVERRIDE,FSETID,FOWNER,MKNOD,NET_RAW,SETGID,SETUID,SETFCAP,SETPCAP,NET_BIND_SERVICE,SYS_CHROOT,KILL,AUDIT_WRITE","yarn.acl.enable":"false","yarn.timeline-service.entity-group-fs-store.done-dir":"/tmp/entity-file-history/done/","mapreduce.task.profile":"false","yarn.resourcemanager.fs.state-store.uri":"${hadoop.tmp.dir}/yarn/system/rmstore","mapreduce.jobhistory.always-scan-user-dir":"false","yarn.nodemanager.opportunistic-containers-use-pause-for-preemption":"false","yarn.nodemanager.linux-container-executor.nonsecure-mode.local-user":"nobody","yarn.timeline-service.reader.class":"org.apache.hadoop.yarn.server.timelineservice.storage.HBaseTimelineReaderImpl","yarn.resourcemanager.configuration.provider-class":"org.apache.hadoop.yarn.LocalConfigurationProvider","yarn.nodemanager.runtime.linux.docker.userremapping-uid-threshold":"1","yarn.resourcemanager.configuration.file-system-based-store":"/yarn/conf","mapreduce.job.cache.limit.max-single-resource-mb":"0","yarn.nodemanager.runtime.linux.docker.stop.grace-period":"10","yarn.resourcemanager.resource-profiles.source-file":"resource-profiles.json","yarn.nodemanager.resource.percentage-physical-cpu-limit":"100","mapreduce.jobhistory.client.thread-count":"10","tfile.fs.input.buffer.size":"262144","mapreduce.client.progressmonitor.pollinterval":"1000","yarn.nodemanager.log-dirs":"${yarn.log.dir}/userlogs","fs.automatic.close":"true","yarn.nodemanager.hostname":"0.0.0.0","yarn.nodemanager.resource.memory.cgroups.swappiness":"0","ftp.stream-buffer-size":"4096","yarn.fail-fast":"false","yarn.timeline-service.app-aggregation-interval-secs":"15","hadoop.security.group.mapping.ldap.search.filter.user":"(&(objectClass=user)(sAMAccountName={0}))","yarn.nodemanager.container-localizer.log.level":"INFO","yarn.timeline-service.address":"${yarn.timeline-service.hostname}:10200","mapreduce.job.ubertask.maxmaps":"9","fs.s3a.threads.keepalivetime":"60","mapreduce.jobhistory.webapp.rest-csrf.methods-to-ignore":"GET,OPTIONS,HEAD","mapreduce.task.files.preserve.failedtasks":"false","yarn.app.mapreduce.client.job.retry-interval":"2000","ha.failover-controller.graceful-fence.connection.retries":"1","yarn.resourcemanager.delegation.token.max-lifetime":"*********(redacted)","yarn.timeline-service.client.drain-entities.timeout.ms":"2000","yarn.nodemanager.resource-plugins.fpga.vendor-plugin.class":"org.apache.hadoop.yarn.server.nodemanager.containermanager.resourceplugin.fpga.IntelFpgaOpenclPlugin","yarn.timeline-service.entity-group-fs-store.summary-store":"org.apache.hadoop.yarn.server.timeline.LeveldbTimelineStore","mapreduce.reduce.cpu.vcores":"1","mapreduce.job.encrypted-intermediate-data.buffer.kb":"128","fs.client.resolve.remote.symlinks":"true","yarn.nodemanager.webapp.https.address":"0.0.0.0:8044","hadoop.http.cross-origin.allowed-origins":"*","mapreduce.job.encrypted-intermediate-data":"false","yarn.timeline-service.entity-group-fs-store.retain-seconds":"604800","yarn.resourcemanager.metrics.runtime.buckets":"60,300,1440","yarn.timeline-service.generic-application-history.max-applications":"10000","yarn.nodemanager.local-dirs":"${hadoop.tmp.dir}/nm-local-dir","mapreduce.shuffle.connection-keep-alive.enable":"false","yarn.node-labels.configuration-type":"centralized","fs.s3a.path.style.access":"false","yarn.nodemanager.aux-services.mapreduce_shuffle.class":"org.apache.hadoop.mapred.ShuffleHandler","yarn.sharedcache.store.in-memory.staleness-period-mins":"10080","fs.adl.impl":"org.apache.hadoop.fs.adl.AdlFileSystem","yarn.resourcemanager.nodemanager.minimum.version":"NONE","mapreduce.jobhistory.webapp.xfs-filter.xframe-options":"SAMEORIGIN","yarn.app.mapreduce.am.staging-dir.erasurecoding.enabled":"false","net.topology.impl":"org.apache.hadoop.net.NetworkTopology","io.map.index.skip":"0","yarn.timeline-service.reader.webapp.https.address":"${yarn.timeline-service.webapp.https.address}","fs.ftp.data.connection.mode":"ACTIVE_LOCAL_DATA_CONNECTION_MODE","mapreduce.job.local-fs.single-disk-limit.check.kill-limit-exceed":"true","yarn.scheduler.maximum-allocation-vcores":"4","hadoop.http.cross-origin.allowed-headers":"X-Requested-With,Content-Type,Accept,Origin","yarn.nodemanager.log-aggregation.compression-type":"none","yarn.timeline-service.version":"1.0f","yarn.ipc.rpc.class":"org.apache.hadoop.yarn.ipc.HadoopYarnProtoRPC","mapreduce.reduce.maxattempts":"4","hadoop.security.dns.log-slow-lookups.enabled":"false","mapreduce.job.committer.setup.cleanup.needed":"true","mapreduce.job.running.reduce.limit":"0","ipc.maximum.response.length":"134217728","yarn.resourcemanager.webapp.rest-csrf.methods-to-ignore":"GET,OPTIONS,HEAD","mapreduce.job.token.tracking.ids.enabled":"*********(redacted)","hadoop.caller.context.max.size":"128","yarn.nodemanager.runtime.linux.docker.host-pid-namespace.allowed":"false","yarn.nodemanager.runtime.linux.docker.delayed-removal.allowed":"false","hadoop.registry.system.acls":"sasl:yarn@, sasl:mapred@, sasl:hdfs@","yarn.nodemanager.recovery.dir":"${hadoop.tmp.dir}/yarn-nm-recovery","fs.s3a.fast.upload.buffer":"disk","mapreduce.jobhistory.intermediate-done-dir":"${yarn.app.mapreduce.am.staging-dir}/history/done_intermediate","yarn.app.mapreduce.shuffle.log.separate":"true","fs.s3a.max.total.tasks":"5","fs.s3a.readahead.range":"64K","hadoop.http.authentication.simple.anonymous.allowed":"true","fs.s3a.attempts.maximum":"20","hadoop.registry.zk.connection.timeout.ms":"15000","yarn.resourcemanager.delegation-token-renewer.thread-count":"*********(redacted)","yarn.nodemanager.health-checker.script.timeout-ms":"1200000","yarn.timeline-service.leveldb-timeline-store.start-time-write-cache-size":"10000","yarn.resourcemanager.resource-profiles.enabled":"false","yarn.timeline-service.hbase-schema.prefix":"prod.","fs.azure.authorization":"false","mapreduce.map.log.level":"INFO","yarn.resourcemanager.decommissioning-nodes-watcher.poll-interval-secs":"20","mapreduce.output.fileoutputformat.compress.type":"RECORD","yarn.resourcemanager.leveldb-state-store.path":"${hadoop.tmp.dir}/yarn/system/rmstore","yarn.timeline-service.webapp.rest-csrf.custom-header":"X-XSRF-Header","mapreduce.ifile.readahead.bytes":"4194304","yarn.sharedcache.app-checker.class":"org.apache.hadoop.yarn.server.sharedcachemanager.RemoteAppChecker","yarn.nodemanager.linux-container-executor.nonsecure-mode.limit-users":"true","yarn.nodemanager.resource.detect-hardware-capabilities":"false","mapreduce.cluster.acls.enabled":"false","mapreduce.job.speculative.retry-after-no-speculate":"1000","hadoop.security.group.mapping.ldap.search.group.hierarchy.levels":"0","yarn.resourcemanager.fs.state-store.retry-interval-ms":"1000","file.stream-buffer-size":"4096","yarn.resourcemanager.application-timeouts.monitor.interval-ms":"3000","mapreduce.map.output.compress.codec":"org.apache.hadoop.io.compress.DefaultCodec","mapreduce.map.speculative":"true","mapreduce.job.speculative.retry-after-speculate":"15000","yarn.nodemanager.linux-container-executor.cgroups.mount":"false","yarn.app.mapreduce.am.container.log.backups":"0","yarn.app.mapreduce.am.log.level":"INFO","mapreduce.job.reduce.slowstart.completedmaps":"0.05","yarn.timeline-service.http-authentication.type":"simple","hadoop.security.group.mapping.ldap.search.attr.group.name":"cn","yarn.nodemanager.resource-plugins.fpga.allowed-fpga-devices":"0,1","yarn.timeline-service.client.internal-timers-ttl-secs":"420","hadoop.http.logs.enabled":"true","fs.s3a.block.size":"32M","yarn.sharedcache.client-server.address":"0.0.0.0:8045","yarn.nodemanager.logaggregation.threadpool-size-max":"100","yarn.resourcemanager.hostname":"master","yarn.resourcemanager.delegation.key.update-interval":"86400000","mapreduce.reduce.shuffle.fetch.retry.enabled":"${yarn.nodemanager.recovery.enabled}","mapreduce.map.memory.mb":"-1","mapreduce.task.skip.start.attempts":"2","fs.AbstractFileSystem.hdfs.impl":"org.apache.hadoop.fs.Hdfs","yarn.nodemanager.disk-health-checker.enable":"true","ipc.client.tcpnodelay":"true","ipc.client.rpc-timeout.ms":"0","yarn.nodemanager.webapp.rest-csrf.methods-to-ignore":"GET,OPTIONS,HEAD","ipc.client.low-latency":"false","mapreduce.input.lineinputformat.linespermap":"1","yarn.router.interceptor.user.threadpool-size":"5","ipc.client.connect.max.retries.on.timeouts":"45","yarn.timeline-service.leveldb-timeline-store.read-cache-size":"104857600","fs.AbstractFileSystem.har.impl":"org.apache.hadoop.fs.HarFs","mapreduce.job.split.metainfo.maxsize":"10000000","yarn.am.liveness-monitor.expiry-interval-ms":"600000","yarn.resourcemanager.container-tokens.master-key-rolling-interval-secs":"*********(redacted)","yarn.timeline-service.entity-group-fs-store.app-cache-size":"10","fs.s3a.socket.recv.buffer":"8192","yarn.resourcemanager.resource-tracker.address":"${yarn.resourcemanager.hostname}:8031","yarn.nodemanager.node-labels.provider.fetch-timeout-ms":"1200000","mapreduce.job.heap.memory-mb.ratio":"0.8","yarn.resourcemanager.leveldb-state-store.compaction-interval-secs":"3600","yarn.resourcemanager.webapp.rest-csrf.custom-header":"X-XSRF-Header","yarn.scheduler.configuration.fs.path":"file://${hadoop.tmp.dir}/yarn/system/schedconf","mapreduce.client.output.filter":"FAILED","hadoop.http.filter.initializers":"org.apache.hadoop.http.lib.StaticUserWebFilter","mapreduce.reduce.memory.mb":"-1","yarn.timeline-service.hostname":"0.0.0.0","file.replication":"1","yarn.nodemanager.container-metrics.unregister-delay-ms":"10000","yarn.nodemanager.container-metrics.period-ms":"-1","mapreduce.fileoutputcommitter.task.cleanup.enabled":"false","yarn.nodemanager.log.retain-seconds":"10800","yarn.timeline-service.entity-group-fs-store.cleaner-interval-seconds":"3600","yarn.resourcemanager.keytab":"/etc/krb5.keytab","hadoop.security.group.mapping.providers.combined":"true","mapreduce.reduce.merge.inmem.threshold":"1000","yarn.timeline-service.recovery.enabled":"false","fs.azure.saskey.usecontainersaskeyforallaccess":"true","yarn.sharedcache.nm.uploader.thread-count":"20","yarn.resourcemanager.nodemanager-graceful-decommission-timeout-secs":"3600","mapreduce.shuffle.ssl.enabled":"false","yarn.timeline-service.hbase.coprocessor.app-final-value-retention-milliseconds":"259200000","fs.s3a.committer.staging.abort.pending.uploads":"true","yarn.nodemanager.opportunistic-containers-max-queue-length":"0","yarn.resourcemanager.state-store.max-completed-applications":"${yarn.resourcemanager.max-completed-applications}","mapreduce.job.speculative.minimum-allowed-tasks":"10","yarn.log-aggregation.retain-seconds":"-1","yarn.nodemanager.disk-health-checker.min-free-space-per-disk-mb":"0","mapreduce.jobhistory.max-age-ms":"604800000","hadoop.http.cross-origin.allowed-methods":"GET,POST,HEAD","yarn.resourcemanager.opportunistic-container-allocation.enabled":"false","mapreduce.jobhistory.webapp.address":"0.0.0.0:19888","hadoop.system.tags":"YARN,HDFS,NAMENODE,DATANODE,REQUIRED,SECURITY,KERBEROS,PERFORMANCE,CLIENT\n ,SERVER,DEBUG,DEPRECATED,COMMON,OPTIONAL","yarn.log-aggregation.file-controller.TFile.class":"org.apache.hadoop.yarn.logaggregation.filecontroller.tfile.LogAggregationTFileController","yarn.client.nodemanager-connect.max-wait-ms":"180000","yarn.resourcemanager.webapp.address":"${yarn.resourcemanager.hostname}:8088","mapreduce.jobhistory.recovery.enable":"false","mapreduce.reduce.shuffle.parallelcopies":"5","fs.AbstractFileSystem.webhdfs.impl":"org.apache.hadoop.fs.WebHdfs","fs.trash.interval":"0","yarn.app.mapreduce.client.max-retries":"3","hadoop.security.authentication":"simple","mapreduce.task.profile.reduce.params":"${mapreduce.task.profile.params}","yarn.app.mapreduce.am.resource.mb":"1536","mapreduce.input.fileinputformat.list-status.num-threads":"1","yarn.nodemanager.container-executor.class":"org.apache.hadoop.yarn.server.nodemanager.DefaultContainerExecutor","io.mapfile.bloom.size":"1048576","yarn.timeline-service.ttl-ms":"604800000","yarn.resourcemanager.nm-container-queuing.min-queue-length":"5","yarn.nodemanager.resource.cpu-vcores":"-1","mapreduce.job.reduces":"1","fs.s3a.multipart.size":"100M","yarn.scheduler.minimum-allocation-vcores":"1","mapreduce.job.speculative.speculative-cap-total-tasks":"0.01","hadoop.ssl.client.conf":"ssl-client.xml","mapreduce.job.queuename":"default","mapreduce.job.encrypted-intermediate-data-key-size-bits":"128","fs.s3a.metadatastore.authoritative":"false","yarn.nodemanager.webapp.xfs-filter.xframe-options":"SAMEORIGIN","ha.health-monitor.sleep-after-disconnect.ms":"1000","yarn.app.mapreduce.shuffle.log.limit.kb":"0","hadoop.security.group.mapping":"org.apache.hadoop.security.JniBasedUnixGroupsMappingWithFallback","yarn.client.application-client-protocol.poll-timeout-ms":"-1","mapreduce.jobhistory.jhist.format":"binary","yarn.resourcemanager.ha.enabled":"false","hadoop.http.staticuser.user":"dr.who","mapreduce.task.exit.timeout.check-interval-ms":"20000","mapreduce.jobhistory.intermediate-user-done-dir.permissions":"770","mapreduce.task.exit.timeout":"60000","yarn.nodemanager.linux-container-executor.resources-handler.class":"org.apache.hadoop.yarn.server.nodemanager.util.DefaultLCEResourcesHandler","mapreduce.reduce.shuffle.memory.limit.percent":"0.25","yarn.resourcemanager.reservation-system.enable":"false","mapreduce.map.output.compress":"false","ha.zookeeper.acl":"world:anyone:rwcda","ipc.server.max.connections":"0","yarn.nodemanager.aux-services":"mapreduce_shuffle","yarn.nodemanager.runtime.linux.docker.default-container-network":"host","yarn.router.webapp.address":"0.0.0.0:8089","yarn.scheduler.maximum-allocation-mb":"8192","yarn.resourcemanager.scheduler.monitor.policies":"org.apache.hadoop.yarn.server.resourcemanager.monitor.capacity.ProportionalCapacityPreemptionPolicy","yarn.sharedcache.cleaner.period-mins":"1440","yarn.nodemanager.resource-plugins.gpu.docker-plugin.nvidia-docker-v1.endpoint":"http://localhost:3476/v1.0/docker/cli","yarn.app.mapreduce.am.container.log.limit.kb":"0","ipc.client.connect.retry.interval":"1000","yarn.timeline-service.http-cross-origin.enabled":"false","fs.wasbs.impl":"org.apache.hadoop.fs.azure.NativeAzureFileSystem$Secure","yarn.federation.subcluster-resolver.class":"org.apache.hadoop.yarn.server.federation.resolver.DefaultSubClusterResolverImpl","yarn.resourcemanager.zk-state-store.parent-path":"/rmstore","mapreduce.jobhistory.cleaner.enable":"true","yarn.timeline-service.client.fd-flush-interval-secs":"10","hadoop.security.kms.client.encrypted.key.cache.expiry":"43200000","yarn.client.nodemanager-client-async.thread-pool-max-size":"500","mapreduce.map.maxattempts":"4","yarn.resourcemanager.nm-container-queuing.sorting-nodes-interval-ms":"1000","fs.s3a.committer.staging.tmp.path":"tmp/staging","yarn.nodemanager.sleep-delay-before-sigkill.ms":"250","yarn.resourcemanager.nm-container-queuing.min-queue-wait-time-ms":"10","mapreduce.job.end-notification.retry.attempts":"0","yarn.nodemanager.resource.count-logical-processors-as-cores":"false","hadoop.registry.zk.root":"/registry","adl.feature.ownerandgroup.enableupn":"false","yarn.resourcemanager.zk-max-znode-size.bytes":"1048576","mapreduce.job.reduce.shuffle.consumer.plugin.class":"org.apache.hadoop.mapreduce.task.reduce.Shuffle","yarn.resourcemanager.delayed.delegation-token.removal-interval-ms":"*********(redacted)","yarn.nodemanager.localizer.cache.target-size-mb":"10240","fs.s3a.committer.staging.conflict-mode":"fail","mapreduce.client.libjars.wildcard":"true","fs.s3a.committer.staging.unique-filenames":"true","yarn.nodemanager.node-attributes.provider.fetch-timeout-ms":"1200000","fs.s3a.list.version":"2","ftp.client-write-packet-size":"65536","fs.AbstractFileSystem.adl.impl":"org.apache.hadoop.fs.adl.Adl","hadoop.security.key.default.cipher":"AES/CTR/NoPadding","yarn.client.failover-retries":"0","fs.s3a.multipart.purge.age":"86400","mapreduce.job.local-fs.single-disk-limit.check.interval-ms":"5000","net.topology.node.switch.mapping.impl":"org.apache.hadoop.net.ScriptBasedMapping","yarn.nodemanager.amrmproxy.address":"0.0.0.0:8049","ipc.server.listen.queue.size":"128","map.sort.class":"org.apache.hadoop.util.QuickSort","fs.viewfs.rename.strategy":"SAME_MOUNTPOINT","hadoop.security.kms.client.authentication.retry-count":"1","fs.permissions.umask-mode":"022","fs.s3a.assumed.role.credentials.provider":"org.apache.hadoop.fs.s3a.SimpleAWSCredentialsProvider","yarn.nodemanager.vmem-check-enabled":"true","yarn.nodemanager.numa-awareness.enabled":"false","yarn.nodemanager.recovery.compaction-interval-secs":"3600","yarn.app.mapreduce.client-am.ipc.max-retries":"3","yarn.federation.registry.base-dir":"yarnfederation/","mapreduce.job.max.map":"-1","mapreduce.job.local-fs.single-disk-limit.bytes":"-1","mapreduce.job.ubertask.maxreduces":"1","hadoop.security.kms.client.encrypted.key.cache.size":"500","hadoop.security.java.secure.random.algorithm":"SHA1PRNG","ha.failover-controller.cli-check.rpc-timeout.ms":"20000","mapreduce.jobhistory.jobname.limit":"50","mapreduce.application.classpath":"$HADOOP_HOME/share/hadoop/mapreduce/*:$HADOOP_HOME/share/hadoop/mapreduce/lib/*","yarn.client.nodemanager-connect.retry-interval-ms":"10000","yarn.timeline-service.state-store-class":"org.apache.hadoop.yarn.server.timeline.recovery.LeveldbTimelineStateStore","yarn.nodemanager.env-whitelist":"JAVA_HOME,HADOOP_COMMON_HOME,HADOOP_HDFS_HOME,HADOOP_CONF_DIR,CLASSPATH_PREPEND_DISTCACHE,HADOOP_YARN_HOME,HADOOP_HOME","yarn.sharedcache.nested-level":"3","yarn.timeline-service.webapp.rest-csrf.methods-to-ignore":"GET,OPTIONS,HEAD","fs.azure.user.agent.prefix":"unknown","yarn.resourcemanager.zk-delegation-token-node.split-index":"*********(redacted)","yarn.nodemanager.numa-awareness.read-topology":"false","yarn.nodemanager.webapp.address":"${yarn.nodemanager.hostname}:8042","rpc.metrics.quantile.enable":"false","yarn.registry.class":"org.apache.hadoop.registry.client.impl.FSRegistryOperationsService","mapreduce.jobhistory.admin.acl":"*","yarn.resourcemanager.system-metrics-publisher.dispatcher.pool-size":"10","yarn.scheduler.queue-placement-rules":"user-group","hadoop.http.authentication.kerberos.keytab":"${user.home}/hadoop.keytab","yarn.resourcemanager.recovery.enabled":"false","yarn.timeline-service.webapp.rest-csrf.enabled":"false"},"System Properties":{"java.io.tmpdir":"/tmp","line.separator":"\n","path.separator":":","sun.management.compiler":"HotSpot 64-Bit Tiered Compilers","SPARK_SUBMIT":"true","sun.cpu.endian":"little","java.specification.version":"1.8","java.vm.specification.name":"Java Virtual Machine Specification","java.vendor":"Oracle Corporation","java.vm.specification.version":"1.8","user.home":"/root","file.encoding.pkg":"sun.io","sun.nio.ch.bugLevel":"","sun.arch.data.model":"64","sun.boot.library.path":"/home/java/jre/lib/amd64","user.dir":"/home/spark-3.0.1","java.library.path":"/usr/java/packages/lib/amd64:/usr/lib64:/lib64:/lib:/usr/lib","sun.cpu.isalist":"","os.arch":"amd64","java.vm.version":"25.162-b12","jetty.git.hash":"e1bc35120a6617ee3df052294e433f3a25ce7097","java.endorsed.dirs":"/home/java/jre/lib/endorsed","java.runtime.version":"1.8.0_162-b12","java.vm.info":"mixed mode","java.ext.dirs":"/home/java/jre/lib/ext:/usr/java/packages/lib/ext","java.runtime.name":"Java(TM) SE Runtime Environment","file.separator":"/","java.class.version":"52.0","java.specification.name":"Java Platform API Specification","sun.boot.class.path":"/home/java/jre/lib/resources.jar:/home/java/jre/lib/rt.jar:/home/java/jre/lib/sunrsasign.jar:/home/java/jre/lib/jsse.jar:/home/java/jre/lib/jce.jar:/home/java/jre/lib/charsets.jar:/home/java/jre/lib/jfr.jar:/home/java/jre/classes","file.encoding":"UTF-8","user.timezone":"Asia/Shanghai","java.specification.vendor":"Oracle Corporation","sun.java.launcher":"SUN_STANDARD","os.version":"3.10.0-1062.el7.x86_64","sun.os.patch.level":"unknown","java.vm.specification.vendor":"Oracle Corporation","user.country":"CN","sun.jnu.encoding":"UTF-8","user.language":"zh","java.vendor.url":"http://java.oracle.com/","java.awt.printerjob":"sun.print.PSPrinterJob","java.awt.graphicsenv":"sun.awt.X11GraphicsEnvironment","awt.toolkit":"sun.awt.X11.XToolkit","os.name":"Linux","java.vm.vendor":"Oracle Corporation","java.vendor.url.bug":"http://bugreport.sun.com/bugreport/","user.name":"root","java.vm.name":"Java HotSpot(TM) 64-Bit Server VM","sun.java.command":"org.apache.spark.deploy.SparkSubmit --master spark://master:7077 --conf spark.driver.memory=512m --class org.apache.spark.examples.SparkPi --num-executors 1 --executor-memory 512m --executor-cores 1 /home/spark-3.0.1/examples/jars/spark-examples_2.12-3.0.1.jar","java.home":"/home/java/jre","java.version":"1.8.0_162","sun.io.unicode.encoding":"UnicodeLittle"},"Classpath Entries":{"/home/spark-3.0.1/jars/jaxb-runtime-2.3.2.jar":"System Classpath","/home/spark-3.0.1/jars/scala-collection-compat_2.12-2.1.1.jar":"System Classpath","/home/spark-3.0.1/jars/spire-platform_2.12-0.17.0-M1.jar":"System Classpath","/home/spark-3.0.1/jars/libfb303-0.9.3.jar":"System Classpath","/home/spark-3.0.1/jars/hadoop-mapreduce-client-jobclient-3.2.0.jar":"System Classpath","/home/spark-3.0.1/jars/arpack_combined_all-0.1.jar":"System Classpath","/home/spark-3.0.1/jars/spark-graphx_2.12-3.0.1.jar":"System Classpath","/home/spark-3.0.1/jars/jersey-container-servlet-core-2.30.jar":"System Classpath","/home/spark-3.0.1/jars/htrace-core4-4.1.0-incubating.jar":"System Classpath","/home/spark-3.0.1/jars/parquet-format-2.4.0.jar":"System Classpath","/home/spark-3.0.1/jars/logging-interceptor-3.12.6.jar":"System Classpath","/home/spark-3.0.1/jars/breeze_2.12-1.0.jar":"System Classpath","/home/spark-3.0.1/jars/kerb-admin-1.0.1.jar":"System Classpath","/home/spark-3.0.1/jars/netty-all-4.1.47.Final.jar":"System Classpath","/home/spark-3.0.1/jars/istack-commons-runtime-3.0.8.jar":"System Classpath","/home/spark-3.0.1/jars/kerb-crypto-1.0.1.jar":"System Classpath","/home/spark-3.0.1/jars/spark-sketch_2.12-3.0.1.jar":"System Classpath","/home/spark-3.0.1/jars/avro-mapred-1.8.2-hadoop2.jar":"System Classpath","/home/spark-3.0.1/jars/hadoop-common-3.2.0.jar":"System Classpath","/home/spark-3.0.1/jars/joda-time-2.10.5.jar":"System Classpath","/home/spark-3.0.1/jars/jpam-1.1.jar":"System Classpath","/home/spark-3.0.1/jars/javax.inject-1.jar":"System Classpath","/home/spark-3.0.1/jars/metrics-graphite-4.1.1.jar":"System Classpath","/home/spark-3.0.1/jars/hive-shims-2.3.7.jar":"System Classpath","/home/spark-3.0.1/jars/chill-java-0.9.5.jar":"System Classpath","/home/spark-3.0.1/jars/okio-1.15.0.jar":"System Classpath","/home/spark-3.0.1/jars/kryo-shaded-4.0.2.jar":"System Classpath","/home/spark-3.0.1/jars/json4s-ast_2.12-3.6.6.jar":"System Classpath","/home/spark-3.0.1/jars/antlr4-runtime-4.7.1.jar":"System Classpath","spark://master:46515/jars/spark-examples_2.12-3.0.1.jar":"Added By User","/home/spark-3.0.1/jars/guice-4.0.jar":"System Classpath","/home/spark-3.0.1/jars/core-1.1.2.jar":"System Classpath","/home/spark-3.0.1/jars/commons-text-1.6.jar":"System Classpath","/home/spark-3.0.1/jars/commons-pool-1.5.4.jar":"System Classpath","/home/spark-3.0.1/jars/hadoop-yarn-client-3.2.0.jar":"System Classpath","/home/spark-3.0.1/jars/kerby-asn1-1.0.1.jar":"System Classpath","/home/spark-3.0.1/jars/hive-cli-2.3.7.jar":"System Classpath","/home/spark-3.0.1/jars/lz4-java-1.7.1.jar":"System Classpath","/home/spark-3.0.1/jars/spark-mesos_2.12-3.0.1.jar":"System Classpath","/home/spark-3.0.1/jars/kerb-simplekdc-1.0.1.jar":"System Classpath","/home/spark-3.0.1/jars/breeze-macros_2.12-1.0.jar":"System Classpath","/home/spark-3.0.1/jars/dnsjava-2.1.7.jar":"System Classpath","/home/spark-3.0.1/jars/okhttp-2.7.5.jar":"System Classpath","/home/spark-3.0.1/jars/transaction-api-1.1.jar":"System Classpath","/home/spark-3.0.1/jars/xbean-asm7-shaded-4.15.jar":"System Classpath","/home/spark-3.0.1/jars/spark-core_2.12-3.0.1.jar":"System Classpath","/home/spark-3.0.1/jars/metrics-jvm-4.1.1.jar":"System Classpath","/home/spark-3.0.1/jars/paranamer-2.8.jar":"System Classpath","/home/spark-3.0.1/jars/mesos-1.4.0-shaded-protobuf.jar":"System Classpath","/home/spark-3.0.1/jars/ST4-4.0.4.jar":"System Classpath","/home/spark-3.0.1/jars/jersey-container-servlet-2.30.jar":"System Classpath","/home/spark-3.0.1/jars/hive-vector-code-gen-2.3.7.jar":"System Classpath","/home/spark-3.0.1/jars/derby-10.12.1.1.jar":"System Classpath","/home/spark-3.0.1/jars/compress-lzf-1.0.3.jar":"System Classpath","/home/spark-3.0.1/jars/jaxb-api-2.2.11.jar":"System Classpath","/home/spark-3.0.1/jars/orc-shims-1.5.10.jar":"System Classpath","/home/spark-3.0.1/jars/jcl-over-slf4j-1.7.30.jar":"System Classpath","/home/spark-3.0.1/jars/spark-kvstore_2.12-3.0.1.jar":"System Classpath","/home/spark-3.0.1/jars/hk2-locator-2.6.1.jar":"System Classpath","/home/spark-3.0.1/jars/spark-network-shuffle_2.12-3.0.1.jar":"System Classpath","/home/spark-3.0.1/jars/hk2-utils-2.6.1.jar":"System Classpath","/home/spark-3.0.1/jars/jersey-server-2.30.jar":"System Classpath","/home/spark-3.0.1/jars/kubernetes-model-4.9.2.jar":"System Classpath","/home/spark-3.0.1/jars/commons-lang3-3.9.jar":"System Classpath","/home/spark-3.0.1/jars/spark-mllib_2.12-3.0.1.jar":"System Classpath","/home/spark-3.0.1/jars/parquet-common-1.10.1.jar":"System Classpath","/home/spark-3.0.1/jars/guice-servlet-4.0.jar":"System Classpath","/home/spark-3.0.1/jars/hadoop-annotations-3.2.0.jar":"System Classpath","/home/spark-3.0.1/jars/audience-annotations-0.5.0.jar":"System Classpath","/home/spark-3.0.1/jars/slf4j-api-1.7.30.jar":"System Classpath","/home/spark-3.0.1/jars/spark-streaming_2.12-3.0.1.jar":"System Classpath","/home/spark-3.0.1/jars/hive-shims-0.23-2.3.7.jar":"System Classpath","/home/spark-3.0.1/jars/avro-ipc-1.8.2.jar":"System Classpath","/home/spark-3.0.1/jars/scala-xml_2.12-1.2.0.jar":"System Classpath","/home/spark-3.0.1/jars/hadoop-yarn-server-common-3.2.0.jar":"System Classpath","/home/spark-3.0.1/jars/stax-api-1.0.1.jar":"System Classpath","/home/spark-3.0.1/jars/gson-2.2.4.jar":"System Classpath","/home/spark-3.0.1/jars/hadoop-yarn-server-web-proxy-3.2.0.jar":"System Classpath","/home/spark-3.0.1/jars/kerb-core-1.0.1.jar":"System Classpath","/home/spark-3.0.1/jars/curator-framework-2.13.0.jar":"System Classpath","/home/spark-3.0.1/jars/stax2-api-3.1.4.jar":"System Classpath","/home/spark-3.0.1/jars/commons-cli-1.2.jar":"System Classpath","/home/spark-3.0.1/jars/commons-collections-3.2.2.jar":"System Classpath","/home/spark-3.0.1/jars/jackson-core-2.10.0.jar":"System Classpath","/home/hadoop-3.2.1/etc/hadoop/":"System Classpath","/home/spark-3.0.1/jars/kerb-client-1.0.1.jar":"System Classpath","/home/spark-3.0.1/jars/generex-1.0.2.jar":"System Classpath","/home/spark-3.0.1/jars/jsr305-3.0.0.jar":"System Classpath","/home/spark-3.0.1/jars/scala-parser-combinators_2.12-1.1.2.jar":"System Classpath","/home/spark-3.0.1/jars/jackson-jaxrs-base-2.9.5.jar":"System Classpath","/home/spark-3.0.1/jars/jta-1.1.jar":"System Classpath","/home/spark-3.0.1/jars/parquet-column-1.10.1.jar":"System Classpath","/home/spark-3.0.1/jars/jackson-module-scala_2.12-2.10.0.jar":"System Classpath","/home/spark-3.0.1/jars/janino-3.0.16.jar":"System Classpath","/home/spark-3.0.1/jars/hive-llap-common-2.3.7.jar":"System Classpath","/home/spark-3.0.1/jars/spark-sql_2.12-3.0.1.jar":"System Classpath","/home/spark-3.0.1/jars/json-smart-2.3.jar":"System Classpath","/home/spark-3.0.1/jars/scala-library-2.12.10.jar":"System Classpath","/home/spark-3.0.1/jars/hive-metastore-2.3.7.jar":"System Classpath","/home/spark-3.0.1/jars/log4j-1.2.17.jar":"System Classpath","/home/spark-3.0.1/jars/parquet-encoding-1.10.1.jar":"System Classpath","/home/spark-3.0.1/jars/xz-1.5.jar":"System Classpath","/home/spark-3.0.1/jars/aircompressor-0.10.jar":"System Classpath","/home/spark-3.0.1/jars/json4s-scalap_2.12-3.6.6.jar":"System Classpath","/home/spark-3.0.1/jars/jakarta.inject-2.6.1.jar":"System Classpath","/home/spark-3.0.1/jars/hive-shims-scheduler-2.3.7.jar":"System Classpath","/home/spark-3.0.1/jars/jackson-annotations-2.10.0.jar":"System Classpath","/home/spark-3.0.1/jars/threeten-extra-1.5.0.jar":"System Classpath","/home/spark-3.0.1/jars/super-csv-2.2.0.jar":"System Classpath","/home/spark-3.0.1/jars/accessors-smart-1.2.jar":"System Classpath","/home/spark-3.0.1/jars/hive-jdbc-2.3.7.jar":"System Classpath","/home/spark-3.0.1/jars/orc-mapreduce-1.5.10.jar":"System Classpath","/home/spark-3.0.1/jars/jersey-media-jaxb-2.30.jar":"System Classpath","/home/spark-3.0.1/jars/scala-compiler-2.12.10.jar":"System Classpath","/home/spark-3.0.1/jars/opencsv-2.3.jar":"System Classpath","/home/spark-3.0.1/conf/":"System Classpath","/home/spark-3.0.1/jars/commons-logging-1.1.3.jar":"System Classpath","/home/spark-3.0.1/jars/spark-repl_2.12-3.0.1.jar":"System Classpath","/home/spark-3.0.1/jars/jackson-dataformat-yaml-2.10.0.jar":"System Classpath","/home/spark-3.0.1/jars/automaton-1.11-8.jar":"System Classpath","/home/spark-3.0.1/jars/jackson-mapper-asl-1.9.13.jar":"System Classpath","/home/spark-3.0.1/jars/shapeless_2.12-2.3.3.jar":"System Classpath","/home/spark-3.0.1/jars/kerb-common-1.0.1.jar":"System Classpath","/home/spark-3.0.1/jars/slf4j-log4j12-1.7.30.jar":"System Classpath","/home/spark-3.0.1/jars/json-1.8.jar":"System Classpath","/home/spark-3.0.1/jars/jersey-common-2.30.jar":"System Classpath","/home/spark-3.0.1/jars/hadoop-client-3.2.0.jar":"System Classpath","/home/spark-3.0.1/jars/metrics-core-4.1.1.jar":"System Classpath","/home/spark-3.0.1/jars/nimbus-jose-jwt-4.41.1.jar":"System Classpath","/home/spark-3.0.1/jars/HikariCP-2.5.1.jar":"System Classpath","/home/spark-3.0.1/jars/jersey-client-2.30.jar":"System Classpath","/home/spark-3.0.1/jars/ivy-2.4.0.jar":"System Classpath","/home/spark-3.0.1/jars/shims-0.7.45.jar":"System Classpath","/home/spark-3.0.1/jars/commons-lang-2.6.jar":"System Classpath","/home/spark-3.0.1/jars/commons-compress-1.8.1.jar":"System Classpath","/home/spark-3.0.1/jars/hadoop-yarn-registry-3.2.0.jar":"System Classpath","/home/spark-3.0.1/jars/stream-2.9.6.jar":"System Classpath","/home/spark-3.0.1/jars/jackson-datatype-jsr310-2.10.3.jar":"System Classpath","/home/spark-3.0.1/jars/javassist-3.25.0-GA.jar":"System Classpath","/home/spark-3.0.1/jars/commons-httpclient-3.1.jar":"System Classpath","/home/spark-3.0.1/jars/machinist_2.12-0.6.8.jar":"System Classpath","/home/spark-3.0.1/jars/spark-launcher_2.12-3.0.1.jar":"System Classpath","/home/spark-3.0.1/jars/arrow-vector-0.15.1.jar":"System Classpath","/home/spark-3.0.1/jars/jersey-hk2-2.30.jar":"System Classpath","/home/spark-3.0.1/jars/hive-common-2.3.7.jar":"System Classpath","/home/spark-3.0.1/jars/zstd-jni-1.4.4-3.jar":"System Classpath","/home/spark-3.0.1/jars/metrics-json-4.1.1.jar":"System Classpath","/home/spark-3.0.1/jars/token-provider-1.0.1.jar":"System Classpath","/home/spark-3.0.1/jars/guava-14.0.1.jar":"System Classpath","/home/spark-3.0.1/jars/zookeeper-3.4.14.jar":"System Classpath","/home/spark-3.0.1/jars/commons-net-3.1.jar":"System Classpath","/home/spark-3.0.1/jars/spark-catalyst_2.12-3.0.1.jar":"System Classpath","/home/spark-3.0.1/jars/commons-math3-3.4.1.jar":"System Classpath","/home/spark-3.0.1/jars/arrow-format-0.15.1.jar":"System Classpath","/home/spark-3.0.1/jars/univocity-parsers-2.9.0.jar":"System Classpath","/home/spark-3.0.1/jars/httpcore-4.4.12.jar":"System Classpath","/home/spark-3.0.1/jars/py4j-0.10.9.jar":"System Classpath","/home/spark-3.0.1/jars/zjsonpatch-0.3.0.jar":"System Classpath","/home/spark-3.0.1/jars/spire-macros_2.12-0.17.0-M1.jar":"System Classpath","/home/spark-3.0.1/jars/spark-yarn_2.12-3.0.1.jar":"System Classpath","/home/spark-3.0.1/jars/macro-compat_2.12-1.1.1.jar":"System Classpath","/home/spark-3.0.1/jars/antlr-runtime-3.5.2.jar":"System Classpath","/home/spark-3.0.1/jars/jodd-core-3.5.2.jar":"System Classpath","/home/spark-3.0.1/jars/jdo-api-3.0.1.jar":"System Classpath","/home/spark-3.0.1/jars/JLargeArrays-1.5.jar":"System Classpath","/home/spark-3.0.1/jars/hive-serde-2.3.7.jar":"System Classpath","/home/spark-3.0.1/jars/jackson-core-asl-1.9.13.jar":"System Classpath","/home/spark-3.0.1/jars/spark-unsafe_2.12-3.0.1.jar":"System Classpath","/home/spark-3.0.1/jars/metrics-jmx-4.1.1.jar":"System Classpath","/home/spark-3.0.1/jars/jakarta.xml.bind-api-2.3.2.jar":"System Classpath","/home/spark-3.0.1/jars/algebra_2.12-2.0.0-M2.jar":"System Classpath","/home/spark-3.0.1/jars/bonecp-0.8.0.RELEASE.jar":"System Classpath","/home/spark-3.0.1/jars/json4s-core_2.12-3.6.6.jar":"System Classpath","/home/spark-3.0.1/jars/datanucleus-core-4.1.17.jar":"System Classpath","/home/spark-3.0.1/jars/osgi-resource-locator-1.0.3.jar":"System Classpath","/home/spark-3.0.1/jars/hk2-api-2.6.1.jar":"System Classpath","/home/spark-3.0.1/jars/cats-kernel_2.12-2.0.0-M4.jar":"System Classpath","/home/spark-3.0.1/jars/datanucleus-api-jdo-4.2.4.jar":"System Classpath","/home/spark-3.0.1/jars/jakarta.validation-api-2.0.2.jar":"System Classpath","/home/spark-3.0.1/jars/jackson-jaxrs-json-provider-2.9.5.jar":"System Classpath","/home/spark-3.0.1/jars/httpclient-4.5.6.jar":"System Classpath","/home/spark-3.0.1/jars/oro-2.0.8.jar":"System Classpath","/home/spark-3.0.1/jars/re2j-1.1.jar":"System Classpath","/home/spark-3.0.1/jars/parquet-jackson-1.10.1.jar":"System Classpath","/home/spark-3.0.1/jars/hadoop-hdfs-client-3.2.0.jar":"System Classpath","/home/spark-3.0.1/jars/kubernetes-client-4.9.2.jar":"System Classpath","/home/spark-3.0.1/jars/jackson-module-paranamer-2.10.0.jar":"System Classpath","/home/spark-3.0.1/jars/flatbuffers-java-1.9.0.jar":"System Classpath","/home/spark-3.0.1/jars/json4s-jackson_2.12-3.6.6.jar":"System Classpath","/home/spark-3.0.1/jars/minlog-1.3.0.jar":"System Classpath","/home/spark-3.0.1/jars/javax.jdo-3.2.0-m3.jar":"System Classpath","/home/spark-3.0.1/jars/commons-crypto-1.0.0.jar":"System Classpath","/home/spark-3.0.1/jars/spark-tags_2.12-3.0.1.jar":"System Classpath","/home/spark-3.0.1/jars/velocity-1.5.jar":"System Classpath","/home/spark-3.0.1/jars/leveldbjni-all-1.8.jar":"System Classpath","/home/spark-3.0.1/jars/pyrolite-4.30.jar":"System Classpath","/home/spark-3.0.1/jars/chill_2.12-0.9.5.jar":"System Classpath","/home/spark-3.0.1/jars/RoaringBitmap-0.7.45.jar":"System Classpath","/home/spark-3.0.1/jars/jackson-databind-2.10.0.jar":"System Classpath","/home/spark-3.0.1/jars/hive-beeline-2.3.7.jar":"System Classpath","/home/spark-3.0.1/jars/hadoop-auth-3.2.0.jar":"System Classpath","/home/spark-3.0.1/jars/orc-core-1.5.10.jar":"System Classpath","/home/spark-3.0.1/jars/hadoop-yarn-api-3.2.0.jar":"System Classpath","/home/spark-3.0.1/jars/curator-recipes-2.13.0.jar":"System Classpath","/home/spark-3.0.1/jars/kerby-util-1.0.1.jar":"System Classpath","/home/spark-3.0.1/jars/parquet-hadoop-1.10.1.jar":"System Classpath","/home/spark-3.0.1/jars/commons-dbcp-1.4.jar":"System Classpath","/home/spark-3.0.1/jars/libthrift-0.12.0.jar":"System Classpath","/home/spark-3.0.1/jars/scala-reflect-2.12.10.jar":"System Classpath","/home/spark-3.0.1/jars/okhttp-3.12.6.jar":"System Classpath","/home/spark-3.0.1/jars/protobuf-java-2.5.0.jar":"System Classpath","/home/spark-3.0.1/jars/javax.servlet-api-3.1.0.jar":"System Classpath","/home/spark-3.0.1/jars/geronimo-jcache_1.0_spec-1.0-alpha-1.jar":"System Classpath","/home/spark-3.0.1/jars/jsp-api-2.1.jar":"System Classpath","/home/spark-3.0.1/jars/kerb-util-1.0.1.jar":"System Classpath","/home/spark-3.0.1/jars/spark-kubernetes_2.12-3.0.1.jar":"System Classpath","/home/spark-3.0.1/jars/kerb-identity-1.0.1.jar":"System Classpath","/home/spark-3.0.1/jars/hadoop-mapreduce-client-core-3.2.0.jar":"System Classpath","/home/spark-3.0.1/jars/jline-2.14.6.jar":"System Classpath","/home/spark-3.0.1/jars/spark-hive-thriftserver_2.12-3.0.1.jar":"System Classpath","/home/spark-3.0.1/jars/spire_2.12-0.17.0-M1.jar":"System Classpath","/home/spark-3.0.1/jars/jackson-module-jaxb-annotations-2.10.0.jar":"System Classpath","/home/spark-3.0.1/jars/jakarta.annotation-api-1.3.5.jar":"System Classpath","/home/spark-3.0.1/jars/activation-1.1.1.jar":"System Classpath","/home/spark-3.0.1/jars/kerby-config-1.0.1.jar":"System Classpath","/home/spark-3.0.1/jars/woodstox-core-5.0.3.jar":"System Classpath","/home/spark-3.0.1/jars/snakeyaml-1.24.jar":"System Classpath","/home/spark-3.0.1/jars/spark-hive_2.12-3.0.1.jar":"System Classpath","/home/spark-3.0.1/jars/kerb-server-1.0.1.jar":"System Classpath","/home/spark-3.0.1/jars/arrow-memory-0.15.1.jar":"System Classpath","/home/spark-3.0.1/jars/dropwizard-metrics-hadoop-metrics2-reporter-0.1.2.jar":"System Classpath","/home/spark-3.0.1/jars/avro-1.8.2.jar":"System Classpath","/home/spark-3.0.1/jars/spark-tags_2.12-3.0.1-tests.jar":"System Classpath","/home/spark-3.0.1/jars/aopalliance-1.0.jar":"System Classpath","/home/spark-3.0.1/jars/jcip-annotations-1.0-1.jar":"System Classpath","/home/spark-3.0.1/jars/curator-client-2.13.0.jar":"System Classpath","/home/spark-3.0.1/jars/aopalliance-repackaged-2.6.1.jar":"System Classpath","/home/spark-3.0.1/jars/hive-shims-common-2.3.7.jar":"System Classpath","/home/spark-3.0.1/jars/hadoop-yarn-common-3.2.0.jar":"System Classpath","/home/spark-3.0.1/jars/hive-exec-2.3.7-core.jar":"System Classpath","/home/spark-3.0.1/jars/jul-to-slf4j-1.7.30.jar":"System Classpath","/home/spark-3.0.1/jars/javolution-5.5.1.jar":"System Classpath","/home/spark-3.0.1/jars/spark-network-common_2.12-3.0.1.jar":"System Classpath","/home/spark-3.0.1/jars/JTransforms-3.1.jar":"System Classpath","/home/spark-3.0.1/jars/commons-beanutils-1.9.4.jar":"System Classpath","/home/spark-3.0.1/jars/datanucleus-rdbms-4.1.19.jar":"System Classpath","/home/spark-3.0.1/jars/commons-io-2.4.jar":"System Classpath","/home/spark-3.0.1/jars/kerby-xdr-1.0.1.jar":"System Classpath","/home/spark-3.0.1/jars/commons-daemon-1.0.13.jar":"System Classpath","/home/spark-3.0.1/jars/kubernetes-model-common-4.9.2.jar":"System Classpath","/home/spark-3.0.1/jars/hadoop-mapreduce-client-common-3.2.0.jar":"System Classpath","/home/spark-3.0.1/jars/commons-codec-1.10.jar":"System Classpath","/home/spark-3.0.1/jars/commons-compiler-3.0.16.jar":"System Classpath","/home/spark-3.0.1/jars/objenesis-2.5.1.jar":"System Classpath","/home/spark-3.0.1/jars/spire-util_2.12-0.17.0-M1.jar":"System Classpath","/home/spark-3.0.1/jars/ehcache-3.3.1.jar":"System Classpath","/home/spark-3.0.1/jars/snappy-java-1.1.7.5.jar":"System Classpath","/home/spark-3.0.1/jars/hive-storage-api-2.7.1.jar":"System Classpath","/home/spark-3.0.1/jars/kerby-pkix-1.0.1.jar":"System Classpath","/home/spark-3.0.1/jars/jakarta.activation-api-1.2.1.jar":"System Classpath","/home/spark-3.0.1/jars/spark-mllib-local_2.12-3.0.1.jar":"System Classpath","/home/spark-3.0.1/jars/jakarta.ws.rs-api-2.1.6.jar":"System Classpath","/home/spark-3.0.1/jars/commons-configuration2-2.1.1.jar":"System Classpath"}}
{"Event":"SparkListenerApplicationStart","App Name":"Spark Pi","App ID":"app-20201019233352-0002","Timestamp":1603121631550,"User":"root"}
{"Event":"SparkListenerJobStart","Job ID":0,"Submission Time":1603121635939,"Stage Infos":[{"Stage ID":0,"Stage Attempt ID":0,"Stage Name":"reduce at SparkPi.scala:38","Number of Tasks":2,"RDD Info":[{"RDD ID":1,"Name":"MapPartitionsRDD","Scope":"{\"id\":\"1\",\"name\":\"map\"}","Callsite":"map at SparkPi.scala:34","Parent IDs":[0],"Storage Level":{"Use Disk":false,"Use Memory":false,"Deserialized":false,"Replication":1},"Barrier":false,"Number of Partitions":2,"Number of Cached Partitions":0,"Memory Size":0,"Disk Size":0},{"RDD ID":0,"Name":"ParallelCollectionRDD","Scope":"{\"id\":\"0\",\"name\":\"parallelize\"}","Callsite":"parallelize at SparkPi.scala:34","Parent IDs":[],"Storage Level":{"Use Disk":false,"Use Memory":false,"Deserialized":false,"Replication":1},"Barrier":false,"Number of Partitions":2,"Number of Cached Partitions":0,"Memory Size":0,"Disk Size":0}],"Parent IDs":[],"Details":"org.apache.spark.rdd.RDD.reduce(RDD.scala:1076)\norg.apache.spark.examples.SparkPi$.main(SparkPi.scala:38)\norg.apache.spark.examples.SparkPi.main(SparkPi.scala)\nsun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)\nsun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)\nsun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)\njava.lang.reflect.Method.invoke(Method.java:498)\norg.apache.spark.deploy.JavaMainApplication.start(SparkApplication.scala:52)\norg.apache.spark.deploy.SparkSubmit.org$apache$spark$deploy$SparkSubmit$$runMain(SparkSubmit.scala:928)\norg.apache.spark.deploy.SparkSubmit.doRunMain$1(SparkSubmit.scala:180)\norg.apache.spark.deploy.SparkSubmit.submit(SparkSubmit.scala:203)\norg.apache.spark.deploy.SparkSubmit.doSubmit(SparkSubmit.scala:90)\norg.apache.spark.deploy.SparkSubmit$$anon$2.doSubmit(SparkSubmit.scala:1007)\norg.apache.spark.deploy.SparkSubmit$.main(SparkSubmit.scala:1016)\norg.apache.spark.deploy.SparkSubmit.main(SparkSubmit.scala)","Accumulables":[]}],"Stage IDs":[0],"Properties":{"spark.rdd.scope":"{\"id\":\"2\",\"name\":\"reduce\"}","spark.rdd.scope.noOverride":"true"}}
{"Event":"SparkListenerStageSubmitted","Stage Info":{"Stage ID":0,"Stage Attempt ID":0,"Stage Name":"reduce at SparkPi.scala:38","Number of Tasks":2,"RDD Info":[{"RDD ID":1,"Name":"MapPartitionsRDD","Scope":"{\"id\":\"1\",\"name\":\"map\"}","Callsite":"map at SparkPi.scala:34","Parent IDs":[0],"Storage Level":{"Use Disk":false,"Use Memory":false,"Deserialized":false,"Replication":1},"Barrier":false,"Number of Partitions":2,"Number of Cached Partitions":0,"Memory Size":0,"Disk Size":0},{"RDD ID":0,"Name":"ParallelCollectionRDD","Scope":"{\"id\":\"0\",\"name\":\"parallelize\"}","Callsite":"parallelize at SparkPi.scala:34","Parent IDs":[],"Storage Level":{"Use Disk":false,"Use Memory":false,"Deserialized":false,"Replication":1},"Barrier":false,"Number of Partitions":2,"Number of Cached Partitions":0,"Memory Size":0,"Disk Size":0}],"Parent IDs":[],"Details":"org.apache.spark.rdd.RDD.reduce(RDD.scala:1076)\norg.apache.spark.examples.SparkPi$.main(SparkPi.scala:38)\norg.apache.spark.examples.SparkPi.main(SparkPi.scala)\nsun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)\nsun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)\nsun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)\njava.lang.reflect.Method.invoke(Method.java:498)\norg.apache.spark.deploy.JavaMainApplication.start(SparkApplication.scala:52)\norg.apache.spark.deploy.SparkSubmit.org$apache$spark$deploy$SparkSubmit$$runMain(SparkSubmit.scala:928)\norg.apache.spark.deploy.SparkSubmit.doRunMain$1(SparkSubmit.scala:180)\norg.apache.spark.deploy.SparkSubmit.submit(SparkSubmit.scala:203)\norg.apache.spark.deploy.SparkSubmit.doSubmit(SparkSubmit.scala:90)\norg.apache.spark.deploy.SparkSubmit$$anon$2.doSubmit(SparkSubmit.scala:1007)\norg.apache.spark.deploy.SparkSubmit$.main(SparkSubmit.scala:1016)\norg.apache.spark.deploy.SparkSubmit.main(SparkSubmit.scala)","Submission Time":1603121636405,"Accumulables":[]},"Properties":{"spark.rdd.scope":"{\"id\":\"2\",\"name\":\"reduce\"}","spark.rdd.scope.noOverride":"true"}}
{"Event":"SparkListenerExecutorAdded","Timestamp":1603121637799,"Executor ID":"0","Executor Info":{"Host":"192.168.153.135","Total Cores":1,"Log Urls":{"stdout":"http://192.168.153.135:8081/logPage/?appId=app-20201019233352-0002&executorId=0&logType=stdout","stderr":"http://192.168.153.135:8081/logPage/?appId=app-20201019233352-0002&executorId=0&logType=stderr"},"Attributes":{},"Resources":{}}}
{"Event":"SparkListenerBlockManagerAdded","Block Manager ID":{"Executor ID":"0","Host":"192.168.153.135","Port":34409},"Maximum Memory":97832140,"Timestamp":1603121637949,"Maximum Onheap Memory":97832140,"Maximum Offheap Memory":0}
{"Event":"SparkListenerTaskStart","Stage ID":0,"Stage Attempt ID":0,"Task Info":{"Task ID":0,"Index":0,"Attempt":0,"Launch Time":1603121638008,"Executor ID":"0","Host":"192.168.153.135","Locality":"PROCESS_LOCAL","Speculative":false,"Getting Result Time":0,"Finish Time":0,"Failed":false,"Killed":false,"Accumulables":[]}}
{"Event":"SparkListenerExecutorAdded","Timestamp":1603121638131,"Executor ID":"2","Executor Info":{"Host":"192.168.153.137","Total Cores":1,"Log Urls":{"stdout":"http://192.168.153.137:8081/logPage/?appId=app-20201019233352-0002&executorId=2&logType=stdout","stderr":"http://192.168.153.137:8081/logPage/?appId=app-20201019233352-0002&executorId=2&logType=stderr"},"Attributes":{},"Resources":{}}}
{"Event":"SparkListenerExecutorAdded","Timestamp":1603121638228,"Executor ID":"1","Executor Info":{"Host":"192.168.153.136","Total Cores":1,"Log Urls":{"stdout":"http://192.168.153.136:8082/logPage/?appId=app-20201019233352-0002&executorId=1&logType=stdout","stderr":"http://192.168.153.136:8082/logPage/?appId=app-20201019233352-0002&executorId=1&logType=stderr"},"Attributes":{},"Resources":{}}}
{"Event":"SparkListenerBlockManagerAdded","Block Manager ID":{"Executor ID":"2","Host":"192.168.153.137","Port":39783},"Maximum Memory":122644070,"Timestamp":1603121638354,"Maximum Onheap Memory":122644070,"Maximum Offheap Memory":0}
{"Event":"SparkListenerTaskStart","Stage ID":0,"Stage Attempt ID":0,"Task Info":{"Task ID":1,"Index":1,"Attempt":0,"Launch Time":1603121638428,"Executor ID":"2","Host":"192.168.153.137","Locality":"PROCESS_LOCAL","Speculative":false,"Getting Result Time":0,"Finish Time":0,"Failed":false,"Killed":false,"Accumulables":[]}}
{"Event":"SparkListenerBlockManagerAdded","Block Manager ID":{"Executor ID":"1","Host":"192.168.153.136","Port":44998},"Maximum Memory":122644070,"Timestamp":1603121638496,"Maximum Onheap Memory":122644070,"Maximum Offheap Memory":0}
{"Event":"SparkListenerTaskEnd","Stage ID":0,"Stage Attempt ID":0,"Task Type":"ResultTask","Task End Reason":{"Reason":"Success"},"Task Info":{"Task ID":0,"Index":0,"Attempt":0,"Launch Time":1603121638008,"Executor ID":"0","Host":"192.168.153.135","Locality":"PROCESS_LOCAL","Speculative":false,"Getting Result Time":0,"Finish Time":1603121639525,"Failed":false,"Killed":false,"Accumulables":[{"ID":5,"Name":"internal.metrics.jvmGCTime","Update":59,"Value":59,"Internal":true,"Count Failed Values":true},{"ID":4,"Name":"internal.metrics.resultSize","Update":842,"Value":842,"Internal":true,"Count Failed Values":true},{"ID":3,"Name":"internal.metrics.executorCpuTime","Update":66570998,"Value":66570998,"Internal":true,"Count Failed Values":true},{"ID":2,"Name":"internal.metrics.executorRunTime","Update":70,"Value":70,"Internal":true,"Count Failed Values":true},{"ID":1,"Name":"internal.metrics.executorDeserializeCpuTime","Update":1104556264,"Value":1104556264,"Internal":true,"Count Failed Values":true},{"ID":0,"Name":"internal.metrics.executorDeserializeTime","Update":1361,"Value":1361,"Internal":true,"Count Failed Values":true}]},"Task Executor Metrics":{"JVMHeapMemory":0,"JVMOffHeapMemory":0,"OnHeapExecutionMemory":0,"OffHeapExecutionMemory":0,"OnHeapStorageMemory":0,"OffHeapStorageMemory":0,"OnHeapUnifiedMemory":0,"OffHeapUnifiedMemory":0,"DirectPoolMemory":0,"MappedPoolMemory":0,"ProcessTreeJVMVMemory":0,"ProcessTreeJVMRSSMemory":0,"ProcessTreePythonVMemory":0,"ProcessTreePythonRSSMemory":0,"ProcessTreeOtherVMemory":0,"ProcessTreeOtherRSSMemory":0,"MinorGCCount":0,"MinorGCTime":0,"MajorGCCount":0,"MajorGCTime":0},"Task Metrics":{"Executor Deserialize Time":1361,"Executor Deserialize CPU Time":1104556264,"Executor Run Time":70,"Executor CPU Time":66570998,"Peak Execution Memory":0,"Result Size":842,"JVM GC Time":59,"Result Serialization Time":0,"Memory Bytes Spilled":0,"Disk Bytes Spilled":0,"Shuffle Read Metrics":{"Remote Blocks Fetched":0,"Local Blocks Fetched":0,"Fetch Wait Time":0,"Remote Bytes Read":0,"Remote Bytes Read To Disk":0,"Local Bytes Read":0,"Total Records Read":0},"Shuffle Write Metrics":{"Shuffle Bytes Written":0,"Shuffle Write Time":0,"Shuffle Records Written":0},"Input Metrics":{"Bytes Read":0,"Records Read":0},"Output Metrics":{"Bytes Written":0,"Records Written":0},"Updated Blocks":[]}}
{"Event":"SparkListenerTaskEnd","Stage ID":0,"Stage Attempt ID":0,"Task Type":"ResultTask","Task End Reason":{"Reason":"Success"},"Task Info":{"Task ID":1,"Index":1,"Attempt":0,"Launch Time":1603121638428,"Executor ID":"2","Host":"192.168.153.137","Locality":"PROCESS_LOCAL","Speculative":false,"Getting Result Time":0,"Finish Time":1603121640774,"Failed":false,"Killed":false,"Accumulables":[{"ID":5,"Name":"internal.metrics.jvmGCTime","Update":69,"Value":128,"Internal":true,"Count Failed Values":true},{"ID":4,"Name":"internal.metrics.resultSize","Update":842,"Value":1684,"Internal":true,"Count Failed Values":true},{"ID":3,"Name":"internal.metrics.executorCpuTime","Update":70763236,"Value":137334234,"Internal":true,"Count Failed Values":true},{"ID":2,"Name":"internal.metrics.executorRunTime","Update":160,"Value":230,"Internal":true,"Count Failed Values":true},{"ID":1,"Name":"internal.metrics.executorDeserializeCpuTime","Update":1007597102,"Value":2112153366,"Internal":true,"Count Failed Values":true},{"ID":0,"Name":"internal.metrics.executorDeserializeTime","Update":2121,"Value":3482,"Internal":true,"Count Failed Values":true}]},"Task Executor Metrics":{"JVMHeapMemory":0,"JVMOffHeapMemory":0,"OnHeapExecutionMemory":0,"OffHeapExecutionMemory":0,"OnHeapStorageMemory":0,"OffHeapStorageMemory":0,"OnHeapUnifiedMemory":0,"OffHeapUnifiedMemory":0,"DirectPoolMemory":0,"MappedPoolMemory":0,"ProcessTreeJVMVMemory":0,"ProcessTreeJVMRSSMemory":0,"ProcessTreePythonVMemory":0,"ProcessTreePythonRSSMemory":0,"ProcessTreeOtherVMemory":0,"ProcessTreeOtherRSSMemory":0,"MinorGCCount":0,"MinorGCTime":0,"MajorGCCount":0,"MajorGCTime":0},"Task Metrics":{"Executor Deserialize Time":2121,"Executor Deserialize CPU Time":1007597102,"Executor Run Time":160,"Executor CPU Time":70763236,"Peak Execution Memory":0,"Result Size":842,"JVM GC Time":69,"Result Serialization Time":0,"Memory Bytes Spilled":0,"Disk Bytes Spilled":0,"Shuffle Read Metrics":{"Remote Blocks Fetched":0,"Local Blocks Fetched":0,"Fetch Wait Time":0,"Remote Bytes Read":0,"Remote Bytes Read To Disk":0,"Local Bytes Read":0,"Total Records Read":0},"Shuffle Write Metrics":{"Shuffle Bytes Written":0,"Shuffle Write Time":0,"Shuffle Records Written":0},"Input Metrics":{"Bytes Read":0,"Records Read":0},"Output Metrics":{"Bytes Written":0,"Records Written":0},"Updated Blocks":[]}}
{"Event":"SparkListenerStageCompleted","Stage Info":{"Stage ID":0,"Stage Attempt ID":0,"Stage Name":"reduce at SparkPi.scala:38","Number of Tasks":2,"RDD Info":[{"RDD ID":1,"Name":"MapPartitionsRDD","Scope":"{\"id\":\"1\",\"name\":\"map\"}","Callsite":"map at SparkPi.scala:34","Parent IDs":[0],"Storage Level":{"Use Disk":false,"Use Memory":false,"Deserialized":false,"Replication":1},"Barrier":false,"Number of Partitions":2,"Number of Cached Partitions":0,"Memory Size":0,"Disk Size":0},{"RDD ID":0,"Name":"ParallelCollectionRDD","Scope":"{\"id\":\"0\",\"name\":\"parallelize\"}","Callsite":"parallelize at SparkPi.scala:34","Parent IDs":[],"Storage Level":{"Use Disk":false,"Use Memory":false,"Deserialized":false,"Replication":1},"Barrier":false,"Number of Partitions":2,"Number of Cached Partitions":0,"Memory Size":0,"Disk Size":0}],"Parent IDs":[],"Details":"org.apache.spark.rdd.RDD.reduce(RDD.scala:1076)\norg.apache.spark.examples.SparkPi$.main(SparkPi.scala:38)\norg.apache.spark.examples.SparkPi.main(SparkPi.scala)\nsun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)\nsun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)\nsun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)\njava.lang.reflect.Method.invoke(Method.java:498)\norg.apache.spark.deploy.JavaMainApplication.start(SparkApplication.scala:52)\norg.apache.spark.deploy.SparkSubmit.org$apache$spark$deploy$SparkSubmit$$runMain(SparkSubmit.scala:928)\norg.apache.spark.deploy.SparkSubmit.doRunMain$1(SparkSubmit.scala:180)\norg.apache.spark.deploy.SparkSubmit.submit(SparkSubmit.scala:203)\norg.apache.spark.deploy.SparkSubmit.doSubmit(SparkSubmit.scala:90)\norg.apache.spark.deploy.SparkSubmit$$anon$2.doSubmit(SparkSubmit.scala:1007)\norg.apache.spark.deploy.SparkSubmit$.main(SparkSubmit.scala:1016)\norg.apache.spark.deploy.SparkSubmit.main(SparkSubmit.scala)","Submission Time":1603121636405,"Completion Time":1603121640779,"Accumulables":[{"ID":2,"Name":"internal.metrics.executorRunTime","Value":230,"Internal":true,"Count Failed Values":true},{"ID":5,"Name":"internal.metrics.jvmGCTime","Value":128,"Internal":true,"Count Failed Values":true},{"ID":4,"Name":"internal.metrics.resultSize","Value":1684,"Internal":true,"Count Failed Values":true},{"ID":1,"Name":"internal.metrics.executorDeserializeCpuTime","Value":2112153366,"Internal":true,"Count Failed Values":true},{"ID":3,"Name":"internal.metrics.executorCpuTime","Value":137334234,"Internal":true,"Count Failed Values":true},{"ID":0,"Name":"internal.metrics.executorDeserializeTime","Value":3482,"Internal":true,"Count Failed Values":true}]}}
{"Event":"SparkListenerJobEnd","Job ID":0,"Completion Time":1603121640792,"Job Result":{"Result":"JobSucceeded"}}
{"Event":"SparkListenerApplicationEnd","Timestamp":1603121640817}
一个日志文件代表了一个Spark App，日志中每行为JSON格式的Spark App事件内容，文件由上而下顺序的输出了整个Spark App执行的事件过程。
事件分析
利用事件日志提供的多个指标，我们可以发现一些Spark App中存在的程序错误和性能故障，大概有如下分析方法：
- 获取Task执行结果，对失败的Task进行异常总结，针对异常形成统一解决方案。
- 比较Task的平均执行时间和最大执行时间，以确定是否有某个Task拖慢整体任务进度。
- 观察Task的平均数据量大小和最大数据量大小，以确定是否存在数据倾斜。
- 分析Task的JVM GC时间，观察是否有GC时间过慢的情况，主要观察GC占用CPU时间的百分比，如果占用过多则需要对GC进行优化。
Spark学习-事件日志EventLog相关推荐
- Windows事件日志监控
大多数数据泄露属内部人员而为,但各企业在监控内部网络活动方面仍存在不足. 无论是大型还是小型企业,监控内部网络活动已成为其主要要求.要保护网络安全以防范泄露和威胁,各企业需要采取积极的措施来保证其网络 ...
- EventLog Analyzer：功能丰富的事件日志管理软件
Windows 设备是大多数企业网络中最受欢迎的选择.为了处理这些设备生成的数 TB 的事件日志数据,安全管理员可以使用EventLog Analyzer,这是一个强大的日志管理工具,提供端到端的事件 ...
- 使用EventLog类写Windows事件日志
此文转自 银河使者 http://www.cnblogs.com/nokiaguy/archive/2009/02/26/1398708.html 多谢作者分享 操作系统: Windows XP ...
- EventLog实现事件日志操作
选中"我的电脑",在其右键菜单中选择"管理",在打开的对话框中包括了如下图所示的"日志"信息: 选中其中的某一条日志,可以看到如下的详细信息 ...
- Windows 事件日志分析管理
Windows 设备是大多数商业网络中最受欢迎的选择.为了处理这些设备生成的数 TB 的事件日志数据,安全管理员需要使用功能强大的日志管理工具(如EventLog Analyzer),该工具可以通过自 ...
- Spark学习之路一——Spark基础及环境搭建
Spark学习之路一--Spark基础及环境搭建 文章目录 一. Spark 概述 1.1 概述 1.2 优势特性 1.2.1 运行速度快 1.2.2 容易使用 1.2.3 通用性 1.2.4 运行模 ...
- 大数据之spark学习记录二: Spark的安装与上手
大数据之spark学习记录二: Spark的安装与上手 文章目录 大数据之spark学习记录二: Spark的安装与上手 Spark安装 本地模式 Standalone 模式 基本配置 步骤1: 复制 ...
- 过程挖掘（Process Mining）5——事件日志（Event Logs）（1）：数据源与事件日志
为何有过程挖掘5没有4?过程挖掘4讲数据挖掘的内容,实际上我们在流程分析时,数据挖掘也是重要的工具,是必须掌握的:其次,很多技术用到的思想是相通的,流程发现.一致性检验等技术,都会用到数据挖掘中一些基 ...
- Apache Spark学习：利用Scala语言开发Spark应用程序
Spark内核是由Scala语言开发的,因此使用Scala语言开发Spark应用程序是自然而然的事情.如果你对Scala语言还不太熟悉,可以阅读网络教程 A Scala Tutorial for Ja ...
最新文章
- 【C++】智能指针（一）入门
- 为什么有些人非技术出身，却能带好团队？
- STL系列：关联容器的操作
- HTML：调用静态页面html 的几种方法
- 阿里巴巴牵手上汽集团：成立汽车科技公司
- Android TV 开发 （1）
- C++标准程序库读书笔记-第二章新的语言特性
- Linux 命令完全手册
- LDA算法和PCA算法的总结(原理和思想)
- html简单网页源代码表格,HTML 表格
- 经纬财富:昭通炒现货白银技巧
- Centos5反p2p模块安装精要
- 史上最全 Vue 前端代码风格指南
- cblas_sgemm和cublasSgemm参数详解
- 麦当劳和肯德鸡的笑话
- 《上瘾：让用户养成使用习惯的四大产品逻辑》读书笔记
- 数据库查询语句遇到:Unknown column 'XXXX' in 'where clause'解决方法
- 学生使用计算机的管理制度,学生使用计算机教室管理制度
- Windows图标显示异常解决方法。桌面图标异常，开始菜单图标异常，任务栏图标异常。图标缓存位置。
- 程序员面试系列（2）非计算机专业程序员