HiveServer2 频繁宕机根本原因分析
2024-06-04 08:02:47
文章目录
- 概要
- 一、进行heap dump
- 1.1 查HiveServer2的进程id，得到219045
- 1.2 根据进程号，查jstat的gc信息
- 二、heap dump分析
- 2.1 把hiveServer2的heap dump回来
- 2.2 使用eclipse memory analyzer进行分析
- 2.3 leak suspects
- 2.4 dominatorTree对象占用空间比例
- 2.5 整个jstat后台脚本，记录gc情况
- 3.结论与解决方法
- 总结
概要
问题描述： hiveServer2跑一段时间后, 一般在凌晨就会宕机（凌晨有很多定时任务），宕机比较有规律，就是跑几天就宕一次，分析一下原因。作业没有特别大的任务。
分析方法：jstat, dump
分析结论：hive2的hiveServer本身内存泄漏造成
解决方法：不解决或升级。
一、进行heap dump
1.1 查HiveServer2的进程id，得到219045
root@tbds-10-232-2-37:/data/var> ps -ef|grep HiveServer2
hive 219045 1 92 Feb17 ? 3-13:55:08 /usr/jdk64/jdk1.8.0_111/bin/java -Xmx16384m -Dhdp.version=2.2.0.0-2041 -Djava.net.preferIPv4Stack=true -Dhdp.version=2.2.0.0-2041 -Dhadoop.log.dir=/data/var/log/hadoop/hive -Dhadoop.log.file=hadoop.log -Dhadoop.home.dir=/data/bigdata/tbds/usr/hdp/2.2.0.0-2041/hadoop -Dhadoop.id.str=hive -Dhadoop.root.logger=INFO,console -Djava.library.path=:/usr/hdp/current/hadoop-client/lib/native/Linux-amd64-64:/data/bigdata/tbds/usr/hdp/2.2.0.0-2041/hadoop/lib/native -Dhadoop.policy.file=hadoop-policy.xml -Djava.net.preferIPv4Stack=true -Xmx16384m -Djava.security.auth.login.config=/etc/hive/conf.server/kafka_client_for_ranger_jaas.conf -XX:+UseG1GC -XX:InitiatingHeapOccupancyPercent=70 -XX:+PrintGCDateStamps -XX:GCLogFileSize=512M -verbose:gc -XX:+PrintGCDetails -Xloggc:/data/var/log/hive/hive_gc.log -Dlog4j.configurationFile=/etc/hive/conf/hive-log4j2.xml -Dhadoop.security.logger=INFO,NullAppender org.apache.hadoop.util.RunJar /usr/hdp/2.2.0.0-2041/hive/lib/hive-service-2.2.0.jar org.apache.hive.service.server.HiveServer2 --hiveconf hive.aux.jars.path=file:///usr/hdp/current/hive-webhcat/share/hcatalog/hive-hcatalog-core-2.2.0.jar,file:///usr/hdp/current/hive-webhcat/share/hcatalog/hive-hcatalog-core.jar,file:///usr/hdp/current/hive-webhcat/share/hcatalog/hive-hcatalog-pig-adapter-2.2.0.jar,file:///usr/hdp/current/hive-webhcat/share/hcatalog/hive-hcatalog-pig-adapter.jar,file:///usr/hdp/current/hive-webhcat/share/hcatalog/hive-hcatalog-server-extensions-2.2.0.jar,file:///usr/hdp/current/hive-webhcat/share/hcatalog/hive-hcatalog-server-extensions.jar,file:///usr/hdp/current/hive-webhcat/share/hcatalog/hive-hcatalog-streaming-2.2.0.jar,file:///usr/hdp/current/hive-webhcat/share/hcatalog/hive-hcatalog-streaming.jar -hiveconf hive.metastore.uris=
1.2 根据进程号，查jstat的gc信息
root@tbds-10-x-xx-x:/data/var>jstat -gcutil 219045 2000S0 S1 E O M CCS YGC YGCT FGC FGCT GCT 0.00 0.00 0.00 88.12 73.03 36.52 14741 3097.156 3 94.687 3191.8430.00 0.00 0.00 88.12 73.03 36.52 14741 3097.156 3 94.687 3191.8430.00 0.00 0.00 88.12 73.03 36.52 14741 3097.156 3 94.687 3191.8430.00 0.00 0.00 88.12 73.03 36.52 14741 3097.156 3 94.687 3191.843
发现fullGC后，老年代，还是很满(FullGC的次数由2变到3)。
艰难的fullGC,
经过28s的fullGC，顺利完成gc,但老年代92.19%，按照目前趋势 应该明天凌晨就跑挂，看看预言成真不。。。
二、heap dump分析
2.1 把hiveServer2的heap dump回来
经过FullGC后进行的heap dump.
-dump:live 只要存活的对象
hive@tbds-x-x-x-x:/data/var/hiveDump>jmap -dump:live,format=b,file=/data/var/hiveDump/hiveserver2.afterFgc.dump.bin 219045
Dumping heap to /data/var/hiveDump/hiveserver2.dump.bin ...
Heap dump file created
dump回来，发现文件大小17G。
2.2 使用eclipse memory analyzer进行分析
修改EMA的MemoryAnalyzer.ini，把Xmx改为28G, -Xmx28672m.(我的电脑32G)
2.3 leak suspects
发现问题：
问题1： 7627个org.apache.hadoop.hive.ql.exec.UDFClassLoader 占用4.1G内存
问题2：16585个log4jd的RandomAccessFile占用 4.1G内存
问题3：java.util.jar.JarFile占用2.9G
这三个对象把都实例放到HashMap上进行缓存，导致实例永远在FullGC时都可达，导致无法进行回收。
Shallow 与 Retained的区别
Shallow Size
Shallow Size是对象本身占据的内存的大小，不包含其引用的对象。对于常规对象（非数组）的Shallow Size由其成员变量的数量和类型来定，而数组的ShallowSize由数组类型和数组长度来决定，它为数组元素大小的总和。
Retained Size
Retained Size=当前对象大小+当前对象可直接或间接引用到的对象的大小总和。(间接引用的含义：A->B->C,C就是间接引用) ，并且排除被GC Roots直接或者间接引用的对象
换句话说，Retained Size就是当前对象被GC后，从Heap上总共能释放掉的内存。
不过，释放的时候还要排除被GC Roots直接或间接引用的对象。他们暂时不会被被当做Garbage。
最大的retain heap
有busy monitor thread
2.4 dominatorTree对象占用空间比例
下图中，大的对象，无法回收
第一个MedatadataProvider一看就是里面用了享元模式（缓存），数据是无法回收的。其他系统类无法回收，最后的UDFClassLoader占用589k，看上去不大，但有7627个（2.3的问题1中分析出这个数字）。7627个*589k/1024/1024=4.28G。恐怖，全部无法回收。
2.5 整个jstat后台脚本，记录gc情况
脚本
root@tbds-x-x-x-37:/data/var>s nohup jstat -gcutil 219045 2000 > /data/var/hiveserver.gcutil.20220221.log 2>&1 &
第二天上班，发现的确凌晨挂了。
挂前的jstat信息：
S0 S1 E O M CCS YGC YGCT FGC FGCT GCT 0.00 0.00 0.00 99.99 71.97 35.45 21507 5736.954 547 16373.339 22110.2930.00 0.00 62.14 99.99 71.97 35.45 21508 5736.954 547 16403.868 22140.8220.00 0.00 0.00 99.99 71.99 35.46 21510 5738.118 548 16403.868 22141.9860.00 0.00 0.00 99.99 71.99 35.46 21510 5738.118 548 16403.868 22141.9860.00 0.00 0.00 99.99 71.99 35.46 21510 5738.118 548 16403.868 22141.9860.00 0.00 0.00 99.99 71.99 35.46 21510 5738.118 548 16403.868 22141.9860.00 0.00 0.00 99.99 71.99 35.46 21510 5738.118 548 16403.868 22141.9860.00 0.00 0.00 99.99 71.99 35.46 21510 5738.118 548 16403.868 22141.9860.00 0.00 0.00 99.99 71.99 35.46 21510 5738.118 548 16403.868 22141.9860.00 0.00 0.00 99.99 71.99 35.46 21510 5738.118 548 16403.868 22141.9860.00 0.00 0.00 99.99 71.99 35.46 21510 5738.118 548 16403.868 22141.9860.00 0.00 0.00 99.99 71.99 35.46 21510 5738.118 548 16403.868 22141.9860.00 0.00 0.00 99.99 71.99 35.46 21510 5738.118 548 16403.868 22141.9860.00 0.00 0.00 99.99 71.99 35.46 21510 5738.118 548 16403.868 22141.9860.00 0.00 0.00 99.99 71.99 35.46 21510 5738.118 548 16403.868 22141.9860.00 0.00 0.00 99.99 71.99 35.46 21510 5738.118 548 16403.868 22141.9860.00 0.00 0.00 99.99 71.99 35.46 21510 5738.118 548 16403.868 22141.9860.00 0.00 0.00 99.99 71.99 35.46 21510 5738.118 548 16403.868 22141.9860.00 0.00 61.76 100.00 71.97 35.46 21511 5738.118 548 16436.419 22174.5380.00 0.00 0.00 100.00 71.98 35.46 21516 5740.012 548 16436.419 22176.4310.00 0.00 0.00 100.00 71.98 35.46 21517 5740.730 549 16436.419 22177.1490.00 0.00 0.00 100.00 71.98 35.46 21517 5740.730 549 16436.419 22177.1490.00 0.00 0.00 100.00 71.98 35.46 21517 5740.730 549 16436.419 22177.1490.00 0.00 0.00 100.00 71.98 35.46 21517 5740.730 549 16436.419 22177.1490.00 0.00 0.00 100.00 71.98 35.46 21517 5740.730 549 16436.419 22177.1490.00 0.00 0.00 100.00 71.98 35.46 21517 5740.730 549 16436.419 22177.1490.00 0.00 0.00 100.00 71.98 35.46 21517 5740.730 549 16436.419 22177.1490.00 0.00 0.00 100.00 71.98 35.46 21517 5740.730 549 16436.419 22177.1490.00 0.00 0.00 100.00 71.98 35.46 21517 5740.730 549 16436.419 22177.1490.00 0.00 0.00 100.00 71.98 35.46 21517 5740.730 549 16436.419 22177.1490.00 0.00 0.00 100.00 71.98 35.46 21517 5740.730 549 16436.419 22177.1490.00 0.00 0.00 100.00 71.98 35.46 21517 5740.730 549 16436.419 22177.1490.00 0.00 0.00 100.00 71.98 35.46 21517 5740.730 549 16436.419 22177.1490.00 0.00 0.00 100.00 71.98 35.46 21517 5740.730 549 16436.419 22177.1490.00 0.00 0.00 100.00 71.98 35.46 21517 5740.730 549 16436.419 22177.1490.00 0.00 0.00 99.98 71.98 35.46 21518 5741.179 550 16466.653 22207.8330.00 0.00 0.00 99.98 71.98 35.46 21518 5741.179 550 16466.653 22207.8330.00 0.00 0.00 99.98 71.98 35.46 21518 5741.179 550 16466.653 22207.8330.00 0.00 0.00 99.98 71.98 35.46 21518 5741.179 550 16466.653 22207.8330.00 0.00 0.00 99.98 71.98 35.46 21518 5741.179 550 16466.653 22207.8330.00 0.00 0.00 99.98 71.98 35.46 21518 5741.179 550 16466.653 22207.8330.00 0.00 0.00 99.98 71.98 35.46 21518 5741.179 550 16466.653 22207.8330.00 0.00 0.00 99.98 71.98 35.46 21518 5741.179 550 16466.653 22207.8330.00 0.00 0.00 99.98 71.98 35.46 21518 5741.179 550 16466.653 22207.8330.00 0.00 0.00 99.98 71.98 35.46 21518 5741.179 550 16466.653 22207.8330.00 0.00 0.00 99.98 71.98 35.46 21518 5741.179 550 16466.653 22207.8330.00 0.00 0.00 99.98 71.98 35.46 21518 5741.179 550 16466.653 22207.8330.00 0.00 0.00 99.98 71.98 35.46 21518 5741.179 550 16466.653 22207.8330.00 0.00 0.00 99.98 71.98 35.46 21518 5741.179 550 16466.653 22207.8330.00 0.00 0.00 99.98 71.98 35.46 21518 5741.179 550 16466.653 22207.8330.00 0.00 62.00 99.98 71.98 35.46 21519 5741.179 550 16496.641 22237.8210.00 0.00 0.00 99.98 71.98 35.46 21520 5741.951 551 16496.641 22238.5920.00 0.00 0.00 99.98 71.98 35.46 21520 5741.951 551 16496.641 22238.5920.00 0.00 0.00 99.98 71.98 35.46 21520 5741.951 551 16496.641 22238.5920.00 0.00 0.00 99.98 71.98 35.46 21520 5741.951 551 16496.641 22238.5920.00 0.00 0.00 99.98 71.98 35.46 21520 5741.951 551 16496.641 22238.5920.00 0.00 0.00 99.98 71.98 35.46 21520 5741.951 551 16496.641 22238.5920.00 0.00 0.00 99.98 71.98 35.46 21520 5741.951 551 16496.641 22238.5920.00 0.00 0.00 99.98 71.98 35.46 21520 5741.951 551 16496.641 22238.5920.00 0.00 0.00 99.98 71.98 35.46 21520 5741.951 551 16496.641 22238.5920.00 0.00 0.00 99.98 71.98 35.46 21520 5741.951 551 16496.641 22238.5920.00 0.00 0.00 99.98 71.98 35.46 21520 5741.951 551 16496.641 22238.5920.00 0.00 0.00 99.98 71.98 35.46 21520 5741.951 551 16496.641 22238.5920.00 0.00 0.00 99.98 71.98 35.46 21520 5741.951 551 16496.641 22238.5920.00 0.00 0.00 99.98 71.98 35.46 21520 5741.951 551 16496.641 22238.5920.00 0.00 0.00 99.98 71.98 35.46 21520 5741.951 551 16496.641 22238.5920.00 0.00 0.00 99.98 71.99 35.46 21524 5742.970 551 16526.578 22269.5490.00 0.00 0.00 99.98 71.99 35.46 21527 5744.093 552 16526.578 22270.6720.00 0.00 0.00 99.98 71.99 35.46 21527 5744.093 552 16526.578 22270.6720.00 0.00 0.00 99.98 71.99 35.46 21527 5744.093 552 16526.578 22270.6720.00 0.00 0.00 99.98 71.99 35.46 21527 5744.093 552 16526.578 22270.6720.00 0.00 0.00 99.98 71.99 35.46 21527 5744.093 552 16526.578 22270.6720.00 0.00 0.00 99.98 71.99 35.46 21527 5744.093 552 16526.578 22270.6720.00 0.00 0.00 99.98 71.99 35.46 21527 5744.093 552 16526.578 22270.6720.00 0.00 0.00 99.98 71.99 35.46 21527 5744.093 552 16526.578 22270.6720.00 0.00 0.00 99.98 71.99 35.46 21527 5744.093 552 16526.578 22270.6720.00 0.00 0.00 99.98 71.99 35.46 21527 5744.093 552 16526.578 22270.6720.00 0.00 0.00 99.98 71.99 35.46 21527 5744.093 552 16526.578 22270.6720.00 0.00 0.00 99.98 71.99 35.46 21527 5744.093 552 16526.578 22270.6720.00 0.00 0.00 99.98 71.99 35.46 21527 5744.093 552 16526.578 22270.6720.00 0.00 0.00 99.98 71.99 35.46 21527 5744.093 552 16526.578 22270.6720.00 0.00 0.00 99.98 71.99 35.46 21527 5744.093 552 16526.578 22270.6720.00 0.00 0.00 99.99 71.99 35.47 21531 5745.369 552 16555.798 22301.1680.00 0.00 0.00 99.99 71.99 35.47 21533 5746.465 553 16555.798 22302.2630.00 0.00 0.00 99.99 71.99 35.47 21533 5746.465 553 16555.798 22302.2630.00 0.00 0.00 99.99 71.99 35.47 21533 5746.465 553 16555.798 22302.2630.00 0.00 0.00 99.99 71.99 35.47 21533 5746.465 553 16555.798 22302.2630.00 0.00 0.00 99.99 71.99 35.47 21533 5746.465 553 16555.798 22302.2630.00 0.00 0.00 99.99 71.99 35.47 21533 5746.465 553 16555.798 22302.2630.00 0.00 0.00 99.99 71.99 35.47 21533 5746.465 553 16555.798 22302.2630.00 0.00 0.00 99.99 71.99 35.47 21533 5746.465 553 16555.798 22302.2630.00 0.00 0.00 99.99 71.99 35.47 21533 5746.465 553 16555.798 22302.2630.00 0.00 0.00 99.99 71.99 35.47 21533 5746.465 553 16555.798 22302.2630.00 0.00 0.00 99.99 71.99 35.47 21533 5746.465 553 16555.798 22302.2630.00 0.00 0.00 99.99 71.99 35.47 21533 5746.465 553 16555.798 22302.2630.00 0.00 0.00 99.99 71.99 35.47 21533 5746.465 553 16555.798 22302.2630.00 0.00 0.00 99.99 71.99 35.47 21533 5746.465 553 16555.798 22302.2630.00 0.00 0.00 99.99 71.99 35.47 21533 5746.465 553 16555.798 22302.2630.00 0.00 17.35 99.99 71.99 35.46 21534 5746.465 553 16586.625 22333.0900.00 0.00 0.00 99.99 71.99 35.47 21540 5748.518 553 16586.625 22335.1430.00 0.00 0.00 99.99 71.99 35.47 21541 5749.257 554 16586.625 22335.8810.00 0.00 0.00 99.99 71.99 35.47 21541 5749.257 554 16586.625 22335.8810.00 0.00 0.00 99.99 71.99 35.47 21541 5749.257 554 16586.625 22335.8810.00 0.00 0.00 99.99 71.99 35.47 21541 5749.257 554 16586.625 22335.8810.00 0.00 0.00 99.99 71.99 35.47 21541 5749.257 554 16586.625 22335.8810.00 0.00 0.00 99.99 71.99 35.47 21541 5749.257 554 16586.625 22335.8810.00 0.00 0.00 99.99 71.99 35.47 21541 5749.257 554 16586.625 22335.8810.00 0.00 0.00 99.99 71.99 35.47 21541 5749.257 554 16586.625 22335.8810.00 0.00 0.00 99.99 71.99 35.47 21541 5749.257 554 16586.625 22335.8810.00 0.00 0.00 99.99 71.99 35.47 21541 5749.257 554 16586.625 22335.8810.00 0.00 0.00 99.99 71.99 35.47 21541 5749.257 554 16586.625 22335.8810.00 0.00 0.00 99.99 71.99 35.47 21541 5749.257 554 16586.625 22335.8810.00 0.00 0.00 99.99 71.99 35.47 21541 5749.257 554 16586.625 22335.8810.00 0.00 0.00 99.99 71.99 35.47 21541 5749.257 554 16586.625 22335.8810.00 0.00 0.00 99.99 71.99 35.47 21541 5749.257 554 16586.625 22335.8810.00 0.00 0.00 99.98 71.99 35.46 21545 5750.336 554 16616.478 22366.8130.00 0.00 0.00 99.98 71.99 35.46 21546 5751.022 555 16616.478 22367.5000.00 0.00 0.00 99.98 71.99 35.46 21546 5751.022 555 16616.478 22367.5000.00 0.00 0.00 99.98 71.99 35.46 21546 5751.022 555 16616.478 22367.5000.00 0.00 0.00 99.98 71.99 35.46 21546 5751.022 555 16616.478 22367.5000.00 0.00 0.00 99.98 71.99 35.46 21546 5751.022 555 16616.478 22367.5000.00 0.00 0.00 99.98 71.99 35.46 21546 5751.022 555 16616.478 22367.5000.00 0.00 0.00 99.98 71.99 35.46 21546 5751.022 555 16616.478 22367.5000.00 0.00 0.00 99.98 71.99 35.46 21546 5751.022 555 16616.478 22367.5000.00 0.00 0.00 99.98 71.99 35.46 21546 5751.022 555 16616.478 22367.5000.00 0.00 0.00 99.98 71.99 35.46 21546 5751.022 555 16616.478 22367.5000.00 0.00 0.00 99.98 71.99 35.46 21546 5751.022 555 16616.478 22367.5000.00 0.00 0.00 99.98 71.99 35.46 21546 5751.022 555 16616.478 22367.5000.00 0.00 0.00 99.98 71.99 35.46 21546 5751.022 555 16616.478 22367.5000.00 0.00 0.00 99.98 71.99 35.46 21546 5751.022 555 16616.478 22367.5000.00 0.00 0.00 99.98 71.99 35.46 21546 5751.022 555 16616.478 22367.5000.00 0.00 0.00 99.99 71.98 35.46 21550 5752.389 556 16646.145 22398.5330.00 0.00 0.00 99.99 71.98 35.46 21550 5752.389 556 16646.145 22398.5330.00 0.00 0.00 99.99 71.98 35.46 21550 5752.389 556 16646.145 22398.5330.00 0.00 0.00 99.99 71.98 35.46 21550 5752.389 556 16646.145 22398.5330.00 0.00 0.00 99.99 71.98 35.46 21550 5752.389 556 16646.145 22398.5330.00 0.00 0.00 99.99 71.98 35.46 21550 5752.389 556 16646.145 22398.5330.00 0.00 0.00 99.99 71.98 35.46 21550 5752.389 556 16646.145 22398.5330.00 0.00 0.00 99.99 71.98 35.46 21550 5752.389 556 16646.145 22398.5330.00 0.00 0.00 99.99 71.98 35.46 21550 5752.389 556 16646.145 22398.5330.00 0.00 0.00 99.99 71.98 35.46 21550 5752.389 556 16646.145 22398.5330.00 0.00 0.00 99.99 71.98 35.46 21550 5752.389 556 16646.145 22398.5330.00 0.00 0.00 99.99 71.98 35.46 21550 5752.389 556 16646.145 22398.5330.00 0.00 0.00 99.99 71.98 35.46 21550 5752.389 556 16646.145 22398.5330.00 0.00 0.00 99.99 71.98 35.46 21550 5752.389 556 16646.145 22398.5330.00 0.00 0.00 99.99 71.98 35.46 21550 5752.389 556 16646.145 22398.5330.00 0.00 0.00 99.99 71.98 35.46 21550 5752.389 556 16646.145 22398.5330.00 0.00 0.00 99.99 71.98 35.46 21550 5752.389 556 16646.145 22398.5330.00 0.00 0.00 99.99 71.98 35.46 21553 5753.008 556 16679.451 22432.4590.00 0.00 0.00 99.99 71.98 35.46 21555 5754.260 557 16679.451 22433.7110.00 0.00 0.00 99.99 71.98 35.46 21555 5754.260 557 16679.451 22433.7110.00 0.00 0.00 99.99 71.98 35.46 21555 5754.260 557 16679.451 22433.7110.00 0.00 0.00 99.99 71.98 35.46 21555 5754.260 557 16679.451 22433.7110.00 0.00 0.00 99.99 71.98 35.46 21555 5754.260 557 16679.451 22433.7110.00 0.00 0.00 99.99 71.98 35.46 21555 5754.260 557 16679.451 22433.7110.00 0.00 0.00 99.99 71.98 35.46 21555 5754.260 557 16679.451 22433.7110.00 0.00 0.00 99.99 71.98 35.46 21555 5754.260 557 16679.451 22433.7110.00 0.00 0.00 99.99 71.98 35.46 21555 5754.260 557 16679.451 22433.7110.00 0.00 0.00 99.99 71.98 35.46 21555 5754.260 557 16679.451 22433.7110.00 0.00 0.00 99.99 71.98 35.46 21555 5754.260 557 16679.451 22433.7110.00 0.00 0.00 99.99 71.98 35.46 21555 5754.260 557 16679.451 22433.7110.00 0.00 0.00 99.99 71.98 35.46 21555 5754.260 557 16679.451 22433.7110.00 0.00 0.00 99.99 71.98 35.46 21555 5754.260 557 16679.451 22433.7110.00 0.00 0.00 99.99 71.98 35.46 21555 5754.260 557 16679.451 22433.7110.00 0.00 0.00 99.98 71.98 35.46 21560 5755.845 558 16708.417 22464.2620.00 0.00 0.00 99.98 71.98 35.46 21560 5755.845 558 16708.417 22464.2620.00 0.00 0.00 99.98 71.98 35.46 21560 5755.845 558 16708.417 22464.2620.00 0.00 0.00 99.98 71.98 35.46 21560 5755.845 558 16708.417 22464.2620.00 0.00 0.00 99.98 71.98 35.46 21560 5755.845 558 16708.417 22464.2620.00 0.00 0.00 99.98 71.98 35.46 21560 5755.845 558 16708.417 22464.2620.00 0.00 0.00 99.98 71.98 35.46 21560 5755.845 558 16708.417 22464.2620.00 0.00 0.00 99.98 71.98 35.46 21560 5755.845 558 16708.417 22464.2620.00 0.00 0.00 99.98 71.98 35.46 21560 5755.845 558 16708.417 22464.2620.00 0.00 0.00 99.98 71.98 35.46 21560 5755.845 558 16708.417 22464.2620.00 0.00 0.00 99.98 71.98 35.46 21560 5755.845 558 16708.417 22464.2620.00 0.00 0.00 99.98 71.98 35.46 21560 5755.845 558 16708.417 22464.2620.00 0.00 0.00 99.98 71.98 35.46 21560 5755.845 558 16708.417 22464.2620.00 0.00 0.00 99.98 71.98 35.46 21560 5755.845 558 16708.417 22464.2620.00 0.00 59.14 99.98 71.98 35.46 21561 5755.845 558 16736.086 22491.9310.00 0.00 0.00 99.98 71.98 35.46 21562 5756.516 559 16736.086 22492.6020.00 0.00 0.00 99.98 71.98 35.46 21562 5756.516 559 16736.086 22492.6020.00 0.00 0.00 99.98 71.98 35.46 21562 5756.516 559 16736.086 22492.6020.00 0.00 0.00 99.98 71.98 35.46 21562 5756.516 559 16736.086 22492.6020.00 0.00 0.00 99.98 71.98 35.46 21562 5756.516 559 16736.086 22492.6020.00 0.00 0.00 99.98 71.98 35.46 21562 5756.516 559 16736.086 22492.6020.00 0.00 0.00 99.98 71.98 35.46 21562 5756.516 559 16736.086 22492.6020.00 0.00 0.00 99.98 71.98 35.46 21562 5756.516 559 16736.086 22492.6020.00 0.00 0.00 99.98 71.98 35.46 21562 5756.516 559 16736.086 22492.6020.00 0.00 0.00 99.98 71.98 35.46 21562 5756.516 559 16736.086 22492.6020.00 0.00 0.00 99.98 71.98 35.46 21562 5756.516 559 16736.086 22492.6020.00 0.00 0.00 99.98 71.98 35.46 21562 5756.516 559 16736.086 22492.6020.00 0.00 0.00 99.98 71.98 35.46 21562 5756.516 559 16736.086 22492.6020.00 100.00 52.22 100.00 71.98 35.46 21563 5756.977 559 16762.848 22519.8250.00 0.00 0.00 99.99 71.97 35.44 21569 5758.853 559 16762.848 22521.7000.00 0.00 0.00 99.99 71.97 35.44 21571 5759.968 560 16762.848 22522.8160.00 0.00 0.00 99.99 71.97 35.44 21571 5759.968 560 16762.848 22522.8160.00 0.00 0.00 99.99 71.97 35.44 21571 5759.968 560 16762.848 22522.8160.00 0.00 0.00 99.99 71.97 35.44 21571 5759.968 560 16762.848 22522.8160.00 0.00 0.00 99.99 71.97 35.44 21571 5759.968 560 16762.848 22522.8160.00 0.00 0.00 99.99 71.97 35.44 21571 5759.968 560 16762.848 22522.8160.00 0.00 0.00 99.99 71.97 35.44 21571 5759.968 560 16762.848 22522.8160.00 0.00 0.00 99.99 71.97 35.44 21571 5759.968 560 16762.848 22522.8160.00 0.00 0.00 99.99 71.97 35.44 21571 5759.968 560 16762.848 22522.8160.00 0.00 0.00 99.99 71.97 35.44 21571 5759.968 560 16762.848 22522.8160.00 0.00 0.00 99.99 71.97 35.44 21571 5759.968 560 16762.848 22522.8160.00 0.00 0.00 99.99 71.97 35.44 21571 5759.968 560 16762.848 22522.8160.00 0.00 0.00 99.99 71.97 35.44 21571 5759.968 560 16762.848 22522.8160.00 0.00 0.00 99.99 71.97 35.44 21571 5759.968 560 16762.848 22522.8160.00 0.00 0.00 99.99 71.97 35.44 21573 5760.294 560 16791.153 22551.4470.00 0.00 0.00 99.99 71.97 35.44 21577 5762.010 560 16791.153 22553.1630.00 0.00 0.00 99.99 71.97 35.44 21578 5762.655 561 16791.153 22553.8080.00 0.00 0.00 99.99 71.97 35.44 21578 5762.655 561 16791.153 22553.8080.00 0.00 0.00 99.99 71.97 35.44 21578 5762.655 561 16791.153 22553.8080.00 0.00 0.00 99.99 71.97 35.44 21578 5762.655 561 16791.153 22553.8080.00 0.00 0.00 99.99 71.97 35.44 21578 5762.655 561 16791.153 22553.8080.00 0.00 0.00 99.99 71.97 35.44 21578 5762.655 561 16791.153 22553.8080.00 0.00 0.00 99.99 71.97 35.44 21578 5762.655 561 16791.153 22553.8080.00 0.00 0.00 99.99 71.97 35.44 21578 5762.655 561 16791.153 22553.8080.00 0.00 0.00 99.99 71.97 35.44 21578 5762.655 561 16791.153 22553.8080.00 0.00 0.00 99.99 71.97 35.44 21578 5762.655 561 16791.153 22553.8080.00 0.00 0.00 99.99 71.97 35.44 21578 5762.655 561 16791.153 22553.8080.00 0.00 0.00 99.99 71.97 35.44 21578 5762.655 561 16791.153 22553.8080.00 0.00 0.00 99.99 71.97 35.44 21578 5762.655 561 16791.153 22553.8080.00 0.00 0.00 99.99 71.97 35.44 21578 5762.655 561 16791.153 22553.8080.00 0.00 58.51 100.00 71.97 35.44 21579 5762.655 561 16820.409 22583.0640.00 0.00 0.00 100.00 71.98 35.44 21584 5764.774 562 16820.409 22585.1830.00 0.00 0.00 100.00 71.98 35.44 21584 5764.774 562 16820.409 22585.1830.00 0.00 0.00 100.00 71.98 35.44 21584 5764.774 562 16820.409 22585.1830.00 0.00 0.00 100.00 71.98 35.44 21584 5764.774 562 16820.409 22585.1830.00 0.00 0.00 100.00 71.98 35.44 21584 5764.774 562 16820.409 22585.1830.00 0.00 0.00 100.00 71.98 35.44 21584 5764.774 562 16820.409 22585.1830.00 0.00 0.00 100.00 71.98 35.44 21584 5764.774 562 16820.409 22585.1830.00 0.00 0.00 100.00 71.98 35.44 21584 5764.774 562 16820.409 22585.1830.00 0.00 0.00 100.00 71.98 35.44 21584 5764.774 562 16820.409 22585.1830.00 0.00 0.00 100.00 71.98 35.44 21584 5764.774 562 16820.409 22585.1830.00 0.00 0.00 100.00 71.98 35.44 21584 5764.774 562 16820.409 22585.1830.00 0.00 0.00 100.00 71.98 35.44 21584 5764.774 562 16820.409 22585.1830.00 0.00 0.00 100.00 71.98 35.44 21584 5764.774 562 16820.409 22585.1830.00 0.00 0.00 100.00 71.98 35.44 21584 5764.774 562 16820.409 22585.1830.00 0.00 0.00 100.00 71.98 35.44 21584 5764.774 562 16820.409 22585.1830.00 0.00 0.00 100.00 71.98 35.44 21584 5764.774 562 16820.409 22585.1830.00 0.00 9.57 99.98 71.98 35.44 21584 5764.774 562 16852.386 22617.1600.00 0.00 0.00 99.98 71.97 35.45 21586 5765.560 563 16852.386 22617.9460.00 0.00 0.00 99.98 71.97 35.45 21586 5765.560 563 16852.386 22617.9460.00 0.00 0.00 99.98 71.97 35.45 21586 5765.560 563 16852.386 22617.9460.00 0.00 0.00 99.98 71.97 35.45 21586 5765.560 563 16852.386 22617.9460.00 0.00 0.00 99.98 71.97 35.45 21586 5765.560 563 16852.386 22617.9460.00 0.00 0.00 99.98 71.97 35.45 21586 5765.560 563 16852.386 22617.9460.00 0.00 0.00 99.98 71.97 35.45 21586 5765.560 563 16852.386 22617.9460.00 0.00 0.00 99.98 71.97 35.45 21586 5765.560 563 16852.386 22617.9460.00 0.00 0.00 99.98 71.97 35.45 21586 5765.560 563 16852.386 22617.9460.00 0.00 0.00 99.98 71.97 35.45 21586 5765.560 563 16852.386 22617.9460.00 0.00 0.00 99.98 71.97 35.45 21586 5765.560 563 16852.386 22617.9460.00 0.00 0.00 99.98 71.97 35.45 21586 5765.560 563 16852.386 22617.9460.00 0.00 0.00 99.98 71.97 35.45 21586 5765.560 563 16852.386 22617.9460.00 0.00 0.00 99.98 71.97 35.45 21586 5765.560 563 16852.386 22617.9460.00 0.00 50.54 99.98 71.96 35.45 21586 5765.560 563 16881.137 22646.6970.00 0.00 0.00 99.98 71.97 35.45 21592 5767.328 563 16881.137 22648.4650.00 0.00 0.00 99.98 71.97 35.45 21592 5767.741 564 16881.137 22648.8780.00 0.00 0.00 99.98 71.97 35.45 21592 5767.741 564 16881.137 22648.8780.00 0.00 0.00 99.98 71.97 35.45 21592 5767.741 564 16881.137 22648.8780.00 0.00 0.00 99.98 71.97 35.45 21592 5767.741 564 16881.137 22648.8780.00 0.00 0.00 99.98 71.97 35.45 21592 5767.741 564 16881.137 22648.8780.00 0.00 0.00 99.98 71.97 35.45 21592 5767.741 564 16881.137 22648.8780.00 0.00 0.00 99.98 71.97 35.45 21592 5767.741 564 16881.137 22648.8780.00 0.00 0.00 99.98 71.97 35.45 21592 5767.741 564 16881.137 22648.8780.00 0.00 0.00 99.98 71.97 35.45 21592 5767.741 564 16881.137 22648.8780.00 0.00 0.00 99.98 71.97 35.45 21592 5767.741 564 16881.137 22648.8780.00 0.00 0.00 99.98 71.97 35.45 21592 5767.741 564 16881.137 22648.8780.00 0.00 0.00 99.98 71.97 35.45 21592 5767.741 564 16881.137 22648.8780.00 0.00 0.00 99.98 71.97 35.45 21592 5767.741 564 16881.137 22648.8780.00 0.00 0.00 99.98 71.97 35.45 21592 5767.741 564 16881.137 22648.8780.00 0.00 0.00 99.99 71.98 35.46 21593 5768.156 565 16909.378 22677.5330.00 0.00 0.00 99.99 71.98 35.46 21593 5768.156 565 16909.378 22677.5330.00 0.00 0.00 99.99 71.98 35.46 21593 5768.156 565 16909.378 22677.5330.00 0.00 0.00 99.99 71.98 35.46 21593 5768.156 565 16909.378 22677.5330.00 0.00 0.00 99.99 71.98 35.46 21593 5768.156 565 16909.378 22677.5330.00 0.00 0.00 99.99 71.98 35.46 21593 5768.156 565 16909.378 22677.5330.00 0.00 0.00 99.99 71.98 35.46 21593 5768.156 565 16909.378 22677.5330.00 0.00 0.00 99.99 71.98 35.46 21593 5768.156 565 16909.378 22677.5330.00 0.00 0.00 99.99 71.98 35.46 21593 5768.156 565 16909.378 22677.5330.00 0.00 0.00 99.99 71.98 35.46 21593 5768.156 565 16909.378 22677.5330.00 0.00 0.00 99.99 71.98 35.46 21593 5768.156 565 16909.378 22677.5330.00 0.00 0.00 99.99 71.98 35.46 21593 5768.156 565 16909.378 22677.5330.00 0.00 0.00 99.99 71.98 35.46 21593 5768.156 565 16909.378 22677.5330.00 0.00 0.00 99.99 71.98 35.46 21593 5768.156 565 16909.378 22677.5330.00 0.00 0.00 99.99 71.98 35.46 21593 5768.156 565 16909.378 22677.5330.00 0.00 0.00 100.00 71.98 35.46 21596 5769.031 566 16938.490 22707.5210.00 0.00 0.00 100.00 71.98 35.46 21596 5769.031 566 16938.490 22707.5210.00 0.00 0.00 100.00 71.98 35.46 21596 5769.031 566 16938.490 22707.5210.00 0.00 0.00 100.00 71.98 35.46 21596 5769.031 566 16938.490 22707.5210.00 0.00 0.00 100.00 71.98 35.46 21596 5769.031 566 16938.490 22707.5210.00 0.00 0.00 100.00 71.98 35.46 21596 5769.031 566 16938.490 22707.5210.00 0.00 0.00 100.00 71.98 35.46 21596 5769.031 566 16938.490 22707.5210.00 0.00 0.00 100.00 71.98 35.46 21596 5769.031 566 16938.490 22707.5210.00 0.00 0.00 100.00 71.98 35.46 21596 5769.031 566 16938.490 22707.5210.00 0.00 0.00 100.00 71.98 35.46 21596 5769.031 566 16938.490 22707.5210.00 0.00 0.00 100.00 71.98 35.46 21596 5769.031 566 16938.490 22707.5210.00 0.00 0.00 100.00 71.98 35.46 21596 5769.031 566 16938.490 22707.5210.00 0.00 0.00 100.00 71.98 35.46 21596 5769.031 566 16938.490 22707.5210.00 0.00 0.00 100.00 71.98 35.46 21596 5769.031 566 16938.490 22707.5210.00 0.00 0.00 100.00 71.98 35.46 21596 5769.031 566 16938.490 22707.5210.00 0.00 0.00 99.99 71.97 35.45 21599 5769.848 566 16968.008 22737.8560.00 0.00 0.00 99.99 71.97 35.45 21604 5771.600 566 16968.008 22739.6080.00 0.00 0.00 99.99 71.97 35.45 21604 5771.878 567 16968.008 22739.8860.00 0.00 0.00 99.99 71.97 35.45 21604 5771.878 567 16968.008 22739.8860.00 0.00 0.00 99.99 71.97 35.45 21604 5771.878 567 16968.008 22739.8860.00 0.00 0.00 99.99 71.97 35.45 21604 5771.878 567 16968.008 22739.8860.00 0.00 0.00 99.99 71.97 35.45 21604 5771.878 567 16968.008 22739.8860.00 0.00 0.00 99.99 71.97 35.45 21604 5771.878 567 16968.008 22739.8860.00 0.00 0.00 99.99 71.97 35.45 21604 5771.878 567 16968.008 22739.8860.00 0.00 0.00 99.99 71.97 35.45 21604 5771.878 567 16968.008 22739.8860.00 0.00 0.00 99.99 71.97 35.45 21604 5771.878 567 16968.008 22739.8860.00 0.00 0.00 99.99 71.97 35.45 21604 5771.878 567 16968.008 22739.8860.00 0.00 0.00 99.99 71.97 35.45 21604 5771.878 567 16968.008 22739.8860.00 0.00 0.00 99.99 71.97 35.45 21604 5771.878 567 16968.008 22739.8860.00 0.00 0.00 99.99 71.97 35.45 21604 5771.878 567 16968.008 22739.8860.00 0.00 0.00 99.99 71.97 35.45 21604 5771.878 567 16968.008 22739.8860.00 0.00 0.00 99.99 71.97 35.45 21604 5771.878 567 16968.008 22739.8860.00 0.00 0.00 99.99 71.97 35.45 21604 5771.878 567 16968.008 22739.8860.00 0.00 57.61 100.00 71.97 35.45 21605 5771.878 567 17001.663 22773.5410.00 0.00 0.00 100.00 71.97 35.45 21610 5773.741 567 17001.663 22775.4040.00 0.00 0.00 100.00 71.97 35.45 21615 5775.840 568 17001.663 22777.5030.00 0.00 0.00 100.00 71.97 35.45 21615 5775.840 568 17001.663 22777.5030.00 0.00 0.00 100.00 71.97 35.45 21615 5775.840 568 17001.663 22777.5030.00 0.00 0.00 100.00 71.97 35.45 21615 5775.840 568 17001.663 22777.5030.00 0.00 0.00 100.00 71.97 35.45 21615 5775.840 568 17001.663 22777.5030.00 0.00 0.00 100.00 71.97 35.45 21615 5775.840 568 17001.663 22777.5030.00 0.00 0.00 100.00 71.97 35.45 21615 5775.840 568 17001.663 22777.5030.00 0.00 0.00 100.00 71.97 35.45 21615 5775.840 568 17001.663 22777.5030.00 0.00 0.00 100.00 71.97 35.45 21615 5775.840 568 17001.663 22777.5030.00 0.00 0.00 100.00 71.97 35.45 21615 5775.840 568 17001.663 22777.5030.00 0.00 0.00 100.00 71.97 35.45 21615 5775.840 568 17001.663 22777.5030.00 0.00 0.00 100.00 71.97 35.45 21615 5775.840 568 17001.663 22777.5030.00 0.00 0.00 100.00 71.97 35.45 21615 5775.840 568 17001.663 22777.5030.00 0.00 0.00 100.00 71.97 35.45 21615 5775.840 568 17001.663 22777.5030.00 0.00 0.00 100.00 71.97 35.45 21615 5775.840 568 17001.663 22777.5030.00 0.00 0.00 100.00 71.97 35.45 21615 5775.840 568 17001.663 22777.5030.00 0.00 0.00 100.00 71.97 35.45 21619 5776.975 568 17032.080 22809.0550.00 0.00 0.00 100.00 71.97 35.45 21620 5777.708 569 17032.080 22809.7880.00 0.00 0.00 100.00 71.97 35.45 21620 5777.708 569 17032.080 22809.7880.00 0.00 0.00 100.00 71.97 35.45 21620 5777.708 569 17032.080 22809.7880.00 0.00 0.00 100.00 71.97 35.45 21620 5777.708 569 17032.080 22809.7880.00 0.00 0.00 100.00 71.97 35.45 21620 5777.708 569 17032.080 22809.7880.00 0.00 0.00 100.00 71.97 35.45 21620 5777.708 569 17032.080 22809.7880.00 0.00 0.00 100.00 71.97 35.45 21620 5777.708 569 17032.080 22809.7880.00 0.00 0.00 100.00 71.97 35.45 21620 5777.708 569 17032.080 22809.7880.00 0.00 0.00 100.00 71.97 35.45 21620 5777.708 569 17032.080 22809.7880.00 0.00 0.00 100.00 71.97 35.45 21620 5777.708 569 17032.080 22809.7880.00 0.00 0.00 100.00 71.97 35.45 21620 5777.708 569 17032.080 22809.7880.00 0.00 0.00 100.00 71.97 35.45 21620 5777.708 569 17032.080 22809.7880.00 0.00 0.00 100.00 71.97 35.45 21620 5777.708 569 17032.080 22809.7880.00 0.00 0.00 100.00 71.97 35.45 21620 5777.708 569 17032.080 22809.7880.00 0.00 0.00 100.00 71.97 35.45 21620 5777.708 569 17032.080 22809.7880.00 0.00 0.00 100.00 71.97 35.45 21620 5777.708 569 17032.080 22809.7880.00 0.00 0.00 99.99 71.96 35.45 21622 5778.416 570 17064.165 22842.5810.00 0.00 0.00 99.99 71.96 35.45 21622 5778.416 570 17064.165 22842.5810.00 0.00 0.00 99.99 71.96 35.45 21622 5778.416 570 17064.165 22842.5810.00 0.00 0.00 99.99 71.96 35.45 21622 5778.416 570 17064.165 22842.5810.00 0.00 0.00 99.99 71.96 35.45 21622 5778.416 570 17064.165 22842.5810.00 0.00 0.00 99.99 71.96 35.45 21622 5778.416 570 17064.165 22842.5810.00 0.00 0.00 99.99 71.96 35.45 21622 5778.416 570 17064.165 22842.5810.00 0.00 0.00 99.99 71.96 35.45 21622 5778.416 570 17064.165 22842.5810.00 0.00 0.00 99.99 71.96 35.45 21622 5778.416 570 17064.165 22842.5810.00 0.00 0.00 99.99 71.96 35.45 21622 5778.416 570 17064.165 22842.5810.00 0.00 0.00 99.99 71.96 35.45 21622 5778.416 570 17064.165 22842.5810.00 0.00 0.00 99.99 71.96 35.45 21622 5778.416 570 17064.165 22842.5810.00 0.00 0.00 99.99 71.96 35.45 21622 5778.416 570 17064.165 22842.5810.00 0.00 0.00 99.99 71.96 35.45 21622 5778.416 570 17064.165 22842.5810.00 0.00 0.00 99.99 71.96 35.45 21622 5778.416 570 17064.165 22842.5810.00 0.00 0.00 99.99 71.96 35.45 21622 5778.416 570 17064.165 22842.5810.00 0.00 0.00 99.99 71.96 35.45 21622 5778.416 570 17064.165 22842.5810.00 0.00 0.00 99.99 71.96 35.45 21626 5779.559 570 17097.383 22876.9410.00 0.00 0.00 99.99 71.96 35.45 21627 5780.418 571 17097.383 22877.8010.00 0.00 0.00 99.99 71.96 35.45 21627 5780.418 571 17097.383 22877.8010.00 0.00 0.00 99.99 71.96 35.45 21627 5780.418 571 17097.383 22877.8010.00 0.00 0.00 99.99 71.96 35.45 21627 5780.418 571 17097.383 22877.8010.00 0.00 0.00 99.99 71.96 35.45 21627 5780.418 571 17097.383 22877.8010.00 0.00 0.00 99.99 71.96 35.45 21627 5780.418 571 17097.383 22877.8010.00 0.00 0.00 99.99 71.96 35.45 21627 5780.418 571 17097.383 22877.8010.00 0.00 0.00 99.99 71.96 35.45 21627 5780.418 571 17097.383 22877.8010.00 0.00 0.00 99.99 71.96 35.45 21627 5780.418 571 17097.383 22877.8010.00 0.00 0.00 99.99 71.96 35.45 21627 5780.418 571 17097.383 22877.8010.00 0.00 0.00 99.99 71.96 35.45 21627 5780.418 571 17097.383 22877.8010.00 0.00 0.00 99.99 71.96 35.45 21627 5780.418 571 17097.383 22877.8010.00 0.00 0.00 99.99 71.96 35.45 21627 5780.418 571 17097.383 22877.8010.00 0.00 0.00 99.99 71.96 35.45 21627 5780.418 571 17097.383 22877.8010.00 0.00 0.00 99.99 71.96 35.45 21627 5780.418 571 17097.383 22877.8010.00 0.00 0.00 99.99 71.96 35.45 21627 5780.418 571 17097.383 22877.8010.00 0.00 0.00 99.99 71.96 35.45 21627 5780.418 571 17097.383 22877.8010.00 0.00 0.00 99.99 71.96 35.45 21627 5780.418 571 17097.383 22877.8010.00 0.00 0.00 99.98 71.96 35.45 21632 5781.952 571 17132.676 22914.6280.00 0.00 0.00 99.98 71.96 35.45 21632 5782.311 572 17132.676 22914.9870.00 0.00 0.00 99.98 71.96 35.45 21632 5782.311 572 17132.676 22914.9870.00 0.00 0.00 99.98 71.96 35.45 21632 5782.311 572 17132.676 22914.9870.00 0.00 0.00 99.98 71.96 35.45 21632 5782.311 572 17132.676 22914.9870.00 0.00 0.00 99.98 71.96 35.45 21632 5782.311 572 17132.676 22914.9870.00 0.00 0.00 99.98 71.96 35.45 21632 5782.311 572 17132.676 22914.9870.00 0.00 0.00 99.98 71.96 35.45 21632 5782.311 572 17132.676 22914.9870.00 0.00 0.00 99.98 71.96 35.45 21632 5782.311 572 17132.676 22914.9870.00 0.00 0.00 99.98 71.96 35.45 21632 5782.311 572 17132.676 22914.9870.00 0.00 0.00 99.98 71.96 35.45 21632 5782.311 572 17132.676 22914.9870.00 0.00 0.00 99.98 71.96 35.45 21632 5782.311 572 17132.676 22914.9870.00 0.00 0.00 99.98 71.96 35.45 21632 5782.311 572 17132.676 22914.9870.00 0.00 0.00 99.98 71.96 35.45 21632 5782.311 572 17132.676 22914.9870.00 0.00 0.00 99.98 71.96 35.45 21632 5782.311 572 17132.676 22914.9870.00 0.00 0.00 99.98 71.96 35.45 21632 5782.311 572 17132.676 22914.9870.00 0.00 0.00 99.98 71.96 35.45 21637 5783.815 572 17162.441 22946.2560.00 0.00 0.00 99.98 71.96 35.45 21637 5784.181 573 17162.441 22946.6220.00 0.00 0.00 99.98 71.96 35.45 21637 5784.181 573 17162.441 22946.6220.00 0.00 0.00 99.98 71.96 35.45 21637 5784.181 573 17162.441 22946.6220.00 0.00 0.00 99.98 71.96 35.45 21637 5784.181 573 17162.441 22946.6220.00 0.00 0.00 99.98 71.96 35.45 21637 5784.181 573 17162.441 22946.6220.00 0.00 0.00 99.98 71.96 35.45 21637 5784.181 573 17162.441 22946.6220.00 0.00 0.00 99.98 71.96 35.45 21637 5784.181 573 17162.441 22946.6220.00 0.00 0.00 99.98 71.96 35.45 21637 5784.181 573 17162.441 22946.6220.00 0.00 0.00 99.98 71.96 35.45 21637 5784.181 573 17162.441 22946.6220.00 0.00 0.00 99.98 71.96 35.45 21637 5784.181 573 17162.441 22946.6220.00 0.00 0.00 99.98 71.96 35.45 21637 5784.181 573 17162.441 22946.6220.00 0.00 0.00 99.98 71.96 35.45 21637 5784.181 573 17162.441 22946.6220.00 0.00 0.00 99.98 71.96 35.45 21637 5784.181 573 17162.441 22946.6220.00 0.00 0.00 99.98 71.96 35.45 21637 5784.181 573 17162.441 22946.6220.00 0.00 0.00 99.98 71.96 35.45 21637 5784.181 573 17162.441 22946.6220.00 0.00 0.00 99.98 71.96 35.45 21637 5784.181 573 17162.441 22946.6220.00 0.00 0.00 99.99 71.96 35.45 21642 5785.868 573 17194.331 22980.1990.00 0.00 0.00 99.99 71.96 35.45 21642 5786.238 574 17194.331 22980.5690.00 0.00 0.00 99.99 71.96 35.45 21642 5786.238 574 17194.331 22980.5690.00 0.00 0.00 99.99 71.96 35.45 21642 5786.238 574 17194.331 22980.5690.00 0.00 0.00 99.99 71.96 35.45 21642 5786.238 574 17194.331 22980.5690.00 0.00 0.00 99.99 71.96 35.45 21642 5786.238 574 17194.331 22980.5690.00 0.00 0.00 99.99 71.96 35.45 21642 5786.238 574 17194.331 22980.5690.00 0.00 0.00 99.99 71.96 35.45 21642 5786.238 574 17194.331 22980.5690.00 0.00 0.00 99.99 71.96 35.45 21642 5786.238 574 17194.331 22980.5690.00 0.00 0.00 99.99 71.96 35.45 21642 5786.238 574 17194.331 22980.5690.00 0.00 0.00 99.99 71.96 35.45 21642 5786.238 574 17194.331 22980.5690.00 0.00 0.00 99.99 71.96 35.45 21642 5786.238 574 17194.331 22980.5690.00 0.00 0.00 99.99 71.96 35.45 21642 5786.238 574 17194.331 22980.5690.00 0.00 0.00 99.99 71.96 35.45 21642 5786.238 574 17194.331 22980.5690.00 0.00 0.00 99.99 71.96 35.45 21642 5786.238 574 17194.331 22980.5690.00 0.00 22.99 100.00 71.96 35.45 21642 5786.238 574 17223.926 23010.1640.00 0.00 0.00 100.00 71.96 35.45 21647 5788.016 575 17223.926 23011.9420.00 0.00 0.00 100.00 71.96 35.45 21647 5788.016 575 17223.926 23011.9420.00 0.00 0.00 100.00 71.96 35.45 21647 5788.016 575 17223.926 23011.9420.00 0.00 0.00 100.00 71.96 35.45 21647 5788.016 575 17223.926 23011.9420.00 0.00 0.00 100.00 71.96 35.45 21647 5788.016 575 17223.926 23011.9420.00 0.00 0.00 100.00 71.96 35.45 21647 5788.016 575 17223.926 23011.9420.00 0.00 0.00 100.00 71.96 35.45 21647 5788.016 575 17223.926 23011.9420.00 0.00 0.00 100.00 71.96 35.45 21647 5788.016 575 17223.926 23011.9420.00 0.00 0.00 100.00 71.96 35.45 21647 5788.016 575 17223.926 23011.9420.00 0.00 0.00 100.00 71.96 35.45 21647 5788.016 575 17223.926 23011.9420.00 0.00 0.00 100.00 71.96 35.45 21647 5788.016 575 17223.926 23011.9420.00 0.00 0.00 100.00 71.96 35.45 21647 5788.016 575 17223.926 23011.9420.00 0.00 0.00 100.00 71.96 35.45 21647 5788.016 575 17223.926 23011.9420.00 0.00 0.00 100.00 71.96 35.45 21647 5788.016 575 17223.926 23011.9420.00 0.00 0.00 100.00 71.96 35.45 21647 5788.016 575 17223.926 23011.9420.00 0.00 0.00 99.98 71.96 35.44 21649 5788.379 575 17253.060 23041.4390.00 0.00 0.00 99.98 71.96 35.44 21653 5790.281 576 17253.060 23043.3410.00 0.00 0.00 99.98 71.96 35.44 21653 5790.281 576 17253.060 23043.3410.00 0.00 0.00 99.98 71.96 35.44 21653 5790.281 576 17253.060 23043.3410.00 0.00 0.00 99.98 71.96 35.44 21653 5790.281 576 17253.060 23043.3410.00 0.00 0.00 99.98 71.96 35.44 21653 5790.281 576 17253.060 23043.3410.00 0.00 0.00 99.98 71.96 35.44 21653 5790.281 576 17253.060 23043.3410.00 0.00 0.00 99.98 71.96 35.44 21653 5790.281 576 17253.060 23043.3410.00 0.00 0.00 99.98 71.96 35.44 21653 5790.281 576 17253.060 23043.3410.00 0.00 0.00 99.98 71.96 35.44 21653 5790.281 576 17253.060 23043.3410.00 0.00 0.00 99.98 71.96 35.44 21653 5790.281 576 17253.060 23043.3410.00 0.00 0.00 99.98 71.96 35.44 21653 5790.281 576 17253.060 23043.3410.00 0.00 0.00 99.98 71.96 35.44 21653 5790.281 576 17253.060 23043.3410.00 0.00 0.00 99.98 71.96 35.44 21653 5790.281 576 17253.060 23043.3410.00 0.00 0.00 99.98 71.96 35.44 21653 5790.281 576 17253.060 23043.3410.00 0.00 0.00 99.98 71.96 35.44 21653 5790.281 576 17253.060 23043.3410.00 0.00 0.00 99.98 71.96 35.44 21653 5790.281 576 17253.060 23043.3410.00 0.00 0.00 99.98 71.96 35.45 21655 5790.690 576 17284.518 23075.2080.00 0.00 0.00 99.98 71.96 35.45 21658 5792.323 577 17284.518 23076.8410.00 0.00 0.00 99.98 71.96 35.45 21658 5792.323 577 17284.518 23076.8410.00 0.00 0.00 99.98 71.96 35.45 21658 5792.323 577 17284.518 23076.8410.00 0.00 0.00 99.98 71.96 35.45 21658 5792.323 577 17284.518 23076.8410.00 0.00 0.00 99.98 71.96 35.45 21658 5792.323 577 17284.518 23076.8410.00 0.00 0.00 99.98 71.96 35.45 21658 5792.323 577 17284.518 23076.8410.00 0.00 0.00 99.98 71.96 35.45 21658 5792.323 577 17284.518 23076.8410.00 0.00 0.00 99.98 71.96 35.45 21658 5792.323 577 17284.518 23076.8410.00 0.00 0.00 99.98 71.96 35.45 21658 5792.323 577 17284.518 23076.8410.00 0.00 0.00 99.98 71.96 35.45 21658 5792.323 577 17284.518 23076.8410.00 0.00 0.00 99.98 71.96 35.45 21658 5792.323 577 17284.518 23076.8410.00 0.00 0.00 99.98 71.96 35.45 21658 5792.323 577 17284.518 23076.8410.00 0.00 0.00 99.98 71.96 35.45 21658 5792.323 577 17284.518 23076.8410.00 0.00 0.00 99.98 71.96 35.45 21658 5792.323 577 17284.518 23076.8410.00 0.00 0.00 99.98 71.96 35.45 21658 5792.323 577 17313.260 23105.5830.00 0.00 0.00 99.98 71.96 35.45 21660 5793.136 578 17313.260 23106.3960.00 0.00 0.00 99.98 71.96 35.45 21660 5793.136 578 17313.260 23106.3960.00 0.00 0.00 99.98 71.96 35.45 21660 5793.136 578 17313.260 23106.3960.00 0.00 0.00 99.98 71.96 35.45 21660 5793.136 578 17313.260 23106.3960.00 0.00 0.00 99.98 71.96 35.45 21660 5793.136 578 17313.260 23106.3960.00 0.00 0.00 99.98 71.96 35.45 21660 5793.136 578 17313.260 23106.3960.00 0.00 0.00 99.98 71.96 35.45 21660 5793.136 578 17313.260 23106.3960.00 0.00 0.00 99.98 71.96 35.45 21660 5793.136 578 17313.260 23106.3960.00 0.00 0.00 99.98 71.96 35.45 21660 5793.136 578 17313.260 23106.3960.00 0.00 0.00 99.98 71.96 35.45 21660 5793.136 578 17313.260 23106.3960.00 0.00 0.00 99.98 71.96 35.45 21660 5793.136 578 17313.260 23106.3960.00 0.00 0.00 99.98 71.96 35.45 21660 5793.136 578 17313.260 23106.3960.00 0.00 0.00 99.98 71.96 35.45 21660 5793.136 578 17313.260 23106.3960.00 0.00 0.00 99.98 71.96 35.45 21660 5793.136 578 17313.260 23106.3960.00 0.00 0.00 99.99 71.97 35.45 21662 5793.492 578 17341.110 23134.6010.00 0.00 0.00 99.99 71.97 35.45 21666 5795.595 579 17341.110 23136.7050.00 0.00 0.00 99.99 71.97 35.45 21666 5795.595 579 17341.110 23136.7050.00 0.00 0.00 99.99 71.97 35.45 21666 5795.595 579 17341.110 23136.7050.00 0.00 0.00 99.99 71.97 35.45 21666 5795.595 579 17341.110 23136.7050.00 0.00 0.00 99.99 71.97 35.45 21666 5795.595 579 17341.110 23136.7050.00 0.00 0.00 99.99 71.97 35.45 21666 5795.595 579 17341.110 23136.7050.00 0.00 0.00 99.99 71.97 35.45 21666 5795.595 579 17341.110 23136.7050.00 0.00 0.00 99.99 71.97 35.45 21666 5795.595 579 17341.110 23136.7050.00 0.00 0.00 99.99 71.97 35.45 21666 5795.595 579 17341.110 23136.7050.00 0.00 0.00 99.99 71.97 35.45 21666 5795.595 579 17341.110 23136.7050.00 0.00 0.00 99.99 71.97 35.45 21666 5795.595 579 17341.110 23136.7050.00 0.00 0.00 99.99 71.97 35.45 21666 5795.595 579 17341.110 23136.7050.00 0.00 0.00 99.99 71.97 35.45 21666 5795.595 579 17341.110 23136.7050.00 0.00 0.00 99.99 71.97 35.45 21666 5795.595 579 17341.110 23136.7050.00 0.00 0.00 99.99 71.97 35.45 21666 5795.595 579 17341.110 23136.7050.00 0.00 0.00 99.99 71.97 35.45 21666 5795.595 579 17341.110 23136.7050.00 0.00 0.00 99.99 71.97 35.45 21666 5795.595 579 17341.110 23136.7050.00 0.00 0.00 99.99 71.97 35.45 21666 5795.595 579 17341.110 23136.7050.00 0.00 0.00 99.99 71.97 35.45 21666 5795.595 579 17341.110 23136.7050.00 0.00 0.00 99.99 71.97 35.45 21666 5795.595 579 17341.110 23136.7050.00 0.00 0.00 99.99 71.97 35.45 21666 5795.595 579 17341.110 23136.7050.00 0.00 0.00 99.99 71.97 35.45 21666 5795.595 579 17341.110 23136.7050.00 0.00 0.00 99.99 71.97 35.45 21666 5795.595 579 17341.110 23136.7050.00 0.00 0.00 99.99 71.97 35.45 21666 5795.595 579 17341.110 23136.7050.00 0.00 0.00 99.99 71.97 35.45 21666 5795.595 579 17341.110 23136.7050.00 0.00 0.00 99.99 71.97 35.45 21666 5795.595 579 17341.110 23136.7050.00 0.00 0.00 99.99 71.97 35.45 21666 5795.595 579 17341.110 23136.7050.00 0.00 0.00 99.99 71.97 35.45 21666 5795.595 579 17341.110 23136.7050.00 0.00 0.00 99.99 71.97 35.45 21666 5795.595 579 17341.110 23136.7050.00 0.00 0.00 99.99 71.97 35.45 21666 5795.595 579 17341.110 23136.7050.00 0.00 0.00 99.99 71.97 35.45 21666 5795.595 579 17341.110 23136.7050.00 0.00 0.00 99.99 71.97 35.45 21666 5795.595 579 17341.110 23136.7050.00 0.00 0.00 99.99 71.97 35.45 21666 5795.595 579 17341.110 23136.7050.00 0.00 0.00 99.99 71.97 35.45 21666 5795.595 579 17341.110 23136.7050.00 0.00 0.00 99.99 71.97 35.45 21666 5795.595 579 17341.110 23136.7050.00 0.00 0.00 99.99 71.97 35.45 21666 5795.595 579 17341.110 23136.7050.00 0.00 0.00 99.99 71.97 35.45 21666 5795.595 579 17341.110 23136.7050.00 0.00 0.00 99.99 71.97 35.45 21666 5795.595 579 17341.110 23136.7050.00 0.00 0.00 99.99 71.97 35.45 21666 5795.595 579 17341.110 23136.7050.00 0.00 0.00 99.99 71.97 35.45 21666 5795.595 579 17341.110 23136.7050.00 0.00 0.00 99.99 71.97 35.45 21666 5795.595 579 17341.110 23136.7050.00 0.00 0.00 99.99 71.97 35.45 21666 5795.595 579 17341.110 23136.7050.00 0.00 0.00 99.99 71.97 35.45 21666 5795.595 579 17341.110 23136.7050.00 0.00 0.00 99.99 71.97 35.45 21666 5795.595 579 17341.110 23136.705最后频繁的几十次FullGC都无法回收1%的老年代内存，最终OOM.
3.结论与解决方法
经过分析，随着程序运行，不可回收的对象肯定会原来越多，最终必然导致OOM。
如何解决？
**解决方法：**不解决，延缓问题，增大jvm，延长宕机间隔。升级版本（查查新版中是否解决了本问题）
目前用的hive版本是2.2版
如何查版本？
看lib的后缀：
root@tbds-x-x-x-x:/usr/hdp/2.2.0.0-2041/hive/lib>ls hive*
hive-accumulo-handler-2.2.0.jar hive-common-2.2.0.jar hive-exec-2.2.0.jar.bak.20200616 hive-hplsql-2.2.0.jar hive-llap-common-2.2.0.jar hive-metastore-2.2.0.jar hive-service-rpc-2.2.0.jar hive-shims-scheduler-2.2.0.jar
hive-ant-2.2.0.jar hive-contrib-2.2.0.jar hive-hbase-handler-2.2.0.jar hive-hwi-2.2.0.jar hive-llap-ext-client-2.2.0.jar hive-orc-2.2.0.jar hive-shims-0.23-2.2.0.jar hive-storage-api-2.3.1.jar
hive-beeline-2.2.0.jar hive-druid-handler-2.2.0.jar hive-hcatalog-core-2.2.0.jar hive-jdbc-2.2.0.jar hive-llap-server-2.2.0.jar hive-serde-2.2.0.jar hive-shims-2.2.0.jar hive-testutils-2.2.0.jar
hive-cli-2.2.0.jar hive-exec-2.2.0.jar hive-hcatalog-server-extensions-2.2.0.jar hive-llap-client-2.2.0.jar hive-llap-tez-2.2.0.jar hive-service-2.2.0.jar hive-shims-common-2.2.0.jar
总结
经过分析，对hiveServer 以一定频率宕机的根本原因，得以知晓。
HiveServer2 频繁宕机根本原因分析相关推荐
- springboot项目 tomcat8.x 频繁宕机 原因分析
大家好,我是烤鸭: 今天分享一次线上tomcat宕机原因. 1 . 最近两次的宕机时间部分日志 1.1 2019-04-04 13:22:27.187 [Thread-10] INFO o.s.co ...
- ORA-04031错误导致宕机案例分析
今天遇到一起ORACLE数据库宕机案例,下面是对这起数据库宕机案例的原因进行分析.解读.分析过程中顺便记录一下这个案例的前因后果,攒点经验值,培养一下分析.解决问题的能力. 案例环境: 操作系统 :O ...
- 中标麒麟v7服务器宕机问题分析
一 问题描述 某行v7服务器出现异常重启现象,故障系统转储vmcore,以及宕机的内核日志. 二 日志分析 通过查看dmesg日志,报错日志可分为两类: 1 python脚本运行过程中出现的" ...
- 某项目Oracle RAC基础库发生ASM磁盘文件头损坏宕机事件分析排查
问题描述: 2021年2月26日收到某现场项目经理电话反馈现场Oracle RAC数据库发生宕机事件,但数据库已恢复正常,需要我方进行故障分析排查原因. 日志分析: 到了现场后通过现场人员对接登录到此 ...
- ORA-04031导致数据库宕机问题分析
背景介绍 2014/6/5接渠道反馈,用户数据库意外宕机,后经过重启服务器数据库恢复正常,用户希望能够排查原因,避免再次出现宕机事故,这种意外宕机原因排查是我们远程处理经常遇到的案例,虽然宕机的原因有 ...
- 如何打造一个频繁宕机的业务系统？
删库跑路专家.宕机行为艺术家.肥胖版吴彦祖--作者本人,曾经说过:"打造一个偶尔宕机的系统并不难,难的是经常宕机,一辈子宕机,从来没有稳定过的系统,这才是最难的." 但是,作为一个 ...
- 解决hbase RegionServer频繁宕机的一些办法
HBase在使用中会遇到非常多的问题,最为常见的就是HBase regionserver挂掉(此文中简称regionserver为RS), 而RS挂掉的原因各不相同.在本文中对遇到过的HBase RS ...
- GitHub 多次宕机的罪魁祸首竟是 MySQL？
整理 | 苏宓 出品 | CSDN(ID:CSDNnews) 宕机时时有,但近期特别多.这边苹果服务器发生大规模宕机,导致 App Store.Apple Music.Books 等十几项服务中断,另 ...
- Redis的KEYS命令引起RDS数据库雪崩，RDS发生两次宕机，造成几百万的资金损失
文章目录 第一次宕机 事故影响 原因分析 改进方案 第二次宕机 原因分析 改进方案 总结 Redis开发建议 1.冷热数据分离,不要将所有数据全部都放到Redis中 2.不同的业务数据要分开存储 3. ...
最新文章
- HDU - 4612 Warm up(边双缩点+树的直径)
- C语言面试题之华为篇2(答案全)
- 管壁式换热器cad图纸_民勤图纸管板加工加工
- 汇编语言常见错误（转载)
- linux 命令find
- python界面-python 可视化界面
- 0112 - 借用纸币发行门票
- oracle数据库SQL防止超长,SQL Server和Oracle防止数据锁定的比较
- opencv画框显示python_Python OpenCV实现鼠标画框效果
- Linux环境下搭建Java Web测试环境的具体步骤
- Java常用软件下载地址
- java号码池_Java常量池详解
- 电脑重装系统需要多少钱？
- UNIX网络编程卷1 回射客户程序 TCP客户程序设计范式
- 1111111111111111111111111111
- 鸿蒙系统内核为什么还是安卓,鸿蒙系统和安卓的区别
- matlab中的graythresh函数的实例
- 教你解包华为U8825D官方固件UPDATE.APP --以B956为例
- 【创业分享】2022年，仅赚几万，但却很踏实？
- 树莓派4b在miniconda下安装kivy，示例报错‘ImportError...libstdc++.so.6: version ‘GLIBCXX_3.4.29‘ not found‘解决办法
热门文章
- Unity3D动画游戏设计算法--序列化与反序列化
- 请转载网站删除本人的文章
- 童年经典回忆 | 从零开始带大家用Python撸一个魔塔小游戏呀（3）
- 腾讯电商mini店铺营销活动策划
- MybatisPlus实现分页效果并解决错误：cant found IPage for args!
- 2021年金属非金属矿山（地下矿山）安全管理人员考试及金属非金属矿山（地下矿山）安全管理人员免费试题
- 奇舞周刊第 324 期：Focusing on focus
- c语言调声音文件大小,Android MP3文件录制 + 声音分贝大小自定义View实现
- PHP资源列表(转)
- 2011级-csdn-java-张侃—第十一周周报