想把hadoop的进程日志导入hive表进行分析，遂做了以下的尝试。

关于hadoop进程日志的解析
使用正则表达式获取四个字段，一个是日期时间，一个是日志级别，一个是类，最后一个是详细信息，
然后在hive中建一个表，可以用来方便查询。

2015-12-18 22:23:23,357 INFO org.apache.hadoop.yarn.server.nodemanager.containermanager.monitor.ContainersMonitorImpl: Memory usage of ProcessTree 32652 for container-id container_1448915696877_26289_01_000158: 110.6 MB of 2 GB physical memory used; 2.1 GB of 4.2 GB virtual memory used
2015-12-18 22:23:23,426 INFO org.apache.hadoop.yarn.server.nodemanager.containermanager.monitor.ContainersMonitorImpl: Memory usage of ProcessTree 32615 for container-id container_1448915696877_26289_01_000102: 104.6 MB of 2 GB physical memory used; 2.1 GB of 4.2 GB virtual memory used
2015-12-18 22:23:23,467 WARN org.apache.hadoop.yarn.server.nodemanager.containermanager.monitor.ContainersMonitorImpl: Uncaught exception in ContainerMemoryManager while managing memory of container_1448915696877_26289_01_000270
java.lang.IllegalArgumentException: disparate values
        at sun.misc.FDBigInt.quoRemIteration(FloatingDecimal.java:2931)
        at sun.misc.FormattedFloatingDecimal.dtoa(FormattedFloatingDecimal.java:922)
        at sun.misc.FormattedFloatingDecimal.<init>(FormattedFloatingDecimal.java:542)
        at java.util.Formatter$FormatSpecifier.print(Formatter.java:3264)
        at java.util.Formatter$FormatSpecifier.print(Formatter.java:3202)
        at java.util.Formatter$FormatSpecifier.printFloat(Formatter.java:2769)
        at java.util.Formatter$FormatSpecifier.print(Formatter.java:2720)
        at java.util.Formatter.format(Formatter.java:2500)
        at java.util.Formatter.format(Formatter.java:2435)
        at java.lang.String.format(String.java:2148)
        at org.apache.hadoop.util.StringUtils.format(StringUtils.java:123)
        at org.apache.hadoop.util.StringUtils$TraditionalBinaryPrefix.long2String(StringUtils.java:758)
        at org.apache.hadoop.yarn.server.nodemanager.containermanager.monitor.ContainersMonitorImpl$MonitoringThread.formatUsageString(ContainersMonitorImpl.java:487)
        at org.apache.hadoop.yarn.server.nodemanager.containermanager.monitor.ContainersMonitorImpl$MonitoringThread.run(ContainersMonitorImpl.java:399)
2015-12-18 22:23:23,498 WARN org.apache.hadoop.yarn.server.nodemanager.containermanager.monitor.ContainersMonitorImpl: Uncaught exception in ContainerMemoryManager while managing memory of container_1448915696877_26289_01_000214

DROP TABLE IF EXISTS hadoop_log; CREATE TABLE hadoop_log (
date1 STRING,time1 STRING,msgtype STRING,classname STRING,msgtext STRING) ROW FORMAT SERDE 'org.apache.hadoop.hive.serde2.RegexSerDe' WITH SERDEPROPERTIES ("input.regex" = "^(\\d{4}-\\d{2}-\\d{2})\\s+(\\d{2}.\\d{2}.\\d{2}.\\d{3})\\s+(\\S+)\\s+(\\S+)\\s+(.*)$", "output.format.string" = "%1$s %2$s %3$s %4$s %5$s"
)
STORED AS TEXTFILE; LOAD DATA LOCAL INPATH "/home/student/hadooplog" INTO TABLE hadoop_log; SELECT date1, time1, msgtext FROM hadoop_log WHERE msgtype='ERROR' OR msgtype='WARN' LIMIT 5; LOAD DATA LOCAL INPATH "/home/student/hadooplog3" OVERWRITE INTO TABLE hadoop_log;

需要注意的一点是，hive以\n做为行分隔符，所以需要对原有的日志文件进行处理，因为原有的日志文件中可能有异常或错误发生，这个时候是多行的。否则hive中会有很

多空的记录。

可以写一段bash shell或python来完成需要的功能。下面是我刚学python写的，很简陋。

import re
p=re.compile(r"^\d{4}-\d{2}-\d{2}\s+\d{2}.\d{2}.\d{2}.\d{3} INFO|WARN|ERROR|DEBUG")
str=""
f2=open('/home/student/hadooplog4','w')
with open('/app/cdh23502/logs/hadoop-student-datanode-nn1.log','r') as f:for l in f:if(str==""):str=l.rstrip()continueif(str!="" and len(p.findall(l))>0):print "\n"+strf2.write(str+"\n")str=l.rstrip()else:str=str+l.rstrip()print "\n" + strf2.write(str+"\n")f2.flush()
f2.close()