Logstash 时间转换（YYYY-MM-dd HH:mm:ss转Unix时间）

适用场景
- 日志时间转Unix时间
示例日志：

2017-03-21 00:00:00,291 INFO [DubboServerHandler-10.135.6.53:20885-thread-98] i.w.w.r.m.RequirementManager [RequirementManager.java:860] fetch no data from oracle
2017-03-21 00:00:00,294 INFO [DubboServerHandler-10.135.6.53:20885-thread-98] i.w.w.r.m.RequirementManager [RequirementManager.java:1549] begin fetch data from oracle
2017-03-21 00:00:00,301 ERROR [DubboServerHandler-10.135.6.53:20885-thread-98] i.w.w.h.d.testDao [testDao.java:265] find test project failed,  error code={}
java.sql.SQLException: ORA-00942: table or view does not existat oracle.jdbc.driver.DatabaseError.throwSqlException(DatabaseError.java:112) ~[ojdbc14-10.2.0.1.0.jar:Oracle JDBC Driver version - "10.2.0.1.0XE"]at oracle.jdbc.driver.T4CTTIoer.processError(T4CTTIoer.java:331) ~[ojdbc14-10.2.0.1.0.jar:Oracle JDBC Driver version - "10.2.0.1.0XE"]at oracle.jdbc.driver.T4CTTIoer.processError(T4CTTIoer.java:288) ~[ojdbc14-10.2.0.1.0.jar:Oracle JDBC Driver version - "10.2.0.1.0XE"]at oracle.jdbc.driver.T4C8Oall.receive(T4C8Oall.java:743) ~[ojdbc14-10.2.0.1.0.jar:Oracle JDBC Driver version - "10.2.0.1.0XE"]at oracle.jdbc.driver.T4CPreparedStatement.doOall8(T4CPreparedStatement.java:216) ~[ojdbc14-10.2.0.1.0.jar:Oracle JDBC Driver version - "10.2.0.1.0XE"]at oracle.jdbc.driver.T4CPreparedStatement.executeForDescribe(T4CPreparedStatement.java:799) ~[ojdbc14-10.2.0.1.0.jar:Oracle JDBC Driver version - "10.2.0.1.0XE"]at oracle.jdbc.driver.OracleStatement.executeMaybeDescribe(OracleStatement.java:1038) ~[ojdbc14-10.2.0.1.0.jar:Oracle JDBC Driver version - "10.2.0.1.0XE"]at oracle.jdbc.driver.T4CPreparedStatement.executeMaybeDescribe(T4CPreparedStatement.java:839) ~[ojdbc14-10.2.0.1.0.jar:Oracle JDBC Driver version - "10.2.0.1.0XE"]at oracle.jdbc.driver.OracleStatement.doExecuteWithTimeout(OracleStatement.java:1133) ~[ojdbc14-10.2.0.1.0.jar:Oracle JDBC Driver version - "10.2.0.1.0XE"]at oracle.jdbc.driver.OraclePreparedStatement.executeInternal(OraclePreparedStatement.java:3285) ~[ojdbc14-10.2.0.1.0.jar:Oracle JDBC Driver version - "10.2.0.1.0XE"]at oracle.jdbc.driver.OraclePreparedStatement.executeQuery(OraclePreparedStatement.java:3329) ~[ojdbc14-10.2.0.1.0.jar:Oracle JDBC Driver version - "10.2.0.1.0XE"]at org.apache.commons.dbcp.DelegatingPreparedStatement.executeQuery(DelegatingPreparedStatement.java:96) ~[commons-dbcp-1.4.jar:1.4]at org.apache.commons.dbcp.DelegatingPreparedStatement.executeQuery(DelegatingPreparedStatement.java:96) ~[commons-dbcp-1.4.jar:1.4]at io.jiankunking.demo.haier.dao.testDao.findBigResourceMaterialByStatus(testDao.java:252) ~[demo-requirement-1.0-SNAPSHOT.jar:na]at io.jiankunking.demo.requirement.manager.RequirementManager.findBigResourceMaterialByStatus(RequirementManager.java:1550) [demo-requirement-1.0-SNAPSHOT.jar:na]at io.jiankunking.demo.requirement.manager.RequirementManager$$FastClassByCGLIB$$b97dbf5c.invoke(<generated>) [spring-core-3.2.6.RELEASE.jar:na]at org.springframework.cglib.proxy.MethodProxy.invoke(MethodProxy.java:204) [spring-core-3.2.6.RELEASE.jar:3.2.6.RELEASE]at org.springframework.aop.framework.CglibAopProxy$CglibMethodInvocation.invokeJoinpoint(CglibAopProxy.java:701) [spring-aop-3.2.6.RELEASE.jar:3.2.6.RELEASE]at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:150) [spring-aop-3.2.6.RELEASE.jar:3.2.6.RELEASE]at org.springframework.transaction.interceptor.TransactionInterceptor$1.proceedWithInvocation(TransactionInterceptor.java:96) [spring-tx-3.2.6.RELEASE.jar:3.2.6.RELEASE]at org.springframework.transaction.interceptor.TransactionAspectSupport.invokeWithinTransaction(TransactionAspectSupport.java:260) [spring-tx-3.2.6.RELEASE.jar:3.2.6.RELEASE]at org.springframework.transaction.interceptor.TransactionInterceptor.invoke(TransactionInterceptor.java:94) [spring-tx-3.2.6.RELEASE.jar:3.2.6.RELEASE]at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:172) [spring-aop-3.2.6.RELEASE.jar:3.2.6.RELEASE]at org.springframework.aop.framework.CglibAopProxy$DynamicAdvisedInterceptor.intercept(CglibAopProxy.java:634) [spring-aop-3.2.6.RELEASE.jar:3.2.6.RELEASE]at io.jiankunking.demo.requirement.manager.RequirementManager$$EnhancerByCGLIB$$de554283.findBigResourceMaterialByStatus(<generated>) [spring-core-3.2.6.RELEASE.jar:na]at io.jiankunking.demo.requirement.service.RequirementServiceImpl.fetchBigResourceMaterial(RequirementServiceImpl.java:4791) [demo-requirement-1.0-SNAPSHOT.jar:na]at io.jiankunking.demo.requirement.service.RequirementServiceImpl$$FastClassByCGLIB$$15e55a9c.invoke(<generated>) [spring-core-3.2.6.RELEASE.jar:na]at org.springframework.cglib.proxy.MethodProxy.invoke(MethodProxy.java:204) [spring-core-3.2.6.RELEASE.jar:3.2.6.RELEASE]at org.springframework.aop.framework.CglibAopProxy$CglibMethodInvocation.invokeJoinpoint(CglibAopProxy.java:701) [spring-aop-3.2.6.RELEASE.jar:3.2.6.RELEASE]at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:150) [spring-aop-3.2.6.RELEASE.jar:3.2.6.RELEASE]at org.springframework.aop.framework.adapter.MethodBeforeAdviceInterceptor.invoke(MethodBeforeAdviceInterceptor.java:51) [spring-aop-3.2.6.RELEASE.jar:3.2.6.RELEASE]at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:172) [spring-aop-3.2.6.RELEASE.jar:3.2.6.RELEASE]at org.springframework.aop.framework.adapter.MethodBeforeAdviceInterceptor.invoke(MethodBeforeAdviceInterceptor.java:51) [spring-aop-3.2.6.RELEASE.jar:3.2.6.RELEASE]at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:172) [spring-aop-3.2.6.RELEASE.jar:3.2.6.RELEASE]at org.springframework.aop.aspectj.AspectJAfterThrowingAdvice.invoke(AspectJAfterThrowingAdvice.java:55) [spring-aop-3.2.6.RELEASE.jar:3.2.6.RELEASE]at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:172) [spring-aop-3.2.6.RELEASE.jar:3.2.6.RELEASE]at org.springframework.aop.aspectj.AspectJAfterThrowingAdvice.invoke(AspectJAfterThrowingAdvice.java:55) [spring-aop-3.2.6.RELEASE.jar:3.2.6.RELEASE]at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:172) [spring-aop-3.2.6.RELEASE.jar:3.2.6.RELEASE]at org.springframework.aop.aspectj.AspectJAfterAdvice.invoke(AspectJAfterAdvice.java:42) [spring-aop-3.2.6.RELEASE.jar:3.2.6.RELEASE]at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:172) [spring-aop-3.2.6.RELEASE.jar:3.2.6.RELEASE]at org.springframework.aop.aspectj.AspectJAfterAdvice.invoke(AspectJAfterAdvice.java:42) [spring-aop-3.2.6.RELEASE.jar:3.2.6.RELEASE]at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:172) [spring-aop-3.2.6.RELEASE.jar:3.2.6.RELEASE]at org.springframework.aop.framework.adapter.MethodBeforeAdviceInterceptor.invoke(MethodBeforeAdviceInterceptor.java:51) [spring-aop-3.2.6.RELEASE.jar:3.2.6.RELEASE]at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:172) [spring-aop-3.2.6.RELEASE.jar:3.2.6.RELEASE]at org.springframework.aop.aspectj.AspectJAfterAdvice.invoke(AspectJAfterAdvice.java:42) [spring-aop-3.2.6.RELEASE.jar:3.2.6.RELEASE]at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:172) [spring-aop-3.2.6.RELEASE.jar:3.2.6.RELEASE]at org.springframework.aop.interceptor.ExposeInvocationInterceptor.invoke(ExposeInvocationInterceptor.java:91) [spring-aop-3.2.6.RELEASE.jar:3.2.6.RELEASE]at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:172) [spring-aop-3.2.6.RELEASE.jar:3.2.6.RELEASE]at org.springframework.aop.framework.CglibAopProxy$DynamicAdvisedInterceptor.intercept(CglibAopProxy.java:634) [spring-aop-3.2.6.RELEASE.jar:3.2.6.RELEASE]at io.jiankunking.demo.requirement.service.RequirementServiceImpl$$EnhancerByCGLIB$$e80e4734.fetchBigResourceMaterial(<generated>) [spring-core-3.2.6.RELEASE.jar:na]at com.alibaba.dubbo.common.bytecode.Wrapper55.invokeMethod(Wrapper55.java) [dubbo-2.5.4.1-20160510.024647-3.jar:2.5.4.1-SNAPSHOT]at com.alibaba.dubbo.rpc.proxy.javassist.JavassistProxyFactory$1.doInvoke(JavassistProxyFactory.java:46) [dubbo-2.5.4.1-20160510.024647-3.jar:2.5.4.1-SNAPSHOT]at com.alibaba.dubbo.rpc.proxy.AbstractProxyInvoker.invoke(AbstractProxyInvoker.java:72) [dubbo-2.5.4.1-20160510.024647-3.jar:2.5.4.1-SNAPSHOT]at com.alibaba.dubbo.rpc.protocol.InvokerWrapper.invoke(InvokerWrapper.java:53) [dubbo-2.5.4.1-20160510.024647-3.jar:2.5.4.1-SNAPSHOT]at com.alibaba.dubbo.rpc.filter.ExceptionFilter.invoke(ExceptionFilter.java:64) [dubbo-2.5.4.1-20160510.024647-3.jar:2.5.4.1-SNAPSHOT]at com.alibaba.dubbo.rpc.protocol.ProtocolFilterWrapper$1.invoke(ProtocolFilterWrapper.java:91) [dubbo-2.5.4.1-20160510.024647-3.jar:2.5.4.1-SNAPSHOT]at com.alibaba.dubbo.monitor.support.MonitorFilter.invoke(MonitorFilter.java:75) [dubbo-2.5.4.1-20160510.024647-3.jar:2.5.4.1-SNAPSHOT]at com.alibaba.dubbo.rpc.protocol.ProtocolFilterWrapper$1.invoke(ProtocolFilterWrapper.java:91) [dubbo-2.5.4.1-20160510.024647-3.jar:2.5.4.1-SNAPSHOT]at com.alibaba.dubbo.rpc.filter.TimeoutFilter.invoke(TimeoutFilter.java:42) [dubbo-2.5.4.1-20160510.024647-3.jar:2.5.4.1-SNAPSHOT]at com.alibaba.dubbo.rpc.protocol.ProtocolFilterWrapper$1.invoke(ProtocolFilterWrapper.java:91) [dubbo-2.5.4.1-20160510.024647-3.jar:2.5.4.1-SNAPSHOT]at com.alibaba.dubbo.rpc.protocol.dubbo.filter.TraceFilter.invoke(TraceFilter.java:78) [dubbo-2.5.4.1-20160510.024647-3.jar:2.5.4.1-SNAPSHOT]at com.alibaba.dubbo.rpc.protocol.ProtocolFilterWrapper$1.invoke(ProtocolFilterWrapper.java:91) [dubbo-2.5.4.1-20160510.024647-3.jar:2.5.4.1-SNAPSHOT]at com.alibaba.dubbo.rpc.filter.ContextFilter.invoke(ContextFilter.java:60) [dubbo-2.5.4.1-20160510.024647-3.jar:2.5.4.1-SNAPSHOT]at com.alibaba.dubbo.rpc.protocol.ProtocolFilterWrapper$1.invoke(ProtocolFilterWrapper.java:91) [dubbo-2.5.4.1-20160510.024647-3.jar:2.5.4.1-SNAPSHOT]at com.alibaba.dubbo.rpc.filter.GenericFilter.invoke(GenericFilter.java:132) [dubbo-2.5.4.1-20160510.024647-3.jar:2.5.4.1-SNAPSHOT]at com.alibaba.dubbo.rpc.protocol.ProtocolFilterWrapper$1.invoke(ProtocolFilterWrapper.java:91) [dubbo-2.5.4.1-20160510.024647-3.jar:2.5.4.1-SNAPSHOT]at com.alibaba.dubbo.rpc.filter.ClassLoaderFilter.invoke(ClassLoaderFilter.java:38) [dubbo-2.5.4.1-20160510.024647-3.jar:2.5.4.1-SNAPSHOT]at com.alibaba.dubbo.rpc.protocol.ProtocolFilterWrapper$1.invoke(ProtocolFilterWrapper.java:91) [dubbo-2.5.4.1-20160510.024647-3.jar:2.5.4.1-SNAPSHOT]at com.alibaba.dubbo.rpc.filter.EchoFilter.invoke(EchoFilter.java:38) [dubbo-2.5.4.1-20160510.024647-3.jar:2.5.4.1-SNAPSHOT]at com.alibaba.dubbo.rpc.protocol.ProtocolFilterWrapper$1.invoke(ProtocolFilterWrapper.java:91) [dubbo-2.5.4.1-20160510.024647-3.jar:2.5.4.1-SNAPSHOT]at com.alibaba.dubbo.rpc.protocol.dubbo.DubboProtocol$1.reply(DubboProtocol.java:108) [dubbo-2.5.4.1-20160510.024647-3.jar:2.5.4.1-SNAPSHOT]at com.alibaba.dubbo.remoting.exchange.support.header.HeaderExchangeHandler.handleRequest(HeaderExchangeHandler.java:84) [dubbo-2.5.4.1-20160510.024647-3.jar:2.5.4.1-SNAPSHOT]at com.alibaba.dubbo.remoting.exchange.support.header.HeaderExchangeHandler.received(HeaderExchangeHandler.java:170) [dubbo-2.5.4.1-20160510.024647-3.jar:2.5.4.1-SNAPSHOT]at com.alibaba.dubbo.remoting.transport.DecodeHandler.received(DecodeHandler.java:52) [dubbo-2.5.4.1-20160510.024647-3.jar:2.5.4.1-SNAPSHOT]at com.alibaba.dubbo.remoting.transport.dispatcher.ChannelEventRunnable.run(ChannelEventRunnable.java:82) [dubbo-2.5.4.1-20160510.024647-3.jar:2.5.4.1-SNAPSHOT]at java.util.concurrent.ThreadPoolExecutor.runWorker(Unknown Source) [na:1.8.0_111]at java.util.concurrent.ThreadPoolExecutor$Worker.run(Unknown Source) [na:1.8.0_111]at java.lang.Thread.run(Unknown Source) [na:1.8.0_111]
2017-03-21 00:00:00,302 INFO [DubboServerHandler-10.135.6.53:20885-thread-98] i.t.s.r.m.RequirementManager [RequirementManager.java:1556] fetch no data from oracle

logstash 配置文件：

input {file {path => [ "/home/jiankunking/software/testdata/test.log" ]tags => [ "test","jiankunking" ]start_position => beginningcodec =>multiline {pattern => "^\d{4}-\d{1,2}-\d{1,2}\s\d{1,2}:\d{1,2}:\d{1,2}"negate => truewhat => "previous"}}
}
filter {grok{match => ["message","%{TIMESTAMP_ISO8601:time}\s{1,}%{NOTSPACE:loglevel}\s{1,}%{NOTSPACE:thread-id}\s{1,}%{NOTSPACE:packageinfo}\s{1,}%{NOTSPACE:classinfo}\s{1,}%{GREEDYDATA:msginfo}"]}date{match=>["time","YYYY-MM-dd HH:mm:ss,SSS"]target=>"logdate"}ruby{code => "event.set('logdateunix',event.get('logdate').to_i)"}mutate {remove_field => ["message"]}
}
output{elasticsearch{hosts => ["111.222.333.444:9200"]index => "logstash-demo"codec => rubydebug}
}

具体效果如下：

从 Logstash-5.0 开始， LogStash::Event 改为 Java 实现，直接使用
event[“parent”][“child”] 形式获取的不是原事件的引用而是复制品。需要改用 event.get(‘[parent][child]’) 和 event.set(‘[parent][child]’,
‘value’) 的方法。

小注：本文演示logstash用的是5.0.1版本

5.0之前版本可以参考（未测试）：
http://www.cnblogs.com/yangwenbo214/p/6189622.html

作者：jiankunking 出处：http://blog.csdn.net/jiankunking

Logstash 时间转换（YYYY-MM-dd HH:mm:ss转Unix时间）相关推荐

java 时间转换工具类 yyyyMMdd HH:mm
获取系统当前时间戳 : System.currentTimeMillis()) 获取系统当前时间任意格式,自己根据生成的格式选择性填写 /*** 获取当前时间* 把需要生成的时间格式替换一下就可以* ...
【怒怼老乔】苹果手机ios系统居然特喵的不支持日期yyyy-MM-dd HH:mm:ss.ms，只支持 yyyy/MM/dd HH:mm:ss
苹果手机ios系统居然特喵的不支持日期yyyy-MM-dd HH:mm:ss.ms, 只支持yyyy/MM/dd HH:mm:ss 我只能用 .replace(/\-/g,'\/'); 来替换掉了, ...
Kettle5.2 couldn't convert string [...] to a date using format [yyyy/MM/dd HH:mm:ss.SSS]
使用kettle出现 couldn't convert string [...] to a date using format [yyyy/MM/dd HH:mm:ss.SSS] 试了各种方法,最后发 ...
C# 中的年月日时分秒 yyyy MM dd HH mm ss
C# 中的年月日时分秒 yyyy MM dd HH mm ss 其中大小写含义不同,不能混用年:yyyy 月:MM 日:dd(1-31) DD(0-365) 时:hh:12小时制 HH:24小时制 ...
时间格式区别之YYYY/MM/dd HH:mm 与 yyyy/MM/dd HH:mm
1. 相应的业务场景 1.1 场景简介正值2019年末,2020年初之际,系统日期时间为2019/12/31而系统报表却出现了2020/12/31的日期. 1.2 代码案例如下 2.Java中YYY ...
spring boot + vue 前后端分离时间戳转换为 yyyy:MM:dd HH:mm:ss格式
后端 1.model(entity)注释直接转换当返回类型为resultMap,在entity类里面相应属性上加上以下注释 @JsonFormat(pattern="yyyy-MM-dd ...
js时间格式化 YYYY/MM/DD HH:MM:SSS
/*** 将"2018-05-19T08:04:52.000+0000"这种格式的时间转化为正常格式* @param time*/ function timeFormat(time ...
java 通配符日期_java – 使用带有通配符支持的SimpleDateFormat解析日期字符串(例如* yyyy * MM * dd * hh * mm * ss)...
首先,我想知道是否存在类似于SimpleDateFormat但是支持通配符的现有库?如果没有,最好的办法是什么? 我有这个问题,我需要匹配并从文件名中提取日期,但我似乎无法找到适合这种情况的方法.虽然 ...
小程序日期格式（yyyy-MM-dd HH:mm:ss）转（yyyy/MM/dd HH:mm:ss）
let newDate = (date).replace(/-/g, '/'); var date = new Date(newDate).getTime(); 转载于:https://www.cnb ...
java date eee_java将 Date原始格式EEE MMM dd HH:mm:ss Z yyyy转成指定格式
java将 Date原始格式"EEE MMM dd HH:mm:ss Z yyyy"转成指定格式 //将 Date原始格式"EEE MMM dd HH:mm:ss Z y ...

Logstash 时间转换（YYYY-MM-dd HH:mm:ss转Unix时间）

Logstash 时间转换（YYYY-MM-dd HH:mm:ss转Unix时间）相关推荐

最新文章

热门文章