利用tail内置插件来获取日志

tail插件相当于tail -f，它会不断的获取更新的日志，

<source>@type     tailpath      /log-dir/*-app.logpos_file  /log-dir/app.log.postagidaasrefresh_interval 10sread_from_head truepath_key path<parse>@type json      #把日志格式直接解析为json格式</parse>
</source>
<source>@type     tailpath      /log-dir/*-debug.logpos_file  /log-dir/debug.log.postagdebugmultiline_flush_interval 2sread_from_head truepath_key path<parse>@type   multiline               #multiline 相当于logstash的multiline format_firstline /^(?<level>(INFO|WARN|ERROR)+)/format1 /(?<level>[a-zA-Z]+)\s*\[(?<date>[0-9/\-: ,]+)\] (?<logger>[a-zA-Z\.]+):(?<message>[\d\D\s]+)/</parse>
</source>
<source>@type     tailpath      /log-dir/*-requests.logpos_file  /log-dir/request.log.postagrequestrefresh_interval 10sread_from_head truepath_key path<parse>@type regexpexpression /(?<message>.*)/</parse>
</source>

第一个filter是为日志添加字段，tag和宿主机的名字，这个可能需要调docker，直接取只会取到docker的ID

<filter *>@type record_transformer             <record>tag ${tag}hostname "#{Socket.gethostname}"</record>
</filter><filter request>@type    grep                         #排除掉一些不需要的日志<exclude>key messagepattern /.*healthcheck.*|.*prometheusMetrics.*|.*(v1+\/)+(configurations)+(\/+versions).*/</exclude>
</filter>
<filter request>@type parserkey_name messagereserve_data yes<parse>   @type regexpexpression  /(?<ip>[^|]+)\|(?<date>[^|]+)\|(?<statusCode>[^|]+)\|(?<contentLength>[^|]+)\|(?<reqURI>[^|]+)\|(?<referer>[^|]+)\|(?<userAgent>[^|]+)\|(?<reqId>[^|]+)\|(?<internalIp>[^|]+)\|(?<reqHost>[^|]+)\|(?<reqOrigin>[^|]+)\|(?<reqTime>[^|]+) \|.*\|(?<requestMethod>[\w]+)/</parse>
</filter>
<match idaas>@type rewrite_tag_filter        #重写tag，匹配的重写tag为app.token,不匹配的重写标app.idaas<rule>key     thread_namepattern /token/tag     app.token</rule><rule>key     thread_namepattern /token/tag     app.idaasinvert  true</rule>
</match>

上面已经把idaas进行分流处理，这里我们把app.token进行一次过滤，然后和app.idaas一起输入到ES中

<filter app.token>@type parserkey_name thread_namereserve_data yes<parse>@type regexpexpression /(?<thread_name>[A-Za-z0-9\.\-_=/\? ]+\.)/</parse>
</filter>
<match request>@type elasticsearchhost elasticsearchlog-lb.elasticsearch-logindex_name    s3-fluentd-request-%Y%m%dtype_name     s3-fluentd-requestinclude_timestamp truessl_verify    false<buffer tag,time>timekey          4s          timekey_wait     1s     #这里的刷新是把buffer的数据及时发送到ES，保持数据的实时性</buffer>
</match>
<match debug>@type elasticsearchhost elasticsearchlog-lb.elasticsearch-logindex_name    s3-fluentd-debug-%Y%m%dtype_name     s3-fluentd-debuginclude_timestamp truessl_verify    false<buffer tag,time>timekey          4s          timekey_wait     1s   </buffer>
</match>
<match app.*>@type elasticsearchhost elasticsearchlog-lb.elasticsearch-logindex_name    s3-fluentd-idaas-%Y%m%dtype_name     s3-fluentd-idaasinclude_timestamp truessl_verify    false<buffer tag,time>timekey          4s          timekey_wait     1s   </buffer>
</match>

fluentd日志处理-安装配置（一）
https://blog.51cto.com/11078047/2316881
Fluentd 日志处理-S3拉取日志处理（二）
https://blog.51cto.com/11078047/2316910
Fluentd日志处理-插件使用和调试问题（四）
https://blog.51cto.com/11078047/2320018