fluent-bit代替filebeat实战(smartgate、nginx)

一、安装fluent-bit ,两种方法

方法一、docker

mkdir -p /data/fluent-bit/etccat > /data/fluent-bit/etc/fluent-bit.conf << 'EOF'
[SERVICE]flush        1daemon       Offlog_level    info#parsers_file parsers.conf#plugins_file plugins.confhttp_server  Offhttp_listen  0.0.0.0http_port    2020storage.metrics on[INPUT]name  tailtag   accessgate        ignore_older   2hbuffer_chunk_size  32kbuffer_max_size  64Kpath        /data/log/accessgate/*.access*.csvdb          /tmp/accessgate.dbexclude_Path *.gz,*.zipkey  message[OUTPUT]name          eshost            192.168.11.100port             9200http_user       elastichttp_passwd     passwordmatch   accessgateindex   sg-accesspipeline   sg-access
EOFcat > /data/fluent-bit/start.sh << 'EOF'
docker run -it \
--name fluent-bit \
--restart always \
-p 2020:2020 \
-v /etc/localtime:/etc/localtime \
-v /data/log/:/data/log/ \
-v `pwd`/etc/:/fluent-bit/etc/ \
fluent/fluent-bit:1.9.3
EOFcd /data/fluent-bit/ && bash start.sh

方法二、 yum
1、准备tg-agent-bit安装源

cat  > /etc/yum.repos.d/fluent-bit.repo << 'EOF'
[td-agent-bit]
name = TD Agent Bit
baseurl = https://packages.fluentbit.io/centos/7/$basearch/
gpgcheck=1
gpgkey=https://packages.fluentbit.io/fluentbit.key
enabled=1
EOF

2、通过yum安装td-agent-bit

yum install td-agent-bit -y
systemctl start td-agent-bit && systemctl enable td-agent-bit   #centos7
service td-agent-bit start  && chkconfig td-agent-bit on        #centos6

二、网关日志
2、fluent-bit代替filebeat

2.1 原filebeat配置

filebeat.inputs:
- type: logenabled: truepaths:- "/data/rio/log/accessgate/*.access*.csv"fields:             type: "accessgate"                                        pipeline: "sg-access"output.elasticsearch:username: elasticpassword: passwordhosts: - http://192.168.11.100:9200worker: 2bulk_max_size: 256indices:- index: "sg-access"when.equals:fields:type: "accessgate"

2.2、fluent-bit配置

fluent-bit不支持elasticsearch集群多主机的配置

cat >  /etc/td-agent-bit/td-agent-bit.conf << 'EOF'[SERVICE]flush        1daemon       Offlog_level    info#parsers_file parsers.conf#plugins_file plugins.confhttp_server  Offhttp_listen  0.0.0.0http_port    2020storage.metrics on[INPUT]name  tailtag   accessgate        ignore_older   2hbuffer_chunk_size  32kbuffer_max_size  64Kpath        /data/log/accessgate/*.access*.csvdb          /tmp/accessgate.dbexclude_Path *.gz,*.zipkey  message[OUTPUT]name          eshost            192.168.11.100port             9200http_user       elastichttp_passwd     passwordmatch   accessgateindex   sg-accesspipeline   sg-access
EOF

三、nginx日志

[SERVICE]flush        1daemon       Offlog_level    infoparsers_file parsers.conf   #引用parsers.conf文件plugins_file plugins.confhttp_server  Offhttp_listen  0.0.0.0http_port    2020storage.metrics on[INPUT]name  tailtag   nginxaccessparser  nginx             #使用parsers.conf文件中的nginx parser.ignore_Older   2hbuffer_Chunk_Size  32kbuffer_Max_Size  64Kpath        /data/nginx/log/access.logdb          /tmp/nginx.dbexclude_Path *.gz,*.zipkey  message[OUTPUT]name            eshost            192.168.11.100port            9200http_user       elastichttp_passwd     passwordmatch   nginxaccessindex   nginxaccess

parsers.conf

[PARSER]Name   nginxFormat regexRegex ^(?<remote>[^ ]*) (?<host>[^ ]*) (?<user>[^ ]*) \[(?<time>[^\]]*)\] "(?<method>\S+)(?: +(?<path>[^\"]*?)(?: +\S*)?)?" (?<code>[^ ]*) (?<size>[^ ]*)(?: "(?<referer>[^\"]*)" "(?<agent>[^\"]*)")Time_Key timeTime_Format %d/%b/%Y:%H:%M:%S %z

验证结果如下

参考：https://blog.csdn.net/u012516914/article/details/106717302

fluent-bit.conf: |[SERVICE]Flush         1Log_Level     infoDaemon        offParsers_File  parsers.confHTTP_Server   OnHTTP_Listen   0.0.0.0HTTP_Port     2020@INCLUDE input-kubernetes.conf@INCLUDE filter-kubernetes.conf@INCLUDE output-elasticsearch.confinput-kubernetes.conf: |[INPUT]Name              tailTag               kube.*Path              /var/log/containers/*.logParser            dockerDB                /var/log/flb_kube.dbMem_Buf_Limit     5MBSkip_Long_Lines   OnRefresh_Interval  10[INPUT]Name           systemdTag             host.*Systemd_Filter  _SYSTEMD_UNIT=kubelet.servicePath            /var/log/journalDB              /var/log/flb_host.dbfilter-kubernetes.conf: |[FILTER]Name                kubernetesMatch               kube.*Kube_URL            https://kubernetes.default.svc.cluster.local:443Merge_Log           OnK8S-Logging.Parser  OnK8S-Logging.Exclude On[FILTER]Name                kubernetesMatch               host.*Kube_URL            https://kubernetes.default.svc.cluster.local:443Merge_Log           OnUse_Journal         Onoutput-elasticsearch.conf: |[OUTPUT]Name            esMatch           *Host            ${FLUENT_ELASTICSEARCH_HOST}Port            ${FLUENT_ELASTICSEARCH_PORT}Logstash_Format OnRetry_Limit     Falseparsers.conf: |[PARSER]Name   apacheFormat regexRegex  ^(?<host>[^ ]*) [^ ]* (?<user>[^ ]*) \[(?<time>[^\]]*)\] "(?<method>\S+)(?: +(?<path>[^\"]*?)(?: +\S*)?)?" (?<code>[^ ]*) (?<size>[^ ]*)(?: "(?<referer>[^\"]*)" "(?<agent>[^\"]*)")?$Time_Key timeTime_Format %d/%b/%Y:%H:%M:%S %z[PARSER]Name   apache2Format regexRegex  ^(?<host>[^ ]*) [^ ]* (?<user>[^ ]*) \[(?<time>[^\]]*)\] "(?<method>\S+)(?: +(?<path>[^ ]*) +\S*)?" (?<code>[^ ]*) (?<size>[^ ]*)(?: "(?<referer>[^\"]*)" "(?<agent>[^\"]*)")?$Time_Key timeTime_Format %d/%b/%Y:%H:%M:%S %z[PARSER]Name   apache_errorFormat regexRegex  ^\[[^ ]* (?<time>[^\]]*)\] \[(?<level>[^\]]*)\](?: \[pid (?<pid>[^\]]*)\])?( \[client (?<client>[^\]]*)\])? (?<message>.*)$[PARSER]Name   nginxFormat regexRegex ^(?<remote>[^ ]*) (?<host>[^ ]*) (?<user>[^ ]*) \[(?<time>[^\]]*)\] "(?<method>\S+)(?: +(?<path>[^\"]*?)(?: +\S*)?)?" (?<code>[^ ]*) (?<size>[^ ]*)(?: "(?<referer>[^\"]*)" "(?<agent>[^\"]*)")?$Time_Key timeTime_Format %d/%b/%Y:%H:%M:%S %z[PARSER]Name   jsonFormat jsonTime_Key timeTime_Format %d/%b/%Y:%H:%M:%S %z[PARSER]Name        dockerFormat      jsonTime_Key    timeTime_Format %Y-%m-%dT%H:%M:%S.%LTime_Keep   On# Command      |  Decoder | Field | Optional Action# =============|==================|=================Decode_Field_As   escaped    log[PARSER]Name        syslogFormat      regexRegex       ^\<(?<pri>[0-9]+)\>(?<time>[^ ]* {1,2}[^ ]* [^ ]*) (?<host>[^ ]*) (?<ident>[a-zA-Z0-9_\/\.\-]*)(?:\[(?<pid>[0-9]+)\])?(?:[^\:]*\:)? *(?<message>.*)$Time_Key    timeTime_Format %b %d %H:%M:%S

https://docs.fluentbit.io/manual/pipeline/outputs/elasticsearch

fluent-bit代替filebeat实战(smartgate、nginx)相关推荐

Nginx实战｜Nginx健康检查
来自:NingG 个人博客链接:https://ningg.top/nginx-series-health-check/ 0. 背景服务治理的一个重要任务是感知服务节点变更,完成服务自动注册及异常 ...
你真的懂Spring Cloud+Nginx秒杀实战，Nginx高性能秒杀和限流吗？
Nginx高性能秒杀和限流从性能上来说,内部网关Zuul限流理论上比外部网关Nginx限流的性能会差一些.和Zuul一样,外部网关Nginx也可以通过Lua脚本的形式执行缓存在Redis内部的令牌桶 ...
实战：Nginx集成Lua脚本并调用memcached
之前在阿里云的centos系统上已经安装过nginx,但是没有安装lua所需的模块,因此重新编译nginx来扩展lua模块. ngx_lua_module 是一个nginx http模块,它把 lua ...
负载均衡环境搭建实战之nginx和tomcat
Linux基本环境负载均衡的环境需要在linux下搭建完成,所以有一个基础的linux系统是必须的,这里建议大家按照http://edu.51cto.com/course/10209.html中的基 ...
微服务架构实战：Nginx的安装及负载均衡配置、开机启动
#日志目录 base path=/tmp #跟踪服务器 tracker server=10.10.10.22:22122tracker server=10.10.10.32:22122#URL中是否有 ...
L09-10老男孩Linux运维实战培训-Nginx服务生产实战应用指南05(架构解决方案)
nginx的多实例设置首先说一下nginx后面加的参数的说明 -s 后面加reload 就是重新加载的意思和apache的graceful同样的效果 -v 小写的v显示版本号后退出 -V大写的V显示 ...
实战：Nginx + FastCGI 程序（C/C++）搭建高性能web server
1.介绍 Nginx - 高性能web server,这个不用多说了,大家都知道. FastCGI程序 - 常驻型CGI程序,它是语言无关的.可伸缩架构的CGI开放扩展,其主要行为是将CGI解释器进程 ...
nginx 获取body参数_分布式实战：Nginx缓存之流量分发层
本文首发于Ressmix个人站点:https://www.tpvlog.com 本章,我将进行Nginx流量分发层的lua代码编写.流量分发层的Nginx服务器,会基于商品id执行流量分发策略: 获取 ...
路由包含#号导致的nginx_分布式实战：Nginx缓存之OpenResty部署
本文首发于Ressmix个人站点:https://www.tpvlog.com 经过前面几章的讲解,我已经通过代码实现了三级缓存架构中的JVM本地缓存和Redis分布式缓存.本章,我将讲解最后的一层- ...

fluent-bit代替filebeat实战(smartgate、nginx)

fluent-bit代替filebeat实战(smartgate、nginx)相关推荐

最新文章

热门文章