docker-compose部署EFKL，存储，过滤laravel日志文件

继上一篇部署EFK之后，由于发现通过fluentd转发到ES的日志并不能实现我们预期的效果，先看看我们预期的效果：

再给大家看一下我要收集的日志格式：

[2021-10-29 03:39:12] saveData.INFO: saveData {"params":{"index":"fulfillments_1","id":5941107,"body":{"id":5941107,"shippingMethodId":null,"shippingMethodName":null,"pluginId":null,"shipToName":"tan2","shipToPhone":null,"shipToSuburb":"FRASER RISE","shipToState":"VIC","shipToPostcode":"3336","shipToCountry":"AU","shipToAddress1":"Second St","shipToAddress2":null,"shipToCompanyName":"eiz","shipToEmail":null,"fromAddress1":"tet-1","fromAddress2":null,"fromSuburb":"Moorabbin","fromState":"VIC","fromCountry":"AU","fromPostcode":"3189","fromCompany_name":"eiz","fromName":"jin2","fromPhone":"47658975","fromEmail":null,"carrierName":null,"labelNumber":[],"fulfillmentStatus":1,"consignments":[],"products":[{"id":4,"account_id":1,"product_id":4,"sku":"124","title":"dsadasds","weight":1,"length":11,"width":11,"height":11,"quantity":0,"location":null,"insured_amount":null,"status":0,"custom_label":null,"custom_label2":null,"custom_label3":null,"img_url":null,"barcode":null,"wms_stock":0,"pivot":{"fulfillment_id":5941107,"product_id":4,"qty":1,"note":null,"sku":"124"}}],"consignmentStatus":0,"picklistStatus":0,"createdAt":"2021-10-26 13:33:03","updatedAt":"2021-10-29 14:39:11","package_info":[{"packObj":[],"qty":"2","weight":"13","length":"6","width":"7","height":"8","package_id":null}],"price":null,"note":null,"tags":[{"id":95,"account_id":1,"parent_id":null,"name":"test","description":"{\"name\":\"test\",\"color\":\"#eb2f96\"}"}],"errors":null,"tracking_status":0,"packageNum":2,"productNum":1,"autoQuoteResult":[],"orders":[],"log":[],"shipToRef":"TJ0000212"}}} []

我们预期的效果是将日志中的内容都格式化显示出来，但是上一篇文章，EFK部署日志系统，搭建完成后日志中的内容还是会都堆在message字段中，这让我们很难查阅，于是便有了第二次尝试（本篇文章依然是根据laravel框架来示例）：

1、docker-compose部署logstash+filebeat,大家可以看到这次我用的是opensearch(等同于Elasticsearch)+opensearch-dashboards(等同于Kibana)，下面是我的docker-compose.yaml文件内容

version: "2.2"
services:opensearch:build:context: dockerfilesdockerfile: opensearch-no-security.dockerfilerestart: alwayscontainer_name: opensearchimage: wangyi/opensearch:latestenvironment:- discovery.type=single-nodeports:- 9200:9200- 9600:9600 # required for Performance Analyzervolumes:- opensearch-data1:/usr/share/opensearch/dataopensearch-dashboards:build:context: dockerfilesdockerfile: opensearch-dashboards-no-security.dockerfileimage: wangyi/opensearch-dashboard:latestcontainer_name: opensearch-dashboardsports:- 5601:5601environment:OPENSEARCH_HOSTS: '["http://opensearch:9200"]' # must be a string with no spaces when specified as an environment variablefilebeat:build: ./filebeatrestart: "always"container_name: filebeatvolumes:- ./storage/logs/:/tools/logs/user: rootlogstash:depends_on:- opensearchimage: "docker.elastic.co/logstash/logstash:7.1.0"volumes:- ./logstash/logstash.yml:/usr/share/logstash/config/logstash.yml- ./logstash/conf.d/:/usr/share/logstash/conf.d/ports:- "5044:5044"links:- opensearchvolumes:opensearch-data1:

2、其次是我的目录文件

3、然后编辑filebeat文件夹下的Dockerfile文件

FROM docker.elastic.co/beats/filebeat-oss:7.11.0# Copy our custom configuration file
COPY ./filebeat.yml /usr/share/filebeat/filebeat.ymlUSER root
# Create a directory to map volume with all docker log files
RUN mkdir /usr/share/filebeat/dockerlogs
RUN chown -R root /usr/share/filebeat/
RUN chmod -R go-w /usr/share/filebeat/

4、然后继续编辑logstash.yml文件

path.config: /usr/share/logstash/conf.d/*.conf
path.logs: /var/log/logstash

5、接下来是本篇文章的关键，编辑filebeat配置文件和logstash配置文件

filebeat.yml

filebeat.inputs:
- type: logenabled: truepaths:- /tools/logs/saveData/*/*/*.logfields:filetype: savedata ## 此处设置的filetype值在logstash配置文件里面需要用到，不同的日志文件创建不同的index,相当于一个标记的作用- type: logenabled: truepaths:- /tools/logs/condition/*/*/*.logfields:filetype: conditionsetup.ilm.enabled: falsesetup.template.settings:index.number_of_shards: 1index.number_of_replicas: 0index.codec: best_compressionoutput.logstash:  ##通过此处来链接logstash服务，将日志分发给logstash，然后再由logstash进行过滤enabled: truehosts: ["logstash:5044"]

配置完filebeat.yml我们就需要配置logstash.conf文件

input {beats {port => 5044}
}
filter {grok {match => {"message" => "\[%{TIMESTAMP_ISO8601:logtime}\] %{WORD:env}\.(?<level>[A-Z]{4,5})\: %{WORD:params} %{GREEDYDATA:msg} " ##此处的正则仅供参考，我的场景是过滤laravel日志文件的正则表达式}}json {source => "msg" ##将过滤完的内容转为json格式，不转的话是string格式的，不加这句话不会达到我们预期的效果}mutate{remove_field => ["message"] ##将原本的message字段删除掉}
}
output {if [fields][filetype] == "savedata" { ##判断来自于哪一个日志文件，filebeat文件配置elasticsearch {index => "savedatas_%{+YYYY.MM.dd}"hosts => ["opensearch:9200"]}}if [fields][filetype] == "condition" {elasticsearch {index => "conditions_%{+YYYY.MM.dd}"hosts => ["opensearch:9200"]}}
}

这里给大家推荐一个在线的grok测试地址，特别好用GROK在线测试

！！！一切配置就绪，点火，启动

docker-compose up -d 服务名称

这里说一下启动顺序
1、opensearch（E）
2、opensearch-dashboards（K）
3、logstash（L）
4、filebeat（F）

我们通过查看日志可以看到服务都已经成功启动

docker logs -f imageId

然后我们查看opensearch-dashboards后台可以看到达到我们文章开头预期的那个效果，所有转存的日志已经被格式化处理。

奥，对忘了贴出来我的opensearch和opensearch-dashboards的配置文件
目录配置：

1、opensearch.yml

cluster.name: docker-cluster# Bind to all interfaces because we don't know what IP address Docker will assign to us.
network.host: 0.0.0.0
compatibility.override_main_response_version: true

2、opensearch-no-security.dockerfile

FROM opensearchproject/opensearch:1.1.0
RUN /usr/share/opensearch/bin/opensearch-plugin remove opensearch-security
COPY --chown=opensearch:opensearch config.d/opensearch.yml /usr/share/opensearch/config/

3、opensearch-dashboards.yml

# Copyright 2021 Amazon.com, Inc. or its affiliates. All Rights Reserved.
#
# Licensed under the Apache License, Version 2.0 (the "License").
# You may not use this file except in compliance with the License.
# A copy of the License is located at
#
#     http://www.apache.org/licenses/LICENSE-2.0
#
# or in the "license" file accompanying this file. This file is distributed
# on an "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either
# express or implied. See the License for the specific language governing
# permissions and limitations under the License.# Description:
# Default configuration for OpenSearch Dashboardsserver.host: "0"
opensearch.hosts: ["https://localhost:9200"]
#opensearch.ssl.verificationMode: none
#opensearch.username: "kibanaserver"
#opensearch.password: "kibanaserver"
#opensearch.requestHeadersWhitelist: [ authorization,securitytenant ]#opensearch_security.multitenancy.enabled: true
##opensearch_security.multitenancy.tenants.preferred: ["Private", "Global"]
#opensearch_security.readonly_mode.roles: ["kibana_read_only"]
# Use this setting if you are running opensearch-dashboards without https
#opensearch_security.cookie.secure: false

4、opensearch-dashboards-no-security.dockerfile

FROM opensearchproject/opensearch-dashboards:1.1.0
RUN /usr/share/opensearch-dashboards/bin/opensearch-dashboards-plugin remove securityDashboards
COPY --chown=opensearch-dashboards:opensearch-dashboards config.d/opensearch_dashboards.yml /usr/share/opensearch-dashboards/config/

其实这两个（opensearch+opensearch-dashboards）完全可以用Elasticsearch+Kibana代替，看大家需求，Elasticsearch+Kibana的配置文件在上一篇文章中有写出来

这期的文章就写到这里，下一期写不用logstash来过滤日志文件，因为后来发现logstash这个玩意太占CPU，仅仅通过filebeat+es的pipeline就可以实现我们预期的效果，且不耗CPU，过几天写

docker-compose部署EFKL，存储，过滤laravel日志文件相关推荐

ZooKeeper ：Docker Compose部署ZooKeeper集群
用于Docker Compose部署ZooKeeper集群的yaml文件: version: '3' networks:zookeeper-networks:driver: bridgeservice ...
Kafka：Docker Compose部署Kafka集群
创建目录用于存放Docker Compose部署Kafka集群的yaml文件: mkdir -p /root/composefile/kafka/ 写入该yaml文件: vim /root/compo ...
使用Docker Compose 部署Nexus后提示：Unable to create directory /nexus-data/instance
场景 Ubuntu Server 上使用Docker Compose 部署Nexus(图文教程): https://blog.csdn.net/BADAO_LIUMANG_QIZHI/article/ ...
Docker - 使用Docker Compose部署应用
简介 Docker Compose是一个基于Docker Engine进行安装的Python工具.该工具使得用户可以在一个声明式的配置文件中定义一个多容器的应用,在Docker节点上,以单引擎模式(S ...
RabbitMQ：Docker Compose部署RabbitMQ
创建目录,用于存放Docker Compose部署RabbitMQ的yaml文件: mkdir -p /root/composefile/rabbitmq 写入该yaml文件: vim /root/c ...
使用docker compose部署MySQL主从复制集群
使用docker compose部署MySQL主从复制集群环境说明宿主机:Ubuntu 14.04.6 LTS Docker Engine: 18.06.3-ce docker compose: ...
docker compose部署服务
1 用docker compose部署服务 - 需求:假如现在我们手里有很多容器,每个容器对应每个服务,有nginx容器,redis容器,mysql容器等.现在我们需要批量化的去管理,批量启动,停止, ...
使用Docker Compose 部署Nexus后初次登录账号密码不正确，并且在nexus-data下没有admin.password
场景 Ubuntu Server 上使用Docker Compose 部署Nexus(图文教程): https://blog.csdn.net/BADAO_LIUMANG_QIZHI/article/ ...
Ubuntu Server 上使用Docker Compose 部署Nexus(图文教程)
场景 Docker-Compose简介与Ubuntu Server 上安装Compose: https://blog.csdn.net/BADAO_LIUMANG_QIZHI/article/deta ...
Docker Compose部署Nexus3时的docker-compose.yml代码
场景 Docker-Compose简介与Ubuntu Server 上安装Compose: https://blog.csdn.net/BADAO_LIUMANG_QIZHI/article/deta ...