网络上推荐了很多部署efk的方案，按照教程尝试很多教程，都无法使用。在学习了efk原理之后，自己搭建了一套可用的efk方案。

efk方案架构这里就不详细介绍了，网上资料一大堆。

通过eck部署es和kibana

参考我的文档：在k8s上搭建elasticsearch服务

可以部署好elasticsearch和elasticsearch

注意，elasticsearch和kibana都不使用https

需要增加配置

spec:http:tls:selfSignedCertificate:disabled: true

部署flunted

fluentd.yaml文件

# fluentd-es-ds.yaml
apiVersion: v1
kind: ServiceAccount
metadata:name: fluentd-esnamespace: elastic-systemlabels:app: fluentd-es
---
kind: ClusterRole
apiVersion: rbac.authorization.k8s.io/v1
metadata:name: fluentd-eslabels:app: fluentd-es
rules:
- apiGroups:- ""resources:- "namespaces"- "pods"verbs:- "get"- "watch"- "list"
---
kind: ClusterRoleBinding
apiVersion: rbac.authorization.k8s.io/v1
metadata:name: fluentd-eslabels:app: fluentd-es
subjects:
- kind: ServiceAccountname: fluentd-esnamespace: elastic-systemapiGroup: ""
roleRef:kind: ClusterRolename: fluentd-esapiGroup: ""
---
apiVersion: apps/v1
kind: DaemonSet
metadata:name: fluentd-esnamespace: elastic-systemlabels:app: fluentd-es
spec:selector:matchLabels:app: fluentd-estemplate:metadata:labels:app: fluentd-esspec:serviceAccount: fluentd-esserviceAccountName: fluentd-estolerations:- key: node-role.kubernetes.io/mastereffect: NoSchedule      containers:- name: fluentd-esimage: fluent/fluentd-kubernetes-daemonset:v1.11.5-debian-elasticsearch7-1.1env:- name:  FLUENT_ELASTICSEARCH_HOSTvalue: quickstart-es-http# default user- name:  FLUENT_ELASTICSEARCH_USERvalue: elastic# is already present from the elasticsearch deployment- name:  FLUENT_ELASTICSEARCH_PASSWORDvalueFrom:secretKeyRef:name: quickstart-es-elastic-userkey: elastic# elasticsearch standard port- name:  FLUENT_ELASTICSEARCH_PORTvalue: "9200"# der elastic operator ist https standard- name: FLUENT_ELASTICSEARCH_SCHEMEvalue: "http"# dont need systemd logs for now- name: FLUENTD_SYSTEMD_CONFvalue: disable# da certs self signt sind muss verify disabled werden- name:  FLUENT_ELASTICSEARCH_SSL_VERIFYvalue: "false"# to avoid issue https://github.com/uken/fluent-plugin-elasticsearch/issues/525- name:  FLUENT_ELASTICSEARCH_RELOAD_CONNECTIONSvalue: "false"resources:limits:memory: 512Mirequests:cpu: 100mmemory: 100MivolumeMounts:- name: varlogmountPath: /var/log- name: varlibdockercontainersmountPath: /var/lib/docker/containersreadOnly: true- name: config-volumemountPath: /fluentd/etcterminationGracePeriodSeconds: 30volumes:- name: varloghostPath:path: /var/log- name: varlibdockercontainershostPath:path: /var/lib/docker/containers- name: config-volumeconfigMap:name: fluentd-es-config

fluentd-es-configmap.yaml

# fluentd-es-configmap
kind: ConfigMap
apiVersion: v1
metadata:name: fluentd-es-confignamespace: elastic-system
data:fluent.conf: |-# https://github.com/fluent/fluentd-kubernetes-daemonset/blob/master/docker-image/v1.11/debian-elasticsearch7/conf/fluent.conf@include "#{ENV['FLUENTD_SYSTEMD_CONF'] || 'systemd'}.conf"@include "#{ENV['FLUENTD_PROMETHEUS_CONF'] || 'prometheus'}.conf"@include kubernetes.conf@include conf.d/*.conf<match kubernetes.**># https://github.com/kubernetes/kubernetes/issues/23001@type elasticsearch_dynamic@id  kubernetes_elasticsearch@log_level infoinclude_tag_key truehost "#{ENV['FLUENT_ELASTICSEARCH_HOST']}"port "#{ENV['FLUENT_ELASTICSEARCH_PORT']}"path "#{ENV['FLUENT_ELASTICSEARCH_PATH']}"scheme "#{ENV['FLUENT_ELASTICSEARCH_SCHEME'] || 'http'}"ssl_verify "#{ENV['FLUENT_ELASTICSEARCH_SSL_VERIFY'] || 'true'}"ssl_version "#{ENV['FLUENT_ELASTICSEARCH_SSL_VERSION'] || 'TLSv1_2'}"user "#{ENV['FLUENT_ELASTICSEARCH_USER'] || use_default}"password "#{ENV['FLUENT_ELASTICSEARCH_PASSWORD'] || use_default}"reload_connections "#{ENV['FLUENT_ELASTICSEARCH_RELOAD_CONNECTIONS'] || 'false'}"reconnect_on_error "#{ENV['FLUENT_ELASTICSEARCH_RECONNECT_ON_ERROR'] || 'true'}"reload_on_failure "#{ENV['FLUENT_ELASTICSEARCH_RELOAD_ON_FAILURE'] || 'true'}"log_es_400_reason "#{ENV['FLUENT_ELASTICSEARCH_LOG_ES_400_REASON'] || 'false'}"logstash_prefix logstash-${record['kubernetes']['namespace_name']}logstash_dateformat "#{ENV['FLUENT_ELASTICSEARCH_LOGSTASH_DATEFORMAT'] || '%Y.%m.%d'}"logstash_format "#{ENV['FLUENT_ELASTICSEARCH_LOGSTASH_FORMAT'] || 'true'}"index_name "#{ENV['FLUENT_ELASTICSEARCH_LOGSTASH_INDEX_NAME'] || 'logstash'}"target_index_key "#{ENV['FLUENT_ELASTICSEARCH_TARGET_INDEX_KEY'] || use_nil}"type_name "#{ENV['FLUENT_ELASTICSEARCH_LOGSTASH_TYPE_NAME'] || 'fluentd'}"include_timestamp "#{ENV['FLUENT_ELASTICSEARCH_INCLUDE_TIMESTAMP'] || 'false'}"template_name "#{ENV['FLUENT_ELASTICSEARCH_TEMPLATE_NAME'] || use_nil}"template_file "#{ENV['FLUENT_ELASTICSEARCH_TEMPLATE_FILE'] || use_nil}"template_overwrite "#{ENV['FLUENT_ELASTICSEARCH_TEMPLATE_OVERWRITE'] || use_default}"sniffer_class_name "#{ENV['FLUENT_SNIFFER_CLASS_NAME'] || 'Fluent::Plugin::ElasticsearchSimpleSniffer'}"request_timeout "#{ENV['FLUENT_ELASTICSEARCH_REQUEST_TIMEOUT'] || '5s'}"suppress_type_name "#{ENV['FLUENT_ELASTICSEARCH_SUPPRESS_TYPE_NAME'] || 'true'}"enable_ilm "#{ENV['FLUENT_ELASTICSEARCH_ENABLE_ILM'] || 'false'}"ilm_policy_id "#{ENV['FLUENT_ELASTICSEARCH_ILM_POLICY_ID'] || use_default}"ilm_policy "#{ENV['FLUENT_ELASTICSEARCH_ILM_POLICY'] || use_default}"ilm_policy_overwrite "#{ENV['FLUENT_ELASTICSEARCH_ILM_POLICY_OVERWRITE'] || 'false'}"<buffer>flush_thread_count "#{ENV['FLUENT_ELASTICSEARCH_BUFFER_FLUSH_THREAD_COUNT'] || '8'}"flush_interval "#{ENV['FLUENT_ELASTICSEARCH_BUFFER_FLUSH_INTERVAL'] || '5s'}"chunk_limit_size "#{ENV['FLUENT_ELASTICSEARCH_BUFFER_CHUNK_LIMIT_SIZE'] || '2M'}"queue_limit_length "#{ENV['FLUENT_ELASTICSEARCH_BUFFER_QUEUE_LIMIT_LENGTH'] || '32'}"retry_max_interval "#{ENV['FLUENT_ELASTICSEARCH_BUFFER_RETRY_MAX_INTERVAL'] || '30'}"retry_forever true</buffer></match><match **>@type elasticsearch@id out_es@log_level infoinclude_tag_key truehost "#{ENV['FLUENT_ELASTICSEARCH_HOST']}"port "#{ENV['FLUENT_ELASTICSEARCH_PORT']}"path "#{ENV['FLUENT_ELASTICSEARCH_PATH']}"scheme "#{ENV['FLUENT_ELASTICSEARCH_SCHEME'] || 'http'}"ssl_verify "#{ENV['FLUENT_ELASTICSEARCH_SSL_VERIFY'] || 'true'}"ssl_version "#{ENV['FLUENT_ELASTICSEARCH_SSL_VERSION'] || 'TLSv1_2'}"user "#{ENV['FLUENT_ELASTICSEARCH_USER'] || use_default}"password "#{ENV['FLUENT_ELASTICSEARCH_PASSWORD'] || use_default}"reload_connections "#{ENV['FLUENT_ELASTICSEARCH_RELOAD_CONNECTIONS'] || 'false'}"reconnect_on_error "#{ENV['FLUENT_ELASTICSEARCH_RECONNECT_ON_ERROR'] || 'true'}"reload_on_failure "#{ENV['FLUENT_ELASTICSEARCH_RELOAD_ON_FAILURE'] || 'true'}"log_es_400_reason "#{ENV['FLUENT_ELASTICSEARCH_LOG_ES_400_REASON'] || 'false'}"logstash_prefix "#{ENV['FLUENT_ELASTICSEARCH_LOGSTASH_PREFIX'] || 'logstash'}"logstash_dateformat "#{ENV['FLUENT_ELASTICSEARCH_LOGSTASH_DATEFORMAT'] || '%Y.%m.%d'}"logstash_format "#{ENV['FLUENT_ELASTICSEARCH_LOGSTASH_FORMAT'] || 'true'}"index_name "#{ENV['FLUENT_ELASTICSEARCH_LOGSTASH_INDEX_NAME'] || 'logstash'}"target_index_key "#{ENV['FLUENT_ELASTICSEARCH_TARGET_INDEX_KEY'] || use_nil}"type_name "#{ENV['FLUENT_ELASTICSEARCH_LOGSTASH_TYPE_NAME'] || 'fluentd'}"include_timestamp "#{ENV['FLUENT_ELASTICSEARCH_INCLUDE_TIMESTAMP'] || 'false'}"template_name "#{ENV['FLUENT_ELASTICSEARCH_TEMPLATE_NAME'] || use_nil}"template_file "#{ENV['FLUENT_ELASTICSEARCH_TEMPLATE_FILE'] || use_nil}"template_overwrite "#{ENV['FLUENT_ELASTICSEARCH_TEMPLATE_OVERWRITE'] || use_default}"sniffer_class_name "#{ENV['FLUENT_SNIFFER_CLASS_NAME'] || 'Fluent::Plugin::ElasticsearchSimpleSniffer'}"request_timeout "#{ENV['FLUENT_ELASTICSEARCH_REQUEST_TIMEOUT'] || '5s'}"suppress_type_name "#{ENV['FLUENT_ELASTICSEARCH_SUPPRESS_TYPE_NAME'] || 'true'}"enable_ilm "#{ENV['FLUENT_ELASTICSEARCH_ENABLE_ILM'] || 'false'}"ilm_policy_id "#{ENV['FLUENT_ELASTICSEARCH_ILM_POLICY_ID'] || use_default}"ilm_policy "#{ENV['FLUENT_ELASTICSEARCH_ILM_POLICY'] || use_default}"ilm_policy_overwrite "#{ENV['FLUENT_ELASTICSEARCH_ILM_POLICY_OVERWRITE'] || 'false'}"<buffer>flush_thread_count "#{ENV['FLUENT_ELASTICSEARCH_BUFFER_FLUSH_THREAD_COUNT'] || '8'}"flush_interval "#{ENV['FLUENT_ELASTICSEARCH_BUFFER_FLUSH_INTERVAL'] || '5s'}"chunk_limit_size "#{ENV['FLUENT_ELASTICSEARCH_BUFFER_CHUNK_LIMIT_SIZE'] || '2M'}"queue_limit_length "#{ENV['FLUENT_ELASTICSEARCH_BUFFER_QUEUE_LIMIT_LENGTH'] || '32'}"retry_max_interval "#{ENV['FLUENT_ELASTICSEARCH_BUFFER_RETRY_MAX_INTERVAL'] || '30'}"retry_forever true</buffer></match>kubernetes.conf: |-# https://github.com/fluent/fluentd-kubernetes-daemonset/blob/master/docker-image/v1.11/debian-elasticsearch7/conf/kubernetes.conf<label @FLUENT_LOG><match fluent.**>@type null@id ignore_fluent_logs</match></label><source>@id fluentd-containers.log@type tailpath /var/log/containers/*.logpos_file /var/log/es-containers.log.postag raw.kubernetes.*read_from_head true<parse>@type multi_format<pattern>format jsontime_key timetime_format %Y-%m-%dT%H:%M:%S.%NZ</pattern><pattern>format /^(?<time>.+) (?<stream>stdout|stderr) [^ ]* (?<log>.*)$/time_format %Y-%m-%dT%H:%M:%S.%N%:z</pattern></parse></source># Detect exceptions in the log output and forward them as one log entry.<match raw.kubernetes.**>@id raw.kubernetes@type detect_exceptionsremove_tag_prefix rawmessage logstream streammultiline_flush_interval 5chunk_limit_size 512mmax_bytes 50000000max_lines 1000</match># Concatenate multi-line logs<filter **>@id filter_concat@type concatkey messagemultiline_end_regexp /\n$/separator ""</filter># Enriches records with Kubernetes metadata<filter kubernetes.**>@id filter_kubernetes_metadata@type kubernetes_metadata</filter># Fixes json fields in Elasticsearch<filter kubernetes.**>@id filter_parser@type parserkey_name logreserve_data trueremove_key_name_field true<parse>@type multi_format<pattern>format json</pattern><pattern>format none</pattern></parse></filter><source>@type tail@id in_tail_minionpath /var/log/salt/minionpos_file /var/log/fluentd-salt.postag salt<parse>@type regexpexpression /^(?<time>[^ ]* [^ ,]*)[^\[]*\[[^\]]*\]\[(?<severity>[^ \]]*) *\] (?<message>.*)$/time_format %Y-%m-%d %H:%M:%S</parse></source><source>@type tail@id in_tail_startupscriptpath /var/log/startupscript.logpos_file /var/log/fluentd-startupscript.log.postag startupscript<parse>@type syslog</parse></source><source>@type tail@id in_tail_dockerpath /var/log/docker.logpos_file /var/log/fluentd-docker.log.postag docker<parse>@type regexpexpression /^time="(?<time>[^)]*)" level=(?<severity>[^ ]*) msg="(?<message>[^"]*)"( err="(?<error>[^"]*)")?( statusCode=($<status_code>\d+))?/</parse></source><source>@type tail@id in_tail_etcdpath /var/log/etcd.logpos_file /var/log/fluentd-etcd.log.postag etcd<parse>@type none</parse></source><source>@type tail@id in_tail_kubeletmultiline_flush_interval 5spath /var/log/kubelet.logpos_file /var/log/fluentd-kubelet.log.postag kubelet<parse>@type kubernetes</parse></source><source>@type tail@id in_tail_kube_proxymultiline_flush_interval 5spath /var/log/kube-proxy.logpos_file /var/log/fluentd-kube-proxy.log.postag kube-proxy<parse>@type kubernetes</parse></source><source>@type tail@id in_tail_kube_apiservermultiline_flush_interval 5spath /var/log/kube-apiserver.logpos_file /var/log/fluentd-kube-apiserver.log.postag kube-apiserver<parse>@type kubernetes</parse></source><source>@type tail@id in_tail_kube_controller_managermultiline_flush_interval 5spath /var/log/kube-controller-manager.logpos_file /var/log/fluentd-kube-controller-manager.log.postag kube-controller-manager<parse>@type kubernetes</parse></source><source>@type tail@id in_tail_kube_schedulermultiline_flush_interval 5spath /var/log/kube-scheduler.logpos_file /var/log/fluentd-kube-scheduler.log.postag kube-scheduler<parse>@type kubernetes</parse></source><source>@type tail@id in_tail_reschedulermultiline_flush_interval 5spath /var/log/rescheduler.logpos_file /var/log/fluentd-rescheduler.log.postag rescheduler<parse>@type kubernetes</parse></source><source>@type tail@id in_tail_glbcmultiline_flush_interval 5spath /var/log/glbc.logpos_file /var/log/fluentd-glbc.log.postag glbc<parse>@type kubernetes</parse></source><source>@type tail@id in_tail_cluster_autoscalermultiline_flush_interval 5spath /var/log/cluster-autoscaler.logpos_file /var/log/fluentd-cluster-autoscaler.log.postag cluster-autoscaler<parse>@type kubernetes</parse></source># Example:# 2017-02-09T00:15:57.992775796Z AUDIT: id="90c73c7c-97d6-4b65-9461-f94606ff825f" ip="104.132.1.72" method="GET" user="kubecfg" as="<self>" asgroups="<lookup>" namespace="default" uri="/api/v1/namespaces/default/pods"# 2017-02-09T00:15:57.993528822Z AUDIT: id="90c73c7c-97d6-4b65-9461-f94606ff825f" response="200"<source>@type tail@id in_tail_kube_apiserver_auditmultiline_flush_interval 5spath /var/log/kubernetes/kube-apiserver-audit.logpos_file /var/log/kube-apiserver-audit.log.postag kube-apiserver-audit<parse>@type multilineformat_firstline /^\S+\s+AUDIT:/# Fields must be explicitly captured by name to be parsed into the record.# Fields may not always be present, and order may change, so this just looks# for a list of key="\"quoted\" value" pairs separated by spaces.# Unknown fields are ignored.# Note: We can't separate query/response lines as format1/format2 because#       they don't always come one after the other for a given query.format1 /^(?<time>\S+) AUDIT:(?: (?:id="(?<id>(?:[^"\\]|\\.)*)"|ip="(?<ip>(?:[^"\\]|\\.)*)"|method="(?<method>(?:[^"\\]|\\.)*)"|user="(?<user>(?:[^"\\]|\\.)*)"|groups="(?<groups>(?:[^"\\]|\\.)*)"|as="(?<as>(?:[^"\\]|\\.)*)"|asgroups="(?<asgroups>(?:[^"\\]|\\.)*)"|namespace="(?<namespace>(?:[^"\\]|\\.)*)"|uri="(?<uri>(?:[^"\\]|\\.)*)"|response="(?<response>(?:[^"\\]|\\.)*)"|\w+="(?:[^"\\]|\\.)*"))*/time_format %Y-%m-%dT%T.%L%Z</parse></source>

注意：

elasticsearch取消了https认证，所以这里配置需要用http，重点！！！

验证

查看部署

登陆kibana

通过kibana查看是否有日志信息

kubectl port-forward service/quickstart-kb-http 5601:5601

通过kubectl port-forward转发，可以通过http://127.0.0.1:5601/访问kibana服务

服务会自动跳转到http://127.0.0.1:5601/app/home#/

注意需要输入账号和密码，具体参考我的文档：k8s上搭建elasticsearch服务

创建索引

如上图有4个步骤

按字母D进入
单机Index Patterns
输入索引pattern：logstash-*
创建索引

查看数据

按照上面三个步骤，现在可以看到容器的日志信息了

参考

注：参考了以下方案，不代表可行，反正我验证之后都没成功过

Kubernetes日志系统设计和实践(1)-EFK日志系统搭建（ECK方式）

Kubernetes日志系统设计和实践(1)-EFK日志系统搭建（ECK方式） | pigpi

实战带你一步步高效进行k8s日志收集

太绝了！实战带你一步步高效进行k8s日志收集

Kubernetes 基于 EFK 技术栈的日志收集实践

Kubernetes 基于 EFK 技术栈的日志收集实践_qhh0205的博客-CSDN博客

基于kubernetes实现efk日志收集--aliyun

基于kubernetes实现efk日志收集--aliyun_我的运维的技术博客_51CTO博客

搭建 EFK 日志系统

搭建 EFK 日志系统 · 从 Docker 到 Kubernetes 进阶手册

日志收集方案

参考：Kubernetes日志采集-EFK - 简书

k8s部署EFK实现日志管理（mac desktop）相关推荐

k8s资源监控和日志管理
k8s资源监控和日志管理 k8s监控资源使用率查看集群资源状态 kubectl get cs显示scheduler Unhealthy,controller-manager Unhealthy 监控 ...
k8s使用EFK实现日志采集存储展示
k8s集群上会跑各种各样的系统和应用程序的pod,而为了快速发现问题和更好的做日志监控,就必须要做日志的采集和集中存储展示了.综合考虑之下,我们推荐使用EFK技术栈来实现这个目的. k8s日志采集架构 ...
微服务nacos简单部署+网关设置+日志管理
目录 1.nacos简单部署 2.网关设置 3日志管理. 1.nacos简单部署 nacos引用的主要依赖是然后在配置文件进行注册(需要提前启动nacos插件,默认密码都是nacos) <dep ...
一文读懂开源日志管理方案 ELK 和 EFK 的区别
公众号关注「奇妙的 Linux 世界」设为「星标」,每天带你玩转 Linux ! 前言主流的 ELK (Elasticsearch, Logstash, Kibana) 目前已经转变为 EFK ...
Logging Operator - 优雅的云原生日志管理方案 (一)
Logging Operator是BanzaiCloud下开源的一个云原生场景下的日志采集方案.之前小白转载过崔大佬介绍的一篇文章,不过由于之前一直认为在单个k8s集群下同时管理Fluent bit和 ...
Kubernetes-基于EFK进行统一的日志管理
1.统一日志管理的整体方案通过应用和系统日志可以了解Kubernetes集群内所发生的事情,对于调试问题和监视集群活动来说日志非常有用.对于大部分的应用来说,都会具有某种日志机制.因此,大多数容器引 ...
容器日志管理 (2) 开源日志管理方案 ELK/EFK
本篇已加入<.NET Core on K8S学习实践系列文章索引>,可以点击查看更多容器化技术相关系列文章.上一篇<容器日志管理(1)>中介绍了Docker自带的logs子命令 ...
你必须知道的容器日志 (2) 开源日志管理方案 ELK/EFK
本篇已加入<.NET Core on K8S学习实践系列文章索引>,可以点击查看更多容器化技术相关系列文章.上一篇<你必须知道的容器日志(1)>中介绍了Docker自带的log ...
Kubernetes-基于EFK进行统一的日志管理原理(kibana查询语法)
全栈工程师开发手册 (作者:栾鹏) 架构系列文章 EFK安装部署参考:https://blog.csdn.net/luanpeng825485697/article/details/83312662 ...

k8s部署EFK实现日志管理（mac desktop）