一图胜千言

• 基础架构

• 工作原理

• Logstash工作原理

• Logstash工作流程

• ELK整体部署图

ELK 安装配置简化过程

1 基本配置vim /etc/hosts192.168.2.61         master-node192.168.2.62       data-node1192.168.2.63       data-node2wget https://artifacts.elastic.co/downloads/elasticsearch/elasticsearch-6.0.0.rpmrpm -ivh elasticsearch-6.0.0.rpmelasticsearch.ymljvm.optionslog4j2.propertiesvim /etc/elasticsearch/elasticsearch.yml cluster.name: master-node  # 集群中的名称node.name: master  # 该节点名称node.master: true  # 意思是该节点为主节点node.data: false  # 表示这不是数据节点network.host: 0.0.0.0  # 监听全部ip，在实际环境中应设置为一个安全的iphttp.port: 9200  # es服务的端口号discovery.zen.ping.unicast.hosts: ["192.168.2.61", "192.168.2.62", "192.168.2.63"] # 配置自动发现scp /etc/elasticsearch/elasticsearch.yml data-node1:/tmp/scp /etc/elasticsearch/elasticsearch.yml data-node2:/tmp/cp /tmp/elasticsearch.yml /etc/elasticsearch/elasticsearch.ymlsystemctl start elasticsearch.servicecurl '192.168.2.61:9200/_cluster/health?pretty'curl '192.168.2.61:9200/_cluster/state?pretty'
2 kibana配置wget https://artifacts.elastic.co/downloads/kibana/kibana-6.0.0-x86_64.rpmrpm -ivh kibana-6.0.0-x86_64.rpmvim /etc/kibana/kibana.ymlserver.port: 5601  # 配置kibana的端口server.host: 192.168.2.61  # 配置监听ip# 配置es服务器的ip，如果是集群则配置该集群中主节点的ipelasticsearch.url: "http://192.168.2.61:9200" # 配置kibana的日志文件路径，不然默认是messages里记录日志 logging.dest: /var/log/kibana.log  touch /var/log/kibana.log; chmod 777 /var/log/kibana.logsystemctl start kibanahttp://192.168.2.61:5601/
3 logstash配置192.168.2.62wget https://artifacts.elastic.co/downloads/logstash/logstash-6.0.0.rpmrpm -ivh logstash-6.0.0.rpmvim /etc/logstash/conf.d/syslog.conf input {  # 定义日志源syslog {type => "system-syslog"  # 定义类型port => 10514    # 定义监听端口}}output {  # 定义日志输出stdout {codec => rubydebug  # 将日志输出到当前的终端上显示}}cd /usr/share/logstash/bin检查配置./logstash --path.settings /etc/logstash/ -f /etc/logstash/conf.d/syslog.conf --config.test_and_exit配置kibana服务器的ip以及配置的监听端口vim /etc/rsyslog.conf#### RULES ####*.* @@192.168.2.62:10514systemctl restart rsyslog指定配置文件，启动logstashcd /usr/share/logstash/bin./logstash --path.settings /etc/logstash/ -f /etc/logstash/conf.d/syslog.conflogstash收集nginx日志
vim /etc/logstash/conf.d/nginx.conf input {file {  # 指定一个文件作为输入源path => "/var/log/nginx/access.log"  # 指定文件的路径start_position => "beginning"  # 指定何时开始收集type => "nginx"  # 定义日志类型，可自定义}
}
filter {  # 配置过滤器grok {match => { "message" => "%{IPORHOST:http_host} %{IPORHOST:clientip} - %{USERNAME:remote_user} \[%{HTTPDATE:timestamp}\] \"(?:%{WORD:http_verb} %{NOTSPACE:http_request}(?: HTTP/%{NUMBER:http_version})?|%{DATA:raw_http_request})\" %{NUMBER:response} (?:%{NUMBER:bytes_read}|-) %{QS:referrer} %{QS:agent} %{QS:xforwardedfor} %{NUMBER:request_time:float}"}  # 定义日志的输出格式}geoip {source => "clientip"}
}
output {stdout { codec => rubydebug }elasticsearch {hosts => ["192.168.2.61:9200"]index => "nginx-test-%{+YYYY.MM.dd}"}
}cd /usr/share/logstash/bin
./logstash --path.settings /etc/logstash/ -f /etc/logstash/conf.d/nginx.conf --config.test_and_exitcd /etc/nginx/http_virtual_host.d
vim elk.conf
server {listen 80;server_name elk.test.com;location / {proxy_pass      http://192.168.2.61:5601;proxy_set_header Host   $host;proxy_set_header X-Real-IP      $remote_addr;proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;}access_log  /tmp/elk_access.log main2;
}vim
log_format main2 '$http_host $remote_addr - $remote_user [$time_local] "$request" ''$status $body_bytes_sent "$http_referer" ''"$http_user_agent" "$upstream_addr" $request_time';
nginx -t
nginx -s reload
配置hosts  192.168.2.62 elk.eichong.comls /var/log/nginx/access.log
wc -l !$
重启logstash服务，生成日志的索引
systemctl restart logstash重启完成后，在es服务器上检查是否有nginx-test开头的索引生成
curl '192.168.2.61:9200/_cat/indices?v'nginx-test索引已经生成了，那么这时就可以到kibana上配置该索引
managent  index patterns   create index patterns
discover
http://192.168.2.61:5601/status 查看状态

最新版本yum安装

001 elasticsearchrpm --import https://artifacts.elastic.co/GPG-KEY-elasticsearch
vim /etc/yum.repos.d/elasticsearch.repo[elasticsearch-6.x]
name=Elasticsearch repository for 6.x packages
baseurl=https://artifacts.elastic.co/packages/6.x/yum
gpgcheck=1
gpgkey=https://artifacts.elastic.co/GPG-KEY-elasticsearch
enabled=1
autorefresh=1
type=rpm-mdyum install elasticsearch -y002 kibana
vim /etc/yum.repos.d/kibana.repo[kibana-6.x]
name=Kibana repository for 6.x packages
baseurl=https://artifacts.elastic.co/packages/6.x/yum
gpgcheck=1
gpgkey=https://artifacts.elastic.co/GPG-KEY-elasticsearch
enabled=1
autorefresh=1
type=rpm-mdyum install kibana -y003 logstash
vim /etc/yum.repos.d/logstash.repo[logstash-6.x]
name=Elastic repository for 6.x packages
baseurl=https://artifacts.elastic.co/packages/6.x/yum
gpgcheck=1
gpgkey=https://artifacts.elastic.co/GPG-KEY-elasticsearch
enabled=1
autorefresh=1
type=rpm-md

主要配置文件

001 elasticsearch
cat /etc/elasticsearch/elasticsearch.yml |grep ^[^#]cluster.name: my-elk
node.name: master
path.data: /var/lib/elasticsearch
path.logs: /var/log/elasticsearch
network.host: 0.0.0.0
http.port: 9200
discovery.zen.ping.unicast.hosts: ["172.19.1.216", "172.19.1.217"]002 kibana
cat /etc/kibana/kibana.yml |grep ^[^#]server.port: 5601
server.host: "172.19.1.216"
elasticsearch.url: "http://172.19.1.216:9200"
logging.dest: /var/log/kibana.log  # 文件需创建并授权003 logstash

汉化

https://github.com/anbai-inc/Kibana_Hanization

其他优秀博客

https://www.cnblogs.com/kevingrace/p/5919021.html
http://blog.51cto.com/zero01/2079879