elk日志管理系统搭建

简介

ELK是指Elasticsearch、Logstash、Kibana的简称

Elasticsearch : 实时全文搜索和分析引擎，提供搜集、分析、存储数据三大功能, 构建于Apache Lucene搜索引擎库之上。

Logstash: 一个用来搜集、分析、过滤日志的工具。它支持几乎任何类型的日志，包括系统日志、错误日志和自定义应用程序日志。它可以从许多来源接收日志，这些来源包括 syslog、消息传递（例如 RabbitMQ）和JMX，它能够以多种方式输出数据，包括电子邮件、websockets和Elasticsearch。

Kibana: 一个基于Web的图形界面，用于搜索、分析和可视化存储在 Elasticsearch指标中的日志数据。它利用Elasticsearch的REST接口来检索数据，不仅允许用户创建他们自己的数据的定制仪表板视图，还允许他们以特殊的方式查询和过滤数据

环境

系统: centos7
JDK: 1.8
三个二进制安装包下载地址:
https://download.csdn.net/download/zhaosongbin/11974259

jdk 安装

安装方法: https://zhaosongbin.blog.csdn.net/article/details/87927937

elasticsearch

安装

mkdir elasticsearch
cd elasticsearch
wget https://artifacts.elastic.co/downloads/elasticsearch/elasticsearch-7.4.2-linux-x86_64.tar.gz
tar -zxvf elasticsearch-7.4.2-linux-x86_64.tar.gz

启动

注意
不可使用root启动，否则会报以下错误
can not run elasticsearch as root

[root@erk bin]# ./elasticsearch
future versions of Elasticsearch will require Java 11; your Java version from [/usr/local/jdk1.8.0_211/jre] does not meet this requirement
[2019-11-12T17:37:39,170][WARN ][o.e.b.ElasticsearchUncaughtExceptionHandler] [gk-erk] uncaught exception in thread [main]
org.elasticsearch.bootstrap.StartupException: java.lang.RuntimeException: can not run elasticsearch as rootat org.elasticsearch.bootstrap.Elasticsearch.init(Elasticsearch.java:163) ~[elasticsearch-7.4.2.jar:7.4.2]at org.elasticsearch.bootstrap.Elasticsearch.execute(Elasticsearch.java:150) ~[elasticsearch-7.4.2.jar:7.4.2]at org.elasticsearch.cli.EnvironmentAwareCommand.execute(EnvironmentAwareCommand.java:86) ~[elasticsearch-7.4.2.jar:7.4.2]at org.elasticsearch.cli.Command.mainWithoutErrorHandling(Command.java:125) ~[elasticsearch-cli-7.4.2.jar:7.4.2]at org.elasticsearch.cli.Command.main(Command.java:90) ~[elasticsearch-cli-7.4.2.jar:7.4.2]at org.elasticsearch.bootstrap.Elasticsearch.main(Elasticsearch.java:115) ~[elasticsearch-7.4.2.jar:7.4.2]at org.elasticsearch.bootstrap.Elasticsearch.main(Elasticsearch.java:92) ~[elasticsearch-7.4.2.jar:7.4.2]
Caused by: java.lang.RuntimeException: can not run elasticsearch as rootat org.elasticsearch.bootstrap.Bootstrap.initializeNatives(Bootstrap.java:105) ~[elasticsearch-7.4.2.jar:7.4.2]at org.elasticsearch.bootstrap.Bootstrap.setup(Bootstrap.java:172) ~[elasticsearch-7.4.2.jar:7.4.2]at org.elasticsearch.bootstrap.Bootstrap.init(Bootstrap.java:349) ~[elasticsearch-7.4.2.jar:7.4.2]at org.elasticsearch.bootstrap.Elasticsearch.init(Elasticsearch.java:159) ~[elasticsearch-7.4.2.jar:7.4.2]... 6 more

添加用户和组 elk

groupadd elk
useradd -g elk elk

切换目录所属的用户组和用户

cd /home/elasticsearch
chgrp elk elasticsearch-7.4.2 -R
chown elk elasticsearch-7.4.2 -R

切换用户elk,然后启动

su elk
cd /home/elasticsearch/elasticsearch-7.4.2/bin/
./elasticsearch

服务器外部访问修改配置文件==elasticsearch.yml==

cd ../config/
pwd
/home/elasticsearch/elasticsearch-7.4.2/config
vim elasticsearch.yml

#network.host: 192.168.0.1
network.host: 192.168.1.1

#@discovery.seed_hosts: [“host1”, “host2”]
discovery.seed_hosts: [“192.168.1.1”, “[::1]”]

此时启动会报错

ERROR: [2] bootstrap checks failed
[1]: max file descriptors [4096] for elasticsearch process is too low, increase to at least [65535]
[2]: max virtual memory areas vm.max_map_count [65530] is too low, increase to at least [262144]

vm.max_map_count 表示虚拟内存大小，它是一个内核参数。elasticsearch 默认要求不低于 262144
nofile 表示进程允许打开的最大文件数。elasticsearch 进程要求可以打开的最大文件数不低于 65536
我们需要修改服务器内容,先切换到root用户

修改配置文件 /etc/sysctl.conf

vm.max_map_count=655360

修改配置文件 /etc/security/limits.conf

* soft     nofile     65536
* hard     nofile     131072
* soft     nproc      2048
* hard     nproc      4096

修改后需要重启服务器

然后重启服务

su elk
cd /home/elasticsearch/elasticsearch-7.4.2/bin/
./elasticsearch

后台启动

nohup ./elasticsearch &

Logstash

安装启动

mkdir  /home/logstash
cd /home/logstash
wget https://artifacts.elastic.co/downloads/logstash/logstash-7.4.2.tar.gz
tar -zxvf logstash-7.4.2.tar.gz
cd logstash-7.4.2/

修改配置

开放http地址,用来接收服务发送来的日志

vim /config/logstash-sample.conf

# Sample Logstash configuration for creating a simple
# Beats -> Logstash -> Elasticsearch pipeline.input {beats {port => 5044}tcp {# 这里其实把logstash作为服务，开启9250端口接收logback发出的消   host => "0.0.0.0" port => 9250 mode => "server" tags => ["tags"] codec => json_lines}}output {elasticsearch {hosts => ["http://192.168.1.1:9200"]index => "%{[@metadata][beat]}-%{[@metadata][version]}-%{+YYYY.MM.dd}"#user => "elastic"#password => "changeme"}
}

启动

nohup bin/logstash -f config/logstash-sample.conf &

Kibana

解压安装

mkdir /home/kibana/
cd /home/kibana/
wget https://artifacts.elastic.co/downloads/kibana/kibana-7.4.2-linux-x86_64.tar.gz
tar -zxvf kibana-7.4.2-linux-x86_64.tar.gz

修改配置

Kibana 服务外部访问还需要修改下配置

vim config/kibana.yml

#server.host: “localhost”
server.host: “192.168.1.1”

#elasticsearch.hosts: [“http://localhost:9200”]
elasticsearch.hosts: [“http://192.168.1.1:9200”]

#i18n.locale: “en”
i18n.locale: “zh-CN”

启动

注意Kibana不能使用root启动,否则会报错

Kibana should not be run as root.  Use --allow-root to continue.

切换目录所属的用户组和用户

cd /home/kibana
chgrp elk kibana-7.4.2-linux-x86_64 -R
chown elk kibana-7.4.2-linux-x86_64 -R

su elk
cd /home/kibana/kibana-7.4.2-linux-x86_64
nohup bin/kibana &

kibana查看进程

ps -ef |grep node

访问地址
http://192.168.1.1:5601