CentOS 7安装ELK日志分析系统

1、Elastic Search安装

本次部署目录/data/elk下

cd /data/elk
wget https://artifacts.elastic.co/downloads/elasticsearch/elasticsearch-8.4.1-linux-x86_64.tar.gz

解压到当前目录

tar -zxvf elasticsearch-8.4.1-linux-x86_64.tar.gz

编辑elasticsearch.yml文件

cd /data/elk/elasticsearch-8.4.1/config/
vi elasticsearch.yml

新增以下内容

network.host: 0.0.0.0 #可远程访问
node.name: node-base #节点名称 这个与下面一点一定要配，不然即使启动成功也会操作超时或发生master_not_discovered_exception
cluster.initial_master_nodes: ["node-base"] #发现当前节点名称
path.conf: /data/ELK/elasticsearch/conf
path.data: /data/ELK/elasticsearch/data
http.port: 9200 #端口号
http.cors.allow-origin: "*" #以下皆是跨域配置
http.cors.enabled: true
http.cors.allow-headers : X-Requested-With,X-Auth-Token,Content-Type,Content-Length,Authorization
http.cors.allow-credentials: true

ElasticSearch启动不能用root账号，需要新建账户
创建新用户及授权

groupadd elsearch #增加用户组
useradd elsearch -g elsearch -p elsearch #新增用户并设置用户组
cd /data/elk
chown -R elsearch:elsearch elasticsearch-7.5.1 #修改文件属所用户
su elsearch
cd /data/elk/elasticsearch-8.4.1/bin/
sh elasticsearch > /data/elk/el.log &

启动过程中出现如下错误

[1]: max file descriptors [4096] for elasticsearch process is too low, increase to at least [65535]
[2]: max virtual memory areas vm.max_map_count [65530] is too low, increase to at least [262144]
[3]: the default discovery settings are unsuitable for production use; at least one of [discovery.seed_hosts, discovery.seed_providers, cluster.initial_master_nodes] must be configured

解决1：将当前用户的软硬限制调大
切换到root用户

su
vi /etc/security/limits.conf

新增以下配置

* soft nofile 65535
* hard nofile 65535
#也可以用@加账号
@elsearch soft nofile 65535
@elsearch hard nofile 65535
#执行
ulimit -n 65535
ulimit -n
#返回 65535
ulimit -H -n 65535
ulimit -H -n
#返回 65535

解决2：调大elasticsearch用户拥有的内存权限
切换到root用户

su
#临时修改
sysctl -w vm.max_map_count=262144
sysctl -a|grep vm.max_map_count #查看修改结果
#永久修改
vi /etc/sysctl.conf
#增加以下内容
vm.max_map_count=262144
#保存并执行
sysctl -p

解决3：没有配置node节点名称和主节点
修改配置文件

cd /data/elk/elasticsearch-8.4.1/config/
vi elasticsearch.yml
#修改以下配置
node.name: node-1
cluster.initial_master_nodes: ["node-1"]
#重新启动
sh elasticsearch &

出现以下画面启动成功

✅ Elasticsearch security features have been automatically configured!
✅ Authentication is enabled and cluster connections are encrypted.ℹ️  Password for the elastic user (reset with `bin/elasticsearch-reset-password -u elastic`):phtF6nPnroGhPCC_n9s9ℹ️  HTTP CA certificate SHA-256 fingerprint:7c6ff0a7ab717859c2d550fc3c06d0128b67791750e08ca70076cc75bd1dafc4ℹ️  Configure Kibana to use this cluster:
• Run Kibana and click the configuration link in the terminal when Kibana starts.
• Copy the following enrollment token and paste it into Kibana in your browser (valid for the next 30 minutes):eyJ2ZXIiOiI4LjQuMSIsImFkciI6WyIxOTIuMTY4LjEwMS4xMDI6OTIwMCJdLCJmZ3IiOiI3YzZmZjBhN2FiNzE3ODU5YzJkNTUwZmMzYzA2ZDAxMjhiNjc3OTE3NTBlMDhjYTcwMDc2Y2M3NWJkMWRhZmM0Iiwia2V5Ijoid01ySS1ZSUJzaEdRSzlZalE2a1g6YmxFMW1TeXpRZFNxcUZHYWUzcHJldyJ9ℹ️ Configure other nodes to join this cluster:
• Copy the following enrollment token and start new Elasticsearch nodes with `bin/elasticsearch --enrollment-token <token>` (valid for the next 30 minutes):eyJ2ZXIiOiI4LjQuMSIsImFkciI6WyIxOTIuMTY4LjEwMS4xMDI6OTIwMCJdLCJmZ3IiOiI3YzZmZjBhN2FiNzE3ODU5YzJkNTUwZmMzYzA2ZDAxMjhiNjc3OTE3NTBlMDhjYTcwMDc2Y2M3NWJkMWRhZmM0Iiwia2V5Ijoid3NySS1ZSUJzaEdRSzlZalE2a2k6Z2FOTzlEZVdSVnE1SGYtSWxLdFRyQSJ9If you're running in Docker, copy the enrollment token and run:

修改配置文件elasticsearch.yml

ingest.geoip.downloader.enabled: false #新增
xpack.security.enabled: true #修改为falsexpack.security.http.ssl:enabled: true #修改为falsekeystore.path: certs/http.p12

重新启动
访问http://ip:9200
如果不能访问请查看防火墙是否开启9200端口

firewall-cmd --zone=public --add-port=9200/tcp --permanent

自启配置
进入进入到 cd /etc/init.d 目录

cd /etc/init.d    　　【进入到目录】
vi elsearch 　　 【创建es系统启动服务文件】

添加以下内容，ES_HOME为es安装目录
su es为运行用户

#!/bin/bash
#chkconfig: 345 63 37
#description: elasticsearch
#processname: elasticsearch-8.4.1export ES_HOME=/usr/local/elasticsearch-8.4.1case $1 instart)su elsearch<<!cd $ES_HOME./bin/elasticsearch -d -p pidexit
!echo "elasticsearch is started";;stop)pid=`cat $ES_HOME/pid`kill -9 $pidecho "elasticsearch is stopped";;restart)pid=`cat $ES_HOME/pid`kill -9 $pidecho "elasticsearch is stopped"sleep 1su es<<!cd $ES_HOME./bin/elasticsearch -d -p pidexit
!echo "elasticsearch is started";;*)echo "start|stop|restart";;
esac
exit 0

修改权限

chmod 777 elsearch

添加和删除服务并设置启动方式

chkconfig --add elsearch　　　　【添加系统服务】
chkconfig --del elsearch　　　　【删除系统服务】

关闭和启动服务

service elsearch start　　　　　【启动】
service elsearch stop　　　　　 【停止】
service elsearch restart　　   【重启】

设置服务是否开机启动

chkconfig elsearch on　　　　　　【开启】
chkconfig elsearch off　　   　 【关闭】

2、Kibana安装

切换到root用户，下载安装包

su
cd /data/elk
wget https://artifacts.elastic.co/downloads/kibana/kibana-7.5.1-linux-x86_64.tar.gz

解压到当前目录

tar -zxvf kibana-7.5.1-linux-x86_64.tar.gz

编辑kibana.yml文件

cd /data/elk/kibana-7.5.1-linux-x86_64/config/
vi kibana.yml

新增以下内容

server.port: 5601
server.host: "0.0.0.0"
server.name: "kibana"
elasticsearch.url: "http://localhost:9200"
kibana.index: ".kibana"

保存并启动kibana

cd /data/elk/kibana-7.5.1-linux-x86_64/bin
#root不能直接启动kibana所以后面加--allow-root
sh kibana --allow-root > /data/elk/kibana.log &

启动出现错误

FATAL  Error: [elasticsearch.url]: definition for this key is missing

修改kibana.yml文件

elasticsearch.url: "http://localhost:9200"
#修改成下面
elasticsearch.hosts: ["http://localhost:9200"]

访问http://ip:5601
如果不能访问请查看防火墙是否开启5601端口

3、Logstash安装

切换到root用户，下载安装包

su
cd /data/elk
wget https://artifacts.elastic.co/downloads/kibana/logstash-7.5.1.tar.gz

解压到当前目录

tar -zxvf logstash-7.5.1.tar.gz

新增logstash.conf文件

cd /data/elk/logstash-7.5.1/bin/
vi logstash.conf

新增以下内容

input {tcp {port => 5044codec => json_lines}
}
output {elasticsearch {hosts => ["localhost:9200"]}
}

保存并启动logstash

cd /data/elk/logstash-7.5.1/bin/
sh logstash -f logstash.conf  &

查看日志，无报错信息，默认启动成功。

4、配置项目

添加logstash依赖

<dependency><groupId>net.logstash.logback</groupId><artifactId>logstash-logback-encoder</artifactId><version>6.3</version>
</dependency>

配置logback-spring.xml文件
加spring后缀是能使用标签
多环境配置

<?xml version="1.0" encoding="UTF-8"?>
<configuration>
<springProfile name="exp"><appender name="LOGSTASH" class="net.logstash.logback.appender.LogstashTcpSocketAppender"><destination>192.168.5.19:5044</destination><queueSize>1048576</queueSize><encoder class="net.logstash.logback.encoder.LoggingEventCompositeJsonEncoder"><providers><timestamp><timeZone>UTC</timeZone></timestamp><pattern><pattern>{"severity":"%level","service": "wbx-gateway","pid": "${PID:-}","thread": "%thread","class": "%logger{40}","rest": "%message->%ex{full}"}</pattern></pattern></providers></encoder></appender>
</springProfile>
<springProfile name="test"><appender name="LOGSTASH" class="net.logstash.logback.appender.LogstashTcpSocketAppender"><destination>192.168.5.19:5044</destination><queueSize>1048576</queueSize><encoder class="net.logstash.logback.encoder.LoggingEventCompositeJsonEncoder"><providers><timestamp><timeZone>UTC</timeZone></timestamp><pattern><pattern>{"severity":"%level","service": "wbx-gateway","pid": "${PID:-}","thread": "%thread","class": "%logger{40}","rest": "%message->%ex{full}"}</pattern></pattern></providers></encoder></appender>
</springProfile><property name="pattern" value="%d{yyyy-MM-dd:HH:mm:ss.SSS} [%thread] %-5level  %msg%n"/><appender name="STDOUT" class="ch.qos.logback.core.ConsoleAppender"><!-- 字符串System.out（默认）或者System.err --><target>System.out</target><!-- 对记录事件进行格式化 --><encoder class="ch.qos.logback.classic.encoder.PatternLayoutEncoder"><pattern>${pattern}</pattern></encoder></appender><springProfile name="dev"><root level="info"><appender-ref ref="STDOUT" />　</root>
</springProfile>
<springProfile name="exp"><root level="info"><appender-ref ref="LOGSTASH" />　</root>
</springProfile>
<springProfile name="test"><root level="info"><appender-ref ref="LOGSTASH" />　</root>
</springProfile>
</configuration>