elk日志分析系统_部署ELK企业内部日志分析系统
2024-03-31 11:11:49
部署ELK企业内部日志分析系统
一、实验环境
二、基本环境部署
1.IP地址配置
2.主机名配置三台节点hosts文件要一致
[root@yichen-els-node1 ~]# cat /etc/hosts
192.168.150.30 yichen-els-node1
192.168.150.31 yichen-els-node2
192.168.150.32 yichen-els-node3
192.168.150.33 yichen-logstash
192.168.150.50 yichen-kibana
192.168.150.51 yichen-web
3.关闭SELinux
[root@yichen-els-node1 ~]# cat /etc/sysconfig/selinux | grep -v '^#' |grep 'SELINUX='
SELINUX=disabled
[root@yichen-els-node2 ~]# cat /etc/sysconfig/selinux | grep -v '^#' |grep 'SELINUX='
SELINUX=disabled
[root@yichen-els-node3 ~]# cat /etc/sysconfig/selinux | grep -v '^#' |grep 'SELINUX='
SELINUX=disabled
[root@yichen-logstash ~]# cat /etc/sysconfig/selinux | grep -v '^#' |grep 'SELINUX='
SELINUX=disabled
[root@yichen-kibana ~]# cat /etc/sysconfig/selinux | grep -v '^#' |grep 'SELINUX='
SELINUX=disabled
[root@yichen-web ~]# cat /etc/sysconfig/selinux | grep -v '^#' |grep 'SELINUX='
SELINUX=disabled
4.同步时间
yum install chrony -y 三个节点安装时间服务器
systemctl enable chronyd.service 开机自启服务
[root@yichen-els-node1 ~]# echo 'allow 192.168.150.0/24' >> /etc/chrony.conf
[root@yichen-els-node1 ~]# systemctl restart chronyd.service
[root@yichen-els-node1 ~]# chronyc sources
210 Number of sources = 4
MS Name/IP address Stratum Poll Reach LastRx Last sample
===============================================================================
^+ 203.107.6.88 2 6 17 11 -7966us[-7966us] +/- 27ms
^- time.cloudflare.com 3 6 144 9 +4070us[+4070us] +/- 79ms
^? tick.ntp.infomaniak.ch 0 6 0 - +0ns[ +0ns] +/- 0ns
^* 111.230.189.174 2 6 17 12 +1529us[+1253us] +/- 58ms[root@yichen-els-node2 ~]# echo 'server 192.168.150.30 iburst' >>/etc/chrony.conf
[root@yichen-els-node2 ~]# systemctl restart chronyd.service
[root@yichen-els-node2 ~]# chronyc sources
210 Number of sources = 4
MS Name/IP address Stratum Poll Reach LastRx Last sample
===============================================================================
^? yichen-els-node1 0 6 0 - +0ns[ +0ns] +/- 0ns
^* 139.199.214.202 2 6 17 1 +716us[ +799us] +/- 58ms
^+ 120.25.115.20 2 6 17 1 -707us[ -624us] +/- 25ms
^- 139.199.215.251 2 6 17 1 -452us[ -452us] +/- 56ms[root@yichen-els-node3 ~]# echo 'server 192.168.150.30 iburst' >>/etc/chrony.conf
[root@yichen-els-node3 ~]# systemctl restart chronyd.service
[root@yichen-els-node3 ~]# chronyc sources
210 Number of sources = 5
MS Name/IP address Stratum Poll Reach LastRx Last sample
===============================================================================
^? yichen-els-node1 0 6 0 - +0ns[ +0ns] +/- 0ns
^? 139.199.214.202 2 6 3 1 +1600us[+1600us] +/- 59ms
^? 203.107.6.88 2 6 7 2 -1847us[-1847us] +/- 22ms
^? h199-182-204-197.ip4.unm> 2 6 1 3 +118ms[ +118ms] +/- 237ms
^? 139.199.215.251 2 6 7 1 +212us[ +212us] +/- 52ms
## 剩余的三个几点都搭建时间服务器同上1.安装搭建elasticsearh集群和head插件
1.需要java环境(三个节点都要安装java环境)
yum install java java-1.8.0-openjdk-devel -y
java -version
openjdk version "1.8.0_242"
OpenJDK Runtime Environment (build 1.8.0_242-b08)
OpenJDK 64-Bit Server VM (build 25.242-b08, mixed mode)
2.YUM安装elasticsearch软件
cat /etc/yum.repos.d/elasticsearch.repo [elasticsearch]
name=Elasticsearch repository for 7.x packages
baseurl=https://artifacts.elastic.co/packages/7.x/yum
gpgcheck=1
gpgkey=https://artifacts.elastic.co/GPG-KEY-elasticsearch
enabled=0
autorefresh=1
type=rpm-md
yum install --enablerepo=elasticsearch elasticsearch -y
3.更改配置文件
##修改JVM的配置
[root@yichen-els-node1 ~]# vim /etc/elasticsearch/jvm.options
-Xms2g
-Xmx2g
默认为1G修改为2G
##修改elasticsearch配置文件
[root@yichen-els-node1 ~]# vim /etc/elasticsearch/elasticsearch.yml
[root@yichen-els-node1 ~]# cat /etc/elasticsearch/elasticsearch.yml | grep -v '^#'
cluster.name: whc-elk-application
node.name: yichen-els-node1 #### 三个节点分别对应自己的主机name
path.data: /var/lib/elasticsearch
path.logs: /var/log/elasticsearch
bootstrap.memory_lock: true
network.host: 192.168.150.30 #### 三个节点分别对应自己的IP地址
http.port: 9200
discovery.seed_hosts: ["192.168.150.30", "192.168.150.31","192.168.150.32"]
cluster.initial_master_nodes: ["192.168.150.30", "192.168.150.31","192.168.150.32"]
[root@yichen-els-node1 ~]# vim /etc/systemd/system.conf
DefaultLimitMEMLOCK=infinity #54行 重启虚拟机生效
4.启动集群并检查集群状态
[root@yichen-els-node1 ~]# systemctl restart elasticsearch.service
Job for elasticsearch.service failed because a fatal signal was delivered to the control process. See "systemctl status elasticsearch.service" and "journalctl -xe" for details.
[root@yichen-els-node1 ~]#
[root@yichen-els-node1 ~]# systemctl status elasticsearch.service
● elasticsearch.service - ElasticsearchLoaded: loaded (/usr/lib/systemd/system/elasticsearch.service; enabled; vendor preset: disabled)Active: failed (Result: signal) since 五 2020-04-10 22:22:22 CST; 12s agoDocs: http://www.elastic.coProcess: 7470 ExecStart=/usr/share/elasticsearch/bin/elasticsearch -p ${PID_DIR}/elasticsearch.pid --quiet (code=killed, signal=KILL)Main PID: 7470 (code=killed, signal=KILL)4月 10 22:22:02 yichen-els-node1 systemd[1]: Starting Elasticsearch...
4月 10 22:22:02 yichen-els-node1 elasticsearch[7470]: OpenJDK 64-Bit Server VM warning: Option UseConcMarkSweepGC was deprecated in version 9.0 and will likely be ...e release.
4月 10 22:22:22 yichen-els-node1 systemd[1]: elasticsearch.service: main process exited, code=killed, status=9/KILL
4月 10 22:22:22 yichen-els-node1 systemd[1]: Failed to start Elasticsearch.
4月 10 22:22:22 yichen-els-node1 systemd[1]: Unit elasticsearch.service entered failed state.
4月 10 22:22:22 yichen-els-node1 systemd[1]: elasticsearch.service failed.
Hint: Some lines were ellipsized, use -l to show in full.
内存原因，调整内存即可
[root@yichen-els-node1 ~]# systemctl restart elasticsearch.service
[root@yichen-els-node1 ~]#
[root@yichen-els-node1 ~]# systemctl status elasticsearch.service
● elasticsearch.service - ElasticsearchLoaded: loaded (/usr/lib/systemd/system/elasticsearch.service; enabled; vendor preset: disabled)Active: active (running) since 五 2020-04-10 22:25:42 CST; 1min 2s agoDocs: http://www.elastic.coMain PID: 7587 (java)CGroup: /system.slice/elasticsearch.service├─7587 /usr/share/elasticsearch/jdk/bin/java -Des.networkaddress.cache.ttl=60 -Des.networkaddress.cache.negative.ttl=10 -XX:+AlwaysPreTouch -Xss1m -Djava.awt.headless=tr...└─7680 /usr/share/elasticsearch/modules/x-pack-ml/platform/linux-x86_64/bin/controller4月 10 22:24:56 yichen-els-node1 systemd[1]: Starting Elasticsearch...
4月 10 22:24:56 yichen-els-node1 elasticsearch[7587]: OpenJDK 64-Bit Server VM warning: Option UseConcMarkSweepGC was deprecated in version 9.0 and will likely be ...e release.
4月 10 22:25:42 yichen-els-node1 systemd[1]: Started Elasticsearch.
Hint: Some lines were ellipsized, use -l to show in full.
启动服务OK
##防火墙放行服务
firewall-cmd --add-service=elasticsearch --permanent
firewall-cmd --reload
##启动服务
systemctl enable elasticsearch.service
systemctl start elasticsearch.service
##检查集群状态
[root@yichen-els-node1 ~]# curl http://192.168.150.30:9200/_cluster/health?pretty
{"cluster_name" : "test-elk-application","status" : "green", ## 显示绿色为正常"timed_out" : false,"number_of_nodes" : 3, ## 节点就为3"number_of_data_nodes" : 3,"active_primary_shards" : 0,"active_shards" : 0,"relocating_shards" : 0,"initializing_shards" : 0,"unassigned_shards" : 0,"delayed_unassigned_shards" : 0,"number_of_pending_tasks" : 0,"number_of_in_flight_fetch" : 0,"task_max_waiting_in_queue_millis" : 0,"active_shards_percent_as_number" : 100.0
}
5.安装elasticsearch-head插件
##先安装npm命令
yum install npm -y
##安装elasticsearch-head
git clone git://github.com/mobz/elasticsearch-head.git
cd elasticsearch-head/
https://github.com/Medium/phantomjs/releases/download/v2.1.1/phantomjs-2.1.1-linux-x86_64.tar.bz2
下载压缩包
tar jxvf /tmp/phantomjs/phantomjs-2.1.1-linux-x86_64.tar.bz2
npm install
##启动elasticsearch-head
npm run start &
[1] 7864
> elasticsearch-head@0.0.0 start /root/elasticsearch-head
> grunt server
Running "connect:server" (connect) task
Waiting forever...
Started connect web server on http://localhost:9100
[root@yichen-els-node1 elasticsearch-head]# netstat -antp | grep 9100
tcp 0 0 0.0.0.0:9100 0.0.0.0:* LISTEN 7874/grunt
[root@yichen-els-node1 elasticsearch-head]# firewall-cmd --add-port=9100/tcp --permanent
[root@yichen-els-node1 elasticsearch-head]# firewall-cmd --reload
##修改elasticsearch主配置文件
[root@yichen-els-node1 ~]# vim /etc/elasticsearch/elasticsearch.yml
http.cors.enabled: true
http.cors.allow-origin: “*”2.安装logstash
1.安装java环境
[root@yichen-logstash ~]# yum install java java-1.8.0-openjdk-devel -y
2.安装logstash软件
##配置yum源
[root@yichen-logstash ~]# vim /etc/yum.repos.d/logstash.repo
[root@yichen-logstash ~]# cat /etc/yum.repos.d/logstash.repo
[logstash-7.x]
name=Elastic repository for 7.x packages
baseurl=https://artifacts.elastic.co/packages/7.x/yum
gpgcheck=1
gpgkey=https://artifacts.elastic.co/GPG-KEY-elasticsearch
enabled=1
autorefresh=1
type=rpm-md
[root@yichen-logstash ~]#
##软件安装
yum install logstash
[root@yichen-logstash ~]# ln -s /usr/share/logstash/bin/logstash /usr/local/bin/
[root@yichen-logstash ~]# systemctl enable logstash.service
Created symlink from /etc/systemd/system/multi-user.target.wants/logstash.service to /etc/systemd/system/logstash.service.
[root@yichen-logstash ~]# systemctl restart logstash.service
[root@yichen-logstash ~]# systemctl status logstash.service
● logstash.service - logstashLoaded: loaded (/etc/systemd/system/logstash.service; enabled; vendor preset: disabled)Active: active (running) since 日 2020-04-12 18:40:21 CST; 10s agoMain PID: 17231 (java)CGroup: /system.slice/logstash.service└─17231 /bin/java -Xms1g -Xmx1g -XX:+UseConcMarkSweepGC -XX:CMSInitiatingOccupancyFraction=75 -XX:+UseCMSInitiatingOccupancyOnly -Djava.awt.headless=true -Dfile.encoding=UTF-8 -Djruby.co...4月 12 18:40:21 yichen-logstash systemd[1]: Started logstash.
[root@yichen-logstash ~]# netstat -antp
Active Internet connections (servers and established)
Proto Recv-Q Send-Q Local Address Foreign Address State PID/Program name
tcp 0 0 0.0.0.0:22 0.0.0.0:* LISTEN 6784/sshd
tcp 0 0 192.168.150.33:22 192.168.150.10:6501 ESTABLISHED 16898/sshd: root@pt
tcp 0 36 192.168.150.33:22 192.168.150.10:5338 ESTABLISHED 14581/sshd: root@pt
tcp6 0 0 :::22 :::* LISTEN 6784/sshd
tcp6 0 0 127.0.0.1:9600 :::* LISTEN 17812/java
[root@yichen-logstash ~]# firewall-cmd --add-port=9600/tcp --permanent
[root@yichen-logstash ~]# firewall-cmd --reload
3.logstash基本使用
[root@yichen-logstash ~]# logstash -e 'input { stdin{} } output { stdout{} }'
WARNING: Could not find logstash.yml which is typically located in $LS_HOME/config or /etc/logstash. You can specify the path using --path.settings. Continuing using the defaults
Could not find log4j2 configuration at path /usr/share/logstash/config/log4j2.properties. Using default config which logs errors to the console
[WARN ] 2020-04-12 18:46:58.043 [LogStash::Runner] multilocal - Ignoring the 'pipelines.yml' file because modules or command line options are specified
[INFO ] 2020-04-12 18:46:58.054 [LogStash::Runner] runner - Starting Logstash {"logstash.version"=>"7.6.2"}
[INFO ] 2020-04-12 18:46:59.718 [Converge PipelineAction::Create<main>] Reflections - Reflections took 36 ms to scan 1 urls, producing 20 keys and 40 values
[WARN ] 2020-04-12 18:47:00.620 [[main]-pipeline-manager] LazyDelegatingGauge - A gauge metric of an unknown type (org.jruby.RubyArray) has been created for key: cluster_uuids. This may result in invalid serialization. It is recommended to log an issue to the responsible developer/development team.
[INFO ] 2020-04-12 18:47:00.622 [[main]-pipeline-manager] javapipeline - Starting pipeline {:pipeline_id=>"main", "pipeline.workers"=>2, "pipeline.batch.size"=>125, "pipeline.batch.delay"=>50, "pipeline.max_inflight"=>250, "pipeline.sources"=>["config string"], :thread=>"#<Thread:0x401747aa run>"}
[INFO ] 2020-04-12 18:47:01.592 [[main]-pipeline-manager] javapipeline - Pipeline started {"pipeline.id"=>"main"}
The stdin plugin is now waiting for input:
[INFO ] 2020-04-12 18:47:01.685 [Agent thread] agent - Pipelines running {:count=>1, :running_pipelines=>[:main], :non_running_pipelines=>[]}
[INFO ] 2020-04-12 18:47:01.920 [Api Webserver] agent - Successfully started Logstash API endpoint {:port=>9600}
www.baidu.com 手动输入
[WARN ] 2020-04-12 18:47:38.485 [[main]<stdin] line - Received an event that has a different character encoding than you configured. {:text=>"wwwxE3xE3xE3.baidu.com", :expected_charset=>"UTF-8"}
/usr/share/logstash/vendor/bundle/jruby/2.5.0/gems/awesome_print-1.7.0/lib/awesome_print/formatters/base_formatter.rb:31: warning: constant ::Fixnum is deprecated
{"message" => "wwwxE3xE3xE3.baidu.com","host" => "yichen-logstash","@version" => "1","@timestamp" => 2020-04-12T10:47:38.498Z
}
www.sina.com.cn 手动输入
{"message" => "www.sina.com.cn","host" => "yichen-logstash","@version" => "1","@timestamp" => 2020-04-12T10:47:51.828Z
}
www.163.com
{"message" => "www.163.com","host" => "yichen-logstash","@version" => "1","@timestamp" => 2020-04-12T10:48:02.445Z
}
www.taobao.com
{"message" => "www.taobao.com","host" => "yichen-logstash","@version" => "1","@timestamp" => 2020-04-12T10:48:09.720Z
}
[root@wanghongcha0-logstash ~]# logstash -e 'input { stdin{} } output { elasticsearch { hosts=> ["192.168.150.30:9200"]} }'
WARNING: Could not find logstash.yml which is typically located in $LS_HOME/config or /etc/logstash. You can specify the path using --path.settings. Continuing using the defaults
Could not find log4j2 configuration at path /usr/share/logstash/config/log4j2.properties. Using default config which logs errors to the console
[WARN ] 2020-04-13 10:37:18.735 [LogStash::Runner] multilocal - Ignoring the 'pipelines.yml' file because modules or command line options are specified
[INFO ] 2020-04-13 10:37:18.743 [LogStash::Runner] runner - Starting Logstash {"logstash.version"=>"7.6.2"}
[INFO ] 2020-04-13 10:37:21.410 [Converge PipelineAction::Create<main>] Reflections - Reflections took 232 ms to scan 1 urls, producing 20 keys and 40 values
[INFO ] 2020-04-13 10:37:22.847 [[main]-pipeline-manager] elasticsearch - Elasticsearch pool URLs updated {:changes=>{:removed=>[], :added=>[http://192.168.150.30:9200/]}}
[WARN ] 2020-04-13 10:37:23.154 [[main]-pipeline-manager] elasticsearch - Restored connection to ES instance {:url=>"http://192.168.150.30:9200/"}
[INFO ] 2020-04-13 10:37:23.333 [[main]-pipeline-manager] elasticsearch - ES Output version determined {:es_version=>7}
[WARN ] 2020-04-13 10:37:23.336 [[main]-pipeline-manager] elasticsearch - Detected a 6.x and above cluster: the `type` event field won't be used to determine the document _type {:es_version=>7}
[INFO ] 2020-04-13 10:37:23.547 [[main]-pipeline-manager] elasticsearch - New Elasticsearch output {:class=>"LogStash::Outputs::ElasticSearch", :hosts=>["//192.168.150.30:9200"]}
[INFO ] 2020-04-13 10:37:23.637 [Ruby-0-Thread-6: :1] elasticsearch - Using default mapping template
[WARN ] 2020-04-13 10:37:23.678 [[main]-pipeline-manager] LazyDelegatingGauge - A gauge metric of an unknown type (org.jruby.specialized.RubyArrayOneObject) has been created for key: cluster_uuids. This may result in invalid serialization. It is recommended to log an issue to the responsible developer/development team.
[INFO ] 2020-04-13 10:37:23.698 [[main]-pipeline-manager] javapipeline - Starting pipeline {:pipeline_id=>"main", "pipeline.workers"=>2, "pipeline.batch.size"=>125, "pipeline.batch.delay"=>50, "pipeline.max_inflight"=>250, "pipeline.sources"=>["config string"], :thread=>"#<Thread:0x6e8e1512 run>"}
[INFO ] 2020-04-13 10:37:23.776 [Ruby-0-Thread-6: :1] elasticsearch - Attempting to install template {:manage_template=>{"index_patterns"=>"logstash-*", "version"=>60001, "settings"=>{"index.refresh_interval"=>"5s", "number_of_shards"=>1, "index.lifecycle.name"=>"logstash-policy", "index.lifecycle.rollover_alias"=>"logstash"}, "mappings"=>{"dynamic_templates"=>[{"message_field"=>{"path_match"=>"message", "match_mapping_type"=>"string", "mapping"=>{"type"=>"text", "norms"=>false}}}, {"string_fields"=>{"match"=>"*", "match_mapping_type"=>"string", "mapping"=>{"type"=>"text", "norms"=>false, "fields"=>{"keyword"=>{"type"=>"keyword", "ignore_above"=>256}}}}}], "properties"=>{"@timestamp"=>{"type"=>"date"}, "@version"=>{"type"=>"keyword"}, "geoip"=>{"dynamic"=>true, "properties"=>{"ip"=>{"type"=>"ip"}, "location"=>{"type"=>"geo_point"}, "latitude"=>{"type"=>"half_float"}, "longitude"=>{"type"=>"half_float"}}}}}}}
[INFO ] 2020-04-13 10:37:23.812 [Ruby-0-Thread-6: :1] elasticsearch - Installing elasticsearch template to _template/logstash
[INFO ] 2020-04-13 10:37:24.247 [Ruby-0-Thread-6: :1] elasticsearch - Creating rollover alias <logstash-{now/d}-000001>
[INFO ] 2020-04-13 10:37:24.675 [Ruby-0-Thread-6: :1] elasticsearch - Installing ILM policy {"policy"=>{"phases"=>{"hot"=>{"actions"=>{"rollover"=>{"max_size"=>"50gb", "max_age"=>"30d"}}}}}} to _ilm/policy/logstash-policy
[INFO ] 2020-04-13 10:37:25.653 [[main]-pipeline-manager] javapipeline - Pipeline started {"pipeline.id"=>"main"}
The stdin plugin is now waiting for input:
[INFO ] 2020-04-13 10:37:25.771 [Agent thread] agent - Pipelines running {:count=>1, :running_pipelines=>[:main], :non_running_pipelines=>[]}
[INFO ] 2020-04-13 10:37:26.088 [Api Webserver] agent - Successfully started Logstash API endpoint {:port=>9600}
www.baidu.com
www.taobao.com
www.jd.com
4.logstash配置文件使用
[root@yichen-logstash ~]# setfacl -m u:logstash:r /var/log/messages
[root@yichen-logstash ~]# cd /etc/logstash/conf.d/
[root@yichen-logstash conf.d]# vim system.conf
[root@wanghongcha0-logstash ~]# cat /etc/logstash/conf.d/system.conf
input {file {path => "/var/log/messages"type => "system"start_position => "beginning"}
}
output {elasticsearch {hosts => ["192.168.150.30:9200"]index => "whc-apache-%{+YYYY.MM.dd}"}
}
## 重启服务，有点慢，耐心等待
[root@yichen-logstash conf.d]# systemctl restart logstash.service
[root@wanghongcha0-logstash ~]# systemctl status logstash.service
● logstash.service - logstashLoaded: loaded (/etc/systemd/system/logstash.service; enabled; vendor preset: disabled)Active: active (running) since 一 2020-04-13 11:30:24 CST; 10min agoMain PID: 12098 (java)CGroup: /system.slice/logstash.service└─12098 /bin/java -Xms3g -Xmx3g -XX:+UseConcMarkSweepGC -XX:CMSInitiatingOccupancyFraction=75 -XX:+UseCMSInitiatingOccupancyOnly -Djava.awt.headless...4月 13 11:30:45 wanghongcha0-logstash logstash[12098]: [2020-04-13T11:30:45,534][INFO ][logstash.outputs.elasticsearch][main] Attempting to install te..._field"=
4月 13 11:30:45 wanghongcha0-logstash logstash[12098]: [2020-04-13T11:30:45,643][WARN ][org.logstash.instrument.metrics.gauge.LazyDelegatingGauge][main] A gau...
4月 13 11:30:45 wanghongcha0-logstash logstash[12098]: [2020-04-13T11:30:45,651][INFO ][logstash.javapipeline ][main] Starting pipeline {:pipeline_id=>"mai...
4月 13 11:30:46 wanghongcha0-logstash logstash[12098]: [2020-04-13T11:30:46,834][INFO ][logstash.inputs.beats ][main] Beats inputs: Starting input ...0:5044"}
4月 13 11:30:47 wanghongcha0-logstash logstash[12098]: [2020-04-13T11:30:47,208][INFO ][logstash.inputs.file ][main] No sincedb_path set, generati...sages"]}
4月 13 11:30:47 wanghongcha0-logstash logstash[12098]: [2020-04-13T11:30:47,240][INFO ][logstash.javapipeline ][main] Pipeline started {"pipeline.id"=>"main"}
4月 13 11:30:47 wanghongcha0-logstash logstash[12098]: [2020-04-13T11:30:47,421][INFO ][org.logstash.beats.Server][main] Starting server on port: 5044
4月 13 11:30:47 wanghongcha0-logstash logstash[12098]: [2020-04-13T11:30:47,432][INFO ][logstash.agent ] Pipelines running {:count=>1, :runn...nes=>[]}
4月 13 11:30:47 wanghongcha0-logstash logstash[12098]: [2020-04-13T11:30:47,476][INFO ][filewatch.observingtail ][main] START, creating Discoverer, W...lections
4月 13 11:30:47 wanghongcha0-logstash logstash[12098]: [2020-04-13T11:30:47,991][INFO ][logstash.agent ] Successfully started Logstash API e...t=>9600}
Hint: Some lines were ellipsized, use -l to show in full.
[root@yichen-logstash ~]# netstat -an | grep 5044
tcp6 0 0 :::5044 :::* LISTEN
[root@yichen-logstash ~]# firewall-cmd --add-port=5044/tcp --permanent
[root@yichen-logstash ~]# firewall-cmd --reload3.Beats轻量型数据采集器
1.下载并安装filebeat
[root@yichen-web ~]# curl -L -O https://artifacts.elastic.co/downloads/beats/filebeat/filebeat-7.6.2-x86_64.rpm
[root@yichen-web ~]# rpm -iv filebeat-7.6.2-x86_64.rpm
##### 66 行
#============================= Filebeat modules ===============================
filebeat.config.modules:path: ${path.config}/modules.d/*.ymlreload.enabled: ture 修改为rure，默认是default
##### 147行
#-------------------------- Elasticsearch output ------------------------------
output.elasticsearch:hosts: ["192.168.150.30:9200"] ### 修改为节点一的IP
### 激活样例配置文件
[root@yichen-web ~]# vim /etc/filebeat/filebeat.yml
[root@yichen-web ~]# cd /etc/filebeat/modules.d/
[root@yichen-web modules.d]# cp apache.yml.disabled apache.yml
[root@yichen-web modules.d]# vim apache.yml
[root@yichen-web modules.d]# cat apache.yml
- module: apache# Access logsaccess:enabled: true# Error logserror:enabled: true
[root@yichen-web ~]# systemctl enable filebeat.service
Created symlink from /etc/systemd/system/multi-user.target.wants/filebeat.service to /usr/lib/systemd/system/filebeat.service.
[root@yichen-web ~]# systemctl restart filebeat.service
[root@yichen-web ~]# echo '<h1>this is test apache </h1>' > /var/www/html/index.html
[root@yichen-web ~]# curl http://192.168.150.50/index.html
<h1>this is test apache </h1>### 完整的ELK:配置filebeat, 目标给logstash, 然后再给elasticsearch。
[root@yichen-web ~]# vim /etc/filebeat/filebeat.yml
### 160行
#----------------------------- Logstash output --------------------------------
output.logstash:# The Logstash hostshosts: ["192.168.150.33:5044"]
[root@wanghongcha0-logstash ~]# cat /etc/logstash/conf.d/apachelogs.conf
input {beats {port => 5044}
}
filter {grok {match => {"message" => "%{HTTPD_COMBINEDLOG}"}}
}
output {elasticsearch {hosts => ["http://192.168.150.30:9200/"]index => "whc-http %{+YYYY.MM.dd}"document_type => "apache_logs"}
}
[root@yichen-web ~]# systemctl status filebeat.service
● filebeat.service - Filebeat sends log files to Logstash or directly to Elasticsearch.Loaded: loaded (/usr/lib/systemd/system/filebeat.service; enabled; vendor preset: disabled)Active: active (running) since 一 2020-04-13 14:11:28 CST; 14s agoDocs: https://www.elastic.co/products/beats/filebeatMain PID: 8497 (filebeat)CGroup: /system.slice/filebeat.service└─8497 /usr/share/filebeat/bin/filebeat -e -c /etc/filebeat/filebeat.yml -path.home /usr/share/filebeat -path.config /etc/filebeat -path.data /var/lib/filebeat -path.logs /var/log/filebeat4月 13 14:11:28 yichen-web filebeat[8497]: 2020-04-13T14:11:28.495+0800 WARN beater/filebeat.go:152 Filebeat is unable to load the Ingest Node pipelines for the configured modules because the Elasticsearch output is not configured...
4月 13 14:11:28 yichen-web filebeat[8497]: 2020-04-13T14:11:28.496+0800 INFO [monitoring] log/log.go:118 Starting metrics logging every 30s
4月 13 14:11:28 yichen-web filebeat[8497]: 2020-04-13T14:11:28.496+0800 INFO instance/beat.go:439 filebeat start running.
4月 13 14:11:28 yichen-web filebeat[8497]: 2020-04-13T14:11:28.496+0800 INFO registrar/registrar.go:145 Loading registrar data from /var/lib/filebeat/registry/filebeat/data.json
4月 13 14:11:28 yichen-web filebeat[8497]: 2020-04-13T14:11:28.496+0800 INFO registrar/registrar.go:152 States Loaded from registrar: 0
4月 13 14:11:28 yichen-web filebeat[8497]: 2020-04-13T14:11:28.496+0800 WARN beater/filebeat.go:368 Filebeat is unable to load the Ingest Node pipelines for the configured modules because the Elasticsearch output is not configured...
4月 13 14:11:28 yichen-web filebeat[8497]: 2020-04-13T14:11:28.496+0800 INFO crawler/crawler.go:72 Loading Inputs: 1
4月 13 14:11:28 yichen-web filebeat[8497]: 2020-04-13T14:11:28.496+0800 INFO crawler/crawler.go:106 Loading and starting Inputs completed. Enabled inputs: 0
4月 13 14:11:28 yichen-web filebeat[8497]: 2020-04-13T14:11:28.496+0800 INFO cfgfile/reload.go:175 Config reloader started
4月 13 14:11:28 yichen-web filebeat[8497]: 2020-04-13T14:11:28.499+0800 INFO cfgfile/reload.go:235 Loading of config files completed.
Hint: Some lines were ellipsized, use -l to show in full.
[root@yichen-web ~]# ps -aux | grep 8497
root 8497 0.3 1.7 451640 36116 ? Ssl 14:11 0:00 /usr/share/filebeat/bin/filebeat -e -c /etc/filebeat/filebeat.yml -path.home /usr/share/filebeat -path.config /etc/filebeat -path.data /var/lib/filebeat -path.logs /var/log/filebeat
root 8512 0.0 0.0 112724 988 pts/0 R+ 14:12 0:00 grep --color=auto 8497
[root@yichen-web ~]#4.Kibana安装配置
1.安装kibana
##配置yum源
[root@yichen-kibana ~]# vim /etc/yum.repos.d/kibana.repo
[root@yichen-kibana ~]# cat /etc/yum.repos.d/kibana.repo
[kibana-7.x]
name=Kibana repository for 7.x packages
baseurl=https://artifacts.elastic.co/packages/7.x/yum
gpgcheck=1
gpgkey=https://artifacts.elastic.co/GPG-KEY-elasticsearch
enabled=1
autorefresh=1
type=rpm-md
##安装
[root@yichen-kibana ~]# yum install kibana -y
2.设置kibana的主配置文件
[root@yichen-kibana ~]# vim /etc/kibana/kibana.yml
[root@yichen-kibana ~]# cat /etc/kibana/kibana.yml | grep -Ev "^$|[#;]"
server.port: 5601
server.host: "0.0.0.0"
elasticsearch.hosts: ["http://192.168.150.30:9200"]
kibana.index: ".kibana"
i18n.locale: "zh-CN"
3.启动服务
[root@yichen-kibana ~]# firewall-cmd --add-service=kibana --permanent
[root@yichen-kibana ~]# firewall-cmd --reload
[root@yichen-kibana ~]# systemctl enable kibana.service
Created symlink from /etc/systemd/system/multi-user.target.wants/kibana.service to /etc/systemd/system/kibana.service.
[root@yichen-kibana ~]# systemctl start kibana.service
[root@yichen-kibana ~]# systemctl status kibana.service
● kibana.service - KibanaLoaded: loaded (/etc/systemd/system/kibana.service; enabled; vendor preset: disabled)Active: active (running) since 日 2020-04-12 20:48:04 CST; 9s agoMain PID: 7547 (node)CGroup: /system.slice/kibana.service└─7547 /usr/share/kibana/bin/../node/bin/node /usr/share/kibana/bin/../src/cli -c /etc/kibana/kibana.yml4月 12 20:48:04 yichen-kibana systemd[1]: Started Kibana.
4月 12 20:48:10 yichen-kibana kibana[7547]: {"type":"log","@timestamp":"2020-04-12T12:48:10Z","tags":["info","plugins-service"],"pid":7547,"message":"Plugin "case" is disabled."}
[root@yichen-kibana ~]# netstat -antp | grep node
tcp 0 0 0.0.0.0:5601 0.0.0.0:* LISTEN 7547/node
tcp 0 0 192.168.150.51:44692 192.168.150.30:9200 ESTABLISHED 7547/node
tcp 0 0 192.168.150.51:44698 192.168.150.30:9200 ESTABLISHED 7547/node
tcp 0 0 192.168.150.51:44720 192.168.150.30:9200 ESTABLISHED 7547/node
tcp 0 0 192.168.150.51:44694 192.168.150.30:9200 ESTABLISHED 7547/node
tcp 0 0 192.168.150.51:44716 192.168.150.30:9200 ESTABLISHED 7547/node
tcp 0 0 192.168.150.51:44696 192.168.150.30:9200 ESTABLISHED 7547/node
tcp 0 0 192.168.150.51:44690 192.168.150.30:9200 ESTABLISHED 7547/node
tcp 0 0 192.168.150.51:44724 192.168.150.30:9200 ESTABLISHED 7547/node
tcp 0 0 192.168.150.51:44722 192.168.150.30:9200 ESTABLISHED 7547/node
tcp 0 0 192.168.150.51:44718 192.168.150.30:9200 ESTABLISHED 7547/node
tcp 0 0 192.168.150.51:44710 192.168.150.30:9200 ESTABLISHED 7547/node
4.配置使用kibana
[root@yichen-kibana ~]# vim 1.sh
[root@yichen-kibana ~]# chmod a+x 1.sh
[root@yichen-kibana ~]# cat 1.sh
#!/bin/bash
for i in `seq 1 100000`
do curl http://192.168.150.50
done4.1.访问kibana
elk日志分析系统_部署ELK企业内部日志分析系统相关推荐
- python wifi探针_【逗老师带你学IT】PRTG监控系统配合树莓派采集企业内部无线网络质量...
本文介绍了一种,如何通过树莓派采集企业内部无线网络质量,将树莓派变成无线探针,并在PRTG网络监控平台上进行显示的方法. 可以监控我们感兴趣的无线网络的各项指标,包括 无线丢包 ping测试最大.最小 ...
- 货物贸易外汇监测系统 企业版_重点耗能企业能耗监测计量系统,能源管控平台方案...
国家在十三五规划中针对重点耗能企业的能耗管控要求在今年年底即将进行检查落实情况,目前时间紧迫,对于各个省份的能管要求也有不同,但也需从根本上落实能源能耗的管控和治理问题. 而目前智慧化的设备和系统是帮 ...
- java tomcat 日志分析工具_设计一个Tomcat访问日志分析工具
常使用web服务器的朋友大都了解,一般的web server有两部分日志: 一是运行中的日志,它主要记录运行的一些信息,尤其是一些异常错误日志信息 二是访问日志信息,它记录的访问的时间,IP,访问的资 ...
- dataguard日志传输模式解析_网络运维基础 日志审计
点击上面蓝字关注我们 综合日志审计平台 综合日志审计平台通过集中采集信息系统中的系统安全事件.用户访问记录.系统运行日志.系统运行状态等各类信息,经过规范化.过滤.归并和告警分析等处理后,以统一格式的 ...
- java日志切割工具_用 Java 实现的日志切割清理工具
对于服务器的日常维护来说,日志清理是非常重要的事情,如果残留日志过多则严重浪费磁盘空间同时影响服务的性能.如果用手工方式进行清理,会花费太多时间,并且很多时候难以满足实际要求.例如:如何在每个星期六凌 ...
- 【逗老师带你学IT】PRTG监控系统配合树莓派采集企业内部无线网络质量
目录 本文介绍了一种,如何通过树莓派采集企业内部无线网络质量,将树莓派变成无线探针,并在PRTG网络监控平台上进行显示的方法. 实现原理 部署方法 一.树莓派无线网络连接 二.编写shell脚本 三. ...
- 日志管理系统排名_目前较好的日志管理系统有哪些？
企业网络日志管理是企业IT管理员的重要工作之一,日志分析对网络安全具有非常重要的意义.随着互联网的飞速发展,企业网络信息安全面临的挑战越来越大,内鬼恶意操作,外部网络攻击,内部误操作等都能破坏企业网络 ...
- MySQL长途售票系统_基于SSH的长途汽车票务售票系统的设计(Struts2,MySQL)(含录像)...
基于SSH的长途汽车票务售票系统的设计(Struts2,MySQL)(含录像)(毕业论文说明书14000字,程序代码,MySQL数据库)摘 要 随着科学技术的不断提高,计算机科学日渐成熟,其强大的功 ...
- labview虚拟心电监测系统_基于LabVIEW实现的心电监护系统
基于 LabVIEW 实现的心电监护系统 王步青 ; 王卫东 ; 李开元 [摘 要] 心电监护是监护系统中的重要组成部分 , 可获得监护对象心电信号的具体 信息 , 对研究不同状态下的心脏状态具有重要 ...
最新文章
- bool变量取反_Task 01：Python基础入门：从变量到异常处理（第1天）
- linux系统root默认密码是多少钱,linux root默认密码忘记后的解决方法
- 三个好用的并发工具类
- Docker CE/EE 原生支持Kubernetes
- sklearn搭建线性模型的总结
- python 字符串 包含 列表_python中包含字符串列表的列
- 郑州大学linux试题,郑州大学Linux讲义 PPT
- centos7 安装node
- Android7.1+查看audio policy使用.conf/.xml(二十七)
- 经典领导选举算法：Bully 算法
- Python：将从chrome中复制的cookies转换为字典的函数
- kitti raw数据处理--跑vins
- nod32 升级方法
- 未来教育 计算机四级题库,未来教育计算机等级考试四级数据库工程师题库.docx...
- matlab 股票分时图_利用Matlab读取股市数据
- lesson5: C++11
- c++switch语言,C++ switch语句
- Flash常用源代码大全
- C语言——函数的声明
- 手游测试之新功能的通用测试点
热门文章
- 1.Spring Security 详细简绍与入门
- python小游戏开发，使用python实现英语打字游戏
- Linux文本搜索工具grep
- 4444端口 linux,Docker Container无法访问本地主机端口4444。为什么？
- Android图片颜色比例,Android开发学习之路-图片颜色获取器开发（1）
- 单元格不规则数据分列_菜鸟记527怎么3秒做完别人零下20度加班处理人工换行的数据？...
- python 画出决策边界_决策边界可视化，让你的分类合理有序
- python多程优化_Python 基本功: 13. 多线程运算提速
- Java实现简单的RPC框架
- 启程 - 《每日五分钟搞定大数据》