SS00011.elasticsearch——|HadoopElasticSearch集中式日志分析系统.v11|—

一、filter插件

### --- Filter插件~~~     Logstash之所以强悍的主要原因是filter插件；
~~~     通过过滤器的各种组合可以得到我们想要的结构化数据。
~~~     官网地址：https://www.elastic.co/guide/en/Logstash/current/plugins-filters-grok.html

### --- grok正则表达式
~~~     grok正则表达式是Logstash非常重要的一个环节；可以通过grok非常方便的将数据拆分和索引~~~     # 语法格式：
~~~     (?<name>pattern)
~~~     ？<name>表示要取出里面的值，pattern就是正则表达式

二、收集控制台输入数据，采集日期时间出来

### --- 开发配置文件~~~     # 开发配置文件
[root@hadoop02 ~]# vim /opt/yanqi/servers/es/Logstash/config/filter.conf
~~~     # 写入配置文件参数
input {stdin{}} filter {grok{ match => {"message" => "(?<date>\d+\.\d+)\s+"}}
}
output {stdout{codec => rubydebug}}

### --- 检查配置文件的完整性~~~     # 检查配置文件完整性
[root@hadoop02 ~]# /opt/yanqi/servers/es/Logstash/bin/logstash \
-f /opt/yanqi/servers/es/Logstash/config/filter.conf -t
~~~     # 输出如下配置参数
Configuration OK
Config Validation Result: OK. Exiting Logstash

### --- 启动Logstash服务~~~     # 启动logstash服务
[root@hadoop02 ~]# /opt/yanqi/servers/es/Logstash/bin/logstash \
-f /opt/yanqi/servers/es/Logstash/config/filter.conf
~~~     # 控制台输入文字
11.11 神棍节！！
~~~     # 输出参数：
{"date" => "11.11","message" => "11.11 神棍节！！","@version" => "1","@timestamp" => 2021-11-26T09:06:02.387Z,"host" => "hadoop02"
}

三、使用grok收集nginx日志数据

### --- nginx一般打印出来的日志格式如下~~~     这种日志是非格式化的，通常，我们获取到日志后，
~~~     还要使用mapreduce 或者spark 做一下清洗操作，就是将非格式化日志编程格式化日志；
~~~     在清洗的时候，如果日志的数据量比较大，那么也是需要花费一定的时间的；
~~~     所以可以使用Logstash 的grok 功能，将nginx 的非格式化数据采集成格式化数据：

~~~     # 插入参数解析后的数据：详见四.6章节36.157.150.1 - - [05/Nov/2019:12:59:28 +0800] "GET/phpmyadmin_8c1019c9c0de7a0f/js/get_scripts.js.php?scripts%5B%5D=jquery/jquery-1.11.1.min.js&scripts%5B%5D=sprintf.js&scripts%5B%5D=ajax.js&scripts%5B%5D=keyhandler.js&scripts%5B%5D=jquery/jquery-ui-1.11.2.min.js&scripts%5B%5D=jquery/jquery.cookie.js&scripts%5B%5D=jquery/jquery.mousewheel.js&scripts%5B%5D=jquery/jquery.event.drag-2.2.js&scripts%5B%5D=jquery/jquery-ui-timepickeraddon.js&scripts%5B%5D=jquery/jquery.ba-hashchange-1.3.js HTTP/1.1" 200 139613 "-" "Mozilla/5.0 (Windows NT 6.1; WOW64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/45.0.2454.101 Safari/537.36"
~~~     # 输出参数
{"time_local" => "05/Nov/2019:12:59:28 +0800","@version" => "1","host" => "hadoop02","message" => "36.157.150.1 - - [05/Nov/2019:12:59:28 +0800] \"GET/phpmyadmin_8c1019c9c0de7a0f/js/get_scripts.js.php?scripts%5B%5D=jquery/jquery-1.11.1.min.js&scripts%5B%5D=sprintf.js&scripts%5B%5D=ajax.js&scripts%5B%5D=keyhandler.js&scripts%5B%5D=jquery/jquery-ui-1.11.2.min.js&scripts%5B%5D=jquery/jquery.cookie.js&scripts%5B%5D=jquery/jquery.mousewheel.js&scripts%5B%5D=jquery/jquery.event.drag-2.2.js&scripts%5B%5D=jquery/jquery-ui-timepickeraddon.js&scripts%5B%5D=jquery/jquery.ba-hashchange-1.3.js HTTP/1.1\" 200 139613 \"-\" \"Mozilla/5.0 (Windows NT 6.1; WOW64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/45.0.2454.101 Safari/537.36\"","rawrequest" => "GET/phpmyadmin_8c1019c9c0de7a0f/js/get_scripts.js.php?scripts%5B%5D=jquery/jquery-1.11.1.min.js&scripts%5B%5D=sprintf.js&scripts%5B%5D=ajax.js&scripts%5B%5D=keyhandler.js&scripts%5B%5D=jquery/jquery-ui-1.11.2.min.js&scripts%5B%5D=jquery/jquery.cookie.js&scripts%5B%5D=jquery/jquery.mousewheel.js&scripts%5B%5D=jquery/jquery.event.drag-2.2.js&scripts%5B%5D=jquery/jquery-ui-timepickeraddon.js&scripts%5B%5D=jquery/jquery.ba-hashchange-1.3.js HTTP/1.1","@timestamp" => 2021-11-26T09:40:40.657Z,"clientip" => "36.157.150.1","http_referer" => "\"-\"","status" => "200","body_bytes_sent" => "139613","agent" => "\"Mozilla/5.0 (Windows NT 6.1; WOW64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/45.0.2454.101 Safari/537.36\""
}

四、在线安装grok插件

### --- 在线安装grok插件~~~     # 更改镜像源地址
[root@hadoop02 ~]# vim /opt/yanqi/servers/es/Logstash/Gemfile
~~~     # 第4/5行配置如下参数
# source "https://rubygems.org"                                 # 将这个镜像源注释掉
source "https://gems.ruby-china.com/"                           # 配置成中国的这个镜像源

### --- 准备在线安装~~~     # 在线安装grok插件
[root@hadoop02 ~]# cd /opt/yanqi/servers/es/Logstash/
[root@hadoop02 Logstash]# bin/logstash-plugin install logstash-filter-grok
~~~     # 输出参数
Validating logstash-filter-grok
Installing logstash-filter-grok
Installation successful

### --- 开发Logstash的配置文件~~~     # 定义Logstash的配置文件如下，我们从控制台输入nginx的日志数据，然后经过filter的过滤，将我们的日志文件转换成为标准的数据格式
[root@hadoop02 ~]# vim /opt/yanqi/servers/es/Logstash/config/monitor_nginx.conf
~~~     # 写入配置参数
input {stdin{}}filter {grok {
match => {"message" => "%{IPORHOST:clientip} \- \- \[%{HTTPDATE:time_local}\] \"(?:%{WORD:method} %{NOTSPACE:request}(?:HTTP/%{NUMBER:httpversion})?|%{DATA:rawrequest})\" %{NUMBER:status} %{NUMBER:body_bytes_sent} %{QS:http_referer} %{QS:agent}"}}
}
output {stdout{codec => rubydebug}}

### --- 检查配置文件完整性~~~     # 检查配置文件完整性
[root@hadoop02 ~]# /opt/yanqi/servers/es/Logstash/bin/logstash \
-f /opt/yanqi/servers/es/Logstash/config/monitor_nginx.conf -t
~~~     # 输出参数
Configuration OK
Config Validation Result: OK. Exiting Logstash

### --- 启动Logstash~~~     # 执行以下命令启动Logstash
[root@hadoop02 ~]# /opt/yanqi/servers/es/Logstash/bin/logstash \
-f /opt/yanqi/servers/es/Logstash/config/monitor_nginx.conf
~~~     # 输出参数：详情查看6数据参数输出

### --- 从控制台输入nginx日志文件数据~~~     # 输入第一条数据
113.31.119.183 - - [05/Nov/2019:12:59:27 +0800] "GET /phpmyadmin_8c1019c9c0de7a0f/js/messages.php? lang=zh_CN&db=&collation_connection=utf8_unicode_ci&token=6a44d72481633c90bffcfd42f11e25a1 HTTP/1.1" 200 8131 "-" "Mozilla/5.0 (Windows NT 6.1; WOW64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/45.0.2454.101 Safari/537.36"
~~~     # 输出参数
{"time_local" => "05/Nov/2019:12:59:27 +0800","@version" => "1","host" => "hadoop02","message" => "113.31.119.183 - - [05/Nov/2019:12:59:27 +0800] \"GET /phpmyadmin_8c1019c9c0de7a0f/js/messages.php? lang=zh_CN&db=&collation_connection=utf8_unicode_ci&token=6a44d72481633c90bffcfd42f11e25a1 HTTP/1.1\" 200 8131 \"-\" \"Mozilla/5.0 (Windows NT 6.1; WOW64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/45.0.2454.101 Safari/537.36\"","rawrequest" => "GET /phpmyadmin_8c1019c9c0de7a0f/js/messages.php? lang=zh_CN&db=&collation_connection=utf8_unicode_ci&token=6a44d72481633c90bffcfd42f11e25a1 HTTP/1.1","@timestamp" => 2021-11-26T09:35:04.242Z,"clientip" => "113.31.119.183","http_referer" => "\"-\"","status" => "200","body_bytes_sent" => "8131","agent" => "\"Mozilla/5.0 (Windows NT 6.1; WOW64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/45.0.2454.101 Safari/537.36\""
}

~~~     # 输入第二条数据36.157.150.1 - - [05/Nov/2019:12:59:28 +0800] "GET /phpmyadmin_8c1019c9c0de7a0f/js/get_scripts.js.php?scripts%5B%5D=jquery/jquery-1.11.1.min.js&scripts%5B%5D=sprintf.js&scripts%5B%5D=ajax.js&scripts%5B%5D=keyhandler.js&scripts%5B%5D=jquery/jquery-ui-1.11.2.min.js&scripts%5B%5D=jquery/jquery.cookie.js&scripts%5B%5D=jquery/jquery.mousewheel.js&scripts%5B%5D=jquery/jquery.event.drag-2.2.js&scripts%5B%5D=jquery/jquery-ui-timepickeraddon.js&scripts%5B%5D=jquery/jquery.ba-hashchange-1.3.js HTTP/1.1" 200 139613 "-" "Mozilla/5.0 (Windows NT 6.1; WOW64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/45.0.2454.101 Safari/537.36"
~~~     # 输出参数
{"time_local" => "05/Nov/2019:12:59:28 +0800","@version" => "1","host" => "hadoop02","message" => "36.157.150.1 - - [05/Nov/2019:12:59:28 +0800] \"GET /phpmyadmin_8c1019c9c0de7a0f/js/get_scripts.js.php?scripts%5B%5D=jquery/jquery-1.11.1.min.js&scripts%5B%5D=sprintf.js&scripts%5B%5D=ajax.js&scripts%5B%5D=keyhandler.js&scripts%5B%5D=jquery/jquery-ui-1.11.2.min.js&scripts%5B%5D=jquery/jquery.cookie.js&scripts%5B%5D=jquery/jquery.mousewheel.js&scripts%5B%5D=jquery/jquery.event.drag-2.2.js&scripts%5B%5D=jquery/jquery-ui-timepickeraddon.js&scripts%5B%5D=jquery/jquery.ba-hashchange-1.3.js HTTP/1.1\" 200 139613 \"-\" \"Mozilla/5.0 (Windows NT 6.1; WOW64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/45.0.2454.101 Safari/537.36\"","rawrequest" => "GET /phpmyadmin_8c1019c9c0de7a0f/js/get_scripts.js.php?scripts%5B%5D=jquery/jquery-1.11.1.min.js&scripts%5B%5D=sprintf.js&scripts%5B%5D=ajax.js&scripts%5B%5D=keyhandler.js&scripts%5B%5D=jquery/jquery-ui-1.11.2.min.js&scripts%5B%5D=jquery/jquery.cookie.js&scripts%5B%5D=jquery/jquery.mousewheel.js&scripts%5B%5D=jquery/jquery.event.drag-2.2.js&scripts%5B%5D=jquery/jquery-ui-timepickeraddon.js&scripts%5B%5D=jquery/jquery.ba-hashchange-1.3.js HTTP/1.1","@timestamp" => 2021-11-26T09:35:28.894Z,"clientip" => "36.157.150.1","http_referer" => "\"-\"","status" => "200","body_bytes_sent" => "139613","agent" => "\"Mozilla/5.0 (Windows NT 6.1; WOW64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/45.0.2454.101 Safari/537.36\""
}

SS00011.elasticsearch——|HadoopElasticSearch集中式日志分析系统.v11|——|Elasticsearch.v11|相关推荐

SS00003.elasticsearch——|HadoopElasticSearch集中式日志分析系统.v03|——|Elasticsearch.v03|
一.Elasticsearch 集群环境准备 ### --- hadoop01~03修改系统配置:修改/etc/sysctl.conf~~~ # 修改/etc/sysctl.conf [root@ha ...
SS00014.elasticsearch——|HadoopElasticSearch集中式日志分析系统.v14|——|Elasticsearch.v14|
一.Filebeat ### --- Filebeat~~~ Filebeat主要是为了解决Logstash工具比较消耗资源比较重的问题, ~~~ 因为Logstash是Java语言编写, ~~~ 所 ...
ELK+Kafka集群日志分析系统
因为是自己本地写好的word文档复制进来的.格式有些出入还望体谅.如有错误请回复.谢谢! 一. 系统介绍 2 二. 版本说明 3 三. 服务部署 3 1) JDK部署 3 2) Elasticsear ...
ES 集中式日志分析平台 Elastic Stack（介绍）
一.ELK 介绍 ELK 构建在开源基础之上,让您能够安全可靠地获取任何来源.任何格式的数据,并且能够实时地对数据进行搜索.分析和可视化. 最近查看 ELK 官方网站,发现新一代的日志采集器 File ...
Logstash+Redis+Elasticsearch+Kibana+Nginx搭建日志分析系统
为什么80%的码农都做不了架构师?>>> 前言: 随着实时分析技术的发展及成本的降低,用户已经不仅仅满足于离线分析.目前我们服务的用户包括微博.微盘.云存储.弹性计算平台等十多 ...
企业级日志分析系统ELK（Elasticsearch , Logstash, Kibana）
企业级日志分析系统ELK(Elasticsearch , Logstash, Kibana) 前言一.ELK概述 1.ELK日志分析系统 2.ELK日志处理特点 3.Elasticsearch概述 ...
中小型研发团队架构实践：集中式日志ELK
一.集中式日志日志可分为系统日志.应用日志以及业务日志,系统日志给运维人员使用,应用日志给研发人员使用,业务日志给业务操作人员使用.我们这里主要讲解应用日志,通过应用日志来了解应用的信息和状态,以及 ...
ELK日志分析系统及相关
目录前言一.ELK概述 1.ELK日志分析系统 2.ELK中日志处理步骤二.Elasticsearch介绍 1.Elasticsearch的基础核心概念 2.mysql关系型数据库与Elasti ...
快速搭建ELK日志分析系统
官网地址:https://www.elastic.co/cn/ 官网权威指南:https://www.elastic.co/guide/cn/elasticsearch/guide/current/i ...
超详细 ELK 日志分析系统
文章目录一.ELK日志分析系统简介 1:ELK日志分析系统组成 2:日志处理步骤二:三款软件简介 1:Elasticsearch (1)Elasticsearch的概述 (2)Elasticsea ...

SS00011.elasticsearch——|HadoopElasticSearch集中式日志分析系统.v11|——|Elasticsearch.v11|

SS00011.elasticsearch——|HadoopElasticSearch集中式日志分析系统.v11|——|Elasticsearch.v11|相关推荐

最新文章

热门文章