CC00051.elasticsearch——|HadoopElasticSearch.V03|—

一、filter插件

### --- Filter插件~~~     Logstash之所以强悍的主要原因是filter插件；
~~~     通过过滤器的各种组合可以得到我们想要的结构化数据。
~~~     官网地址：https://www.elastic.co/guide/en/Logstash/current/plugins-filters-grok.html

### --- grok正则表达式
~~~     grok正则表达式是Logstash非常重要的一个环节；可以通过grok非常方便的将数据拆分和索引~~~     # 语法格式：
~~~     (?<name>pattern)
~~~     ？<name>表示要取出里面的值，pattern就是正则表达式

二、收集控制台输入数据，采集日期时间出来

### --- 开发配置文件[root@hadoop02 ~]# vim /opt/yanqi/servers/es/Logstash/config/filter.conf~~~写入配置文件参数
input {stdin{}} filter {grok{ match => {"message" => "(?<date>\d+\.\d+)\s+"}}
}
output {stdout{codec => rubydebug}}

### --- 检查配置文件完整性[root@hadoop02 ~]# /opt/yanqi/servers/es/Logstash/bin/logstash \
-f /opt/yanqi/servers/es/Logstash/config/filter.conf -t
~~~输出如下配置参数
Configuration OK
Config Validation Result: OK. Exiting Logstash

### --- 启动logstash服务[root@hadoop02 ~]# /opt/yanqi/servers/es/Logstash/bin/logstash \
-f /opt/yanqi/servers/es/Logstash/config/filter.conf~~~ 控制台输入文字
11.11 神棍节！！~~~输出参数：
{"date" => "11.11","message" => "11.11 神棍节！！","@version" => "1","@timestamp" => 2021-11-26T09:06:02.387Z,"host" => "hadoop02"
}

三、使用grok收集nginx日志数据

### --- nginx一般打印出来的日志格式如下~~~     这种日志是非格式化的，通常，我们获取到日志后，
~~~     还要使用mapreduce 或者spark 做一下清洗操作，就是将非格式化日志编程格式化日志；
~~~     在清洗的时候，如果日志的数据量比较大，那么也是需要花费一定的时间的；
~~~     所以可以使用Logstash 的grok 功能，将nginx 的非格式化数据采集成格式化数据：

### --- 插入参数解析后的数据：详见四.6章节36.157.150.1 - - [05/Nov/2019:12:59:28 +0800] "GET/phpmyadmin_8c1019c9c0de7a0f/js/get_scripts.js.php?scripts%5B%5D=jquery/jquery-1.11.1.min.js&scripts%5B%5D=sprintf.js&scripts%5B%5D=ajax.js&scripts%5B%5D=keyhandler.js&scripts%5B%5D=jquery/jquery-ui-1.11.2.min.js&scripts%5B%5D=jquery/jquery.cookie.js&scripts%5B%5D=jquery/jquery.mousewheel.js&scripts%5B%5D=jquery/jquery.event.drag-2.2.js&scripts%5B%5D=jquery/jquery-ui-timepickeraddon.js&scripts%5B%5D=jquery/jquery.ba-hashchange-1.3.js HTTP/1.1" 200 139613 "-" "Mozilla/5.0 (Windows NT 6.1; WOW64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/45.0.2454.101 Safari/537.36"
~~~输出参数
{"time_local" => "05/Nov/2019:12:59:28 +0800","@version" => "1","host" => "hadoop02","message" => "36.157.150.1 - - [05/Nov/2019:12:59:28 +0800] \"GET/phpmyadmin_8c1019c9c0de7a0f/js/get_scripts.js.php?scripts%5B%5D=jquery/jquery-1.11.1.min.js&scripts%5B%5D=sprintf.js&scripts%5B%5D=ajax.js&scripts%5B%5D=keyhandler.js&scripts%5B%5D=jquery/jquery-ui-1.11.2.min.js&scripts%5B%5D=jquery/jquery.cookie.js&scripts%5B%5D=jquery/jquery.mousewheel.js&scripts%5B%5D=jquery/jquery.event.drag-2.2.js&scripts%5B%5D=jquery/jquery-ui-timepickeraddon.js&scripts%5B%5D=jquery/jquery.ba-hashchange-1.3.js HTTP/1.1\" 200 139613 \"-\" \"Mozilla/5.0 (Windows NT 6.1; WOW64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/45.0.2454.101 Safari/537.36\"","rawrequest" => "GET/phpmyadmin_8c1019c9c0de7a0f/js/get_scripts.js.php?scripts%5B%5D=jquery/jquery-1.11.1.min.js&scripts%5B%5D=sprintf.js&scripts%5B%5D=ajax.js&scripts%5B%5D=keyhandler.js&scripts%5B%5D=jquery/jquery-ui-1.11.2.min.js&scripts%5B%5D=jquery/jquery.cookie.js&scripts%5B%5D=jquery/jquery.mousewheel.js&scripts%5B%5D=jquery/jquery.event.drag-2.2.js&scripts%5B%5D=jquery/jquery-ui-timepickeraddon.js&scripts%5B%5D=jquery/jquery.ba-hashchange-1.3.js HTTP/1.1","@timestamp" => 2021-11-26T09:40:40.657Z,"clientip" => "36.157.150.1","http_referer" => "\"-\"","status" => "200","body_bytes_sent" => "139613","agent" => "\"Mozilla/5.0 (Windows NT 6.1; WOW64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/45.0.2454.101 Safari/537.36\""
}

四、在线安装frok插件

### --- 在线安装grok插件~~~     # 更改镜像源地址
[root@hadoop02 ~]# vim /opt/yanqi/servers/es/Logstash/Gemfile~第4/5行配置如下参数
# source "https://rubygems.org"                                 # 将这个镜像源注释掉
source "https://gems.ruby-china.com/"                           # 配置成中国的这个镜像源

### --- 准备在线安装~~~     # 在线安装grok插件
[root@hadoop02 ~]# cd /opt/yanqi/servers/es/Logstash/
[root@hadoop02 Logstash]# bin/logstash-plugin install logstash-filter-grok
~~~输出参数
Validating logstash-filter-grok
Installing logstash-filter-grok
Installation successful

### --- 开发Logstash的配置文件~~~     # 定义Logstash的配置文件如下，我们从控制台输入nginx的日志数据，然后经过filter的过滤，将我们的日志文件转换成为标准的数据格式
[root@hadoop02 ~]# vim /opt/yanqi/servers/es/Logstash/config/monitor_nginx.conf~~~写入配置参数
input {stdin{}}filter {grok {
match => {"message" => "%{IPORHOST:clientip} \- \- \[%{HTTPDATE:time_local}\] \"(?:%{WORD:method} %{NOTSPACE:request}(?:HTTP/%{NUMBER:httpversion})?|%{DATA:rawrequest})\" %{NUMBER:status} %{NUMBER:body_bytes_sent} %{QS:http_referer} %{QS:agent}"}}
}
output {stdout{codec => rubydebug}}

### --- 检查配置文件完整性[root@hadoop02 ~]# /opt/yanqi/servers/es/Logstash/bin/logstash \
-f /opt/yanqi/servers/es/Logstash/config/monitor_nginx.conf -t~~~输出参数
Configuration OK
Config Validation Result: OK. Exiting Logstash

### --- 启动Logstash~~~     # 执行以下命令启动Logstash
[root@hadoop02 ~]# /opt/yanqi/servers/es/Logstash/bin/logstash \
-f /opt/yanqi/servers/es/Logstash/config/monitor_nginx.conf~~~输出参数：详情查看6数据参数输出

### --- 从控制台输入nginx日志文件数据~~~     # 输入第一条数据
113.31.119.183 - - [05/Nov/2019:12:59:27 +0800] "GET /phpmyadmin_8c1019c9c0de7a0f/js/messages.php? lang=zh_CN&db=&collation_connection=utf8_unicode_ci&token=6a44d72481633c90bffcfd42f11e25a1 HTTP/1.1" 200 8131 "-" "Mozilla/5.0 (Windows NT 6.1; WOW64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/45.0.2454.101 Safari/537.36"
~~~输出参数
{"time_local" => "05/Nov/2019:12:59:27 +0800","@version" => "1","host" => "hadoop02","message" => "113.31.119.183 - - [05/Nov/2019:12:59:27 +0800] \"GET /phpmyadmin_8c1019c9c0de7a0f/js/messages.php? lang=zh_CN&db=&collation_connection=utf8_unicode_ci&token=6a44d72481633c90bffcfd42f11e25a1 HTTP/1.1\" 200 8131 \"-\" \"Mozilla/5.0 (Windows NT 6.1; WOW64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/45.0.2454.101 Safari/537.36\"","rawrequest" => "GET /phpmyadmin_8c1019c9c0de7a0f/js/messages.php? lang=zh_CN&db=&collation_connection=utf8_unicode_ci&token=6a44d72481633c90bffcfd42f11e25a1 HTTP/1.1","@timestamp" => 2021-11-26T09:35:04.242Z,"clientip" => "113.31.119.183","http_referer" => "\"-\"","status" => "200","body_bytes_sent" => "8131","agent" => "\"Mozilla/5.0 (Windows NT 6.1; WOW64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/45.0.2454.101 Safari/537.36\""
}

~~~     # 输入第二条数据36.157.150.1 - - [05/Nov/2019:12:59:28 +0800] "GET /phpmyadmin_8c1019c9c0de7a0f/js/get_scripts.js.php?scripts%5B%5D=jquery/jquery-1.11.1.min.js&scripts%5B%5D=sprintf.js&scripts%5B%5D=ajax.js&scripts%5B%5D=keyhandler.js&scripts%5B%5D=jquery/jquery-ui-1.11.2.min.js&scripts%5B%5D=jquery/jquery.cookie.js&scripts%5B%5D=jquery/jquery.mousewheel.js&scripts%5B%5D=jquery/jquery.event.drag-2.2.js&scripts%5B%5D=jquery/jquery-ui-timepickeraddon.js&scripts%5B%5D=jquery/jquery.ba-hashchange-1.3.js HTTP/1.1" 200 139613 "-" "Mozilla/5.0 (Windows NT 6.1; WOW64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/45.0.2454.101 Safari/537.36"
~~~输出参数
{"time_local" => "05/Nov/2019:12:59:28 +0800","@version" => "1","host" => "hadoop02","message" => "36.157.150.1 - - [05/Nov/2019:12:59:28 +0800] \"GET /phpmyadmin_8c1019c9c0de7a0f/js/get_scripts.js.php?scripts%5B%5D=jquery/jquery-1.11.1.min.js&scripts%5B%5D=sprintf.js&scripts%5B%5D=ajax.js&scripts%5B%5D=keyhandler.js&scripts%5B%5D=jquery/jquery-ui-1.11.2.min.js&scripts%5B%5D=jquery/jquery.cookie.js&scripts%5B%5D=jquery/jquery.mousewheel.js&scripts%5B%5D=jquery/jquery.event.drag-2.2.js&scripts%5B%5D=jquery/jquery-ui-timepickeraddon.js&scripts%5B%5D=jquery/jquery.ba-hashchange-1.3.js HTTP/1.1\" 200 139613 \"-\" \"Mozilla/5.0 (Windows NT 6.1; WOW64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/45.0.2454.101 Safari/537.36\"","rawrequest" => "GET /phpmyadmin_8c1019c9c0de7a0f/js/get_scripts.js.php?scripts%5B%5D=jquery/jquery-1.11.1.min.js&scripts%5B%5D=sprintf.js&scripts%5B%5D=ajax.js&scripts%5B%5D=keyhandler.js&scripts%5B%5D=jquery/jquery-ui-1.11.2.min.js&scripts%5B%5D=jquery/jquery.cookie.js&scripts%5B%5D=jquery/jquery.mousewheel.js&scripts%5B%5D=jquery/jquery.event.drag-2.2.js&scripts%5B%5D=jquery/jquery-ui-timepickeraddon.js&scripts%5B%5D=jquery/jquery.ba-hashchange-1.3.js HTTP/1.1","@timestamp" => 2021-11-26T09:35:28.894Z,"clientip" => "36.157.150.1","http_referer" => "\"-\"","status" => "200","body_bytes_sent" => "139613","agent" => "\"Mozilla/5.0 (Windows NT 6.1; WOW64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/45.0.2454.101 Safari/537.36\""
}

CC00051.elasticsearch——|HadoopElasticSearch.V03|——|ELK.v03Logstash部署.V3|相关推荐

CC00006.elasticsearch——|HadoopElasticSearch.V06|——|ELK.v06|集群|ElasticSearch集群部署.V1|
一.Elasticsearch 集群环境准备 ### --- hadoop01~03修改系统配置:修改/etc/sysctl.conf~~~ # 修改/etc/sysctl.conf [root@ha ...
CC00027.elasticsearch——|HadoopElasticSearch.V27|——|ELK.v27|集群|QueryDSL|高亮|
一.高亮 ### --- Elasticsearch中实现高亮的语法比较简单:~~~ # 在使用match查询的同时,加上一个highlight属性: ~~~ pre_tags:前置标签 ~~~ po ...
CC00045.elasticsearch——|HadoopElasticSearch.V45|——|ELK.v45|原理剖析|并发冲突处理机制剖析|
一.并发冲突处理机制剖析 ### --- 详解并发冲突~~~ # 在电商场景下,工作流程为: ~~~ 读取商品信息,包括库存数量 ~~~ 用户下单购买 ~~~ 更新商品信息,将库存数减一 ~~~ 如果 ...
CC00023.elasticsearch——|HadoopElasticSearch.V23|——|ELK.v23|集群|QueryDSL|复合搜索|
一.复合搜索(compound query) ### --- 布尔搜索(bool query)~~~ # bool 查询用bool操作来组合多个查询子句为一个查询. 可用的关键字: ~~~ must: ...
Docker ElK安装部署使用教程
Docker ElK安装部署使用教程原文:Docker ElK安装部署使用教程一.简介 1.核心组成 ELK由Elasticsearch.Logstash和Kibana三部分组件组成: Elast ...
CentOS 7.2下Filebeat+Kafka+ELK生产部署（安全加固）
01架构说明在需要采集日志的服务器上部署Filebeat服务,它将采集到的日志数据推送到Kafka集群: Logstash服务通过input插件读取Kafka集群对应主题的数据,期间可以使用filt ...
1. ELK docker部署
1 拉取镜像,启动 docker run -p 5601:5601 -p 9200:9200 -p 5044:5044 -it --name elk sebp/elk 由于我本机内存不符合安装要求,为 ...
开源实时日志分析ELK平台部署
开源实时日志分析ELK平台部署日志主要包括系统日志.应用程序日志和安全日志.系统运维和开发人员可以通过日志了解服务器软硬件信息.检查配置过程中的错误及错误发生的原因.经常分析日志可以了解服务器的负荷 ...
SS00003.elasticsearch——|HadoopElasticSearch集中式日志分析系统.v03|——|Elasticsearch.v03|
一.Elasticsearch 集群环境准备 ### --- hadoop01~03修改系统配置:修改/etc/sysctl.conf~~~ # 修改/etc/sysctl.conf [root@ha ...
Elasticsearch 5.0 —— Head插件部署指南
使用ES的基本都会使用过head,但是版本升级到5.0后,head插件就不好使了.下面就看看如何在5.0中启动Head插件吧! 官方粗略教程 Running with built in server ...

CC00051.elasticsearch——|HadoopElasticSearch.V03|——|ELK.v03Logstash部署.V3|

CC00051.elasticsearch——|HadoopElasticSearch.V03|——|ELK.v03Logstash部署.V3|相关推荐

最新文章

热门文章