检测解析的日志是否包含单个或多个警告消息，然后添加一个字段来说明这两种情况。在很多的情形下，我们在测试 Logstash 的过滤器时，并不急于把实际的 input 的数据接入到过滤器中来进行测试。我们首先来选择一个比较容易理解的 input 方式，使用一个文档来进行解析，并测试管道。在今天的文章中，我来详细介绍两种常用的方法来如何测试 Logstash 的管道/过滤器。

方法一：使用 generator

方法如下：

logstash.conf

input {generator {message => '{"id":2,"timestamp":"2019-08-11T17:55:56Z","paymentType":"Visa","name":"Darby Dacks","gender":"Female","ip_address":"77.72.239.47","purpose":"Shoes","country":"Poland","age":55}'count => 1}
}filter {json {source => "message"}if [paymentType] == "Mastercard" {drop {}}mutate {remove_field => ["message", "@timestamp", "path", "host", "@version", "log", "event"]}}output {stdout {codec => rubydebug}
}

在上面，我们使用 generator 的方法来生成一个文档，并让这个文档经过 filter 部分，并最终在 console 中进行展示。我们可以通过如下的命令来运行上面的 Logstash 管道：

$ pwd
/Users/liuxg/elastic/logstash-8.6.1
$ ls logstash.conf
logstash.conf
$ ./bin/logstash -f logstash.conf

从上面，我们可以看出来 json filter 工作正常。在本示例中，为了说明问题的方便，我仅使用了几个过滤器。在时间的使用中，我们可以有很多的过滤器来组成这个 pipeline。一旦我们确定了这些过滤器能完成我们所需要的功能，我们可以把所需要的 input 换进来即可，比如：

logstash_filter.conf

input {file {path => "/Users/liuxg/elastic/logstash-8.6.1/sample.json"type    => "applog"start_position => "beginning"sincedb_path => "/dev/null"}
}filter {json {source => "message"}if [paymentType] == "Mastercard" {drop {}}mutate {remove_field => ["message", "@timestamp", "path", "host", "@version", "log", "event"]}}output {stdout { codec => rubydebug }
}

我们可以使用诸如如下格式的测试文件来进行测试：

sample.json

{"id":1,"timestamp":"2019-09-12T13:43:42Z","paymentType":"Amex","name":"Merrill Duffield","gender":"Female","ip_address":"132.150.218.21","purpose":"Toys","country":"United Arab Emirates","age":33}
{"id":2,"timestamp":"2019-08-11T17:55:56Z","paymentType":"Visa","name":"Darby Dacks","gender":"Female","ip_address":"77.72.239.47","purpose":"Shoes","country":"Poland","age":55}
{"id":3,"timestamp":"2019-07-14T04:48:25Z","paymentType":"Visa","name":"Harri Cayette","gender":"Female","ip_address":"227.6.210.146","purpose":"Sports","country":"Canada","age":27}
{"id":4,"timestamp":"2020-02-29T12:41:59Z","paymentType":"Mastercard","name":"Regan Stockman","gender":"Male","ip_address":"139.224.15.154","purpose":"Home","country":"Indonesia","age":34}
{"id":5,"timestamp":"2019-08-03T19:37:51Z","paymentType":"Mastercard","name":"Wilhelmina Polle","gender":"Female","ip_address":"252.254.68.68","purpose":"Health","country":"Ukraine","age":51}

当然实际的文档可能比这个要长很多。

更多关于 generator 方面的示例，请阅读我之前的文章 “Logstash：Data 转换，分析，提取，丰富及核心操作”。

方法二：使用 stdin input

假设我们有以下代表上述两种情况的日志文件：

$ pwd
/Users/liuxg/elastic/logstash-8.6.1
$ cat multivaluewarn.json
{"waf": {"ver": "2.0","warnRules": "3000030;3000057;950001;950109;959073;973335;981173;981244;981318","denyMsg": "Anomaly Score Exceeded for SQL Injection","denyActions": "3","warnMsg": "Basic SQL Authentication Bypass Attempts 3/3;Cross-site Scripting (XSS) common keywords;SQL Injection Attack;Multiple URL Encoding Detected;SQL Injection Attack;IE XSS Filters - Attack Detected;Restricted SQL Character Anomaly Detection Alert - Total # of special characters exceeded;Basic SQL Authentication Bypass Attempts 1/3;SQL Injection Attack: Common Injection Testing Detected"}}
$
$ cat singlevaluewarn.json
{"waf": {"ver": "2.0","warnRules": "681984","policy": "api_89894","warnMsg": "Alert rq without DEVICEID header","warnTags": "DEVICEID_Detection","warnActions": "2"}}

multivaluewarn.json

{"waf": {"ver": "2.0","warnRules": "3000030;3000057;950001;950109;959073;973335;981173;981244;981318","denyMsg": "Anomaly Score Exceeded for SQL Injection","denyActions": "3","warnMsg": "Basic SQL Authentication Bypass Attempts 3/3;Cross-site Scripting (XSS) common keywords;SQL Injection Attack;Multiple URL Encoding Detected;SQL Injection Attack;IE XSS Filters - Attack Detected;Restricted SQL Character Anomaly Detection Alert - Total # of special characters exceeded;Basic SQL Authentication Bypass Attempts 1/3;SQL Injection Attack: Common Injection Testing Detected"}}

{"waf":{"ver":"2.0","warnRules":"3000030;3000057;950001;950109;959073;973335;981173;981244;981318","denyMsg":"Anomaly Score Exceeded for SQL Injection","denyActions":"3","warnMsg":"Basic SQL Authentication Bypass Attempts 3/3;Cross-site Scripting (XSS) common keywords;SQL Injection Attack;Multiple URL Encoding Detected;SQL Injection Attack;IE XSS Filters - Attack Detected;Restricted SQL Character Anomaly Detection Alert - Total # of special characters exceeded;Basic SQL Authentication Bypass Attempts 1/3;SQL Injection Attack: Common Injection Testing Detected"}
}

singlevaluewarn.json

{"waf": {"ver": "2.0","warnRules": "681984","policy": "api_89894","warnMsg": "Alert rq without DEVICEID header","warnTags": "DEVICEID_Detection","warnActions": "2"}}

{"waf":{"ver":"2.0","warnRules":"681984","policy":"api_89894","warnMsg":"Alert rq without DEVICEID header","warnTags":"DEVICEID_Detection","warnActions":"2"}
}

阅读日志我们可以看到字段 [waf][warnMsg] 使用分号分隔警告消息；在多次警告的情况下。
将收集到的信息转换为 Logstash 管道将导致：

logstash_warning.conf

input {stdin { codec => json }
}filter {if ";" in [waf][warnMsg]{mutate {add_field =>  [ "wafWarningMSG", "multi warnings" ]}}else {mutate {add_field =>  [ "wafWarningMSG", "single" ]}}
}output {stdout {codec => rubydebug}
}

将管道添加到 conf 文件（称为 logstash_warning.conf ），然后使用如下的命令来测试 pipeline：

$ pwd
/Users/liuxg/elastic/logstash-8.6.1
$ ls logstash_warning.conf
logstash_warning.conf
$ ./bin/logstash -f logstash_warning.conf < multivaluewarn.json

输出显示一个名为 wafWarningMSG 的新字段，其中包含 "multi warnings"：

当然，我们也可以使用如下的命令来进行测试：

./bin/logstash -f logstash_warning.conf < singlevaluewarn.json

从上面的输出中，我们可以看到 wafWarningMSG 字段的值为 single。

一旦我们测试好 pipeline 中的过滤器，我们就可以直接把 input 部分换成我们想要的格式即可。

希望你觉得它有用，如有任何问题，请随时联系我！

Logstash：在实施之前测试 Logstash 管道/过滤器相关推荐

批处理与管道-过滤器
三种典型的数据流风格 § Batch Sequential (批处理) § Pipe-and-Filter (管道-过滤器) § Process Control(过程控制,3.7) 批处理风格: 直观 ...
管道/过滤器架构风格的优点和不足
管道/过滤器风格的软件架构具有许多很好的特点: (1)使得软构件具有良好的隐蔽性和高内聚.低耦合的特点: (2)允许设计者将整个系统的输入/输出行为看成是多个过滤器的行为的简单合成: (3)支持软件重 ...
系统架构设计 2.1 管道-过滤器风格
一. 管道-过滤风格过滤器:功能组件管道:数据流之间的通路 1.2 特点管道/过滤器结构将数据流处理分为几个顺序的步骤来实现,一个步骤的输出是另一个步骤的输入,每个步骤由一个过滤器来实现每个过 ...
管道-过滤器软件架构
每个构件都有一组输出和输出,构件读输入的数据流,经过内部处理,然后产生输出数据流.因此,这里的构件称为过滤器,这种风格的连接件就像是数据流传输的管道,将一个过滤顺的输出传到另一个过滤器的输入. 一个典 ...
docker logstash_学会这一招，轻松测试 logstash 的配置文件
配置文件本身非常脆弱!所以修改配置文件自然会引入部署失败的风险.如果能够对配置文件进行自动化测试将会极大的降低这种风险.本文将介绍一个可以自动化测试 logstash 配置文件的工具,让大家可以像写单 ...
filebeat+redis+logstash+elasticsearch filebeat+kafka+zookeeper+logstash+elasticsearch
收集日志的工具日志易(收费) splunk(国外,按流量收费) 介绍发展史:使用java语言,在luncen的基础上做二次封装,提供restful接口搜索的原理:倒排索引特点:水平扩展方便.提 ...
管道过滤器风格 java_完成基于管道过滤器风格的KWI实现.doc
完成基于管道过滤器风格的KWI实现.doc 实验2:软件体系结构风格实现一.实验目的初步了解不同的体系结构风格掌握不同体系结构风格的实现二.实验学时 4学时. 三.实验方法根据KWIC的描述 ...
【Logstash】记录一次logstash拉取数据很慢的问题
文章目录 1.背景 1.背景就是感觉logstash拉取数据很慢,然后看看logstash日志如下 [2020-09-10 00:25:07][ERROR]
管道-过滤器体系结构风格
特点:单向流:数据源源不断地产生:有一部分数据产生或处理后立马发往下一个处理构件:可能有缓冲(防止处理速度不一致造成数据丢失) 应用实例:编译器,UNIX管道,图像处理,信号处理等优点: 1.良好的 ...

Logstash：在实施之前测试 Logstash 管道/过滤器

方法一：使用 generator

方法二：使用 stdin input

Logstash：在实施之前测试 Logstash 管道/过滤器相关推荐

最新文章

热门文章