logstash7.8 apache日志解析 grok
2024-06-02 09:00:43
1. apache日志
27.60.18.21 - - [21/Jul/2020:10:10:10 +0530] "GET /api/v1.2/places/search/json?username=liusg&location=28.5359586,77.3677936&query=elasticsearch&explain=true&bridge=true HTTP/1.1" 200 3284
27.60.18.22 - - [22/Jul/2020:10:11:11 +0530] "POST /api/v1.0/places/search/json?username=liushangguo&location=28.5359586,77.3677936&query=docker&explain=true&bridge=true HTTP/1.1" 200 1452
27.60.18.23 - - [23/Jul/2020:12:12:12 +0530] "HEAD /api/v1.2/places/nearby/json?&refLocation=28.5359586,77.3677936&keyword=FINATM HTTP/1.1" 200 3283
27.60.18.24 - - [24/Jul/2020:13:13:13 +0530] "POST /api/v2.0/places/search/json?username=liu.sg&location=28.5359586,77.3677936&query=iphone&explain=true&bridge=true HTTP/1.1" 200 3415
27.60.18.25 - - [25/Jul/2020:16:16:16 +0530] "GET /api/v1.2/places/search/json?username=pradeep.pgu&location=28.5359586,77.3677936&query=15000227329&explain=true&bridge HTTP/1.1" 200 2476
2. logstash配置
input {file {path => "/var/log/aaa.log"stat_interval => 1start_position => "beginning"}
}filter {grok {match => { "message" => "%{IPORHOST:client_ip} %{HTTPDUSER:ident} %{HTTPDUSER:auth} \[%{HTTPDATE:req_time}\] \"(?:%{WORD:method} /api/v%{NUMBER:api_version}/.*/json\?%{NOTSPACE:req_data}(?: HTTP/%{NUMBER:http_version})?|%{DATA:rawrequest})\" %{NUMBER:resp_code} (?:%{NUMBER:req_byte}|-)"}}kv { source => "req_data"field_split => "&"}if [query] {mutate {add_field => { "search" => "%{query}" }}} else if [keyword] {mutate {add_field => { "search" => "%{keyword}" }}}if [refLocation] {mutate {rename => { "refLocation" => "location" }}}# 新增timestamp字段，将@timestamp时间增加8小时ruby { code => "event.set('timestamp', event.get('@timestamp').time.localtime + 8*60*60)" }ruby { code => "event.set('@timestamp',event.get('timestamp'))" }mutate{split=>["location",","]add_field => { "jing" => "%{[location][0]}" }add_field => { "wei" => "%{[location][1]}" }}mutate {# 删除指定字段,messageremove_field => ["@version","host","path","tags"]# 如果将@timestamp字段删除，自动生成索引的日期配置就为空 # remove_field => ["@timestamp"]}
}output {stdout {}elasticsearch {hosts => ["http://192.168.1.58:9200"]index => "httpd-%{+YYYY.MM.dd}"user => "elastic"password => "123456"}
}3. 追加日志
echo '20.10.20.22 - - [25/Aug/2010:20:20:20 +0530] "GET /api/v2.2/places/search/json?username=sdd.pgu&location=28.2222233,77.3333333&query=elasticssss&explain=true&bridge HTTP/1.1" 200 2223' >> /var/log/aaa.log4. 索引查询结果
{"took" : 4,"timed_out" : false,"_shards" : {"total" : 1,"successful" : 1,"skipped" : 0,"failed" : 0},"hits" : {"total" : {"value" : 1,"relation" : "eq"},"max_score" : 1.0,"hits" : [{"_index" : "httpd-2020.07.23","_type" : "_doc","_id" : "S415enMB5upHvde6CV8J","_score" : 1.0,"_source" : {"auth" : "-","api_version" : "2.2","ident" : "-","method" : "GET","location" : ["28.2222233","77.3333333"],"@timestamp" : "2020-07-23T15:00:19.003Z","req_time" : "25/Aug/2010:20:20:20 +0530","username" : "sdd.pgu","query" : "elasticssss","message" : """20.10.20.22 - - [25/Aug/2010:20:20:20 +0530] "GET /api/v2.2/places/search/json?username=sdd.pgu&location=28.2222233,77.3333333&query=elasticssss&explain=true&bridge HTTP/1.1" 200 2223""","wei" : "77.3333333","explain" : "true","req_byte" : "2223","search" : "elasticssss","http_version" : "1.1","client_ip" : "20.10.20.22","timestamp" : "2020-07-23T15:00:19.003Z","resp_code" : "200","req_data" : "username=sdd.pgu&location=28.2222233,77.3333333&query=elasticssss&explain=true&bridge","jing" : "28.2222233"}}]}
}
5. grok表达式
6. grok表达式-案例
6.1. 日志
Jan 1 06:25:43 mailserver14 postfix/cleanup[21403]: BEF25A72965: message-id=<20130101142543.5828399CCAF@mailserver14.example.com> Alert
6.2. grok表达式
%{MONTH:month} %{NUMBER:day} %{TIME:time} %{WORD:ms} (?:%{WORD:postfix}/%{WORD:status}\[%{NUMBER:bbyte}\]: %{WORD:ms22}: (?<sssd>.*>)) %{LOGLEVEL:level}
6.3. 拆分结果
{"level": "Alert","ms": "mailserver14","sssd": "message-id=<20130101142543.5828399CCAF@mailserver14.example.com>","month": "Jan","ms22": "BEF25A72965","bbyte": "21403","time": "06:25:43","postfix": "postfix","day": "1","status": "cleanup"
}
logstash7.8 apache日志解析 grok相关推荐
- elk系列7之通过grok分析apache日志
preface 说道分析日志,我们知道的采集方式有2种: 通过grok在logstash的filter里面过滤匹配. logstash --> redis --> python(py脚本过 ...
- kjb文件 解析_在Linux上使用lnav监控和分析Apache日志文件工具
请关注本头条号,每天坚持更新原创干货技术文章. 如需学习视频,请在微信搜索公众号"智传网优"直接开始自助视频学习 1. 前言 本文主要讲解如何在Linux上使用lnav监控和分析A ...
- Cascading(一)之日志解析
此例子为官网例子,所以直接上代码: 1 package com.wyf.cascade; 2 3 import java.util.Properties; 4 5 import cascading.f ...
- Spark HistoryServer日志解析清理异常
Spark HistoryServer日志解析&清理异常 一.背景介绍 用户在使用 Spark 提交任务时,经常会出现任务完成后在 HistoryServer(Spark 1.6 和 Spar ...
- FTP服务器日志解析
2019独角兽企业重金招聘Python工程师标准>>> FTP服务器日志解析 FTP是老牌的文件传输协议,在网络中应用非常广泛.本节就Vsftp服务器的日志进行重点讨论,在本书的FT ...
- Apache日志分割及分析
相关软件及下载地址: 1. cronolog-1.6.2.tar.gz 2. awstats-6.95.tar.gz 3. GeoIP-1.4.6.tar.gz 4. Geo-IP-1.38.tar. ...
- GitHub--logparser(日志解析器)
Apache HTTPD和NGINX访问日志解析器 这是一个Logparsing框架,旨在简化Apache HTTPD和NGINX访问日志文件的解析. 基本思想是,您应该能够拥有一个解析器,可以通过简 ...
- java 解析hiveserver2日志 解析HiveSQL 获取表的使用次数 热度
java 解析hiveserver2日志 解析HiveSQL 获取表的使用次数 热度 首先逐行读取hiveserver2日志 日志里每个sql之前都会包含Executing command关键字 故先 ...
- 电商数仓DWD层用户行为日志解析
文章目录 前言 一.页面埋点日志.启动日志结构 二.日志解析的流程 2.1 启动日志表解析(包括注意事项) 2.1.1 解析思路 2.1.2 建表语句 2.1.3 数据导入 2.1.4 注意事项 2. ...
最新文章
- 在tomcat下部署两个或多个项目时 log4j和web.xml配置webAppRootKey 的问题(转)
- Bootstrap – 1.认识
- java druid mysql连接池_java使用Druid连接池连接mysql
- python stdin和stdout_stdin似乎比stdout(python)慢得多.为什么？
- 【工具】Xshell安装注册以及简单属性配置
- C++中时间相关函数的使用
- datetime类型的取年月日 sql_SQL2005怎么截取datetime类型字段的年月日，并以截取后的（年月日）字段排序...
- 图神经网络的可解释性
- 最大似然估计_机器学习最大似然估计
- oracle 中的or,oracle语句查询 or和and
- 20款最优秀的JavaScript编辑器 哪家强你说了算！
- 迪普交换机恢复出厂设置_LSW交换机初始化配置指导
- 网站备案需要买服务器吗,域名备案需要购买服务器吗
- “(null)” is of a model that is not supported by this version of Xcode. Ple
- 【sdx62】XBL设置共享内存变量，然后内核层获取变量实现
- shell实现大批量word转码然后分析相关字段
- BZOJ 2708 木偶
- ＜数据库＞ if 条件语句的使用 SQL26 计算25岁以上和以下的用户数量
- Git和Mercurial(Hg)的分析
- Jetpack Compose之手势使用