logstash 配置文件
nginx access 日志格式
nginx access 拆分效果
nginx error 日志格式：
nginx error 拆分效果：

logstash 配置文件

input {file {type => "nginx-access"  path => [ "/home/jiankunking/software/testdata/nginx/access.log" ]tags => [ "nginx","access"]start_position => beginning}file {type => "nginx-error" path => [ "/home/jiankunking/software/testdata/nginx/error.log" ]tags => [ "nginx","error"]start_position => beginning
}
}
filter {if [type] == "nginx-access" {grok{match =>    ["message","%{IPORHOST:client_ip}\s{1,}\-\s\-\s\[%{HTTPDATE:time}\]\s{1,}\"(?:%{WORD:verb}\s{1,}%{NOTSPACE:request}(?:\s{1,}HTTP/%{NUMBER:http_version})?|-)\" %{NUMBER:response}\s{1,}(?:%{NUMBER:bytes}|-)\s{1,}%{QS:referrer}\s{1,}%{QS:agent}"]}date{match=>["time","dd/MMM/yyyy:HH:mm:ss Z"]target=>"logdate"}ruby{code => "event.set('logdateunix',event.get('logdate').to_i)"}} else if [type] == "nginx-error" { grok {match => ["message", "(?<time>\d{4}/\d{2}/\d{2}\s{1,}\d{2}:\d{2}:\d{2})\s{1,}\[%{DATA:err_severity}\]\s{1,}(%{NUMBER:pid:int}#%{NUMBER}:\s{1,}\*%{NUMBER}|\*%{NUMBER}) %{DATA:err_message}(?:,\s{1,}client:\s{1,}(?<client_ip>%{IP}|%{HOSTNAME}))(?:,\s{1,}server:\s{1,}%{IPORHOST:server})(?:, request: %{QS:request})?(?:, host: %{QS:client_ip})?(?:, referrer: \"%{URI:referrer})?","message", "(?<time>\d{4}/\d{2}/\d{2}\s{1,}\d{2}:\d{2}:\d{2})\s{1,}\[%{DATA:err_severity}\]\s{1,}%{GREEDYDATA:err_message}"]}date{match=>["time","yyyy/MM/dd HH:mm:ss"]target=>"logdate"}ruby{code => "event.set('logdateunix',event.get('logdate').to_i)"}}
}
output{elasticsearch{hosts => ["10.10.10.10:9200"]index => "logstash-nginx-%{+YYYY.MM.dd}"}
}#或者output中access日志和error日志也可以分开存取
output{if [type] == "nginx-access" {elasticsearch{hosts => ["localhost:9200"] #本机地址localhost或者远程地址ipindex => "access-nginx-%{+YYYY.MM.dd}"}} else if [type] == "nginx-error" {elasticsearch{hosts => ["localhost:9200"]index => "error-nginx-%{+YYYY.MM.dd}"}}
}

nginx access 日志格式

10.10.10.10 - - [28/Mar/2017:13:21:04 +0800] "GET / HTTP/1.1" 200 612 "-" "Mozilla/5.0 (Windows NT 6.1; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/57.0.2987.110 Safari/537.36" 10.10.10.10 - - [28/Mar/2017:13:21:04 +0800] "GET /favicon.ico HTTP/1.1" 404 572 "http://10.11.11.11/" "Mozilla/5.0 (Windows NT 6.1; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/57.0.2987.110 Safari/537.36"

nginx access 拆分效果

{"_index": "logstash-nginx-access-2017.04.07","_type": "logs","_id": "AVtHLpaYYixw4Fpnef2F","_score": 1,"_source": {"request": "/favicon.ico","agent": ""Mozilla/5.0 (Windows NT 6.1; WOW64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/50.0.2661.102 Safari/537.36"","verb": "GET","http_version": "1.1","message": "10.11.11.12 - - [28/Mar/2017:13:33:42 +0800] "GET /favicon.ico HTTP/1.1" 404 572 "http://log.c.haier.net/" "Mozilla/5.0 (Windows NT 6.1; WOW64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/50.0.2661.102 Safari/537.36"","tags": ["nginx","access"],"path": "/home/jiankunking/software/testdata/nginx/access.log","referrer": ""http://log.c.haier.net/"","@timestamp": "2017-04-07T06:51:26.135Z","response": "404","bytes": "572","logdate": "2017-03-28T05:33:42.000Z","@version": "1","host": "ubuntu","client_ip": "10.11.11.12","logdateunix": 1490679222,"timestamp": "28/Mar/2017:13:33:42 +0800"}}

nginx error 日志格式：

2017/03/28 13:34:21 [error] 17627#0: *8 open() "/usr/local/nginx/html/favicon.ico" failed (2: No such file or directory), client: 10.10.10.10, server: localhost, request: "GET /favicon.ico HTTP/1.1", host: "10.11.11.11"
2017/03/28 14:15:10 [notice] 20260#0: signal process started
2017/03/28 15:04:32 [emerg] 21321#0: invalid URL prefix in /usr/local/nginx/conf/nginx.conf:47

nginx error 拆分效果：

{"_index": "logstash-nginx-error-2017.04.07","_type": "logs","_id": "AVtHEFkyYixw4FpnefNL","_score": 1,"_source": {"path": "/home/jiankunking/software/testdata/nginx/error.log","@timestamp": "2017-04-07T06:18:24.601Z","err_severity": "notice","logdate": "2017-03-28T06:15:10.000Z","@version": "1","host": "ubuntu","time": "2017/03/28 14:15:10","message": "2017/03/28 14:15:10 [notice] 20260#0: signal process started","logdateunix": 1490681710,"tags": ["nginx","error"],"err_message": "20260#0: signal process started"}
},
{"_index": "logstash-nginx-error-2017.04.07","_type": "logs","_id": "AVtHEFkyYixw4FpnefNK","_score": 1,"_source": {"server": "localhost","request": ""GET/favicon.icoHTTP/1.1"","pid": 17627,"message": "2017/03/28 13:34:21 [error] 17627#0: *8 open() "/usr/local/nginx/html/favicon.ico" failed (2: No such file or directory), client: 10.11.11.11, server: localhost, request: "GET/favicon.icoHTTP/1.1", host: "10.11.11.12"","tags": ["nginx","error"],"err_message": "open() "/usr/local/nginx/html/favicon.ico" failed (2: No such file or directory)","path": "/home/jiankunking/software/testdata/nginx/error.log","@timestamp": "2017-04-07T06:18:24.595Z","err_severity": "error","logdate": "2017-03-28T05:34:21.000Z","@version": "1","host": "ubuntu","client_ip": ["10.11.11.11",""10.11.11.12""],"time": "2017/03/28 13:34:21","logdateunix": 1490679261}
},
{"_index": "logstash-nginx-error-2017.04.07","_type": "logs","_id": "AVtHEFkyYixw4FpnefNM","_score": 1,"_source": {"path": "/home/jiankunking/software/testdata/nginx/error.log","@timestamp": "2017-04-07T06:18:24.610Z","err_severity": "emerg","logdate": "2017-03-28T07:04:32.000Z","@version": "1","host": "ubuntu","time": "2017/03/28 15:04:32","message": "2017/03/28 15:04:32 [emerg] 21321#0: invalid URL prefix in /usr/local/nginx/conf/nginx.conf:47","logdateunix": 1490684672,"tags": ["nginx","error"],"err_message": "21321#0: invalid URL prefix in /usr/local/nginx/conf/nginx.conf:47"}
}

logstash nginx error access 日志处理相关推荐

filebeat获取nginx的access日志配置
filebeat获取nginx的access日志配置产生nginx日志的服务器即生产者服务器配置: 拿omp.chinasoft.com举例: 1.nginx.conf主配置文件添加日志格式log_f ...
nginx 关闭access日志_可视化实时Web日志分析工具，堪称神器！
说到web服务器就不得不说Nginx,目前已成为企业建站的首选.但由于种种历史原因,Nginx日志分析工具相较于传统的apache.lighthttp等还是少很多. 今天就和大家分享一个非常强大的实时 ...
【mapreudce】6.对Nginx的access日志进行数据清洗，我们提取出文件数据的ip，时间，url...
1.首先我们需要一个util辅助类 package cn.cutter.demo.hadoop.mapreduce.nginxlog.util;import java.text.ParseExcept ...
nginx关闭access日志
进入nginx的主配置文件nginx.conf 找到access_log on; 修改为access_log off; 即可
nginx php 错误日志,PHP 错误与异常的日志记录
提到 Nginx + PHP 服务的错误日志,我们通常能想到的有 Nginx 的 access 日志.error 日志以及 PHP 的 error 日志.虽然看起来是个很简单的问题,但里面其实又牵扯到 ...
nginx php访问日志配置,nginx php-fpm 输出php错误日志的配置方法
由于nginx仅是一个web 服务器,因此 nginx的access日志只有对访问页面的记录,不会有php 的 error log信息. nginx把对php的请求发给php-fpm fastcgi进 ...
nginx的error.log日志常见的几个错误解决方法
nginx.conf里会有两个日志,分为access.log 和 error.log.其中这两个日志可以细化,一般来说在nginx目录下会有一个logs会保存,然后也可以在对应的server目录里可以 ...
ELK之日志收集filebeat，并对nginx，tomcat access日志JSON格式化
2019独角兽企业重金招聘Python工程师标准>>> 一:ELK日志收集器组件filebeat下载官方下载地址:https://www.elastic.co/downloads/ ...
Nginx的access.log日志分析工具-goaccess
一.安装goaccess wget http://tar.goaccess.io/goaccess-1.3.tar.gz --no-check-certificate #下载tar包 tar -xz ...
使用 elasticsearch、LogStash、Kibana完成网站流量的监控系统(基于nginx的访问日志实现流量监控)
分布式带来的变革: 多节点.日志分散.运维成本高先看几个实际的案例. 各自的解决方案一些比较主流的集中式日志管理系统简单的Rsyslog 商业化的 Splunk 开源的有 Facebook 公司 ...

logstash nginx error access 日志处理

logstash 配置文件

nginx access 日志格式

nginx access 拆分效果

nginx error 日志格式：

nginx error 拆分效果：

logstash nginx error access 日志处理相关推荐

最新文章

热门文章