nginx 获取body参数_日志分析系列(外传二):Nginx日志统一化

本系列故事纯属虚构，如有雷同实属巧合

为了完成对Nginx服务器的日志分析，小B对Q公司的Nginx日志做了统一化要求。下面是小B在统一化过程中遇到的一些知识点：

Nginx日志与字段解析

Q公司的Nginx版本信息是：1.17.6，使用编译安装，安装过程如下：

yum install zlib-devel.x86_64 zlib.x86_64 openssl.x86_64 openssl-devel.x86_64 pcre-devel.x86_64 -y# 安装lua支持，后续的response_body需要
# 这里有坑，解决方案见最后
wget http://luajit.org/download/LuaJIT-2.0.5.tar.gz
tar -xf LuaJIT-2.0.5.tar.gz -C /opt/ && cd /opt/LuaJIT-2.0.5/
make install PREFIX=/usr/local/luajitecho 'export LUAJIT_LIB=/usr/local/luajit/lib' >> /etc/profile
echo 'export LUAJIT_INC=/usr/local/luajit/include/luajit-2.0' >> /etc/profile
source /etc/profile# 下载ngx_devel_kit
wget https://github.com/simpl/ngx_devel_kit/archive/v0.3.1.tar.gz
tar -xf v0.3.1.tar.gz -C /opt/# 下载lua-nginx-module
# 这里有坑，解决方案见最后
wget https://github.com/openresty/lua-nginx-module/archive/v0.10.15.tar.gz
tar -xf v0.10.15.tar.gz -C /opt/# 下载nginx
wget https://nginx.org/download/nginx-1.17.6.tar.gz
tar -xf nginx-1.17.6.tar.gz && cd nginx-1.17.6/# 安装nginx
./configure --prefix=/opt/nginx --with-http_realip_module --with-http_ssl_module --with-pcre --with-ld-opt=-Wl,-rpath,/usr/local/luajit/lib --add-module=/opt/lua-nginx-module-0.10.15 --add-module=/opt/ngx_devel_kit-0.3.1make -j2 && make install

Nginx原始日志格式：vim /opt/nginx/conf/nginx.conf

log_format  main  '$remote_addr - $remote_user [$time_local] "$request" ''$status $body_bytes_sent "$http_referer" ''"$http_user_agent" "$http_x_forwarded_for"';access_log  logs/access.log  main;

原始Nginx的日志为：tail -n 1 -f /opt/nginx/logs/access.log

10.10.10.1 - - [18/Dec/2019:13:27:27 +0800] "GET / HTTP/1.1" 200 612 "-" "Mozilla/5.0 (Macintosh; Intel Mac OS X 10_15_2) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/79.0.3945.79 Safari/537.36" "-"

Nginx日志字段解析

首先小B需要弄明白Nginx日志中每个字段的含义：

Nginx获取body信息

在Q公司目前的架构中，使用GET传递参数的方式已经很少了，为了了解攻击者是否在body中嵌入攻击payload以及了解攻击者获取到了什么结果，小B需要采集body的日志信息。

打印request_body

打印request_body有两种方式：一种是使用nginx的模块；另外一种是使用lua编写脚本，如果需要限制nginx收集request_body的长度，最好使用后者。

使用nginx ngx_http_core模块采集request_body

ngx_http_core模块官网地址 http://nginx.org/en/docs/http/ngx_http_core_module.html 中有一段关于采集request_body的说明，内容如下：

The variable’s value is made available in locations processed by the proxy_pass, fastcgi_pass, uwsgi_pass, and scgi_pass directives when the request body was read to a memory buffer.

# 首先修改配置文件，我这里采集的日志只有request_body字段
vim /opt/nginx/conf/nginx.conf
log_format  main    $request_body;access_log  logs/access.log  main;location / {root   /opt/nginx/html;# 以下为添加内容fastcgi_pass 127.0.0.1:9000;fastcgi_index index.php;fastcgi_param SCRIPT_FILENAME       $document_root$fastcgi_script_name;include fastcgi_params;}

此时Nginx的日志为：

使用ngx_http_core模块收集日志有没有办法限制request_body的长度呢？其实是有的。

在nginx配置文件中的http{}里面添加client_max_body_size 1k;即可。

但是这个配置是不允许用户上传超过1K大小的body内容，如果用户需要上传图片，业务可能就无法正常运行，所以不推荐使用此种方法。

我们在这里测试一下ngx_http_core的内容限制：

# 我这里使用1024个A来测试，我的配置是否有效
curl -XPOST "http://10.10.10.13/test.php?id=123" -H "X-Forwarded-For: 10.10.10.5" -H "Referer: http://10.10.10.13" --data "AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA"

此时Nginx正常打印出了request_body。

# 如果我们超过1024个字节，就会报错，注意1024个A后面有一个1
curl -XPOST "http://10.10.10.13/test.php?id=123" -H "X-Forwarded-For: 10.10.10.5" -H "Referer: http://10.10.10.13" --data "AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA1"

此时Nginx无法打印request_body。

使用lua编写脚本采集request_body

# 修改nginx的配置文件
vim /opt/nginx/conf/nginx.conf
log_format  main    $request_body_head;access_log  logs/access.log  main;location / {root   /opt/nginx/html;# 以下为添加内容set $request_body_head      "";content_by_lua_block {ngx.req.read_body()local   req_body = ngx.req.get_body_data()# 这里的1000代表我们截取request_body的长度，不要取的太长，否则容易导致日志过大ngx.var.request_body_head =  req_body:sub(1,1000)}}

此时Nginx的日志为：

打印response_body

对于response_body我们只有使用lua编写脚本来采集。

修改nginx的配置：vim /opt/nginx/conf/nginx.conf

server {listen       80;server_name  localhost;# 以下为添加内容lua_need_request_body   on;set $response_body      "";body_filter_by_lua      '# 这里的1000就代表截取response_body的长度，不要取的太长，否则容易导致日志过大local       response_body = string.sub(ngx.arg[1],1,1000)ngx.ctx.buffered =  (ngx.ctx.buffered or "")        ..      response_bodyif ngx.arg[2] thenngx.var.response_body = ngx.ctx.bufferedend';}

此时Nginx的日志为：

request_body与response_body如果在上传文件或者下载文件时，内容会很大，采集全部内容需要考虑对系统、Nginx性能和日志存储等方面的影响。如果采集的body内容太短，也会导致采取不到我们想要的信息，所以根据业务取一个合理阈值。

nginx-->syslog-->logstash

小B在测试中与朋友交流得知，可以将nginx日志直接传输到logstash中而不落盘，但是这种方法传输的日志不可靠，并且会对nginx产生性能影响，可以在测试的时候使用。

nginx配置文件：vim /opt/nginx/conf/nginx.conf

log_format  logstash  '$remote_addr - $remote_user [$time_local] "$request" ''$status $body_bytes_sent "$http_referer" ''"$http_user_agent" "$http_x_forwarded_for"';# access_log  logs/access.log  main;access_log  syslog:server=127.0.0.1:514,nohostname,tag=nginx_access logstash;

logstash配置文件：vim /etc/logstash/conf.d/nginx.conf

input {udp {host => "127.0.0.1"port => 514}
}
output {stdout {}
}

执行logstash，并查看效果：/usr/share/logstash/bin/logstash -f /etc/logstash/conf.d

统一Nginx日志格式

在完成了调研之后，小B就开始统一所有的日志字段了，为了方便后续查询分析操作，小B决定所有的字段采用JSON格式存盘，并且添加了运维、安全、研发都关注的字段。

修改配置文件：vim /opt/nginx/conf/nginx.conf

# 完整的nginx配置worker_processes  1;
events {worker_connections  1024;
}http {include       mime.types;default_type  application/octet-stream;log_format main     escape=json '{''"timestamp": $time_local ''"remote_addr": $remote_addr ''"remote_user": "$remote_user ''"request_method": $request_method ''"request_uri": "$request_uri" ''"request_protocol": "$server_protocol" ''"request_length": $request_length ''"request_time": $request_time ''"request_body_head": "$request_body_head" ''"response_status": $status ''"body_bytes_sent": $body_bytes_sent ''"bytes_sent": $bytes_sent ''"response_body": "$response_body" ''"http_referer": "$http_referer" ''"http_user_agent": "$http_user_agent" ''"http_x_forwarded_for": "$http_x_forwarded_for" ''"http_host": "$http_host" ''"server_name": "$server_name" ''"upstream_addr": "$upstream_addr" ''"upstream_status": $upstream_status''}';access_log  logs/access.log  main;sendfile        on;keepalive_timeout  65;server {listen       80;server_name  localhost;lua_need_request_body   on;set $response_body      "";body_filter_by_lua      'local       response_body = string.sub(ngx.arg[1],1,1000)ngx.ctx.buffered =  (ngx.ctx.buffered or "")        ..      response_bodyif ngx.arg[2] thenngx.var.response_body = ngx.ctx.bufferedend';location /lua {root    html;index   index.html;set $request_body_head      "";content_by_lua_block {ngx.req.read_body()local   req_body = ngx.req.get_body_data()ngx.var.request_body_head =  req_body:sub(1,1000)}}error_page   500 502 503 504  /50x.html;location = /50x.html {root   html;}}
}

此时Nginx的日志为：

小结一下

对于timestamp参数，可以不使用time_local而使用time_iso8601。
如果运维对于网络的性能有要求，可以考虑使用$tcpinfo_rtt, $tcpinfo_rttvar, $tcpinfo_snd_cwnd, $tcpinfo_rcv_space这些参数可能需要我们给nginx添加其他模块。

踩坑记录

启动nginx报错

由于luajit导致的报错

nginx: [alert] detected a LuaJIT version which is not OpenResty's; many optimizations will be disabled and performance will be compromised (see https://github.com/openresty/luajit2 for OpenResty's LuaJIT or, even better, consider using the OpenResty releases from https://openresty.org/en/download.html)

解决方案：卸载原有的luajit

wget https://github.com/openresty/luajit2/archive/v2.1-20190912.tar.gz
tar -xf v2.1-20190912.tar.gz && cd luajit2-2.1-20190912/
make install PREFIX=/usr/local/luajitecho 'export LUAJIT_LIB=/usr/local/luajit/lib' >> /etc/profile
echo 'export LUAJIT_INC=/usr/local/luajit/include/luajit-2.1' >> /etc/profile
source /etc/profile

由于lua-nginx-module导致的报错

nginx: [error] lua_load_resty_core failed to load the resty.core module from https://github.com/openresty/lua-resty-core; ensure you are using an OpenResty release from https://openresty.org/en/download.html (rc: 2, reason: module 'resty.core' not found:

解决方案：不要使用v0.10.15，使用14就没有问题了

wget https://github.com/openresty/lua-nginx-module/archive/v0.10.14.tar.gz
tar -xf v0.10.14.tar.gz -C /opt/# 安装nginx
./configure --prefix=/opt/nginx --with-http_realip_module --with-http_ssl_module --with-pcre --with-ld-opt=-Wl,-rpath,/usr/local/luajit/lib --add-module=/opt/lua-nginx-module-0.10.14 --add-module=/opt/ngx_devel_kit-0.3.1make -j2 && make install

参考资料

Nginx安装Lua支持：http://1t.click/b6ru
Nginx安装lua-nginx-module：http://1t.click/b6tw

History

日志分析系列(一)：介绍篇

日志分析系列(外传一)：Nginx透过代理获取真实客户端IP