Loki 收集Nginx日志以 grafana 可视化展示
背景
通常用ELK来收集Nginx日志的,对于服务器较少的用elk则显得太重了,可以用loki+Promtail+grafana 代替。
Loki类似elasticsearch,用于存储;Promtail类似fluent,用于收集;grafana类似kibana,用于展示
先放几张图来展示一下成果:
这里是用grafana查询日志
Nginx 配置
首先修改Nginx日志为json格式:
log_format json_analytics escape=json '{''"msec": "$msec", ' # request unixtime in seconds with a milliseconds resolution'"connection": "$connection", ' # connection serial number'"connection_requests": "$connection_requests", ' # number of requests made in connection'"pid": "$pid", ' # process pid'"request_id": "$request_id", ' # the unique request id'"request_length": "$request_length", ' # request length (including headers and body)'"remote_addr": "$remote_addr", ' # client IP'"remote_user": "$remote_user", ' # client HTTP username'"remote_port": "$remote_port", ' # client port'"time_local": "$time_local", ''"time_iso8601": "$time_iso8601", ' # local time in the ISO 8601 standard format'"request": "$request", ' # full path no arguments if the request'"request_uri": "$request_uri", ' # full path and arguments if the request'"args": "$args", ' # args'"status": "$status", ' # response status code'"body_bytes_sent": "$body_bytes_sent", ' # the number of body bytes exclude headers sent to a client'"bytes_sent": "$bytes_sent", ' # the number of bytes sent to a client'"http_referer": "$http_referer", ' # HTTP referer'"http_user_agent": "$http_user_agent", ' # user agent'"http_x_forwarded_for": "$http_x_forwarded_for", ' # http_x_forwarded_for'"http_host": "$http_host", ' # the request Host: header'"server_name": "$server_name", ' # the name of the vhost serving the request'"request_time": "$request_time", ' # request processing time in seconds with msec resolution'"upstream": "$upstream_addr", ' # upstream backend server for proxied requests'"upstream_connect_time": "$upstream_connect_time", ' # upstream handshake time incl. TLS'"upstream_header_time": "$upstream_header_time", ' # time spent receiving upstream headers'"upstream_response_time": "$upstream_response_time", ' # time spend receiving upstream body'"upstream_response_length": "$upstream_response_length", ' # upstream response length'"upstream_cache_status": "$upstream_cache_status", ' # cache HIT/MISS where applicable'"ssl_protocol": "$ssl_protocol", ' # TLS protocol'"ssl_cipher": "$ssl_cipher", ' # TLS cipher'"scheme": "$scheme", ' # http or https'"request_method": "$request_method", ' # request method'"server_protocol": "$server_protocol", ' # request protocol, like HTTP/1.1 or HTTP/2.0'"pipe": "$pipe", ' # "p" if request was pipelined, "." otherwise'"gzip_ratio": "$gzip_ratio", ''"http_cf_ray": "$http_cf_ray",''"geoip_country_code": "$geoip_country_code"''}';access_log /usr/local/nginx/logs/json_access.log json_analytics;
注意这里的geoip_country_code,用于展示哪个国家的哪个城市访问。需要在Nginx添加geoip的支持。
首先通过yum安装geoip及lib库
yum -y install GeoIP GeoIP-data GeoIP-devel
重新编译Nginx,加入--with-http_geoip_module进行添加模块
yum安装的geoip内置的数据较老,需要下载最新的数据
wget https://dl.miyuru.lk/geoip/dbip/country/dbip.dat.gz
gunzip dbip.dat.gz && mv dbip.dat country.dat
wget https://dl.miyuru.lk/geoip/dbip/city/dbip.dat.gz
gunzip dbip.dat.gz && mv dbip.dat city.dat
nginx.conf增加下面两行
geoip_city /usr/local/nginx/html/ip/city.dat;
geoip_country /usr/local/nginx/html/ip/country.dat;
后执行 /usr/local/nginx/sbin/nginx -s reload;现在Nginx的日志是这样的:
{"msec": "1654072604.175", "connection": "92799", "connection_requests": "1", "pid": "4251", "request_id": "47e117ff2287fb32fb8382aff742334b", "request_length": "1105", "remote_addr": "11.125.67.88", "remote_user": "", "remote_port": "18154", "time_local": "01/Jun/2022:16:36:44 +0800", "time_iso8601": "2022-06-01T16:36:44+08:00", "request": "POST /xx/xx.html HTTP/1.1", "request_uri": "/xx/xx.html", "args": "", "status": "200", "body_bytes_sent": "146", "bytes_sent": "308", "http_referer": "http://172.16.124.225:8080/", "http_user_agent": "Mozilla/5.0 (iPhone; CPU iPhone OS 13_2_3 like Mac OS X) AppleWebKit/605.1.15 (KHTML, like Gecko) Version/13.0.3 Mobile/15E148 Safari/604.1", "http_x_forwarded_for": "172.16.124.225, 218.104.146.57", "http_host": "xx.com", "server_name": "xx.com", "request_time": "0.704", "upstream": "192.168.96.13:80", "upstream_connect_time": "0.000", "upstream_header_time": "0.700", "upstream_response_time": "0.700", "upstream_response_length": "135", "upstream_cache_status": "", "ssl_protocol": "", "ssl_cipher": "", "scheme": "http", "request_method": "POST", "server_protocol": "HTTP/1.1", "pipe": ".", "gzip_ratio": "", "http_cf_ray": "","geoip_country_code": "US"}
Loki部署
下载软件
wget https://github.com/grafana/loki/releases/download/v2.5.0/loki-linux-amd64.zip
unzip loki-linux-amd64.zip
mv loki-linux-amd64 loki
loki的配置文件
auth_enabled: falseserver:http_listen_port: 3100grpc_listen_port: 9096common:path_prefix: /data/lokistorage:filesystem:chunks_directory: /data/loki/chunksrules_directory: /data/loki/rulesreplication_factor: 1ring:instance_addr: 127.0.0.1kvstore:store: inmemoryschema_config:configs:- from: 2020-10-24store: boltdb-shipperobject_store: filesystemschema: v11index:prefix: index_period: 24hruler:alertmanager_url: http://localhost:9093
启动loki
nohup ./loki -config.file=loki-local-config.yaml &
Promtail 部署
下载软件
wget https://github.com/grafana/loki/releases/download/v2.5.0/promtail-linux-amd64.zip
unzip promtail-linux-amd64.zip
mv promtail-linux-amd64 promtail
配置文件
server:http_listen_port: 9080grpc_listen_port: 0positions:filename: /tmp/positions.yamlclients:- url: http://192.168.96.239:3100/loki/api/v1/pushscrape_configs:
- job_name: nginxpipeline_stages:- replace:expression: '(?:[0-9]{1,3}\.){3}([0-9]{1,3})'replace: '***'static_configs:- targets:- localhostlabels:job: nginx_access_loghost: appfelstrudelagent: promtail__path__: /usr/local/nginx/logs/json_access.log
启动
nohup ./promtail --config.file=promtail-local-config.yaml &
grafana展示
下载安装
wget https://mirrors.cloud.tencent.com/grafana/yum/rpm/grafana-8.5.3-1.x86_64.rpm
yum install -y grafana-8.5.3-1.x86_64.rpm
systemctl start grafana-server
grafana添加数据源,选Loki。
输入url:http://localhost:3100 即可
然后导入 13865 模板即可
下面这篇文章没实践过,不过里面地图可以对应到城市,本篇的地图只能对应到国家:
搭建日志聚合grafana&loki 收集Nginx日志_Zz_糖小七的博客-CSDN博客_loki收集nginx
nginx 设置变量获取国家、城市:
科技常识:Nginx服务器中配置GeoIP模块来拦截指定国家IP_绿色消费网
grafana 面板报 "too many outstanding requests" 错误解决方案
Grafana dashboard shows "too many outstanding requests" after upgrade to v2.4.2 - bytemeta
grafana 面板报 "maximum of series (500) reached for a single query" 错误解决方案
maximum of series (500) reached for a single query - Installation - Grafana Labs Community Forums
grafana 面板报 " net/http: timeout awaiting response headers (Client.Timeout exceeded while awaiting headers)" 错误解决方案
Configure Grafana | Grafana documentation
loki 查询超时解决方案
Loki datasource timeout at 30s · Issue #42801 · grafana/grafana · GitHub
loki 用缓存加速查询
巧用缓存加速Loki查询 - 腾讯云开发者社区-腾讯云
loki 配置文件范例参考
Loki datasource timeout at 30s · Issue #42801 · grafana/grafana · GitHub
Loki 收集Nginx日志以 grafana 可视化展示相关推荐
- 【lnmp+ELK+Nginx+GeoIP】超详细新版ELK8.5.0解析收集Nginx日志并实现数据可视化与IP地址地图可视化教程-2022年11月版
超详细新版ELK8.5.0+lnmp1.9解析收集Nginx日志并实现数据可视化与IP地址地图可视化教程-2022年11月版 前言 1.基础环境和所需要的软件包 1.1服务器配置(仅供参考) 1.2 ...
- ELK日志系统之使用Rsyslog快速方便的收集Nginx日志
常规的日志收集方案中Client端都需要额外安装一个Agent来收集日志,例如logstash.filebeat等,额外的程序也就意味着环境的复杂,资源的占用,有没有一种方式是不需要额外安装程序就能实 ...
- logstash收集nginx日志
1.配置nginx日志 编辑nginx.conf文件 vim /etc/nginx/nginx.conf 在http节点下配置如下 log_format json '{"@timestamp ...
- centos6.5下安装配置ELK及收集nginx日志
Elasticsearch 是个开源分布式搜索引擎,它的特点有:分布式,零配置,自动发现,索引自动分片,索引副本机制,restful风格接口,多数据源,自动搜索负载等. Logstash 是一个完全开 ...
- 安装logstash5.4.1,并使用grok表达式收集nginx日志
关于收集日志的方式,最简单性能最好的应该是修改nginx的日志存储格式为json,然后直接采集就可以了. 但是实际上会有一个问题,就是如果你之前有很多旧的日志需要全部导入elk上查看,这时就有两个问题 ...
- CentOS下ELK收集Nginx日志
1. ELK收集Nginx普通格式的日志 1.1 测试服务器架构 1.2 ab工具使用 yum install httpd-tools -y# -n 总共发送多少条请求,注意,最后"/&qu ...
- centos7 单机安装ELK7收集nginx日志 无坑版,肯定出图
系统:centos7 ELK版本:7.8.0 java环境:11 1. 配置JDK #配置JDK11 mkdir /root/source/cd /root/sourcewget https://re ...
- 阿里云日志服务grafana可视化探索
一.背景 网站前端启用了阿里云的负载均衡,各类日志很方便的接入阿里云的日志服务,目前对于日志服务后台提供的可视化不甚满意,故探索grafana对其可视化. 二.目的 1.根据nginx日志(slb)来 ...
- ELK 收集 Nginx 日志
01 安装 Nginx 和 ab 工具 1.1 安装 nginx sudo apt-get install nginx -y # 安装Nginx sudo apt-get install apache ...
最新文章
- 爬虫的系统框架组成-资源库
- js判断时间两小时之内_js判断两个时间的大小
- 【干货】吴甘沙:你是数据,我即生意
- android 自定义取色器,【Android自定义View】仿Photoshop取色器ColorPicker(二)
- Ubuntu更换apt镜像源
- java channel源码_java nio ServerSocketChannel源码分析
- 通过什么途径能够深入了解JavaScript引擎是如何工作的?
- 《哪吒》票房超25亿元 进入中国电影票房总榜前十
- centos 减少tty数量的方法
- network 一级,二级,子域名
- 电量分析 —— 优化耗电
- Rational Rose 2007安装使用教程
- 中兴服务器iSAC接口默认账号密码和IP
- Android Q版本实现自动连接WiFi
- SO逆向入门实战教程四:mfw
- MySQL中统计函数的使用
- 你所不知的有趣投影方法
- 成也苹果败也苹果,曾经女首富身价缩水一半
- 从模拟信号到数字信号
- Linux画图工具gnuplot
热门文章
- 2021哔哩哔哩1024程序员节日第一弹:算法与安全
- 如何关闭 Intel RST
- agv调度系统3.0技术介绍
- 爱上开源之一款查询docker容器启动命令的工具
- request.querystring php,ASP_ASP读取Request.QueryString编码的函数代码,1. 支持参数纯汉字 ?a=深山老熊 - phpStudy...
- android 九宫格 拉伸,CAScale9ImageView(九宫格图片拉伸)
- Opegnl ES之四边形绘制
- 如何应对数千微服务组件带来的挑战?
- 智能验证码 阿里云人机验证的使用(PHP)
- CVE-2012-4792 漏洞利用学习笔记