Prometheus-监控主机基础指标配置及告警

1、监控主机指标

这是一篇介绍主机使用Prometheus监控CPU、磁盘、内存、负载等基础数据的文章，目前生产可用，使用的是node_exporter-0.18.1版本，操作系统是centos7.X版本，使用之前请修改job="gt-dwz-node-exporter"的值对应自己在Prometheus配置的job名称。

2、Prometheus配置项

在prometheus.yml配置文件中添加如下配置：

############gt-dwz#################- job_name: "gt-dwz-node-exporter"static_configs:- targets: ['10.1.5.123:9100','10.1.5.124:9100','10.1.5.125:9100','10.1.5.126:9100']labels:service: gt-dwz-monitor

3、PromQL判断rules文件

[root@gtcq-gt-monitor-prometheus-01 rules]# more    gt-dwz-monitor.rules
groups:
- name: dwz-gt-monitorrules:- alert: "node-Agent告警"expr: up{job="gt-dwz-node-exporter"} == 0for: 120slabels:severity: "重要"team: dwz-gt-monitoralert_type: "Agent告警" alert_host: "{{ reReplaceAll \":(.*)\" \"\" $labels.instance }}"annotations:summary: "{{ $labels.instance }} 已停止采集监控数据 30s！"description: "{{ $labels.instance }} job {{ $labels.job }} 暴露监控数据已停止."- alert: "CPU使用率监控"expr: ceil(100 - sum(increase(node_cpu_seconds_total{job="gt-dwz-node-exporter",mode="idle"}[5m]))  by(instance) / sum(increase(node_cpu_seconds_total{job="gt-dwz-node-exporter"}[5m])) by(instance)*100) > 80for: 2mlabels:severity: "重要"team: bdfbalert_type: "CPU告警"alert_host: "{{ reReplaceAll \":(.*)\" \"\" $labels.instance }}"annotations:summary: "{{ reReplaceAll \":(.*)\" \"\" $labels.instance }} CPU使用率过高"description: "IP:{{ reReplaceAll \":(.*)\" \"\" $labels.instance }}的CPU使用大于80% (当前值: {{ $value }})"- alert: "磁盘使用率监控"expr: round((1 - (node_filesystem_avail_bytes{fstype=~"ext3|ext4|xfs|nfs",job="gt-dwz-node-exporter"} / node_filesystem_size_bytes{fstype=~"ext3|ext4|xfs|nfs",job="gt-dwz-node-exporter"
})) * 100)  > 80for: 2mlabels:severity: "重要"team: dwz-gt-monitoralert_type: "Disk告警"alert_host: "{{ reReplaceAll \":(.*)\" \"\" $labels.instance }}"annotations:summary: "{{ reReplaceAll \":(.*)\" \"\" $labels.instance }} : {{ $labels.mountpoint }} 分区使用率过高"description: "{{ reReplaceAll \":(.*)\" \"\" $labels.instance }}的{{ $labels.mountpoint }} 分区使用大于80% (当前值: {{ $value }}%)"- alert: "内存使用率监控"expr: ceil( (1 - (node_memory_MemAvailable_bytes{job="gt-dwz-node-exporter"} / (node_memory_MemTotal_bytes{job="gt-dwz-node-exporter"})))* 100 ) > 80for: 2mlabels:severity: "重要"team: dwz-gt-monitoralert_type: "MEM告警"alert_host: "{{ reReplaceAll \":(.*)\" \"\" $labels.instance }}"annotations:summary: "{{ reReplaceAll \":(.*)\" \"\" $labels.instance }}内存使用率过高"description: "{{ reReplaceAll \":(.*)\" \"\" $labels.instance }}内存使用大于80% (当前值: {{ $value }})"- alert: "服务器大法宝CPULoad5"expr: node_load5{job="gt-dwz-node-exporter"} > 100for: 2mlabels:severity: "重要"team: dwz-gt-monitoralert_type: "负载告警"alert_host: "{{ reReplaceAll \":(.*)\" \"\" $labels.instance }}"annotations:summary: "{{ reReplaceAll \":(.*)\" \"\" $labels.instance }}CPU负载过高"description: "{{ reReplaceAll \":(.*)\" \"\" $labels.instance }} CPU负载load大于100 (当前值: {{ $value }})"- alert: "服务器文件句柄监控"expr: node_filefd_allocated{job="gt-dwz-node-exporter"} > 50000for: 2mlabels:severity: "重要"team: dwz-gt-monitoralert_type: "文件句柄告警"alert_host: "{{ reReplaceAll \":(.*)\" \"\" $labels.instance }}"annotations:summary: "{{ reReplaceAll \":(.*)\" \"\" $labels.instance }} 文件句柄使用过高"description: "{{ reReplaceAll \":(.*)\" \"\" $labels.instance }} 文件句柄使用过高大于50000 (当前值: {{ $value }})"- alert: "服务器TCP连接数监控"expr: node_sockstat_TCP_tw{job="gt-dwz-node-exporter"} > 15000for: 2mlabels:severity: "重要"team: dwz-gt-monitoralert_type: "TCP连接数告警"alert_host: "{{ reReplaceAll \":(.*)\" \"\" $labels.instance }}"annotations:summary: "{{ reReplaceAll \":(.*)\" \"\" $labels.instance }} 等待关闭的TCP连接数过高"description: "{{ reReplaceAll \":(.*)\" \"\" $labels.instance }} 等待关闭的TCP连接数TIME_WAIT过高大于15000 (当前值: {{ $value }})"- alert: "服务器入口流量监控"expr: round((sum by (instance)  (irate(node_network_receive_bytes_total{job="gt-dwz-node-exporter",device!~'tap.*|veth.*|br.*|docker.*|virbr*|lo*'}[5m])))/1024/1024) > 50for: 2mlabels:severity: "重要"team: dwz-gt-monitoralert_type: "流量告警"alert_host: "{{ reReplaceAll \":(.*)\" \"\" $labels.instance }}"annotations:summary: "{{ reReplaceAll \":(.*)\" \"\" $labels.instance }}监控入口流量过高"description: "{{ reReplaceAll \":(.*)\" \"\" $labels.instance }} 监控入口流量过高过高大于50MB (告警值: {{ $value }}MB)"- alert: "服务器出口流量监控"expr: round((sum by (instance)  (irate(node_network_transmit_bytes_total{job="gt-dwz-node-exporter",device!~'tap.*|veth.*|br.*|docker.*|virbr*|lo*'}[5m])))/1024/1024) > 50for: 2mlabels:severity: "重要"team: dwz-gt-monitoralert_type: "流量告警"alert_host: "{{ reReplaceAll \":(.*)\" \"\" $labels.instance }}"annotations:summary: "{{ reReplaceAll \":(.*)\" \"\" $labels.instance }} 监控出口流量过高"description: "{{ reReplaceAll \":(.*)\" \"\" $labels.instance }} 监控出口流量过高过高大于50MB (告警值: {{ $value }}MB)"
[root@gtcq-gt-monitor-prometheus-01 rules]#

4、测试告警

修改磁盘阈值如下：

  - alert: "磁盘使用率监控"expr: round((1 - (node_filesystem_avail_bytes{fstype=~"ext3|ext4|xfs|nfs",job="gt-dwz-node-exporter"} / node_filesystem_size_bytes{fstype=~"ext3|ext4|xfs|nfs",job="gt-dwz-node-exporter"})) * 100)  > 10for: 2mlabels:severity: "重要"team: dwz-gt-monitoralert_type: "Disk告警"alert_host: "{{ reReplaceAll \":(.*)\" \"\" $labels.instance }}"annotations:summary: "{{ reReplaceAll \":(.*)\" \"\" $labels.instance }} : {{ $labels.mountpoint }} 分区使用率过高"description: "{{ reReplaceAll \":(.*)\" \"\" $labels.instance }}的{{ $labels.mountpoint }} 分区使用大于10% (当前值: {{ $value }}%)"

Prometheus-监控主机基础指标配置及告警相关推荐

CentOS7安装可移植Prometheus+grafana--alertmanager配置邮件告警
背景前两篇博文我们介绍了Prometheus及其相关的监控组件,本次我们记录一下告警组件:alertmanager 安装配置获取安装包官方下载地址:https://prometheus.io/d ...
实用干货丨如何使用Prometheus配置自定义告警规则
前言 Prometheus是一个用于监控和告警的开源系统.一开始由Soundcloud开发,后来在2016年,它迁移到CNCF并且称为Kubernetes之后最流行的项目之一.从整个Linux服务器 ...
Prometheus监控主机的配置
Prometheus监控主机的配置 exporter介绍 node_exporter 安装部署 Prometheus配置重启Prometheus服务查看Prometheus的UI界面 export ...
Prometheus+Grafana监控安装及配置JVM实现企业微信告警
背景本人Java开发工程师一枚,主攻后端,需要搭建一套Prometheus+Grafana的监控系统,采用企业微信告警通知.在网上各种查阅资料的同时,发现很多资料都大致相通且不完整,踩坑无数,经过多 ...
Prometheus Operator 配置PrometheusRule告警规则
PrometheusRule 用于配置 Prometheus 的 Rule 规则文件,包括 recording rules 和 alerting,可以自动被 Prometheus 加载. 配置 Pro ...
Prometheus监控系部署配置过程
一.前言 Prometheus是由SoundCloud开发的开源监控报警系统和时序列数据库(TSDB),基于Go语言开发,是Google BorgMon监控系统的开源版本.Prometheus在201 ...
Prometheus 监控基础
Prometheus 监控基础监控系统基础监控系统发展史 SNMP监控时代第一代主要是监控网络流量和网络设置为主的时代,在那个时代主要是通过SNMP协议监控交换机.路由器.网关.操作系统等,通过 ...
Prometheus 监控报警系统 AlertManager 之邮件告警
文章目录 1.Prometheus & AlertManager 介绍 2.环境.软件准备 3.启动并配置 Prometheus 3.1.Docker 启动 Prometheus 3.2.Do ...
基于prometheus的网络指标监控
1 网络指标监控网络设备(交换机.路由器.防火墙)本身并没有太多的监控方式,常规的是使用snmp,通过oid去采集网络设备的指标,例如,流量或者错包,不过近年来也出现了新的采集方式,例如,telem ...

Prometheus-监控主机基础指标配置及告警

1、监控主机指标

2、Prometheus配置项

3、PromQL判断rules文件

4、测试告警

Prometheus-监控主机基础指标配置及告警相关推荐

最新文章

热门文章