Prometheus + AlertManager + Prometheus-webhook-dingtalk的告警通知

基于mysql的监控配置，使用钉钉推送告警通知，grafana图形化展示

1.docker拉取相关镜像

docker pull prom/mysqld-exporter
docker pull prom/prometheus
docker pull prom/alertmanager
docker pull timonwong/prometheus-webhook-dingtalk
##redis-exporter
docker pull oliver006/redis_exporter

2.启动镜像，配置配置文件

docker run -d -p 9104:9104 --name mysqld-exporter -e DATA_SOURCE_NAME="username:password@(localhost:3306)/db_name" prom/mysqld-exporter
docker run -d --name prometheus -p 9090:9090 prom/prometheus
docker run -d --name alertmanager -p 9093:9093 prom/alertmanager
docker run -d --name webhook -p 8060:8060 timonwong/prometheus-webhook-dingtalk
##redis-exporter启动
docker run -d --name redis_exporter -p 9121:9121 oliver006/redis_exporter --redis.addr redis://192.168.10.66:6379

prometheus 配置

执行 docker exec -it -u root prometheus sh
执行 cd /etc/prometheus/
执行 vi prometheus.yml##shift+i 开启编辑模式
##修改alertmanager地址
alerting:alertmanagers:- static_configs:- targets:- 192.168.10.66:9093      ##修改链接的alertmanager服务地址：本机IP+端口（IP不能填localhost或127.0.0.1，会被Prometheus自己的localhost地址覆盖，导致识别不到）##在Prometheus Job下添加监控mysql Job- job_name: 'mysqld'static_configs:- targets: ['192.168.10.66:9104']        ##链接到mysqld-exporterlabels:name: mysql-5.7instance: 192.168.10.66:3306##esc退出编辑模式；:wq保存退出
##docker restart prometheus重启

webhook 配置

钉钉群组配置钉钉机器人

执行 docker exec -it webhook sh
执行 cd /etc/prometheus-webhook-dingtalk/
执行 vi config.yml##shift+i 开启编辑模式
##添加钉钉机器人的webhook地址（可添加多个）
targets:webhook1:url: https://oapi.dingtalk.com/robot/send?access_token=xxxxxxxxxxxx    # secret for signature (使用加签安全设置)secret: SECxxxxxxxxxxxxxxxxxwebhook2:url: https://oapi.dingtalk.com/robot/send?access_token=xxxxxxxxxxxx##esc退出编辑模式；:wq保存退出
##docker restart webhook重启

alertmanager 配置

执行 docker exec -it alertmanager sh
执行 cd /etc/alertmanager/
执行 vi alertmanager.yml##shift+i 开启编辑模式
##设置webhook地址（可设置多个路由地址）
route:group_by: ['alertname']group_wait: 10sgroup_interval: 10srepeat_interval: 1hroutes:
# webhook1 webhook- receiver: web.hook1group_wait: 10smatch:team: webhook1
# webhook2 webhook- receiver: web.hook2group_wait: 10smatch:team: webhook2
receivers:
- name: 'web.hook1'webhook_configs:- url: http://192.168.10.66:8060/dingtalk/webhook1/sendsend_resolved: true
- name: 'web.hook2'webhook_configs:- url: http://192.168.10.66:8060/dingtalk/webhook2/sendsend_resolved: true##esc退出编辑模式；:wq保存退出
##docker restart alertmanager重启

3.设置告警规则

(1) 宿主机创建规则文件mysql-alerts.yml

groups:
- name: MySQLStatsAlertrules:- alert: MySQL is downexpr: mysql_up == 0for: 1mlabels:severity: criticalannotations:summary: "Instance {{ $labels.instance }} MySQL is down"description: "MySQL database is down. This requires immediate action!"- alert: open files highexpr: mysql_global_status_innodb_num_open_files > (mysql_global_variables_open_files_limit) * 0.75for: 1mlabels:severity: warningannotations:summary: "Instance {{ $labels.instance }} open files high"description: "Open files is high. Please consider increasing open_files_limit."- alert: Read buffer size is bigger than max. allowed packet sizeexpr: mysql_global_variables_read_buffer_size > mysql_global_variables_slave_max_allowed_packet for: 1mlabels:severity: warningannotations:summary: "Instance {{ $labels.instance }} Read buffer size is bigger than max. allowed packet size"description: "Read buffer size (read_buffer_size) is bigger than max. allowed packet size (max_allowed_packet).This can break your replication."- alert: Sort buffer possibly missconfiguredexpr: mysql_global_variables_innodb_sort_buffer_size <256*1024 or mysql_global_variables_read_buffer_size > 4*1024*1024 for: 1mlabels:severity: warningannotations:summary: "Instance {{ $labels.instance }} Sort buffer possibly missconfigured"description: "Sort buffer size is either too big or too small. A good value for sort_buffer_size is between 256k and 4M."- alert: Thread stack size is too smallexpr: mysql_global_variables_thread_stack <196608for: 1mlabels:severity: warningannotations:summary: "Instance {{ $labels.instance }} Thread stack size is too small"description: "Thread stack size is too small. This can cause problems when you use Stored Language constructs for example. A typical is 256k for thread_stack_size."- alert: Used more than 80% of max connections limited expr: mysql_global_status_max_used_connections > mysql_global_variables_max_connections * 0.8for: 1mlabels:severity: warningannotations:summary: "Instance {{ $labels.instance }} Used more than 80% of max connections limited"description: "Used more than 80% of max connections limited"- alert: InnoDB Force Recovery is enabledexpr: mysql_global_variables_innodb_force_recovery != 0 for: 1mlabels:severity: warningannotations:summary: "Instance {{ $labels.instance }} InnoDB Force Recovery is enabled"description: "InnoDB Force Recovery is enabled. This mode should be used for data recovery purposes only. It prohibits writing to the data."- alert: InnoDB Log File size is too smallexpr: mysql_global_variables_innodb_log_file_size < 16777216 for: 1mlabels:severity: warningannotations:summary: "Instance {{ $labels.instance }} InnoDB Log File size is too small"description: "The InnoDB Log File size is possibly too small. Choosing a small InnoDB Log File size can have significant performance impacts."- alert: InnoDB Flush Log at Transaction Commitexpr: mysql_global_variables_innodb_flush_log_at_trx_commit != 1for: 1mlabels:severity: warningannotations:summary: "Instance {{ $labels.instance }} InnoDB Flush Log at Transaction Commit"description: "InnoDB Flush Log at Transaction Commit is set to a values != 1. This can lead to a loss of commited transactions in case of a power failure."- alert: Table definition cache too smallexpr: mysql_global_status_open_table_definitions > mysql_global_variables_table_definition_cachefor: 1mlabels:severity: pageannotations:summary: "Instance {{ $labels.instance }} Table definition cache too small"description: "Your Table Definition Cache is possibly too small. If it is much too small this can have significant performance impacts!"- alert: Table open cache too smallexpr: mysql_global_status_open_tables >mysql_global_variables_table_open_cache * 99/100for: 1mlabels:severity: pageannotations:summary: "Instance {{ $labels.instance }} Table open cache too small"description: "Your Table Open Cache is possibly too small (old name Table Cache). If it is much too small this can have significant performance impacts!"- alert: Thread stack size is possibly too smallexpr: mysql_global_variables_thread_stack < 262144for: 1mlabels:severity: pageannotations:summary: "Instance {{ $labels.instance }} Thread stack size is possibly too small"description: "Thread stack size is possibly too small. This can cause problems when you use Stored Language constructs for example. A typical is 256k for thread_stack_size."- alert: InnoDB Buffer Pool Instances is too smallexpr: mysql_global_variables_innodb_buffer_pool_instances == 1for: 1mlabels:severity: pageannotations:summary: "Instance {{ $labels.instance }} InnoDB Buffer Pool Instances is too small"description: "If you are using MySQL 5.5 and higher you should use several InnoDB Buffer Pool Instances for performance reasons. Some rules are: InnoDB Buffer Pool Instance should be at least 1 Gbyte in size. InnoDB Buffer Pool Instances you can set equal to the number of cores of your machine."- alert: InnoDB Plugin is enabledexpr: mysql_global_variables_ignore_builtin_innodb == 1for: 1mlabels:severity: pageannotations:summary: "Instance {{ $labels.instance }} InnoDB Plugin is enabled"description: "InnoDB Plugin is enabled"- alert: Binary Log is disabledexpr: mysql_global_variables_log_bin != 1for: 1mlabels:severity: warningannotations:summary: "Instance {{ $labels.instance }} Binary Log is disabled"description: "Binary Log is disabled. This prohibits you to do Point in Time Recovery (PiTR)."- alert: Binlog Cache size too smallexpr: mysql_global_variables_binlog_cache_size < 1048576for: 1mlabels:severity: pageannotations:summary: "Instance {{ $labels.instance }} Binlog Cache size too small"description: "Binlog Cache size is possibly to small. A value of 1 Mbyte or higher is OK."- alert: Binlog Statement Cache size too smallexpr: mysql_global_variables_binlog_stmt_cache_size <1048576 and mysql_global_variables_binlog_stmt_cache_size > 0for: 1mlabels:severity: pageannotations:summary: "Instance {{ $labels.instance }} Binlog Statement Cache size too small"description: "Binlog Statement Cache size is possibly to small. A value of 1 Mbyte or higher is typically OK."- alert: Binlog Transaction Cache size too smallexpr: mysql_global_variables_binlog_cache_size  <1048576for: 1mlabels:severity: pageannotations:summary: "Instance {{ $labels.instance }} Binlog Transaction Cache size too small"description: "Binlog Transaction Cache size is possibly to small. A value of 1 Mbyte or higher is typically OK."- alert: Sync Binlog is enabledexpr: mysql_global_variables_sync_binlog == 1for: 1mlabels:severity: pageannotations:summary: "Instance {{ $labels.instance }} Sync Binlog is enabled"description: "Sync Binlog is enabled. This leads to higher data security but on the cost of write performance."- alert: IO thread stoppedexpr: mysql_slave_status_slave_io_running != 1for: 1mlabels:severity: criticalannotations:summary: "Instance {{ $labels.instance }} IO thread stopped"description: "IO thread has stopped. This is usually because it cannot connect to the Master any more."- alert: SQL thread stopped expr: mysql_slave_status_slave_sql_running == 0for: 1mlabels:severity: criticalannotations:summary: "Instance {{ $labels.instance }} SQL thread stopped"description: "SQL thread has stopped. This is usually because it cannot apply a SQL statement received from the master."- alert: SQL thread stoppedexpr: mysql_slave_status_slave_sql_running != 1for: 1mlabels:severity: criticalannotations:summary: "Instance {{ $labels.instance }} Sync Binlog is enabled"description: "SQL thread has stopped. This is usually because it cannot apply a SQL statement received from the master."- alert: Slave lagging behind Masterexpr: rate(mysql_slave_status_seconds_behind_master[1m]) >30 for: 1mlabels:severity: warning annotations:summary: "Instance {{ $labels.instance }} Slave lagging behind Master"description: "Slave is lagging behind Master. Please check if Slave threads are running and if there are some performance issues!"- alert: Slave is NOT read only(Please ignore this warning indicator.)expr: mysql_global_variables_read_only != 0for: 1mlabels:severity: pageannotations:summary: "Instance {{ $labels.instance }} Slave is NOT read only"description: "Slave is NOT set to read only. You can accidentally manipulate data on the slave and get inconsistencies..."

(2) 使用docker cp命令复制yml文件到容器内

docker cp D:\mysql-alerts.yml prometheus:/etc/prometheus/

(3)Prometheus配置文件配置规则文件路径

rule_files:- "/etc/prometheus/mysql-alerts.yml"# - "first_rules.yml"# - "second_rules.yml"

重启访问http://localhost:9090/alerts （有下图列表代表配置成功）

4.配置grafana图形化展示

docker pull grafana/grafanadocker run -d --name grafana -p 3000:3000 grafana/grafana

访问http://localhost:3000 （账号密码都是admin）

首先配置数据源datasource

导入模板7362，显示如下

Prometheus + AlertManager + Prometheus-webhook-dingtalk的告警通知相关推荐

Prometheus+Alertmanager+webhook-dingtalk实现钉钉告警
文章目录一.前提准备及规划二.安装及启动 2.1 Prometheus安装启动 2.2 Node_export安装启动 2.3 Alertmanager安装启动 2.4 Webhook-dingt ...
prometheus 发送恢复值_基于prometheus+grafana+alertmanager监控系统配置钉钉告警
概述因为目前工作基本都是用钉钉办公,所以今天主要介绍一下怎么在prometheus配置钉钉告警,这里的前提是已经部署了alertmanager. 一.配置go 由于Prometheus 是用gola ...
prometheus+alertmanager+webhook实现自定义监控报警系统
文章目录 1.概述 2.prometheus配置报警规则 3.alertmanager配置及部署 3.1.alertmanager配置 3.2.alertmanager部署 4.prometheus关 ...
使用Prometheus+Alertmanager告警JVM异常情况
本文介绍如何使用Prometheus+Alertmanager来对JVM的某些情况作出告警. 本文所提到的脚本可以在这里下载. 摘要用到的工具: Docker,本文大量使用了Docker来启动各个应 ...
prometheus+alertmanager 企业微信告警
一.应用安装启动 1.软件准备 alertmanager-0.24.0.linux-amd64.tar.gz blackbox_exporter-0.22.0.linux-amd64.tar.gz n ...
prometheus alertmanager告警pending
prometheus alertmanager告警在配置了for标签后指标达到告警阈值一直处于pending状态无法变成firing状态配置如下最后定位问题是因为将 value字段放到了lab ...
Prometheus Alertmanager报警组件
全栈工程师开发手册 (作者:栾鹏) 架构系列文章 Prometheus Alertmanager 概述 Alertmanager与Prometheus是相互分离的两个组件.Prometheus服务器根 ...
运维监控系列（15）-Alertmanager添加163邮箱、钉钉、微信告警通知功能
163邮箱可用通过邮件方式进行告警通知的发送,接着上篇文档,我们配置发送邮件告警方式. 1. 设置163邮箱发送邮件,需要smtp邮件服务器,首先需要对网易邮箱进行设置. 首先登陆163邮箱,进入 ...
prometheus+grafana监控以及企业微信告警
prometheus+grafana监控以及企业微信告警(单机二进制部署) 一.下载部署包,更改其中两个包名称,放到/data下 1.安装包以及解压步骤 grafana-enterprise-8.4. ...