[kubernetes]-k8s安装alertmanager和prometheus-webhook-dingtalk

安装alertmanager

创建存放数据及插件的文件夹

# 在指定的node上创建文件夹
mkdir -p /data/k8s/alertmanager
chown -R 65534:root  alertmanager

创建alertmanager-cm.yaml

---
apiVersion: v1
kind: ConfigMap
metadata:name: alertmanager-confignamespace: kube-ops
data:alertmanager.yml: |-# 全局配置项global:resolve_timeout: 5m # 处理超时时间，默认为5minsmtp_from: 'it@hz-health.cn'smtp_smarthost: 'smtp.exmail.qq.com:465'#smtp_smarthost: 'smtp.aliyun.com:465'smtp_auth_username: 'it@hz-health.cn'smtp_auth_password: 'yourpassword'smtp_require_tls: false# 定义路由树信息route:group_by: [alertname]  # 报警分组依据receiver: ops_notify   # 设置默认接收人group_wait: 30s        # 最初即第一次等待多久时间发送一组警报的通知group_interval: 60s    # 在发送新警报前的等待时间 下一次报警开车时间repeat_interval: 1h    # 重复发送告警时间。默认1h       第一次报警时间为group_interval  重复报警的时间为group_interval+repeat_intervalroutes:- receiver: ops_notify  # 基础告警通知group_wait: 10smatch_re:alertname: 实例存活告警|磁盘使用率告警   # 匹配告警规则中的名称发送- receiver: info_notify  # 消息告警通知group_wait: 10smatch_re:alertname: 内存使用率告警|CPU使用率告警# 定义基础告警接收者receivers:- name: ops_notifywebhook_configs:- url: http://prometheus-webhook-dingtalk.ihaozhuo.com/dingtalk/ops_dingding/sendsend_resolved: true  # 警报被解决之后是否通知email_configs:# - to: '423308591@qq.com‘  如果两个邮箱中间需要空格- to: 'it@hz-health.cn, xujiamin@hz-health.cn'send_resolved: true# 定义消息告警接收者- name: info_notifywebhook_configs:- url: http://prometheus-webhook-dingtalk.ihaozhuo.com/dingtalk/info_dingding/sendsend_resolved: true# 一个inhibition规则是在与另一组匹配器匹配的警报存在的条件下，使匹配一组匹配器的警报失效的规则。两个警报必须具有一组相同的标签。inhibit_rules:- source_match:severity: 'critical'target_match:severity: 'warning'equal: ['alertname', 'dev', 'instance']

创建alertmanager-deploy.yaml

---
apiVersion: apps/v1
kind: Deployment
metadata:name: alertmanagernamespace: kube-opslabels:app: alertmanager
spec:replicas: 1revisionHistoryLimit: 2selector:matchLabels:app: alertmanagerminReadySeconds: 0strategy:type: RollingUpdaterollingUpdate:maxUnavailable: 1maxSurge: 1template:metadata:labels:app: alertmanagerspec:affinity:         #亲和性的调度设置nodeAffinity:   #策略为节点亲和性requiredDuringSchedulingIgnoredDuringExecution:      #亲和性的硬策略nodeSelectorTerms:   #这里不再使用nodeselector，使用这个参数可以进行相对简单的逻辑运算- matchExpressions:     #匹配表达式- key: kubernetes.io/hostname      #具体匹配规则(可以通过kubectl get node --show-labels找到相应规则)operator: In  #不在，简单的来说就是不在k8s-04节点values:- prod-k8s-n006containers:- name: alertmanagerimage: prom/alertmanager:v0.21.0args:- "--config.file=/etc/alertmanager/alertmanager.yml"- "--storage.path=/alertmanager/data"resources:limits:memory: "256Mi"cpu: "100m"readinessProbe:tcpSocket:port: 9093initialDelaySeconds: 30periodSeconds: 10livenessProbe:tcpSocket:port: 9093initialDelaySeconds: 60periodSeconds: 10ports:- containerPort: 9093protocol: TCPname: httpvolumeMounts:- name: datamountPath: /alertmanager/data- name: config-volumemountPath: "/etc/alertmanager/alertmanager.yml"subPath: alertmanager.ymlvolumes:- name: datahostPath:# 宿主上目录位置path: /data/k8s/alertmanagertype: DirectoryOrCreate- configMap:name: alertmanager-configname: config-volume

创建alertmanager-svc.yaml

---
apiVersion: v1
kind: Service
metadata:name: alertmanagernamespace: kube-opslabels:app: alertmanager
spec:selector:app: alertmanagertype: NodePortports:- port: 80protocol: TCPtargetPort: 9093

创建alertmanager-ingress.yaml

---
apiVersion: extensions/v1beta1
kind: Ingress
metadata:
# 通过添加下面的annotations 来开启白名单
# 关闭80强制跳转443 为ingress配置增加注解（annotations）：nginx.ingress.kubernetes.io/ssl-redirect: 'false' 就可以禁止http强制跳转至httpsannotations:#nginx.ingress.kubernetes.io/whitelist-source-range: "60.191.70.64/29, xx.xxx.0.0/16"nginx.ingress.kubernetes.io/ssl-redirect: 'false'name: prod-alertmanagernamespace: kube-ops
spec:rules:- host: alertmanager.ihaozhuo.comhttp:paths:- path: /backend:serviceName: alertmanagerservicePort: 80

运行alertmanager

kubectl apply -f ./

安装prometheus-webhook-dingtalk

先看一下prometheus-webhook-dingtalk的参数打算通过configmap的方式挂载config.yml并开启ui。

之前都是直接启动的时候通过命令/srv/alertmanager/prometheus-webhook-dingtalk/prometheus-webhook-dingtalk --ding.profile=ops_dingding=https://oapi.dingtalk.com/robot/send?access_token=1234567890 --web.enable-ui 启动，研究这个config花了点时间

制作一个prometheus-webhook-dingtalk的镜像

创建Dockerfile

FROM centos:centos7.4.1708
MAINTAINER PDABC Enterprise Container Images <jiaminxu@hz-health.cn>ADD  prometheus-webhook-dingtalk /etc/prometheus-webhook-dingtalk
RUN cp /usr/share/zoneinfo/Asia/Shanghai /etc/localtimeWORKDIR /etc/prometheus-webhook-dingtalk#公开端口
EXPOSE 8060USER root
#设置启动命令
ENTRYPOINT ["sh", "-c","/etc/prometheus-webhook-dingtalk/prometheus-webhook-dingtalk"]
# 想写死的话 也是可以的。
#ENTRYPOINT ["sh", "-c","/etc/prometheus-webhook-dingtalk/prometheus-webhook-dingtalk --ding.profile=ops_dingding=https://oapi.dingtalk.com/robot/send?access_token=1234567890   --web.enable-ui "]

编译并上床镜像

docker build -t registry.cn-shanghai.aliyuncs.com/yjk-datag/prometheus-webhook-dingtalk:v3  .
docker push registry.cn-shanghai.aliyuncs.com/yjk-datag/prometheus-webhook-dingtalk:v3

创建prometheus-webhook-dingtalk-cm.yaml

https://github.com/timonwong/prometheus-webhook-dingtalk

apiVersion: v1
kind: ConfigMap
metadata:name: prometheus-webhook-dingtalk-confignamespace: kube-ops
data:config.yml: |targets:# 这个是我用到的ops_dingding:url: https://oapi.dingtalk.com/robot/send?access_token=10bda98979ae2155b6822b699cde1841d4fbd8514c0441bbbb4485caddf3a388x #这是是钉钉机器人的webhookwebhook2:url: https://oapi.dingtalk.com/robot/send?access_token=10bda98979ae2155b6822b699cde1841d4fbd8514c0441bbbb4485caddf3a388x #这是是钉钉机器人的webhookwebhook_legacy:url: https://oapi.dingtalk.com/robot/send?access_token=10bda98979ae2155b6822b699cde1841d4fbd8514c0441bbbb4485caddf3a388x #这是是钉钉机器人的webhook# Customize template contentmessage:# Use legacy templatetitle: '{{ template "default.title" . }}'text: '{{ template "default.content" . }}'webhook_mention_all:url: https://oapi.dingtalk.com/robot/send?access_token=10bda98979ae2155b6822b699cde1841d4fbd8514c0441bbbb4485caddf3a388x #这是是钉钉机器人的webhookmention:all: truewebhook_mention_users:url: https://oapi.dingtalk.com/robot/send?access_token=10bda98979ae2155b6822b699cde1841d4fbd8514c0441bbbb4485caddf3a388x #这是是钉钉机器人的webhookmention:mobiles: ['15xxxxxxxx0']

创建prometheus-webhook-dingtalk-deploy.yaml

apiVersion: apps/v1
kind: Deployment
metadata:name: prometheus-webhook-dingtalknamespace: kube-opslabels:app: prometheus-webhook-dingtalk
spec:replicas: 1revisionHistoryLimit: 2selector:matchLabels:app: prometheus-webhook-dingtalkminReadySeconds: 0strategy:type: RollingUpdaterollingUpdate:maxUnavailable: 1maxSurge: 1template:metadata:labels:app: prometheus-webhook-dingtalkspec:affinity:         #亲和性的调度设置nodeAffinity:   #策略为节点亲和性requiredDuringSchedulingIgnoredDuringExecution:      #亲和性的硬策略nodeSelectorTerms:   #这里不再使用nodeselector，使用这个参数可以进行相对简单的逻辑运算- matchExpressions:     #匹配表达式- key: kubernetes.io/hostname      #具体匹配规则(可以通过kubectl get node --show-labels找到相应规则)operator: In  #不在，简单的来说就是不在k8s-04节点values:- prod-k8s-n006containers:- name: prometheus-webhook-dingtalkimage: registry.cn-shanghai.aliyuncs.com/yjk-datag/prometheus-webhook-dingtalk:v3args:# - "--web.enable-ui" # 这个web ui貌似在这里添加不生效。我也用不上 可以在构建镜像的时候加上试试- "----config.file=/etc/prometheus-webhook-dingtalk/config.yml"resources:limits:memory: "256Mi"cpu: "100m"readinessProbe:tcpSocket:port: 8060initialDelaySeconds: 30periodSeconds: 10livenessProbe:tcpSocket:port: 8060initialDelaySeconds: 60periodSeconds: 10ports:- containerPort: 8060protocol: TCPname: httpvolumeMounts:- name: config-volumemountPath: "/etc/prometheus-webhook-dingtalk/config.yml"subPath: config.ymlvolumes:- configMap:name: prometheus-webhook-dingtalk-configname: config-volume

创建prometheus-webhook-dingtalk-svc.yaml

---
apiVersion: v1
kind: Service
metadata:name: prometheus-webhook-dingtalknamespace: kube-opslabels:app: prometheus-webhook-dingtalk
spec:selector:app: prometheus-webhook-dingtalktype: NodePortports:- port: 80protocol: TCPtargetPort: 8060

创建prometheus-webhook-dingtalk-ingress.yaml

---
apiVersion: extensions/v1beta1
kind: Ingress
metadata:
# 通过添加下面的annotations 来开启白名单
# 关闭80强制跳转443 为ingress配置增加注解（annotations）：nginx.ingress.kubernetes.io/ssl-redirect: 'false' 就可以禁止http强制跳转至httpsannotations:#nginx.ingress.kubernetes.io/whitelist-source-range: "60.191.70.64/29, xx.xxx.0.0/16"nginx.ingress.kubernetes.io/ssl-redirect: 'false'name: prod-prometheus-webhook-dingtalknamespace: kube-ops
spec:rules:- host: prometheus-webhook-dingtalk.ihaozhuo.comhttp:paths:- path: /backend:serviceName: prometheus-webhook-dingtalkservicePort: 80

运行prometheus-webhook-dingtalk

kubectl apply -f ./

测试告警

修改prometheus-webhook-dingtalk的svc为headless

好处是可以不用域名访问可以使用prometheus-webhook-dingtalk-headless.kube-ops.svc.cluster.local:8060 来请求

创建prometheus-webhook-dingtalk-headless-svc.yaml

---
apiVersion: v1
kind: Service
metadata:name: prometheus-webhook-dingtalk-headlessnamespace: kube-opslabels:app: prometheus-webhook-dingtalk
spec:ports:- port: 80name: http# clusterIP 设置为 NoneclusterIP: Noneselector:app: prometheus-webhook-dingtalk

kubectl apply -f ./

应用之后通过其他容器测试