k8s部署Kube Prometheus（Prometheus Operator）

摘要

本文通过Prometheus-operator框架一键化安装prometheus、alertmanage、granfana，并配置企业微信api以及告警推送，搭建 prometheus 的前提环境是已有 k8s 环境并可用。

Prometheus Operator是什么

Prometheus Operator是运行在Kubernetes之上的监控和告警工具。部署时不用创建和修改prometheus的配置文件，所有的操作通过创建prometheus自己的资源对象来实现。对于监控配置的修改可以做到实时生效。

Prometheus Operator的自定义资源(CustomResourceDefinitions CRD)

Prometheus: 定义Prometheus监控系统的部署。
ServiceMonitor：监控一组service。该service需要暴露监控数据，供prometheus收集。
PodMonitor：监控一组pod。
PrometheusRule：Prometheus的规则文件。包含告警规则。
AlertManager：定义告警管理器的部署。

下载yaml配置文件

mkdir -p /home/prometheus
cd /home/prometheus
wget https://github.com/prometheus-operator/kube-prometheus/archive/refs/tags/v0.8.0.tar.gz
tar xf v0.8.0.tar.gz

prometheus

1. 修改配置

为prometheus配置pvc

 root@K8s-master:~/test/prometheus# vim prometheus-prometheus.yaml
...serviceMonitorSelector: {}version: 2.26.0storage:volumeClaimTemplate:spec:storageClassName: managed-nfs-storage   #注意修改为对应的scresources:requests:storage: 5Gi    #适当扩大

配置grafana持久化存储

 root@K8s-master:~/test/prometheus# vim grafana-deployment.yaml
...- name: grafana-storagepersistentVolumeClaim:claimName: grafana-data     # storyclass 注意替换
...

添加grafana的pvc

root@K8s-master:~/test/prometheus# vim grafana-pvc.yaml
kind: PersistentVolumeClaim
apiVersion: v1
metadata:name: grafana-datanamespace: monitoring
spec:accessModes:- ReadWriteManyresources:requests:storage: 5Gi    # 扩大容量storageClassName: aws-gps   # 对应sc

配置prometheus的svc为外部可访问

root@K8s-master:~/test/prometheus# vim prometheus-service.yaml
...
spec:type: LoadBalancerexternalIPs:- 0.0.0.0     #prometheus访问的IP地址自行指定ports:- name: webport: 9090targetPort: webnodePort: 9090selector:app: prometheusapp.kubernetes.io/component: prometheusapp.kubernetes.io/name: prometheusapp.kubernetes.io/part-of: kube-prometheusprometheus: k8ssessionAffinity: None

配置grafana的svc为外部可访问

root@K8s-master:~/test/prometheus# vim grafana-service.yaml
...
spec:type: LoadBalancerexternalIPs:- 0.0.0.0       # 外网访问地址ports:- name: httpport: 3000targetPort: httpnodePort: 3000selector:app.kubernetes.io/component: grafanaapp.kubernetes.io/name: grafanaapp.kubernetes.io/part-of: kube-prometheu

2. 一键化部署：

进入setup目录，并执行 kubectl apply -f .
回到setup的上级目录下，执行 kubectl apply -f .

至此prometheus、granfana、alertmanagey已经完成安装并可访问，接下来进行服务自动发现配置

3. 服务发现

创建prometheus-additional.yaml

 - job_name: 'kubernetes-service-endpoints'kubernetes_sd_configs:- role: endpointsrelabel_configs:- source_labels: [__meta_kubernetes_service_annotation_prometheus_io_scrape]action: keepregex: true- source_labels: [__meta_kubernetes_service_annotation_prometheus_io_scheme]action: replacetarget_label: __scheme__regex: (https?)- source_labels: [__meta_kubernetes_service_annotation_prometheus_io_path]action: replacetarget_label: __metrics_path__regex: (.+)- source_labels: [__address__, __meta_kubernetes_service_annotation_prometheus_io_port]action: replacetarget_label: __address__regex: ([^:]+)(?::\d+)?;(\d+)replacement: $1:$2- action: labelmapregex: __meta_kubernetes_service_label_(.+)- source_labels: [__meta_kubernetes_namespace]action: replacetarget_label: kubernetes_namespace- source_labels: [__meta_kubernetes_service_name]action: replacetarget_label: kubernetes_name

创建secret对象

kubectl create secret generic additional-configs --from-file=prometheus-additional.yaml -n monitoring

修改 prometheus-prometheus.yaml 添加两行

name: additional-configskey: prometheus-additional.yaml

添加之后：
apiVersion: monitoring.coreos.com/v1
kind: Prometheus
metadata:labels:prometheus: k8sname: k8snamespace: monitoring
spec:alerting:alertmanagers:- name: alertmanager-mainnamespace: monitoringport: webbaseImage: quay.io/prometheus/prometheusnodeSelector:beta.kubernetes.io/os: linuxreplicas: 2secrets:- etcd-certsresources:requests:memory: 400MiruleSelector:matchLabels:prometheus: k8srole: alert-rulessecurityContext:fsGroup: 2000runAsNonRoot: truerunAsUser: 1000additionalScrapeConfigs:name: additional-configskey: prometheus-additional.yamlserviceAccountName: prometheus-k8sserviceMonitorNamespaceSelector: {}serviceMonitorSelector: {}version: v2.5.0

执行 kubectl apply -f prometheus-prometheus.yaml
修改 prometheus-clusterRole.yaml 文件，添加权限

apiVersion: rbac.authorization.k8s.io/v1
kind: ClusterRole
metadata:name: prometheus-k8s
rules:- apiGroups:- ""resources:- nodes- services- endpoints- pods- nodes/proxyverbs:- get- list- watch- apiGroups:- ""resources:- configmaps- nodes/metricsverbs:- get- nonResourceURLs:- /metricsverbs:- get

执行 kubectl apply -f prometheus-clusterRole.yaml
以上配置完成服务自动发现，如需添加 target ，只需在相关service添加：

annotations:prometheus.io/scrape: "true"

告警阈值 rule_file.yaml 模板

直接复制以下模板，修改一些关键字即可使用

apiVersion: monitoring.coreos.com/v1
kind: PrometheusRule
metadata:labels:app.kubernetes.io/component: alert-routerapp.kubernetes.io/name: alertmanagerapp.kubernetes.io/part-of: kube-prometheusapp.kubernetes.io/version: 0.21.0prometheus: k8srole: alert-rulesname: <规则名称>namespace: monitorin
spec:groups:- name: <组名>rules:- alert: <监控的项目名称>expr: <promql语句>  # 例如 total_error > 0for: <第一次匹配到规则后的持续时间，到达持续时间之后又匹配到规则则触发告警>  # 例如 5mlabels:severity: pageannotations:summary: <主题>description: <描述>

alertmanage 集成企业微信配置

配置 alertmanager-secret.yaml 文件

apiVersion: v1
kind: Secret
metadata:labels:alertmanager: mainapp.kubernetes.io/component: alert-routerapp.kubernetes.io/name: alertmanagerapp.kubernetes.io/part-of: kube-prometheusapp.kubernetes.io/version: 0.21.0name: alertmanager-mainnamespace: monitoring
stringData:alertmanager.yaml: |-"global":"resolve_timeout": "5m""wechat_api_url": "https://qyapi.weixin.qq.com/cgi-bin/"    #直接使用"wechat_api_secret": ""   #凭证"wechat_api_corp_id": ""  #企业id"receivers":- "name": "wechat""wechat_configs":- "send_resolved": true"to_party": ""    #接收消息的群id"agent_id": """corp_id": """route":"group_by": ["alertname"]"group_interval": "30m""repeat_interval": "30m""group_wait": "3s""receiver": "wechat""repeat_interval": "1m"type: Opaque

官方地址

Kube Prometheus项目地址
https://github.com/coreos/kube-prometheus

项目的Helm安装包地址
https://github.com/helm/charts/blob/master/stable/prometheus-operator

Prometheus官网地址
https://prometheus.io/

Prometheus Operator项目地址
https://github.com/coreos/prometheus-operator/