Prometheus Operator开始

文章目录

Prometheus Operator
- Operator模式
- Prometheus CRD
部署Prometheus Operator
- 1. 下载Prometheus Operator
- 2. images
- 3. 创建被监控应用
- 4. prometheus pod RBAC
- 5. Prometheus 对象
- 6. 观察Target
卸载prometheus operator

来自 Prometheus Operator Get Started实验记录

Prometheus Operator

https://prometheus-operator.dev/docs/prologue/quick-start/

Operator模式

Operator模式是用户声明一个状态，一个控制器循环检测，直到目标达到状态。在kubernetes原生生态里，控制器
只能控制Deployment,Service etc 原生资源对象，而Core OS公司通过扩展kubernetes使得用户可以自定义控制器来控制kubernetes CRD（自定义资源对象）的状态。

Prometheus Operator的能够根据CRD对象自动生成Prometheus的scrape配置。

因此不需要每次修改prometheus.yaml中的配置，杀死Pod，进行重启使得配置生效。只需要像原来修改deployment.yaml,重新apply一下，控制器不断循环操作使得配置生效。

Prometheus CRD

通过在原生Service对象上添加Label，CRD ServiceMonitor会通过Label选择Service. Operator会根据 ServiceMonitor配置生成Prometheus的prometheus.yaml配置。

CRD ServiceMonitor对象也会有label，划分不同的监控组，Prometheus对象通过Label选择ServiceMonitor对象。

部署Prometheus Operator

1. 下载Prometheus Operator

git clone https://github.com/prometheus-operator/kube-prometheus.git

然后使用预编译好的yaml文件。

Operator使用jsonnet，一种json模板框架来渲染生成yaml文件，生成的文件在manifest文件夹

# Create the namespace and CRDs, and then wait for them to be availble before creating the remaining resources
kubectl create -f manifests/setup
until kubectl get servicemonitors --all-namespaces ; do date; sleep 1; echo ""; done
kubectl create -f manifests/

2. images

部分镜像被墙

image list

docker pull bitnami/kube-state-metrics:2.3.0
docker pull selina5288/prometheus-adapter:v0.9.1

tag list

docker tag bitnami/kube-state-metrics:2.3.0 k8s.gcr.io/kube-state-metrics/kube-state-metrics:v2.3.0
docker tag selina5288/prometheus-adapter:v0.9.1 k8s.gcr.io/prometheus-adapter/prometheus-adapter:v0.9.1

3. 创建被监控应用

apiVersion: apps/v1
kind: Deployment
metadata:name: example-app
spec:replicas: 3selector:matchLabels:app: example-apptemplate:metadata:labels:app: example-appspec:containers:- name: example-appimage: fabxc/instrumented_appports:- name: webcontainerPort: 8080
---
kind: Service
apiVersion: v1
metadata:name: example-applabels:app: example-app
spec:selector:app: example-appports:- name: webport: 8080
---
apiVersion: monitoring.coreos.com/v1
kind: ServiceMonitor
metadata:name: example-applabels:team: frontend
spec:selector:matchLabels:app: example-appendpoints:- port: web

4. prometheus pod RBAC

为了可以访问到default 命名空间的权限。

apiVersion: v1
kind: ServiceAccount
metadata:name: prometheus
---
apiVersion: rbac.authorization.k8s.io/v1
kind: ClusterRole
metadata:name: prometheus
rules:
- apiGroups: [""]resources:- nodes- nodes/metrics- services- endpoints- podsverbs: ["get", "list", "watch"]
- apiGroups: [""]resources:- configmapsverbs: ["get"]
- apiGroups:- networking.k8s.ioresources:- ingressesverbs: ["get", "list", "watch"]
- nonResourceURLs: ["/metrics"]verbs: ["get"]
---
apiVersion: rbac.authorization.k8s.io/v1
kind: ClusterRoleBinding
metadata:name: prometheus
roleRef:apiGroup: rbac.authorization.k8s.iokind: ClusterRolename: prometheus
subjects:
- kind: ServiceAccountname: prometheusnamespace: default

5. Prometheus 对象

一个Prometheus对象通过serviceMonitorSelector定义包含哪个ServiceMonitors。

上面的ServiceMonitors 指定了标签team: frontend，所以在Prometheus对象中选择这个标签。

apiVersion: monitoring.coreos.com/v1
kind: Prometheus
metadata:name: prometheuslabels:prometheus: prometheus
spec:replicas: 2serviceAccountName: prometheusserviceMonitorSelector:matchLabels:team: frontend  # 在Prometheus对象中选择 team: frontend 标签alerting:alertmanagers:- namespace: defaultname: alertmanagerport: web

6. 观察Target

为了确定prometheus收集到了example应用的指标，查看prometheus的target列表：

使用grafana观察效果：

# example app每秒CPU分片占比
rate(process_cpu_seconds_total{namespace="default",job="example-app"}[1m])*100

卸载prometheus operator

kubectl delete --ignore-not-found=true -f manifests/ -f manifests/setup