K8S 1.23 metrics-server及cadvisor 杂记

metrics-server

metrics-server是取代heapster，需要开启aggregate（默认开启），k8s建议生产环境开启tls认证以保障node安全。

aggregate可以查看top pod及node :kubectl -n kube-system top pod

metrics-server 还支持custom metric和external metric （比如keda），这些metric都可以用于HPA

metric-server 还增加了summary 查看，如以下命令：
kubectl get --raw /api/v1/nodes/ubuntu/proxy/stats/summary
ps：这里可以看到host所有网卡（含容器）的流量

Node Pressure

使用kubectl describe node可以看到内容

Conditions:Type                 Status  LastHeartbeatTime                 LastTransitionTime                Reason                       Message----                 ------  -----------------                 ------------------                ------                       -------NetworkUnavailable   False   Mon, 06 Jun 2022 08:08:15 +0800   Mon, 06 Jun 2022 08:08:15 +0800   FlannelIsUp                  Flannel is running on this nodeMemoryPressure       False   Mon, 06 Jun 2022 08:10:53 +0800   Sun, 05 Jun 2022 15:01:59 +0800   KubeletHasSufficientMemory   kubelet has sufficient memory availableDiskPressure         False   Mon, 06 Jun 2022 08:10:53 +0800   Sun, 05 Jun 2022 16:32:42 +0800   KubeletHasNoDiskPressure     kubelet has no disk pressurePIDPressure          False   Mon, 06 Jun 2022 08:10:53 +0800   Sun, 05 Jun 2022 15:01:59 +0800   KubeletHasSufficientPID      kubelet has sufficient PID availableReady                True    Mon, 06 Jun 2022 08:10:53 +0800   Sun, 05 Jun 2022 15:02:09 +0800   KubeletReady                 kubelet is posting ready status. AppArmor enabled

这些pressure可以参考：https://kubernetes.io/zh/docs/concepts/scheduling-eviction/node-pressure-eviction/
及以下说明：
Metric to alert on: Disk utilization
Like memory, disk space is a non-compressible resource, so if a kubelet detects low disk space on its root volume, it can cause problems with scheduling pods. If a node’s remaining disk capacity crosses a certain resource threshold, it will get flagged as under disk pressure. The following are the default resource thresholds for a node to come under disk pressure:
Disk pressure signal Threshold Description

imagefs.available 15% Available disk space for the imagefs
filesystem, used for images and container-writable layers
imagefs.inodesFree 5% Available index nodes for the imagefs
filesystem
nodefs.available 10% Available disk space for the root
filesystem
nodefs.inodesFree 5% Available index nodes for the root
filesystem

要特别注意，空间不管的时候不仅仅container会被驱逐，docker image也会被删除。

cadvisor 及其它API

cadvisor 默认已经集成在k8s可，可以直接使用：kubectl get --raw /api/v1/nodes/ubuntu/proxy/metrics/cadvisor 来看待container的指标
其它有用api：

kubectl get --raw /api/v1/nodes/ubuntu/proxy/metrics/resource
kubectl get --raw /api/v1/nodes/ubuntu/proxy/metrics/probes
kubectl get --raw /api/v1/nodes/ubuntu/|jq|more
image、container及磁盘、cpu、memory 压力指标（非具体值）
kubectl get --raw /api/v1/nodes/ubuntu/proxy/stats/summary 可以看到网络流量，cpu及内存使用
有关api里面的container cpu、memory、disk and network metrics 解读：
https://copyfuture.com/blogs-details/2020030216533745477avcct5emgy4bl
https://zhuanlan.zhihu.com/p/342823695
https://kubernetes.io/docs/reference/generated/kubernetes-api/v1.23/

一个应用场景：https://stackoverflow.com/questions/53328104/how-to-get-containers-disk-usage-in-kubernetes-without-docker-command