kubernetes基于k8s-prometheus-adapter实现 pod 的hpa(自动伸缩)
2024-05-19 23:21:02
系统
[root@k8s-build ~]# uname -a
Linux k8s-build 3.10.0-862.11.6.el7.x86_64 #1 SMP Tue Aug 14 21:49:04 UTC 2018 x86_64 x86_64 x86_64 GNU/Linuxkubernetes的版本
[root@master-47-35 custom-metrics-api]# kubectl version
Client Version: version.Info{Major:"1", Minor:"11+", GitVersion:"v1.11.0-1+feebbe2b16f546", GitCommit:"feebbe2b16f546ac289f7afdae40f1721e0656fe", GitTreeState:"clean", BuildDate:"2018-08-23T02:06:55Z", GoVersion:"go1.10.2", Compiler:"gc", Platform:"linux/amd64"}
Server Version: version.Info{Major:"1", Minor:"11+", GitVersion:"v1.11.0-1+feebbe2b16f546", GitCommit:"feebbe2b16f546ac289f7afdae40f1721e0656fe", GitTreeState:"clean", BuildDate:"2018-08-23T02:06:55Z", GoVersion:"go1.10.2", Compiler:"gc", Platform:"linux/amd64"}获取代码k8s-prometheus-adapter
yum install git -y
cd /root/gopath/src/
git clone https://github.com/DirectXMan12/k8s-prometheus-adapter.git切换分支到v0.2.1
[root@k8s-build k8s-prometheus-adapter]# git checkout v0.2.1
Note: checking out 'v0.2.1'.You are in 'detached HEAD' state. You can look around, make experimental
changes and commit them, and you can discard any commits you make in this
state without impacting any branches by performing another checkout.If you want to create a new branch to retain commits you create, you may
do so (now or later) by using -b with the checkout command again. Example:git checkout -b new_branch_nameHEAD 目前位于 3fc2dbd... Explicitly build on tags in Travis
[root@k8s-build k8s-prometheus-adapter]# git status
# 头指针分离于 v0.2.1
无文件要提交，干净的工作区构建k8s-prometheus-adapter镜像
[root@k8s-build k8s-prometheus-adapter]# make docker-build
cp deploy/Dockerfile /tmp/tmp.FbNVQfpHCg
cd /tmp/tmp.FbNVQfpHCg && sed -i "s|BASEIMAGE|busybox|g" Dockerfile
docker run -it -v /tmp/tmp.FbNVQfpHCg:/build -v /root/gopath/src/k8s-prometheus-adapter:/go/src/github.com/directxman12/k8s-prometheus-adapter -e GOARCH=amd64 golang:1.10 /bin/bash -c "\CGO_ENABLED=0 go build -tags netgo -o /build/adapter github.com/directxman12/k8s-prometheus-adapter/cmd/adapter"
Unable to find image 'golang:1.10' locally
Trying to pull repository docker.io/library/golang ...
1.10: Pulling from docker.io/library/golang
05d1a5232b46: Pull complete
5cee356eda6b: Pull complete
89d3385f0fd3: Pull complete
80ae6b477848: Pull complete
94ebfeaaddf3: Pull complete
e132030a369d: Pull complete
c67c5750c788: Pull complete
Digest: sha256:322a64a16553516761f8e71eb94136140031e6a7fcfe794e579de14a7da21ee7
Status: Downloaded newer image for docker.io/golang:1.10
docker build -t directxman12/k8s-prometheus-adapter-amd64:latest /tmp/tmp.FbNVQfpHCg
Sending build context to Docker daemon 55.61 MB
Step 1/4 : FROM busybox
Trying to pull repository docker.io/library/busybox ...
latest: Pulling from docker.io/library/busybox
8c5a7da1afbc: Already exists
Digest: sha256:032ddd66f10483436e8a252e69fdfd20d0164e9953585c10d378183a0924db34
Status: Downloaded newer image for docker.io/busybox:latest---> e1ddd7948a1c
Step 2/4 : COPY adapter /---> a39120e24ddb
Removing intermediate container 6358600308cb
Step 3/4 : USER 1001:1001---> Running in 9755e39bed35---> 718d846e413b
Removing intermediate container 9755e39bed35
Step 4/4 : ENTRYPOINT /adapter---> Running in 68554cb44316---> cb309c2eb2f4
Removing intermediate container 68554cb44316
Successfully built cb309c2eb2f4
rm -rf /tmp/tmp.FbNVQfpHCg
查看k8s-prometheus-adapter镜像
[root@k8s-build k8s-prometheus-adapter]# docker images | grep k8s-prometheus-adapter-amd64
directxman12/k8s-prometheus-adapter-amd64 latest cb309c2eb2f4 About a minute ago 56.8 MB修改镜像tag,指定到私有的镜像仓库
[root@k8s-build k8s-prometheus-adapter]# docker tag cb309c2eb2f4 harbor.enncloud.cn/enncloud/k8s-prometheus-adapter-amd64:v0.2.1
[root@k8s-build k8s-prometheus-adapter]# docker push harbor.enncloud.cn/enncloud/k8s-prometheus-adapter-amd64:v0.2.1
The push refers to a repository [harbor.enncloud.cn/enncloud/k8s-prometheus-adapter-amd64]
d4821deae84a: Pushed
f9d9e4e6e2f0: Pushed
v0.2.1: digest: sha256:32ead17195963566630d9a8286773ddff92f9ccc126cf9a6e2cdcf686d163aef size: 739生成名字叫cm-adapter-serving-certs的secret,值为serving.crt和 serving.key，实际上就是证书
证书生成请参考requestheader-authentication
这里有生成证书的脚本,供参考，这个证书生成脚本需要gcc golang环境，安装gcc go
yum install go gcc -y脚本内容
[root@k8s-build certs]# cat gencerts.sh
#!/usr/bin/env bash
# Detect if we are on mac or should use GNU base64 options
case $(uname) inDarwin)b64_opts='-b=0';; *)b64_opts='--wrap=0'
esacgo get -v -u github.com/cloudflare/cfssl/cmd/...
export PATH=$GOPATH/bin:$PATH
export PURPOSE=metrics
openssl req -x509 -sha256 -new -nodes -days 365 -newkey rsa:2048 -keyout ${PURPOSE}-ca.key -out ${PURPOSE}-ca.crt -subj "/CN=ca"
echo '{"signing":{"default":{"expiry":"43800h","usages":["signing","key encipherment","'${PURPOSE}'"]}}}' > "${PURPOSE}-ca-config.json"export SERVICE_NAME=custom-metrics-apiserver
export ALT_NAMES='"custom-metrics-apiserver.custom-metrics","custom-metrics-apiserver.custom-metrics.svc"'
echo "{\"CN\":\"${SERVICE_NAME}\", \"hosts\": [${ALT_NAMES}], \"key\": {\"algo\": \"rsa\",\"size\": 2048}}" | \cfssl gencert -ca=metrics-ca.crt -ca-key=metrics-ca.key -config=metrics-ca-config.json - | cfssljson -bare apiservercat <<-EOF > cm-adapter-serving-certs.yaml
apiVersion: v1
kind: Secret
metadata:name: cm-adapter-serving-certsnamespace: custom-metrics
data:serving.crt: $(base64 ${b64_opts} < apiserver.pem)serving.key: $(base64 ${b64_opts} < apiserver-key.pem)
EOF生成证书
[root@k8s-build certs]# ./gencerts.sh
github.com/cloudflare/cfssl (download)
Generating a 2048 bit RSA private key
.+++
......+++
writing new private key to 'metrics-ca.key'
-----
2018/08/25 20:36:58 [INFO] generate received request
2018/08/25 20:36:58 [INFO] received CSR
2018/08/25 20:36:58 [INFO] generating key: rsa-2048
2018/08/25 20:36:59 [INFO] encoded CSR
2018/08/25 20:36:59 [INFO] signed certificate with serial number 573231977150840198182111747492918974864906787988
查看脚本生成的内容
[root@k8s-build certs]# ls -l
总用量 32
-rw-r--r-- 1 root root 1066 8月 25 20:36 apiserver.csr
-rw------- 1 root root 1679 8月 25 20:36 apiserver-key.pem
-rw-r--r-- 1 root root 1277 8月 25 20:36 apiserver.pem
-rw-r--r-- 1 root root 4053 8月 25 20:36 cm-adapter-serving-certs.yaml
-rwxr-xr-x 1 root root 1198 8月 25 20:36 gencerts.sh
-rw-r--r-- 1 root root 94 8月 25 20:36 metrics-ca-config.json
-rw-r--r-- 1 root root 1074 8月 25 20:36 metrics-ca.crt
-rw-r--r-- 1 root root 1704 8月 25 20:36 metrics-ca.key查看证书内容
[root@master-47-35 custom-metrics]# cat cm-adapter-serving-certs.yaml
apiVersion: v1
data:serving.crt: LS0tLS1CRUdJTiBDRVJUSUZJQ0FURS0tLS0tCk1JSURHRENDQWdDZ0F3SUJBZ0lCQVRBTkJna3Foa2lHOXcwQkFRc0ZBREFmTVIwd0d3WURWUVFEREJSc2IyTmgKYkdodmMzUkFNVFV4TXpjME5qZzBNREFlRncweE56RXlNakF3TlRFME1EQmFGdzB4T0RFeU1qQXdOVEUwTURCYQpNQjh4SFRBYkJnTlZCQU1NRkd4dlkyRnNhRzl6ZEVBeE5URXpOelEyT0RRd01JSUJJakFOQmdrcWhraUc5dzBCCkFRRUZBQU9DQVE4QU1JSUJDZ0tDQVFFQW9mZE4zd1RqVldlZFo2YlBRN3ZrMTNPeUkwMEVwOGcvYzlTTCtneWMKQ3RHK3hYSkhrVlN4eW1zVlhsNnF4T2tWTWkrL2JqRFVqc1VsSnY5a2JjZkhHemYwYmxiQWdocVhvQ20ycjkvRwp0TVNlczlTL0NER1FNQUhPbkxsemdjVkxVQ1kyUVVXdS9HZDVwQU96dkJSUVBWczNBNUNZcDMzS0grTFVTTExOCnVNMjcvSkVnclUwczdBQ29ZaDk3KzQrSU1qb1pmbHRiK0JOQzRaU3BvemhQZ0EwQWhXRXl0SXkrRDdVWWdIR2cKNm8ybWJpTkV1QUNrQmk5TmFITm8zdGdieVN3OGQzYU9vd3JQQ3VlNEV0UTNIZGlNMHB5WUtuSUhsTFhyNjA1Sgpra25ENWs0OWFoaktQUFJhTlpjNlZpZlEvTzNiekVnNUwvVld6K2lneFJrSyt3SURBUUFCbzE4d1hUQU9CZ05WCkhROEJBZjhFQkFNQ0FxUXdFd1lEVlIwbEJBd3dDZ1lJS3dZQkJRVUhBd0V3RHdZRFZSMFRBUUgvQkFVd0F3RUIKL3pBbEJnTlZIUkVFSGpBY2dnbHNiMk5oYkdodmMzU0NDV3h2WTJGc2FHOXpkSWNFZndBQUFUQU5CZ2txaGtpRwo5dzBCQVFzRkFBT0NBUUVBWERVV3ozVW0zVFg4VFJCVmQ1VVF5Z1ZhKzloMlRtVWJMU0JpdlBURDVRTWFBZUJzCkRYQU0vbTZrWXZIcFlTTi9MOWhraWd1QzhTazdlVFZqeWZabjlsYmNoSVV1NytoYys1UThJTDZVb0VaQ2cxVHQKNDNlYzBhZlc1YUU2bm5lejVhMWs3UDFhY0xhL1ZFUzVWM25wUmg2YUl4a0xVVHNMaUFsdGRGcURpeTF4aVlORgorL2pWMGFHRXhWWWJMbFVQMkIxaTlQODZneGFjNkJRTCtSQTNZZjFISWl5ZkliemMrbjV4Z2U1T0NIbC9XVjV1CkIzTHdZdkh1TGdpTDNZbyswN0gycWZ3MlNIUmZxdlJHdlpmS2tnWmcrTVQrZElYR3ZTTE0ybUhhdElxQnNXZEwKS1VFZnorREdYbzBIam1LTmhuWThRM0kwN1IxY29ta05KUFMxL3c9PQotLS0tLUVORCBDRVJUSUZJQ0FURS0tLS0tCg== serving.key: LS0tLS1CRUdJTiBSU0EgUFJJVkFURSBLRVktLS0tLQpNSUlFb2dJQkFBS0NBUUVBb2ZkTjN3VGpWV2VkWjZiUFE3dmsxM095STAwRXA4Zy9jOVNMK2d5Y0N0Ryt4WEpICmtWU3h5bXNWWGw2cXhPa1ZNaSsvYmpEVWpzVWxKdjlrYmNmSEd6ZjBibGJBZ2hxWG9DbTJyOS9HdE1TZXM5Uy8KQ0RHUU1BSE9uTGx6Z2NWTFVDWTJRVVd1L0dkNXBBT3p2QlJRUFZzM0E1Q1lwMzNLSCtMVVNMTE51TTI3L0pFZwpyVTBzN0FDb1loOTcrNCtJTWpvWmZsdGIrQk5DNFpTcG96aFBnQTBBaFdFeXRJeStEN1VZZ0hHZzZvMm1iaU5FCnVBQ2tCaTlOYUhObzN0Z2J5U3c4ZDNhT293clBDdWU0RXRRM0hkaU0wcHlZS25JSGxMWHI2MDVKa2tuRDVrNDkKYWhqS1BQUmFOWmM2VmlmUS9PM2J6RWc1TC9WV3oraWd4UmtLK3dJREFRQUJBb0lCQURUci82L05neEtXNXFIRgpxWmV5UDFYKzRjZ1RGR0JXV0JWYTc5bm1ZTjVRSm91Umliejg5WkwwWm1rTmViTVpqZWwySEpDK3Q5Vm1xWHNTCm5CZ2EzUDRSekhaNU5qSE15cHAweHRLMHhkS3ZKRVFTVWRFMWtLMS9DYW5BRnZQaUtYZW81TFZ6TlhnWHYzRk8KSEY1cUxNVy9sanc5OVFOVllNUUdjK0xGT2Q4V0pRRm9KZVh3bFBKdC9vK3NWUCtHK3dtYmp3VUNtc3ZqTFc5NAoxbS9XeWtwVlMrMCt3QVRJTGJCeTJqUUVtK3hwZk1IT0xhU01Hem9oUnEySHFvY1d2NE10OWI4bmFPNURyQ3dCCmZIQnIxd3VtL3duWGVESUFQay8yVVYxSllzdDQveVFjSENUS1V0Qzh3WFd6V3RabEE3dFNmM1huNUM0MXRhakoKaTIxMzNuRUNnWUVBd1BUOVRUS3R4NzZHYVovZE5WWStLY1ExYnplQlA5NlI0SFp0M1d4NllUbEJZZ2oxQWg4dQpZVGJCYmhYdDExcTQ0b0h4eXk0eThCVC9xWU55dWhIUGJjMExCYW4zamtwRVZEZ3VaempMSHc3YUUwTlVkNFNDCmFzd0FjTGlnMEdiTU1RMVJnZndHdVpOc3ZXZ2wrVXZWdVB3SVRKUzBUYkdQeVVGTnpZSmlsUjBDZ1lFQTF1STMKaFB1VjBoOGUyZEVHT3g5L0FXb1hCU3lhajNVZ2xTV2VnNTNBOFJ1OU9POTl5OGxHSXZxUDdsVXpub01xZzlwMwpEdXF3TFdCSnQwYU11SG1yZVF6RzhiME9ieG1UTmwreTdUaTIwOU9nUTBaZ0ZhSFhtbng2UStaWlovNFd0UXlMCmxzUUxPMDNqcjUwb3dQV3FJSnRmdEVXckZsYXBCd01IdFBxY0hQY0NnWUJVT1J0dlhOV1p3RUh2VE1aRGg3ZHUKaDZSRFBWai9SZWtxcnNiWnhZWE5kaTVoMmtNSGdCREFnaVFBSnpZL3IrdlBncVRGNlE1MjJOT1hiRGJoblBDTApudHBXeUkrejNocmhIL3l4TllzRHJYbm11cGJkN0VyNHpyN0tkT3hNeXZJSjN5UmF0bVZCRDJhQmJqcFZESkxuClFhdGxzZXJabHMyS0xTVUd0UWpPeVFLQmdDTUVpbmVwYUNqdDJXRTFwd1pMZzY3L1V2RU02OE1sRzBJeTl6b0wKb3oybmY1YWZzS2hkRElqMnZDM2l4SjY0VVYxYlpMdVJrMVJSdExHaFZaMWFkbzhyZmVBM0p2c0Q5L0dNZHVRaApzZW1qT3lLN2ZUMEpPdEY2SjVLWkgwakxKM2dwcXpYT2pPWUZNMjJTOXVrN0g1Wmd1NS90R294eFl1bzVuQy9xCkJ6YlJBb0dBVHUwc2Rnb2dxL2RHdDFIVjNmZnhyVGhVazBMdVZxM0x4Ni9mN0tXWGN4b21qUWlQc0RZUTVkU3IKb21Kb2wvVUxWUnRQbGJtcmZKN1VpcHBlNXh1eUFNQUVocldIeG15SlR2TWZQSks3UGs4eHgrcUlSUGd0YklobwpWcjN5ZFoyYnRka2h4OXo5dGY4WXJ5RkxqNnNIWTY4V1Z2T3hETkcwYTZPYUhWWFJJMUE9Ci0tLS0tRU5EIFJTQSBQUklWQVRFIEtFWS0tLS0tCg==
kind: Secret
metadata:name: cm-adapter-serving-certsnamespace: custom-metrics创建k8s-prometheus-adapter configmap,可以使用这个方式
kubectl create secret generic cm-adapter-serving-certs --from-file=serving.crt=apiserver.crt --from-file=serving.key=apiserver.key -n custom-metrics也可以使用这种方式
kubectl create -f cm-adapter-serving-certs.yaml创建custom-metrics namespace
kubectl create namespace custom-metrics部署
修改镜像custom-metrics-apiserver-deployment.yaml的镜像
cd /opt/ssl/custom-metrics/k8s-prometheus-adapter/deploy/manifests修改custom-metrics-apiserver-deployment.yamlimage: directxman12/k8s-prometheus-adapter-amd64to image: harbor.enncloud.cn/enncloud/k8s-prometheus-adapter-amd64:v0.2.1- --prometheus-url=http://prometheus.prom.svc:9090/to - --prometheus-url=http://prometheus-monitor.kube-system:9090/- --config=/default-config.yaml to - --config=/etc/adapter/config.yaml
添加这个
- --requestheader-client-ca-file=/var/run/secrets/kubernetes.io/serviceaccount/ca.crt修改之后
apiVersion: apps/v1
kind: Deployment
metadata:labels:app: custom-metrics-apiservername: custom-metrics-apiservernamespace: custom-metrics
spec:replicas: 1selector:matchLabels:app: custom-metrics-apiservertemplate:metadata:labels:app: custom-metrics-apiservername: custom-metrics-apiserverspec:serviceAccountName: custom-metrics-apiservercontainers:- name: custom-metrics-apiserverimage: directxman12/k8s-prometheus-adapter-amd64args:- /adapter- --secure-port=6443- --tls-cert-file=/var/run/serving-cert/serving.crt- --tls-private-key-file=/var/run/serving-cert/serving.key- --logtostderr=true- --prometheus-url=http://prometheus-monitor.kube-system:9090/- --metrics-relist-interval=1m- --requestheader-client-ca-file=/var/run/secrets/kubernetes.io/serviceaccount/ca.crt- --v=10- --config=/etc/adapter/config.yamlports:- containerPort: 6443volumeMounts:- mountPath: /var/run/serving-certname: volume-serving-certreadOnly: true- mountPath: /etc/adapter/name: configreadOnly: true- mountPath: /tmpname: tmp-volvolumes:- name: volume-serving-certsecret:secretName: cm-adapter-serving-certs- name: configconfigMap:name: adapter-config- name: tmp-volemptyDir: {}添加kube-apiserver的启动参数
--requestheader-client-ca-file=/etc/kubernetes/ssl/ca.pem \--requestheader-allowed-names=aggregator \--requestheader-extra-headers-prefix=X-Remote-Extra- \--requestheader-group-headers=X-Remote-Group \--requestheader-username-headers=X-Remote-User \--proxy-client-cert-file=/etc/kubernetes/ssl/aggregator-proxy.pem \--proxy-client-key-file=/etc/kubernetes/ssl/aggregator-proxy-key.pem \--enable-aggregator-routing=true \添加kube-controller-manager的启动参数
"--horizontal-pod-autoscaler-use-rest-clients","--horizontal-pod-autoscaler-sync-period=60s",cd /opt/ssl/custom-metrics/k8s-prometheus-adapter/deploy/manifests
kubectl create -f .查看部署结果
[root@master-47-35 custom-metrics]# kubectl get all -n custom-metrics
NAME READY STATUS RESTARTS AGE
pod/custom-metrics-apiserver-5d8dd96b5-8s9rq 1/1 Running 0 16hNAME TYPE CLUSTER-IP EXTERNAL-IP PORT(S) AGE
service/custom-metrics-apiserver ClusterIP 10.254.40.72 <none> 443/TCP 16hNAME DESIRED CURRENT UP-TO-DATE AVAILABLE AGE
deployment.apps/custom-metrics-apiserver 1 1 1 1 16hNAME DESIRED CURRENT READY AGE
replicaset.apps/custom-metrics-apiserver-5d8dd96b5 1 1 1 16h查看api-versions
[root@master-47-35 k8s-prometheus-adapter]# kubectl api-versions
admissionregistration.k8s.io/v1beta1
apiextensions.k8s.io/v1beta1
apiregistration.k8s.io/v1
apiregistration.k8s.io/v1beta1
apps/v1
apps/v1beta1
apps/v1beta2
authentication.k8s.io/v1
authentication.k8s.io/v1beta1
authorization.k8s.io/v1
authorization.k8s.io/v1beta1
autoscaling/v1
autoscaling/v2beta1
batch/v1
batch/v1beta1
certificates.k8s.io/v1beta1
custom.metrics.k8s.io/v1beta1
events.k8s.io/v1beta1
extensions/v1beta1
monitoring.coreos.com/v1
networking.k8s.io/v1
policy/v1beta1
rbac.authorization.k8s.io/v1
rbac.authorization.k8s.io/v1beta1
scheduling.k8s.io/v1beta1
storage.k8s.io/v1
storage.k8s.io/v1beta1
v1
确认这个custom.metrics.k8s.io/v1beta1 api存在
root@master-47-35 ~]# kubectl get --raw "/apis/custom.metrics.k8s.io/v1beta1"
{"kind":"APIResourceList","apiVersion":"v1","groupVersion":"custom.metrics.k8s.io/v1beta1","resources":[{"name":"pods/fs_usage_bytes","singularName":"","namespaced":true,"kind":"MetricValueList","verbs":["get"]},{"name":"pods/spec_memory_reservation_limit_bytes","singularName":"","namespaced":true,"kind":"MetricValueList","verbs":["get"]},{"name":"pods/fs_io_time","singularName":"","namespaced":true,"kind":"MetricValueList","verbs":["get"]},{"name":"namespaces/fs_reads_bytes","singularName":"","namespaced":false,"kind":"MetricValueList","verbs":["get"]},{"name":"namespaces/fs_writes","singularName":"","namespaced":false,"kind":"MetricValueList","verbs":["get"]},{"name":"pods/network_tcp_usage","singularName":"","namespaced":true,"kind":"MetricValueList","verbs":["get"]},{"name":"pods/fs_sector_writes","singularName":"","namespaced":true,"kind":"MetricValueList","verbs":["get"]},{"name":"namespaces/fs_writes_bytes","singularName":"","namespaced":false,"kind":"MetricValueList","verbs":["get"]},{"name":"namespaces/spec_memory_limit_bytes","singularName":"","namespaced":false,"kind":"MetricValueList","verbs":["get"]},{"name":"namespaces/start_time_seconds","singularName":"","namespaced":false,"kind":"MetricValueList","verbs":["get"]},{"name":"namespaces/spec_cpu_shares","singularName":"","namespaced":false,"kind":"MetricValueList","verbs":["get"]},{"name":"namespaces/spec_memory_reservation_limit_bytes","singularName":"","namespaced":false,"kind":"MetricValueList","verbs":["get"]},{"name":"pods/fs_read","singularName":"","namespaced":true,"kind":"MetricValueList","verbs":["get"]},{"name":"pods/cpu_cfs_throttled_periods","singularName":"","namespaced":true,"kind":"MetricValueList","verbs":["get"]},{"name":"pods/cpu_load_average_10s","singularName":"","namespaced":true,"kind":"MetricValueList","verbs":["get"]},{"name":"namespaces/memory_swap","singularName":"","namespaced":false,"kind":"MetricValueList","verbs":["get"]},{"name":"pods/memory_failures","singularName":"","namespaced":true,"kind":"MetricValueList","verbs":["get"]},{"name":"pods/start_time_seconds","singularName":"","namespaced":true,"kind":"MetricValueList","verbs":["get"]},{"name":"pods/memory_max_usage_bytes","singularName":"","namespaced":true,"kind":"MetricValueList","verbs":["get"]},{"name":"pods/spec_cpu_period","singularName":"","namespaced":true,"kind":"MetricValueList","verbs":["get"]},{"name":"pods/cpu_schedstat_run","singularName":"","namespaced":true,"kind":"MetricValueList","verbs":["get"]},{"name":"pods/fs_io_time_weighted","singularName":"","namespaced":true,"kind":"MetricValueList","verbs":["get"]},{"name":"namespaces/network_tcp_usage","singularName":"","namespaced":false,"kind":"MetricValueList","verbs":["get"]},{"name":"namespaces/fs_io_current","singularName":"","namespaced":false,"kind":"MetricValueList","verbs":["get"]},{"name":"namespaces/spec_cpu_quota","singularName":"","namespaced":false,"kind":"MetricValueList","verbs":["get"]},{"name":"pods/cpu_cfs_throttled","singularName":"","namespaced":true,"kind":"MetricValueList","verbs":["get"]},{"name":"pods/cpu_cfs_periods","singularName":"","namespaced":true,"kind":"MetricValueList","verbs":["get"]},{"name":"pods/fs_io_current","singularName":"","namespaced":true,"kind":"MetricValueList","verbs":["get"]},{"name":"namespaces/memory_usage_bytes","singularName":"","namespaced":false,"kind":"MetricValueList","verbs":["get"]},{"name":"pods/fs_reads_merged","singularName":"","namespaced":true,"kind":"MetricValueList","verbs":["get"]},{"name":"namespaces/fs_reads","singularName":"","namespaced":false,"kind":"MetricValueList","verbs":["get"]},{"name":"namespaces/fs_writes_merged","singularName":"","namespaced":false,"kind":"MetricValueList","verbs":["get"]},{"name":"namespaces/memory_failures","singularName":"","namespaced":false,"kind":"MetricValueList","verbs":["get"]},{"name":"namespaces/cpu_user","singularName":"","namespaced":false,"kind":"MetricValueList","verbs":["get"]},{"name":"pods/fs_write","singularName":"","namespaced":true,"kind":"MetricValueList","verbs":["get"]},{"name":"namespaces/cpu_cfs_throttled_periods","singularName":"","namespaced":false,"kind":"MetricValueList","verbs":["get"]},{"name":"pods/cpu_schedstat_run_periods","singularName":"","namespaced":true,"kind":"MetricValueList","verbs":["get"]},{"name":"namespaces/cpu_load_average_10s","singularName":"","namespaced":false,"kind":"MetricValueList","verbs":["get"]},{"name":"pods/last_seen","singularName":"","namespaced":true,"kind":"MetricValueList","verbs":["get"]},{"name":"namespaces/spec_cpu_period","singularName":"","namespaced":false,"kind":"MetricValueList","verbs":["get"]},{"name":"pods/cpu_system","singularName":"","namespaced":true,"kind":"MetricValueList","verbs":["get"]},{"name":"namespaces/fs_read","singularName":"","namespaced":false,"kind":"MetricValueList","verbs":["get"]},{"name":"namespaces/cpu_schedstat_run_periods","singularName":"","namespaced":false,"kind":"MetricValueList","verbs":["get"]},{"name":"pods/fs_sector_reads","singularName":"","namespaced":true,"kind":"MetricValueList","verbs":["get"]},{"name":"namespaces/memory_rss","singularName":"","namespaced":false,"kind":"MetricValueList","verbs":["get"]},{"name":"pods/memory_rss","singularName":"","namespaced":true,"kind":"MetricValueList","verbs":["get"]},{"name":"pods/memory_working_set_bytes","singularName":"","namespaced":true,"kind":"MetricValueList","verbs":["get"]},{"name":"pods/fs_inodes","singularName":"","namespaced":true,"kind":"MetricValueList","verbs":["get"]},{"name":"pods/fs_reads_bytes","singularName":"","namespaced":true,"kind":"MetricValueList","verbs":["get"]},{"name":"pods/fs_writes","singularName":"","namespaced":true,"kind":"MetricValueList","verbs":["get"]},{"name":"namespaces/memory_cache","singularName":"","namespaced":false,"kind":"MetricValueList","verbs":["get"]},{"name":"pods/spec_cpu_shares","singularName":"","namespaced":true,"kind":"MetricValueList","verbs":["get"]},{"name":"namespaces/fs_reads_merged","singularName":"","namespaced":false,"kind":"MetricValueList","verbs":["get"]},{"name":"namespaces/fs_sector_reads","singularName":"","namespaced":false,"kind":"MetricValueList","verbs":["get"]},{"name":"namespaces/fs_usage_bytes","singularName":"","namespaced":false,"kind":"MetricValueList","verbs":["get"]},{"name":"pods/memory_failcnt","singularName":"","namespaced":true,"kind":"MetricValueList","verbs":["get"]},{"name":"pods/spec_memory_swap_limit_bytes","singularName":"","namespaced":true,"kind":"MetricValueList","verbs":["get"]},{"name":"namespaces/spec_memory_swap_limit_bytes","singularName":"","namespaced":false,"kind":"MetricValueList","verbs":["get"]},{"name":"namespaces/cpu_schedstat_run","singularName":"","namespaced":false,"kind":"MetricValueList","verbs":["get"]},{"name":"pods/cpu_usage","singularName":"","namespaced":true,"kind":"MetricValueList","verbs":["get"]},{"name":"namespaces/last_seen","singularName":"","namespaced":false,"kind":"MetricValueList","verbs":["get"]},{"name":"pods/memory_cache","singularName":"","namespaced":true,"kind":"MetricValueList","verbs":["get"]},{"name":"pods/fs_writes_bytes","singularName":"","namespaced":true,"kind":"MetricValueList","verbs":["get"]},{"name":"namespaces/fs_limit_bytes","singularName":"","namespaced":false,"kind":"MetricValueList","verbs":["get"]},{"name":"namespaces/memory_max_usage_bytes","singularName":"","namespaced":false,"kind":"MetricValueList","verbs":["get"]},{"name":"namespaces/memory_working_set_bytes","singularName":"","namespaced":false,"kind":"MetricValueList","verbs":["get"]},{"name":"namespaces/cpu_schedstat_runqueue","singularName":"","namespaced":false,"kind":"MetricValueList","verbs":["get"]},{"name":"namespaces/cpu_system","singularName":"","namespaced":false,"kind":"MetricValueList","verbs":["get"]},{"name":"namespaces/cpu_cfs_periods","singularName":"","namespaced":false,"kind":"MetricValueList","verbs":["get"]},{"name":"namespaces/fs_sector_writes","singularName":"","namespaced":false,"kind":"MetricValueList","verbs":["get"]},{"name":"namespaces/cpu_usage","singularName":"","namespaced":false,"kind":"MetricValueList","verbs":["get"]},{"name":"pods/fs_writes_merged","singularName":"","namespaced":true,"kind":"MetricValueList","verbs":["get"]},{"name":"namespaces/network_udp_usage","singularName":"","namespaced":false,"kind":"MetricValueList","verbs":["get"]},{"name":"pods/memory_swap","singularName":"","namespaced":true,"kind":"MetricValueList","verbs":["get"]},{"name":"namespaces/cpu_cfs_throttled","singularName":"","namespaced":false,"kind":"MetricValueList","verbs":["get"]},{"name":"pods/cpu_user","singularName":"","namespaced":true,"kind":"MetricValueList","verbs":["get"]},{"name":"namespaces/fs_inodes","singularName":"","namespaced":false,"kind":"MetricValueList","verbs":["get"]},{"name":"pods/network_udp_usage","singularName":"","namespaced":true,"kind":"MetricValueList","verbs":["get"]},{"name":"pods/spec_memory_limit_bytes","singularName":"","namespaced":true,"kind":"MetricValueList","verbs":["get"]},{"name":"pods/tasks_state","singularName":"","namespaced":true,"kind":"MetricValueList","verbs":["get"]},{"name":"namespaces/fs_io_time","singularName":"","namespaced":false,"kind":"MetricValueList","verbs":["get"]},{"name":"namespaces/fs_write","singularName":"","namespaced":false,"kind":"MetricValueList","verbs":["get"]},{"name":"pods/memory_usage_bytes","singularName":"","namespaced":true,"kind":"MetricValueList","verbs":["get"]},{"name":"pods/spec_cpu_quota","singularName":"","namespaced":true,"kind":"MetricValueList","verbs":["get"]},{"name":"pods/fs_inodes_free","singularName":"","namespaced":true,"kind":"MetricValueList","verbs":["get"]},{"name":"pods/fs_limit_bytes","singularName":"","namespaced":true,"kind":"MetricValueList","verbs":["get"]},{"name":"namespaces/memory_failcnt","singularName":"","namespaced":false,"kind":"MetricValueList","verbs":["get"]},{"name":"namespaces/tasks_state","singularName":"","namespaced":false,"kind":"MetricValueList","verbs":["get"]},{"name":"pods/cpu_schedstat_runqueue","singularName":"","namespaced":true,"kind":"MetricValueList","verbs":["get"]},{"name":"namespaces/fs_io_time_weighted","singularName":"","namespaced":false,"kind":"MetricValueList","verbs":["get"]},{"name":"pods/fs_reads","singularName":"","namespaced":true,"kind":"MetricValueList","verbs":["get"]},{"name":"namespaces/fs_inodes_free","singularName":"","namespaced":false,"kind":"MetricValueList","verbs":["get"]}]}接下来就是验证了
获取CPU cpu_usage
[root@master-47-35 ~]# kubectl get --raw "/apis/custom.metrics.k8s.io/v1beta1/namespaces/default/pods/*/cpu_usage"
{"kind":"MetricValueList","apiVersion":"custom.metrics.k8s.io/v1beta1","metadata":{"selfLink":"/apis/custom.metrics.k8s.io/v1beta1/namespaces/default/pods/%2A/cpu_usage"},"items":[{"describedObject":{"kind":"Pod","namespace":"default","name":"nginx-dm-fff68d674-qg2gj","apiVersion":"/__internal"},"metricName":"cpu_usage","timestamp":"2018-09-12T14:49:30Z","value":"0"},{"describedObject":{"kind":"Pod","namespace":"default","name":"nginx-dm-fff68d674-t4xf2","apiVersion":"/__internal"},"metricName":"cpu_usage","timestamp":"2018-09-12T14:49:30Z","value":"0"},{"describedObject":{"kind":"Pod","namespace":"default","name":"nginx-dm-fff68d674-w26gb","apiVersion":"/__internal"},"metricName":"cpu_usage","timestamp":"2018-09-12T14:49:30Z","value":"0"}]}获取内存 memory_usage_bytes
[root@master-47-35 ~]# kubectl get --raw "/apis/custom.metrics.k8s.io/v1beta1/namespaces/default/pods/*/memory_usage_bytes"
{"kind":"MetricValueList","apiVersion":"custom.metrics.k8s.io/v1beta1","metadata":{"selfLink":"/apis/custom.metrics.k8s.io/v1beta1/namespaces/default/pods/%2A/memory_usage_bytes"},"items":[{"describedObject":{"kind":"Pod","namespace":"default","name":"nginx-dm-fff68d674-qg2gj","apiVersion":"/__internal"},"metricName":"memory_usage_bytes","timestamp":"2018-09-12T14:52:46Z","value":"3547136"},{"describedObject":{"kind":"Pod","namespace":"default","name":"nginx-dm-fff68d674-t4xf2","apiVersion":"/__internal"},"metricName":"memory_usage_bytes","timestamp":"2018-09-12T14:52:46Z","value":"4435968"},{"describedObject":{"kind":"Pod","namespace":"default","name":"nginx-dm-fff68d674-w26gb","apiVersion":"/__internal"},"metricName":"memory_usage_bytes","timestamp":"2018-09-12T14:52:46Z","value":"6172672"}]}
查看v1beta1.custom.metrics.k8s.io是否存在
[root@master-47-35 ~]# kubectl get apiservice v1beta1.custom.metrics.k8s.io -o yaml
apiVersion: apiregistration.k8s.io/v1
kind: APIService
metadata:creationTimestamp: 2018-09-12T08:57:08ZdeletionGracePeriodSeconds: 0deletionTimestamp: 2018-09-12T10:34:21Zfinalizers:- foregroundDeletionname: v1beta1.custom.metrics.k8s.ioresourceVersion: "3189364"selfLink: /apis/apiregistration.k8s.io/v1/apiservices/v1beta1.custom.metrics.k8s.iouid: d470a606-b669-11e8-aec0-5254e98192ae
spec:group: custom.metrics.k8s.iogroupPriorityMinimum: 100insecureSkipTLSVerify: trueservice:name: custom-metrics-apiservernamespace: custom-metricsversion: v1beta1versionPriority: 100
status:conditions:- lastTransitionTime: 2018-09-12T10:14:40Zmessage: all checks passedreason: Passedstatus: "True"type: Available
基于内存
apiVersion: autoscaling/v1
kind: HorizontalPodAutoscaler
metadata:annotations:autoscaling.alpha.kubernetes.io/conditions: '[{"type":"AbleToScale","status":"True","lastTransitionTime":"2018-09-13T02:23:13Z","reason":"SucceededGetScale","message":"theHPA controller was able to get the target''s current scale"},{"type":"ScalingActive","status":"False","lastTransitionTime":"2018-09-13T02:24:13Z","reason":"ScalingDisabled","message":"scalingis disabled since the replica count of the target is zero"},{"type":"ScalingLimited","status":"True","lastTransitionTime":"2018-09-13T02:03:10Z","reason":"TooManyReplicas","message":"thedesired replica count is more than the maximum replica count"}]'autoscaling.alpha.kubernetes.io/metrics: '[{"type":"Pods","pods":{"metricName":"memory_usage_bytes","targetAverageValue":"1M"}}]'creationTimestamp: 2018-09-13T02:01:10Zname: example2namespace: demoresourceVersion: "3299536"selfLink: /apis/autoscaling/v1/namespaces/demo/horizontalpodautoscalers/example2uid: e2be2af9-b6f8-11e8-bf72-5254e98192ae
spec:maxReplicas: 2minReplicas: 1scaleTargetRef:apiVersion: apps/v1beta1kind: Deploymentname: example2
status:currentReplicas: 0desiredReplicas: 0lastScaleTime: 2018-09-13T02:17:13Z
基于cpu cpu_usage的使用
apiVersion: autoscaling/v1
kind: HorizontalPodAutoscaler
metadata:annotations:autoscaling.alpha.kubernetes.io/conditions: '[{"type":"AbleToScale","status":"True","lastTransitionTime":"2018-09-13T02:32:34Z","reason":"ReadyForNewScale","message":"thelast scale time was sufficiently old as to warrant a new scale"},{"type":"ScalingActive","status":"True","lastTransitionTime":"2018-09-13T02:23:33Z","reason":"ValidMetricFound","message":"theHPA was able to successfully calculate a replica count from pods metric cpu_usage"},{"type":"ScalingLimited","status":"True","lastTransitionTime":"2018-09-13T02:23:33Z","reason":"TooManyReplicas","message":"thedesired replica count is more than the maximum replica count"}]'autoscaling.alpha.kubernetes.io/current-metrics: '[{"type":"Pods","pods":{"metricName":"cpu_usage","currentAverageValue":"322m"}}]'autoscaling.alpha.kubernetes.io/metrics: '[{"type":"Pods","pods":{"metricName":"cpu_usage","targetAverageValue":"30m"}}]'creationTimestamp: 2018-09-13T02:21:33Zname: examplenamespace: demoresourceVersion: "3301437"selfLink: /apis/autoscaling/v1/namespaces/demo/horizontalpodautoscalers/exampleuid: bbc859b6-b6fb-11e8-bf72-5254e98192ae
spec:maxReplicas: 4minReplicas: 2scaleTargetRef:apiVersion: apps/v1beta1kind: Deploymentname: example
status:currentReplicas: 4desiredReplicas: 4lastScaleTime: 2018-09-13T02:26:34Z不过有个问题issues，但是不影响使用
end
参考
部署方式
https://github.com/DirectXMan12/k8s-prometheus-adapter/blob/v0.2.1/docs/walkthrough.md
kubernetes基于k8s-prometheus-adapter实现 pod 的hpa(自动伸缩)相关推荐
- 运维实操——kubernetes（十九）k8s中部署Prometheus、监控nginx、HPA自动伸缩
k8s中部署Prometheus.监控nginx.HPA自动伸缩 1.什么是Prometheus? 2.k8s中部署Prometheus监控 3.prometheus监控nginx 4.基于prome ...
- 基于k8s+Prometheus+Alertmanager+Grafana构建企业级监控告警系统
特别提醒: 下文实验需要的yaml文件和压缩包可加我微信获取 微信: luckylucky421302 1.1 深度解读Prometheus 1.1.1 什么是Prometheus? Prometh ...
- 基于 Knative 低成本部署在线应用，灵活自动伸缩
作者 | 冬岛 阿里巴巴高级技术专家 导读:Serverless 如今已是万众期待未来可期的状态,但一个系统到底具备怎样的能力才能更好地支撑 Serverless 应用?随着 Kubernetes 和 ...
- 【网址收藏】k8s HPA自动伸缩异常：failed to get cpu utilization: missing request
参考链接: Kubernetes:HPA 详解-基于 CPU.内存和自定义指标自动扩缩容
- Prometheus 基于k8s服务发现通过Cadvisor监控Kubernetes
Prometheus服务发现 Prometheus添加被监控端支持两种方式: • 静态配置:手动配置 • 服务发现:动态发现需要监控的Target实例 支持服务发现的来源 • azure_sd_con ...
- Kubernetes（K8s）优势究竟是什么？
Kubernetes(K8s)原先来自于谷歌内部的小组提案,在2000年后,谷歌先后开发了Borg.Omega和Kubernetes这三个容器管理系统. 第一个在谷歌被开发出来的统一的容器管理系统,称 ...
- Istio 自动注入 sidecar 不成功及k8s安装istio后pod无法创建解决方案
环境 Kubernetes v1.15.6 源码安装 Istio v1.2.5 Helm 安装 Istio v1.2.5 Helm 安装 Istio Helm安装 问题 安装完后,做官方 bookin ...
- K8S有状态静态Pod经典示例
研究K8S有一阵子了,这个东西很好用,但是也有很多坑,个人觉得很多地方还有待改进,K8S的静态Pod是一个利器,但也必须对它有全面的了解才能运作. 今天给大家示例一个从头到尾自己研究出来的项目,需要一 ...
- K8S水平伸缩器 - 自动伸缩微服务实例数量
作者:justmine 头条号:大数据达摩院 微信公众号:大数据处理系统 创作不易,在满足创作共用版权协议的基础上可以转载,但请以超链接形式注明出处. 为了方便大家阅读,可以关注头条号或微信公众号,后 ...
最新文章
- form表单提交前进行ajax或js验证，校验不通过不提交
- Wind River Helix系统及物联网解决方案 简化企业IOT部署
- linux驱动 打印变量,linux驱动 内核函数 变量 宏定义
- java基础学完就直接学ssm_java基础学习笔记3（SSM基础）
- linux增加分区swap,linux中Centos7增加swap分区详解
- 为什么下拉框拉不下来_为什么体重降不下来？4个饮食方法降低热量摄入，让体重降下来...
- 解决：ubuntu18.xx 更新nvidia驱动后，重新启动后黑屏
- 自己动手写CPU(5)简单算术操作指令实现_1
- The 2019 ICPC Asia Shanghai Regional Contest
- 【前端面试题】关于一些js的一些面试题（金融行业），我和面试官扯了三个小时
- STM32之SPI原理
- 自注意力机制不一定是灵丹妙药？？？基于MLP的sMLPNet！MSRA出品
- Unity中的layer和tag
- （08）System Verilog 队列详解
- Qemu-6.1.0多热补丁管理
- classcastexception异常_让你为之颤抖的Java常见的异常exception
- loadrunner 一个诡异问题
- Arcgis软件应用（四）网络分析
- linux清空临时文件命令,Linux系统下临时文件TMP清理 ( tmpwatch 命令)
- java根据年份获取年份下的所有月份