Kubernetes集群资源监控

概述

监控指标

一个好的系统，主要监控以下内容

集群监控
- 节点资源利用率
- 节点数
- 运行Pods
Pod监控
- 容器指标
- 应用程序【程序占用多少CPU、内存】

监控平台

使用普罗米修斯【prometheus】 + Grafana 搭建监控平台

prometheus【定时搜索被监控服务的状态】
- 开源的
- 监控、报警、数据库
- 以HTTP协议周期性抓取被监控组件状态
- 不需要复杂的集成过程，使用http接口接入即可
Grafana
- 开源的数据分析和可视化工具
- 支持多种数据源

部署prometheus

首先需要部署一个守护进程

vi node-exporter.yaml

复制一下内容

apiVersion: apps/v1
kind: DaemonSet
metadata:name: node-exporternamespace: kube-systemlabels:k8s-app: node-exporter
spec:selector:matchLabels:k8s-app: node-exportertemplate:metadata:labels:k8s-app: node-exporterspec:containers:- image: prom/node-exportername: node-exporterports:- containerPort: 9100protocol: TCPname: http
---
apiVersion: v1
kind: Service
metadata:labels:k8s-app: node-exportername: node-exporternamespace: kube-system
spec:ports:- name: httpport: 9100nodePort: 31672protocol: TCPtype: NodePortselector:k8s-app: node-exporter

执行命令

kubectl create -f node-exporter.yaml

然后通过yaml的方式部署prometheus

configmap.yaml
生成配置文件,定义一个configmap：存储一些配置文件【不加密】

vi configmap.yaml

apiVersion: v1
kind: ConfigMap
metadata:name: prometheus-confignamespace: kube-system
data:prometheus.yml: |global:scrape_interval:     15sevaluation_interval: 15sscrape_configs:- job_name: 'kubernetes-apiservers'kubernetes_sd_configs:- role: endpointsscheme: httpstls_config:ca_file: /var/run/secrets/kubernetes.io/serviceaccount/ca.crtbearer_token_file: /var/run/secrets/kubernetes.io/serviceaccount/tokenrelabel_configs:- source_labels: [__meta_kubernetes_namespace, __meta_kubernetes_service_name, __meta_kubernetes_endpoint_port_name]action: keepregex: default;kubernetes;https- job_name: 'kubernetes-nodes'kubernetes_sd_configs:- role: nodescheme: httpstls_config:ca_file: /var/run/secrets/kubernetes.io/serviceaccount/ca.crtbearer_token_file: /var/run/secrets/kubernetes.io/serviceaccount/tokenrelabel_configs:- action: labelmapregex: __meta_kubernetes_node_label_(.+)- target_label: __address__replacement: kubernetes.default.svc:443- source_labels: [__meta_kubernetes_node_name]regex: (.+)target_label: __metrics_path__replacement: /api/v1/nodes/${1}/proxy/metrics- job_name: 'kubernetes-cadvisor'kubernetes_sd_configs:- role: nodescheme: httpstls_config:ca_file: /var/run/secrets/kubernetes.io/serviceaccount/ca.crtbearer_token_file: /var/run/secrets/kubernetes.io/serviceaccount/tokenrelabel_configs:- action: labelmapregex: __meta_kubernetes_node_label_(.+)- target_label: __address__replacement: kubernetes.default.svc:443- source_labels: [__meta_kubernetes_node_name]regex: (.+)target_label: __metrics_path__replacement: /api/v1/nodes/${1}/proxy/metrics/cadvisor- job_name: 'kubernetes-service-endpoints'kubernetes_sd_configs:- role: endpointsrelabel_configs:- source_labels: [__meta_kubernetes_service_annotation_prometheus_io_scrape]action: keepregex: true- source_labels: [__meta_kubernetes_service_annotation_prometheus_io_scheme]action: replacetarget_label: __scheme__regex: (https?)- source_labels: [__meta_kubernetes_service_annotation_prometheus_io_path]action: replacetarget_label: __metrics_path__regex: (.+)- source_labels: [__address__, __meta_kubernetes_service_annotation_prometheus_io_port]action: replacetarget_label: __address__regex: ([^:]+)(?::\d+)?;(\d+)replacement: $1:$2- action: labelmapregex: __meta_kubernetes_service_label_(.+)- source_labels: [__meta_kubernetes_namespace]action: replacetarget_label: kubernetes_namespace- source_labels: [__meta_kubernetes_service_name]action: replacetarget_label: kubernetes_name- job_name: 'kubernetes-services'kubernetes_sd_configs:- role: servicemetrics_path: /probeparams:module: [http_2xx]relabel_configs:- source_labels: [__meta_kubernetes_service_annotation_prometheus_io_probe]action: keepregex: true- source_labels: [__address__]target_label: __param_target- target_label: __address__replacement: blackbox-exporter.example.com:9115- source_labels: [__param_target]target_label: instance- action: labelmapregex: __meta_kubernetes_service_label_(.+)- source_labels: [__meta_kubernetes_namespace]target_label: kubernetes_namespace- source_labels: [__meta_kubernetes_service_name]target_label: kubernetes_name- job_name: 'kubernetes-ingresses'kubernetes_sd_configs:- role: ingressrelabel_configs:- source_labels: [__meta_kubernetes_ingress_annotation_prometheus_io_probe]action: keepregex: true- source_labels: [__meta_kubernetes_ingress_scheme,__address__,__meta_kubernetes_ingress_path]regex: (.+);(.+);(.+)replacement: ${1}://${2}${3}target_label: __param_target- target_label: __address__replacement: blackbox-exporter.example.com:9115- source_labels: [__param_target]target_label: instance- action: labelmapregex: __meta_kubernetes_ingress_label_(.+)- source_labels: [__meta_kubernetes_namespace]target_label: kubernetes_namespace- source_labels: [__meta_kubernetes_ingress_name]target_label: kubernetes_name- job_name: 'kubernetes-pods'kubernetes_sd_configs:- role: podrelabel_configs:- source_labels: [__meta_kubernetes_pod_annotation_prometheus_io_scrape]action: keepregex: true- source_labels: [__meta_kubernetes_pod_annotation_prometheus_io_path]action: replacetarget_label: __metrics_path__regex: (.+)- source_labels: [__address__, __meta_kubernetes_pod_annotation_prometheus_io_port]action: replaceregex: ([^:]+)(?::\d+)?;(\d+)replacement: $1:$2target_label: __address__- action: labelmapregex: __meta_kubernetes_pod_label_(.+)- source_labels: [__meta_kubernetes_namespace]action: replacetarget_label: kubernetes_namespace- source_labels: [__meta_kubernetes_pod_name]action: replacetarget_label: kubernetes_pod_name

rbac-setup.yaml
分配一些角色的权限

vi rbac-setup.yaml

apiVersion: rbac.authorization.k8s.io/v1
kind: ClusterRole
metadata:name: prometheus
rules:
- apiGroups: [""]resources:- nodes- nodes/proxy- services- endpoints- podsverbs: ["get", "list", "watch"]
- apiGroups:- extensionsresources:- ingressesverbs: ["get", "list", "watch"]
- nonResourceURLs: ["/metrics"]verbs: ["get"]
---
apiVersion: v1
kind: ServiceAccount
metadata:name: prometheusnamespace: kube-system
---
apiVersion: rbac.authorization.k8s.io/v1
kind: ClusterRoleBinding
metadata:name: prometheus
roleRef:apiGroup: rbac.authorization.k8s.iokind: ClusterRolename: prometheus
subjects:
- kind: ServiceAccountname: prometheusnamespace: kube-system

prometheus.deploy.yml
prometheus.deploy.yml:部署一个deployment【包括端口号，资源限制】

vi prometheus.deploy.yml

apiVersion: apps/v1
kind: Deployment
metadata:labels:name: prometheus-deploymentname: prometheusnamespace: kube-system
spec:replicas: 1selector:matchLabels:app: prometheustemplate:metadata:labels:app: prometheusspec:containers:- image: prom/prometheus:v2.0.0name: prometheuscommand:- "/bin/prometheus"args:- "--config.file=/etc/prometheus/prometheus.yml"- "--storage.tsdb.path=/prometheus"- "--storage.tsdb.retention=24h"ports:- containerPort: 9090protocol: TCPvolumeMounts:- mountPath: "/prometheus"name: data- mountPath: "/etc/prometheus"name: config-volumeresources:requests:cpu: 100mmemory: 100Milimits:cpu: 500mmemory: 2500MiserviceAccountName: prometheus    volumes:- name: dataemptyDir: {}- name: config-volumeconfigMap:name: prometheus-config

prometheus.svc.yaml
prometheus.svc.yaml：对外暴露的端口

vi prometheus.svc.yaml

kind: Service
apiVersion: v1
metadata:labels:app: prometheusname: prometheusnamespace: kube-system
spec:type: NodePortports:- port: 9090targetPort: 9090nodePort: 30003selector:app: prometheus

然后分别部署

# 首先部署 rbac-setup.yaml
kubectl create -f rbac-setup.yaml
# 部署configmap
kubectl create -f configmap.yaml
# 部署deployment
kubectl create -f prometheus.deploy.yml
# 部署svc
kubectl create -f prometheus.svc.yaml

部署完成后，我们使用下面命令查看

kubectl get pods -n kube-system

在我们部署完成后，即可看到 prometheus 的 pod了，然后通过下面命令，能够看到对应的端口

kubectl get svc -n kube-system

通过这个，我们可以看到 prometheus 对外暴露的端口为 30003，访问页面即可对应的图形化界面

部署grafana

在上面我们部署完prometheus后，我们还需要来部署grafana
grafana-deploy.yaml

vi grafana-deploy.yaml


apiVersion: apps/v1
kind: Deployment
metadata:name: grafana-corenamespace: kube-systemlabels:app: grafanacomponent: core
spec:replicas: 1selector:matchLabels:app: grafanatemplate:metadata:labels:app: grafanacomponent: corespec:containers:- image: grafana/grafana:4.2.0name: grafana-coreimagePullPolicy: IfNotPresent# env:resources:# keep request = limit to keep this container in guaranteed classlimits:cpu: 100mmemory: 100Mirequests:cpu: 100mmemory: 100Mienv:# The following env variables set up basic auth twith the default admin user and admin password.- name: GF_AUTH_BASIC_ENABLEDvalue: "true"- name: GF_AUTH_ANONYMOUS_ENABLEDvalue: "false"# - name: GF_AUTH_ANONYMOUS_ORG_ROLE#   value: Admin# does not really work, because of template variables in exported dashboards:# - name: GF_DASHBOARDS_JSON_ENABLED#   value: "true"readinessProbe:httpGet:path: /loginport: 3000# initialDelaySeconds: 30# timeoutSeconds: 1volumeMounts:- name: grafana-persistent-storagemountPath: /varvolumes:- name: grafana-persistent-storageemptyDir: {}

grafana-svc.yaml

vi grafana-svc.yaml

apiVersion: v1
kind: Service
metadata:name: grafananamespace: kube-systemlabels:app: grafanacomponent: core
spec:type: NodePortports:- port: 3000selector:app: grafanacomponent: core

grafana-ing.yaml

vi grafana-ing.yaml

apiVersion: extensions/v1beta1
kind: Ingress
metadata:name: grafananamespace: kube-system
spec:rules:- host: k8s.grafanahttp:paths:- path: /backend:serviceName: grafanaservicePort: 3000

分别部署

# 创建deployment
kubectl create -f grafana-deploy.yaml
# 创建svc
kubectl create -f grafana-svc.yaml
# 创建 ing
kubectl create -f grafana-ing.yaml

执行命令kubectl create -f grafana-ing.yaml的时候报版本错误
error: unable to recognize "grafana-ing.yaml": no matches for kind "Ingress" in version "extensions/v1beta1"
但是这个版本就是这个extensions/v1beta1，我查看pod,发现也创建好了，

我们能看到，我们的grafana正在运行

配置数据源

下面我们需要开始打开 Grafana，然后配置数据源，导入数据显示模板

kubectl get svc -n kube-system

我们可以通过 ip + 30718访问我们的 grafana 图形化页面

然后输入账号和密码：admin admin

进入后，我们就需要配置 prometheus 的数据源

设置显示数据的模板

选择Dashboard，导入我们的模板

然后输入 315 号模板

然后选择 prometheus数据源 mydb，导入即可

导入后的效果如下所示

普罗米修斯监控k8s集群相关推荐

普罗米修斯监控服务之-监控携带metric接口的服务
第一部分:课前小菜: 例子: 分布式:将多台Linux主机组合在一起,做成像一台主机一样来使用微服务:将一台Linux主机拆分成多台Linux使用淘宝:搜索页.商品详情页.订单页.支付页.物流.评 ...
普罗米修斯监控_接近完美的监控系统—普罗米修斯
普罗米修斯(Prometheus)是一个SoundCloud公司开源的监控系统.当年,由于SoundCloud公司生产了太多的服务,传统的监控已经无法满足监控需求,于是他们在2012年决定着手开发新的 ...
一篇文章搞定Prometheus普罗米修斯监控系统的部署
Prometheus普罗米修斯监控系统一.普罗米修斯监控概述 1.1 什么是普罗米修斯监控二.时间序列数据 2.1 什么是序列数据 2.2 时间序列数据特点 2.3 普罗米修斯特征三.普罗米修斯 ...
普罗米修斯监控linux,Prometheus（普罗米修斯）搭建监控
Prometheus(普罗米修斯) 实验环境:(各个主机,ip,所需服务) docker01 docker02 docker03 1.10 1.20 1.30 NodeEXporter NodeEXp ...
阿尔萨斯监控平台普罗米修斯监控平台对服务器资源的监控
一.arthas-阿尔萨斯监控平台内存监控: 1.打开RDMS,选择应用实例,在'快捷方式'下拉选项中选择arthas 2.打开监控界面,输入:dashboard -i 5000 备注:-i 刷新实 ...
普罗米修斯监控docker 服务
环境 192.168.254.127 (监控服务器) 192.168.254.128 (被监控机器) 首先在监控服务器安装: 1.安装Node Exporter 来收集硬件信息所有节点运行以下命令安 ...
普罗米修斯监控服务之PromQL语法和常用计算
PromQL QL顾名思义,Query language即查询语言.Prometheus作为强大的开源监控系统,最大的依赖便是PromQL.是监控数据个性化查询.展示的基础.所以要掌握Promethe ...
普罗米修斯监控_完美的分布式监控系统——普罗米修斯
DevOps工程师或SRE工程师,可能都知道Prometheus普罗米修斯.Prometheus于2012年由SoundCloud创建,目前已经已发展为最热门的分布式监控系统.Prometheus完全 ...
普罗米修斯监控搭建大全
linux系统监控监控集群架构图一:单节点监控 1:前往官方网站下载相关软件包普罗米修斯(prometheus)官方网站:https://prometheus.io/ 2:将下载的软件包上传到需 ...

普罗米修斯监控k8s集群