Kubernetes K8S之kube-prometheus概述与部署
Kubernetes K8S之kube-prometheus概述与部署
主机配置规划
服务器名称(hostname) | 系统版本 | 配置 | 内网IP | 外网IP(模拟) |
---|---|---|---|---|
k8s-master | CentOS7.7 | 2C/4G/20G | 172.16.1.110 | 10.0.0.110 |
k8s-node01 | CentOS7.7 | 2C/4G/20G | 172.16.1.111 | 10.0.0.111 |
k8s-node02 | CentOS7.7 | 2C/4G/20G | 172.16.1.112 | 10.0.0.112 |
prometheus概述
Prometheus是一个开源的系统监控和警报工具包,自2012成立以来,许多公司和组织采用了Prometheus。它现在是一个独立的开源项目,并独立于任何公司维护。在2016年,Prometheus加入云计算基金会作为Kubernetes之后的第二托管项目。
Prometheus性能也足够支撑上万台规模的集群。
Prometheus的关键特性
- 多维度数据模型
- 灵活的查询语言
- 不依赖于分布式存储;单服务器节点是自治的
- 通过基于HTTP的pull方式采集时序数据
- 可以通过中间网关进行时序列数据推送
- 通过服务发现或者静态配置来发现目标服务对象
- 支持多种多样的图表和界面展示,比如Grafana等
架构图
基本原理
Prometheus的基本原理是通过HTTP协议周期性抓取被监控组件的状态,任意组件只要提供对应的HTTP接口就可以接入监控。不需要任何SDK或者其他的集成过程。
这样做非常适合做虚拟化环境监控系统,比如VM、Docker、Kubernetes等。输出被监控组件信息的HTTP接口被叫做exporter 。目前互联网公司常用的组件大部分都有exporter可以直接使用,比如Varnish、Haproxy、Nginx、MySQL、Linux系统信息(包括磁盘、内存、CPU、网络等等)。
Prometheus三大套件
- Server 主要负责数据采集和存储,提供PromQL查询语言的支持。
- Alertmanager 警告管理器,用来进行报警。
- Push Gateway 支持临时性Job主动推送指标的中间网关。
服务过程
- Prometheus Daemon负责定时去目标上抓取metrics(指标)数据,每个抓取目标需要暴露一个http服务的接口给它定时抓取。Prometheus支持通过配置文件、文本文件、Zookeeper、Consul、DNS SRV Lookup等方式指定抓取目标。Prometheus采用PULL的方式进行监控,即服务器可以直接通过目标PULL数据或者间接地通过中间网关来Push数据。
- Prometheus在本地存储抓取的所有数据,并通过一定规则进行清理和整理数据,并把得到的结果存储到新的时间序列中。
- Prometheus通过PromQL和其他API可视化地展示收集的数据。Prometheus支持很多方式的图表可视化,例如Grafana、自带的Promdash以及自身提供的模版引擎等等。Prometheus还提供HTTP API的查询方式,自定义所需要的输出。
- PushGateway支持Client主动推送metrics到PushGateway,而Prometheus只是定时去Gateway上抓取数据。
- Alertmanager是独立于Prometheus的一个组件,可以支持Prometheus的查询语句,提供十分灵活的报警方式。
kube-prometheus部署
kube-prometheus的GitHub地址:
https://github.com/coreos/kube-prometheus/
本次我们选择release-0.2版本,而不是其他版本。
kube-prometheus下载与配置修改
下载
[root@k8s-master prometheus]# pwd
/root/k8s_practice/prometheus
[root@k8s-master prometheus]#
[root@k8s-master prometheus]# wget https://github.com/coreos/kube-prometheus/archive/v0.2.0.tar.gz
[root@k8s-master prometheus]# tar xf v0.2.0.tar.gz
[root@k8s-master prometheus]# ll
total 432
drwxrwxr-x 10 root root 4096 Sep 13 2019 kube-prometheus-0.2.0
-rw-r--r-- 1 root root 200048 Jul 19 11:41 v0.2.0.tar.gz
配置修改
# 当前所在目录
[root@k8s-master manifests]# pwd
/root/k8s_practice/prometheus/kube-prometheus-0.2.0/manifests
[root@k8s-master manifests]#
# 配置修改1
[root@k8s-master manifests]# vim grafana-service.yaml
apiVersion: v1
kind: Service
metadata:labels:app: grafananame: grafananamespace: monitoring
spec:type: NodePort # 添加内容ports:- name: httpport: 3000targetPort: httpnodePort: 30100 # 添加内容selector:app: grafana
[root@k8s-master manifests]#
# 配置修改2
[root@k8s-master manifests]# vim prometheus-service.yaml
apiVersion: v1
kind: Service
metadata:labels:prometheus: k8sname: prometheus-k8snamespace: monitoring
spec:type: NodePort # 添加内容ports:- name: webport: 9090targetPort: webnodePort: 30200 # 添加内容selector:app: prometheusprometheus: k8ssessionAffinity: ClientIP
[root@k8s-master manifests]#
# 配置修改3
[root@k8s-master manifests]# vim alertmanager-service.yaml
apiVersion: v1
kind: Service
metadata:labels:alertmanager: mainname: alertmanager-mainnamespace: monitoring
spec:type: NodePort # 添加内容ports:- name: webport: 9093targetPort: webnodePort: 30300 # 添加内容selector:alertmanager: mainapp: alertmanagersessionAffinity: ClientIP
[root@k8s-master manifests]#
# 配置修改4
[root@k8s-master manifests]# vim grafana-deployment.yaml
# 将apps/v1beta2 改为 apps/v1
apiVersion: apps/v1
kind: Deployment
metadata:labels:app: grafananame: grafananamespace: monitoring
spec:replicas: 1selector:
………………
kube-prometheus镜像版本查看与下载
由于镜像都在国外,因此经常会下载失败。为了快速下载镜像,这里我们下载国内的镜像,然后tag为配置文件中的国外镜像名即可。
查看kube-prometheus的镜像信息
# 当前工作目录
[root@k8s-master manifests]# pwd
/root/k8s_practice/prometheus/kube-prometheus-0.2.0/manifests
[root@k8s-master manifests]#
# 所有镜像信息如下
[root@k8s-master manifests]# grep -riE 'quay.io|k8s.gcr|grafana/' *
0prometheus-operator-deployment.yaml: - --config-reloader-image=quay.io/coreos/configmap-reload:v0.0.1
0prometheus-operator-deployment.yaml: - --prometheus-config-reloader=quay.io/coreos/prometheus-config-reloader:v0.33.0
0prometheus-operator-deployment.yaml: image: quay.io/coreos/prometheus-operator:v0.33.0
alertmanager-alertmanager.yaml: baseImage: quay.io/prometheus/alertmanager
grafana-deployment.yaml: - image: grafana/grafana:6.2.2
kube-state-metrics-deployment.yaml: image: quay.io/coreos/kube-rbac-proxy:v0.4.1
kube-state-metrics-deployment.yaml: image: quay.io/coreos/kube-rbac-proxy:v0.4.1
kube-state-metrics-deployment.yaml: image: quay.io/coreos/kube-state-metrics:v1.7.2
kube-state-metrics-deployment.yaml: image: k8s.gcr.io/addon-resizer:1.8.4
node-exporter-daemonset.yaml: image: quay.io/prometheus/node-exporter:v0.18.1
node-exporter-daemonset.yaml: image: quay.io/coreos/kube-rbac-proxy:v0.4.1
prometheus-adapter-deployment.yaml: image: quay.io/coreos/k8s-prometheus-adapter-amd64:v0.4.1
prometheus-prometheus.yaml: baseImage: quay.io/prometheus/prometheus
##### 由上可知alertmanager和prometheus的镜像版本未显示
### 获取alertmanager镜像版本信息
[root@k8s-master manifests]# cat alertmanager-alertmanager.yaml
apiVersion: monitoring.coreos.com/v1
kind: Alertmanager
metadata:labels:alertmanager: mainname: mainnamespace: monitoring
spec:baseImage: quay.io/prometheus/alertmanagernodeSelector:kubernetes.io/os: linuxreplicas: 3securityContext:fsGroup: 2000runAsNonRoot: truerunAsUser: 1000serviceAccountName: alertmanager-mainversion: v0.18.0
##### 由上可见alertmanager的镜像版本为v0.18.0
### 获取prometheus镜像版本信息
[root@k8s-master manifests]# cat prometheus-prometheus.yaml
apiVersion: monitoring.coreos.com/v1
kind: Prometheus
metadata:labels:prometheus: k8sname: k8snamespace: monitoring
spec:alerting:alertmanagers:- name: alertmanager-mainnamespace: monitoringport: webbaseImage: quay.io/prometheus/prometheusnodeSelector:kubernetes.io/os: linuxpodMonitorSelector: {}replicas: 2resources:requests:memory: 400MiruleSelector:matchLabels:prometheus: k8srole: alert-rulessecurityContext:fsGroup: 2000runAsNonRoot: truerunAsUser: 1000serviceAccountName: prometheus-k8sserviceMonitorNamespaceSelector: {}serviceMonitorSelector: {}version: v2.11.0
##### 由上可见prometheus的镜像版本为v2.11.0
执行脚本:镜像下载并重命名【集群所有机器执行】
[root@k8s-master software]# vim download_prometheus_image.sh
#!/bin/sh##### 在 master 节点和 worker 节点都要执行 【所有机器执行】# 加载环境变量
. /etc/profile
. /etc/bashrc###############################################
# 从国内下载 prometheus 所需镜像,并对镜像重命名
src_registry="registry.cn-beijing.aliyuncs.com/cloud_registry"
# 定义镜像集合数组
images=(kube-rbac-proxy:v0.4.1kube-state-metrics:v1.7.2k8s-prometheus-adapter-amd64:v0.4.1configmap-reload:v0.0.1prometheus-config-reloader:v0.33.0prometheus-operator:v0.33.0
)
# 循环从国内获取的Docker镜像
for img in ${images[@]};
do# 从国内源下载镜像docker pull ${src_registry}/$img# 改变镜像名称docker tag ${src_registry}/$img quay.io/coreos/$img# 删除源始镜像docker rmi ${src_registry}/$img# 打印分割线echo "======== $img download OK ========"
done##### 其他镜像下载
image_name="alertmanager:v0.18.0"
docker pull ${src_registry}/${image_name} && docker tag ${src_registry}/${image_name} quay.io/prometheus/${image_name} && docker rmi ${src_registry}/${image_name}
echo "======== ${image_name} download OK ========"image_name="node-exporter:v0.18.1"
docker pull ${src_registry}/${image_name} && docker tag ${src_registry}/${image_name} quay.io/prometheus/${image_name} && docker rmi ${src_registry}/${image_name}
echo "======== ${image_name} download OK ========"image_name="prometheus:v2.11.0"
docker pull ${src_registry}/${image_name} && docker tag ${src_registry}/${image_name} quay.io/prometheus/${image_name} && docker rmi ${src_registry}/${image_name}
echo "======== ${image_name} download OK ========"image_name="grafana:6.2.2"
docker pull ${src_registry}/${image_name} && docker tag ${src_registry}/${image_name} grafana/${image_name} && docker rmi ${src_registry}/${image_name}
echo "======== ${image_name} download OK ========"image_name="addon-resizer:1.8.4"
docker pull ${src_registry}/${image_name} && docker tag ${src_registry}/${image_name} k8s.gcr.io/${image_name} && docker rmi ${src_registry}/${image_name}
echo "======== ${image_name} download OK ========"echo "********** prometheus docker images OK! **********"
执行脚本后得到如下镜像
[root@k8s-master software]# docker images | grep 'quay.io/coreos'
quay.io/coreos/kube-rbac-proxy v0.4.1 a9d1a87e4379 6 days ago 41.3MB
quay.io/coreos/flannel v0.12.0-amd64 4e9f801d2217 4 months ago 52.8MB ## 之前已存在
quay.io/coreos/kube-state-metrics v1.7.2 3fd71b84d250 6 months ago 33.1MB
quay.io/coreos/prometheus-config-reloader v0.33.0 64751efb2200 8 months ago 17.6MB
quay.io/coreos/prometheus-operator v0.33.0 8f2f814d33e1 8 months ago 42.1MB
quay.io/coreos/k8s-prometheus-adapter-amd64 v0.4.1 5f0fc84e586c 15 months ago 60.7MB
quay.io/coreos/configmap-reload v0.0.1 3129a2ca29d7 3 years ago 4.79MB
[root@k8s-master software]#
[root@k8s-master software]# docker images | grep 'quay.io/prometheus'
quay.io/prometheus/node-exporter v0.18.1 d7707e6f5e95 11 days ago 22.9MB
quay.io/prometheus/prometheus v2.11.0 de242295e225 2 months ago 126MB
quay.io/prometheus/alertmanager v0.18.0 30594e96cbe8 10 months ago 51.9MB
[root@k8s-master software]#
[root@k8s-master software]# docker images | grep 'grafana'
grafana/grafana 6.2.2 a532fe3b344a 9 months ago 248MB
[root@k8s-node01 software]#
[root@k8s-node01 software]# docker images | grep 'addon-resizer'
k8s.gcr.io/addon-resizer 1.8.4 5ec630648120 20 months ago 38.3MB
kube-prometheus启动
启动prometheus
[root@k8s-master kube-prometheus-0.2.0]# pwd
/root/k8s_practice/prometheus/kube-prometheus-0.2.0
[root@k8s-master kube-prometheus-0.2.0]#
### 如果出现异常,可以再重复执行一次或多次
[root@k8s-master kube-prometheus-0.2.0]# kubectl apply -f manifests/
启动后svc与pod状态查看
[root@k8s-master ~]# kubectl top node
NAME CPU(cores) CPU% MEMORY(bytes) MEMORY%
k8s-master 152m 7% 1311Mi 35%
k8s-node01 100m 5% 928Mi 54%
k8s-node02 93m 4% 979Mi 56%
[root@k8s-master ~]#
[root@k8s-master ~]# kubectl get svc -n monitoring
NAME TYPE CLUSTER-IP EXTERNAL-IP PORT(S) AGE
alertmanager-main NodePort 10.97.249.249 <none> 9093:30300/TCP 7m21s
alertmanager-operated ClusterIP None <none> 9093/TCP,9094/TCP,9094/UDP 7m13s
grafana NodePort 10.101.183.103 <none> 3000:30100/TCP 7m20s
kube-state-metrics ClusterIP None <none> 8443/TCP,9443/TCP 7m20s
node-exporter ClusterIP None <none> 9100/TCP 7m20s
prometheus-adapter ClusterIP 10.105.174.86 <none> 443/TCP 7m19s
prometheus-k8s NodePort 10.109.179.233 <none> 9090:30200/TCP 7m19s
prometheus-operated ClusterIP None <none> 9090/TCP 7m3s
prometheus-operator ClusterIP None <none> 8080/TCP 7m21s
[root@k8s-master ~]#
[root@k8s-master ~]# kubectl get pod -n monitoring -o wide
NAME READY STATUS RESTARTS AGE IP NODE NOMINATED NODE READINESS GATES
alertmanager-main-0 2/2 Running 0 2m11s 10.244.4.164 k8s-node01 <none> <none>
alertmanager-main-1 2/2 Running 0 2m11s 10.244.2.225 k8s-node02 <none> <none>
alertmanager-main-2 2/2 Running 0 2m11s 10.244.4.163 k8s-node01 <none> <none>
grafana-5cd56df4cd-6d75r 1/1 Running 0 29s 10.244.2.227 k8s-node02 <none> <none>
kube-state-metrics-7d4bb66d8d-gx7w4 4/4 Running 0 2m18s 10.244.2.223 k8s-node02 <none> <none>
node-exporter-pl47v 2/2 Running 0 2m17s 172.16.1.110 k8s-master <none> <none>
node-exporter-tmmbw 2/2 Running 0 2m17s 172.16.1.111 k8s-node01 <none> <none>
node-exporter-w8wd9 2/2 Running 0 2m17s 172.16.1.112 k8s-node02 <none> <none>
prometheus-adapter-c676d8764-phj69 1/1 Running 0 2m17s 10.244.2.224 k8s-node02 <none> <none>
prometheus-k8s-0 3/3 Running 1 2m1s 10.244.2.226 k8s-node02 <none> <none>
prometheus-k8s-1 3/3 Running 0 2m1s 10.244.4.165 k8s-node01 <none> <none>
prometheus-operator-7559d67ff-lk86l 1/1 Running 0 2m18s 10.244.4.162 k8s-node01 <none> <none>
kube-prometheus访问
prometheus-service访问
访问地址如下:
http://172.16.1.110:30200/
通过访问如下地址,可以看到prometheus已经成功连接上了k8s的apiserver。
http://172.16.1.110:30200/targets
查看service-discovery
http://172.16.1.110:30200/service-discovery
prometheus自己指标查看
http://172.16.1.110:30200/metrics
prometheus的WEB界面上提供了基本的查询,例如查询K8S集群中每个POD的CPU使用情况,可以使用如下查询条件查询:
# 直接使用 container_cpu_usage_seconds_total 可以看见有哪些字段信息
sum(rate(container_cpu_usage_seconds_total{image!="", pod!=""}[1m] )) by (pod)
列表页面
图形页面
grafana-service访问
访问地址如下:
http://172.16.1.110:30100/
首次登录时账号密码默认为:admin/admin
添加数据来源
得到如下页面
如上,数据来源默认是已经添加好了的
点击进入,拉到下面,再点击Test按钮,测验数据来源是否正常
之后可导入一些模板
数据信息图像化查看
异常问题解决
如果 kubectl apply -f manifests/
出现类似如下提示:
unable to recognize "manifests/alertmanager-alertmanager.yaml": no matches for kind "Alertmanager" in version "monitoring.coreos.com/v1"
unable to recognize "manifests/alertmanager-serviceMonitor.yaml": no matches for kind "ServiceMonitor" in version "monitoring.coreos.com/v1"
unable to recognize "manifests/grafana-serviceMonitor.yaml": no matches for kind "ServiceMonitor" in version "monitoring.coreos.com/v1"
unable to recognize "manifests/kube-state-metrics-serviceMonitor.yaml": no matches for kind "ServiceMonitor" in version "monitoring.coreos.com/v1"
unable to recognize "manifests/node-exporter-serviceMonitor.yaml": no matches for kind "ServiceMonitor" in version "monitoring.coreos.com/v1"
unable to recognize "manifests/prometheus-operator-serviceMonitor.yaml": no matches for kind "ServiceMonitor" in version "monitoring.coreos.com/v1"
unable to recognize "manifests/prometheus-prometheus.yaml": no matches for kind "Prometheus" in version "monitoring.coreos.com/v1"
unable to recognize "manifests/prometheus-rules.yaml": no matches for kind "PrometheusRule" in version "monitoring.coreos.com/v1"
unable to recognize "manifests/prometheus-serviceMonitor.yaml": no matches for kind "ServiceMonitor" in version "monitoring.coreos.com/v1"
unable to recognize "manifests/prometheus-serviceMonitorApiserver.yaml": no matches for kind "ServiceMonitor" in version "monitoring.coreos.com/v1"
unable to recognize "manifests/prometheus-serviceMonitorCoreDNS.yaml": no matches for kind "ServiceMonitor" in version "monitoring.coreos.com/v1"
unable to recognize "manifests/prometheus-serviceMonitorKubeControllerManager.yaml": no matches for kind "ServiceMonitor" in version "monitoring.coreos.com/v1"
unable to recognize "manifests/prometheus-serviceMonitorKubeScheduler.yaml": no matches for kind "ServiceMonitor" in version "monitoring.coreos.com/v1"
unable to recognize "manifests/prometheus-serviceMonitorKubelet.yaml": no matches for kind "ServiceMonitor" in version "monitoring.coreos.com/v1"
那么再次 kubectl apply -f manifests/
即可;因为存在依赖。
但如果使用的是kube-prometheus:v0.3.0、v0.4.0、v0.5.0版本并出现了上面的提示【反复执行kubectl apply -f manifests/
,但一直存在】,原因暂不清楚。
完毕!
———END——— 如果觉得不错就关注下呗 (-^O^-) !
Kubernetes K8S之kube-prometheus概述与部署相关推荐
- Kubernetes(k8s)之在集群环境部署Prometheus(普罗米修斯监控)和集群的ui管理工具Grafana
Prometheus Prometheus 演示环境 部署Prometheus和Grafana 测试 Prometheus Prometheus是一个开源系统监控和警报工具包. 现在是一个独立的开源项 ...
- Mac下的Docker及Kubernetes(k8s)本地环境搭建与应用部署、管理界面kubernetes-dashboard
Mac下的Docker及Kubernetes环境搭建与应用部署 Mac安装docker: brew cask install docker 当然也可以直接去官网下载docker的pkg文件安装 Mac ...
- Kubernetes - - k8s - v1.12.3 Helm持久化部署Harbor集成OpenLDAP登录
1,基本概念 前几章部署了harbor和ldap,本节将部署harbor使用openLDAP验证,部署方式与之前相同,只是改了adminserver-cm.yaml的AUTH_MODE: " ...
- k8s部署Kube Prometheus(Prometheus Operator)
摘要 本文通过Prometheus-operator框架一键化安装prometheus.alertmanage.granfana,并配置企业微信api以及告警推送,搭建 prometheus 的前提环 ...
- 【CentOS】利用Kubeadm部署Kubernetes (K8s)
[CentOS]利用Kubeadm部署Kubernetes (K8s)[阅读时间:约10分钟] 一.概述 二.系统环境&项目介绍 1.系统环境 2.项目的任务要求 三.具体实验流程 1 系统准 ...
- Centos7 安装部署Kubernetes(k8s)集群过程
1.系统环境 服务器版本 docker软件版本 CPU架构 CentOS Linux release 7.9 Docker version 20.10.12 x86_64 2.前言 如下图描述了软件部 ...
- kubernetes(k8s) 安装 Prometheus + Grafana
kubernetes(k8s) 安装 Prometheus + Grafana 组件说明 MetricServer:是kubernetes集群资源使用情况的聚合器,收集数据给kubernetes集群内 ...
- 基于kubeadm快速部署kubernetes K8S V1.17.4集群-无坑完整版
基于kubeadm快速部署kubernetes K8S V1.17.4集群,并部署Dashboard Web页面,实现可视化查看Kubernetes资源 主机配置规划 服务器名称(hostname) ...
- Kubernetes(k8s)集群部署(k8s企业级Docker容器集群管理)系列目录
0.目录 整体架构目录:ASP.NET Core分布式项目实战-目录 k8s架构目录:Kubernetes(k8s)集群部署(k8s企业级Docker容器集群管理)系列目录 一.感谢 在此感谢.net ...
最新文章
- java 调用word插件_java一键生成word操作,比poi简单
- ERP项目实施的原则
- TCP往返时延的估计和超时
- jQuery 利用 $.getJson() 实现跨域
- java 设置两个方法互斥_分享两个操作Java枚举的实用方法
- java 二叉堆_【数据结构】二叉堆:Java实现最大堆及堆排序
- Java Thread类最终同步的void join(long time_in_ms)方法,带有示例
- git命令:全局设置用户名邮箱配置
- Java中String的next()和nextLine()方法
- 当技术遇上管理,Linaro 与 CSDN 掌舵人对话开源现状及下一代技术趋势
- python编程入门 适合于零基础朋友-Python不能帮你找到女朋友,却能让你成为有钱的单身狗。...
- Codeforces Round #FF(255) DIV2
- java collection自定义comparator进行排序
- 访问网站403错误解决方法(apache)
- Python 京东抢购茅台脚本(亲测可用),github脚本24小时内删除
- 常用控件的使用有哪些?如何操作控件?
- Java之Eclipse实现——在海绵宝宝的比基尼海滩里投喂鱼食
- 双足机器人课设报告_双足竞步机器人-智能步行者设计-技术报告
- 2018领航杯awd简单复现
- rails 构建高性能web