新钛云服已累计为您分享711篇技术干货

前言：

• blackbox_exporter

是Prometheus 官方提供的 exporter 之一,主要提供http、dns、tcp、icmp 的监控数据采集。

• Consul

主要提供，服务发现，健康检查，等功能，本次集成主要使用到服务发现功能。

本文主要实现，基于consul_sd_config  & consul 的 prometheus  服务发现，实现网路设备ping监控，站点可用行监控，以及证书相关信息监控。

安装环境：

• k8s

• consul

• Prometheus

• blackbox_exporter

1:  Consul 安装

1.1：使用helm 安装 consul

Bash
# 添加 consul helm 源
helm repo add hashicorp https://helm.releases.hashicorp.com
# 安装consul
helm -n consul  install  \
--set storageClass=alicloud-disk-efficiency  \
consul hashicorp/consul \
--version=0.32.1

1.2：查看服务安装状态

Bash
[root@xxxxxxxx consul_install]# kubectl -n consul get pods
NAME                     READY   STATUS    RESTARTS   AGE
consul-consul-9lxfc      1/1     Running   0          6d1h
consul-consul-ntqcf      1/1     Running   0          6d1h
consul-consul-q7c6f      1/1     Running   0          6d1h
consul-consul-server-0   1/1     Running   0          6d1h
consul-consul-server-1   1/1     Running   0          6d1h
consul-consul-server-2   1/1     Running   0          6d1h

1.3：nginx-ingress consul

• consul_ingress.yml

Bash
# consul.xxxxxx.cn  ----->  替换为正确域名apiVersion: networking.k8s.io/v1
kind: Ingress
metadata:name: consul-ingressnamespace: consulannotations:kubernetes.io/ingress.class: nginxnginx.ingress.kubernetes.io/rewrite-target: /
spec:rules:- host: consul.xxxxxx.cnhttp:paths:- path: /pathType: Prefixbackend:service:name: consul-consul-uiport: number: 80

• 执行部署

Bash
kubectl apply -f consul_ingress.yml

1.4：访问测试

2: Blackbox_export

2.1：blackbox 安装

• blackbox-exporter-config.yaml

Bash
apiVersion: v1
kind: ConfigMap
metadata:name: blackbox-exporterlabels:app: blackbox-exporter
data:blackbox.yml: |-modules:## ----------- DNS 检测配置 -----------dns_tcp:  prober: dnsdns:transport_protocol: "tcp"preferred_ip_protocol: "ip4"query_name: "kubernetes.default.svc.cluster.local" # 用于检测域名可用的网址query_type: "A" ## ----------- TCP 检测模块配置 -----------tcp_connect:prober: tcptimeout: 5s## ----------- ICMP 检测配置 -----------ping:prober: icmptimeout: 5sicmp:preferred_ip_protocol: "ip4"## ----------- HTTP GET 2xx 检测模块配置 -----------http_get_2xx:  prober: httptimeout: 10shttp:method: GETpreferred_ip_protocol: "ip4"valid_http_versions: ["HTTP/1.1","HTTP/2"]valid_status_codes: [200]           # 验证的HTTP状态码,默认为2xxno_follow_redirects: false          # 是否不跟随重定向## ----------- HTTP GET 3xx 检测模块配置 -----------http_get_3xx:  prober: httptimeout: 10shttp:method: GETpreferred_ip_protocol: "ip4"valid_http_versions: ["HTTP/1.1","HTTP/2"]valid_status_codes: [301,302,304,305,306,307]  # 验证的HTTP状态码,默认为2xxno_follow_redirects: false                     # 是否不跟随重定向## ----------- HTTP POST 监测模块 -----------http_post_2xx: prober: httptimeout: 10shttp:method: POSTpreferred_ip_protocol: "ip4"valid_http_versions: ["HTTP/1.1", "HTTP/2"]#headers:                             # HTTP头设置#  Content-Type: application/json#body: '{}'                           # 请求体设置

• blackbox-exporter-deploy.yaml

Bash
apiVersion: v1
kind: Service
metadata:name: blackbox-exporterlabels:k8s-app: blackbox-exporter
spec:type: ClusterIPports:- name: httpport: 9115targetPort: 9115selector:k8s-app: blackbox-exporter
---
apiVersion: apps/v1
kind: Deployment
metadata:name: blackbox-exporterlabels:k8s-app: blackbox-exporter
spec:replicas: 1selector:matchLabels:k8s-app: blackbox-exportertemplate:metadata:labels:k8s-app: blackbox-exporterspec:containers:- name: blackbox-exporterimage: prom/blackbox-exporter:v0.19.0args:- --config.file=/etc/blackbox_exporter/blackbox.yml- --web.listen-address=:9115- --log.level=infoports:- name: httpcontainerPort: 9115resources:limits:cpu: 3memory: 6000Mirequests:cpu: 100mmemory: 50MilivenessProbe:tcpSocket:port: 9115initialDelaySeconds: 5timeoutSeconds: 5periodSeconds: 10successThreshold: 1failureThreshold: 3readinessProbe:tcpSocket:port: 9115initialDelaySeconds: 5timeoutSeconds: 5periodSeconds: 10successThreshold: 1failureThreshold: 3volumeMounts:- name: configmountPath: /etc/blackbox_exportervolumes:- name: configconfigMap:name: blackbox-exporterdefaultMode: 420

• 执行安装

Bash
kubectl apply -f blackbox-exporter-deploy.yaml
kubectl apply -f blackbox-exporter-config.yaml

2.2：nginx ingress blackbox-exporter • blackbox_ingress.yml

Bash
apiVersion: networking.k8s.io/v1
kind: Ingress
metadata:name: blackbox-ingressnamespace: monitoringannotations:kubernetes.io/ingress.class: nginxnginx.ingress.kubernetes.io/rewrite-target: /
spec:rules:- host: xxxhttp:paths:- path: /pathType: Prefixbackend:service:name: blackbox-exporterport:number: 9115

• 执行安装

Bash
kubectl apply -f blackbox_ingress.yml

3: rometheus 添加 服务动态发现

Bash
#####  http_get_2xx 数据获取- job_name: http_get_2xxparams:module:- http_get_2xxscrape_interval: 2sscrape_timeout: 2smetrics_path: /probeconsul_sd_configs:# consul 服务地址- server: consul-consul-server.consul.svc.cluster.local:8500tag_separator: ','services:- http_get_2xxrelabel_configs:- source_labels: ['__meta_consul_service_address']target_label: __param_target- source_labels: ['__meta_consul_service_address']target_label: instance- target_label: __address__## blackbox-export  地址replacement: blackbox-exporter.monitoring.svc.cluster.local:9115####### icmp 配置 - job_name: blackbox_icmpparams:module:- pingscrape_interval: 2sscrape_timeout: 2smetrics_path: /probeconsul_sd_configs:# consul 服务地址- server: consul-consul-server.consul.svc.cluster.local:8500tag_separator: ','services:- pingrelabel_configs:- source_labels: ['__meta_consul_service_address']target_label: __param_target- source_labels: ['__meta_consul_service_address']target_label: instance- target_label: __address__## blackbox-export  地址replacement: blackbox-exporter.monitoring.svc.cluster.local:9115

4：添加 icmp 监控

4.1：添加监控地址到consul

• icmp_list

Bash
192.168.1.1
192.168.1.2

• add_consul_service_icmp.sh

Bash
#!/usr/bin/env baship_addr=$1if test "$ip_addr";thencurl -X PUT -d '{"id": "icmp_'${ip_addr}'","name": "ping","address": "'${ip_addr}'","port": 443,"Meta": {"env": "prod","team": "network","project": "network","owner": "Mike"},"tags": ["node"],"checks": [{"http": "http://blackbox-exporter.monitoring.svc.cluster.local:9115/","interval": "15s"}]}' \http://consul-consul-server:8500/v1/agent/service/register
elseecho "请输入参数"
fi

• 添加service ping

Bash
for i in `cat icmp_list`;do bash add_consul_service_icmp.sh  $i;done

4.2：查看consul 服务

4.3：删除ping 监控地址脚本

Bash
#!/usr/bin/env bash
ip_addr=$1curl -X PUT http://consul-consul-server:8500/v1/agent/service/deregister/icmp_${ip_addr}

5: 添加http_get_2xx

5.1：添加监控域名

• domain_name_list

Bash
wwww.baidu.com
wwww.1111.com
wwww.2222.com

• add_consul_service_http_get_2xx.sh

Bash
#!/usr/bin/env bashservice_name=$1if test "$service_name";thencurl -X PUT -d '{"id": "http_get_2xx_'${service_name}'","name": "http_get_2xx","address": "https://'${service_name}'","port": 443,"Meta": {"env": "prod","team": "web","project": "web","owner": "Devops"},"tags": ["node"],"checks": [{"http": "http://blackbox-exporter.monitoring.svc.cluster.local:9115/","interval": "15s"}]}' \http://consul-consul-server:8500/v1/agent/service/register
elseecho "请输入参数"
fi

• 添加 service  http_get_2xx

Bash
for i  in `cat domain_name_list`;do bash  add_consul_service_http_get_2xx.sh  $i;done

5.2：查看consul 服务

5.3：删除域名监控脚本

• del_consul_service_http_get_2xx.sh

Bash
#!/usr/bin/env bash
ip_addr=$1curl -X PUT http://consul-consul-server:8500/v1/agent/service/deregister/http_get_2xx_${ip_addr}

6：查看prometheus 监控

总结：

使用上述方案，黑盒监控与自建cmdb 平台很容易进行集成，使其监控自动化，不需要过多的人工干预，可以省去大量的人工成本，grafana 的配置这里就不进行过多介绍，自行通过谷歌完成。

    推荐阅读   

    推荐视频