blackbox_exporter 黑盒监测

一、简介

blackbox_exporter:
blackbox_exporter是Prometheus 官方提供的 exporter 之一，可以提供 http、dns、tcp、icmp 的监控数据采集，blackbox_exporter 可用于以下监测：
HTTP 测试：定义Request Header信息、判断 Http status/Http Respones Header/HttpBody 内容
TCP 测试：业务组件端口状态监听、应用层协议定义与监听
ICMP 测试：主机探活机制
POST 测试：接口联通性
SSL证书过期时间

github地址：GitHub - prometheus/blackbox_exporter: Blackbox prober exporterhttps://github.com/prometheus/blackbox_exporter

二、安装

二进制安装

$ wget https://github.com/prometheus/blackbox_exporter/releases/download/v0.16.0/blackbox_exporter-0.16.0.linux-amd64.tar.gz
$ tar -zxvf blackbox_exporter-0.16.0.linux-amd64.tar.gz -C /data
$ mv /data/blackbox_exporter-0.16.0.linux-amd64 /data/blackbox_exporter
$ cd /data/blackbox_exporter/
$ ./blackbox_exporter --version
$ nohup ./blackbox_exporter &

使用blackbox_exporter 检测目标，使用语法如下：

语法结构（debug模式）：
http://localhost:9115/probe?target=xxx&module=xxx&debug=true例如：
$ curl "http://localhost:9115/probe?target=10.50.10.7:22022&module=tcp_connect&debug=true"
$ curl "http://localhost:9115/probe?target=10.50.10.7&module=icmp&debug=true"

K8S安装

配置文件

$ cat config.yml
apiVersion: v1
kind: ConfigMap
metadata:name: blackbox-exporternamespace: kube-mon
data:blackbox.yml: |-modules:http_2xx:prober: httptimeout: 2shttp:valid_http_versions: ["HTTP/1.1", "HTTP/2"]valid_status_codes: [200,301,302]method: GETpreferred_ip_protocol: "ipv4"tcp_connect:prober: tcptimeout: 2shttp_403:prober: httptimeout: 2shttp:valid_http_versions: ["HTTP/1.1", "HTTP/2"]valid_status_codes: [403]method: GETpreferred_ip_protocol: "ipv4"https_403:prober: httpstimeout: 2shttp:valid_http_versions: ["HTTP/1.1", "HTTP/2"]valid_status_codes: [403]method: GETpreferred_ip_protocol: "ipv4"icmp:prober: icmp

deploy部署

$ cat deploy.yml
kind: Deployment
apiVersion: apps/v1
metadata:name: blackbox-exporternamespace: kube-monlabels:app.kubernetes.io/name: blackboxapp.kubernetes.io/version: 0.16.0
spec:replicas: 1selector:matchLabels:app.kubernetes.io/name: blackboxtemplate:metadata:labels:app.kubernetes.io/name: blackboxspec:volumes:- name: configconfigMap:name: blackbox-exporterdefaultMode: 420containers:- name: blackbox-exporterimage: prom/blackbox-exporter:v0.16.0imagePullPolicy: IfNotPresentargs:- --config.file=/etc/blackbox_exporter/blackbox.yml- --log.level=info- --web.listen-address=:9115ports:- name: blackbox-portcontainerPort: 9115protocol: TCPresources:limits:cpu: 30mmemory: 100Mirequests:cpu: 10mmemory: 50MivolumeMounts:- name: configmountPath: /etc/blackbox_exporterreadinessProbe:tcpSocket:port: 9115initialDelaySeconds: 5timeoutSeconds: 5periodSeconds: 10successThreshold: 1failureThreshold: 3

svc 服务发现

$ cat svc.yml
apiVersion: v1
kind: Service
metadata:labels:app.kubernetes.io/name: blackboxname: blackbox-exporternamespace: kube-mon
spec:ports:- name: balckboxport: 9115protocol: TCPtargetPort: 9115selector:app.kubernetes.io/name: blackbox

创建以上资源：

$ kubectl apply -f .

如更改配置文件需要重新加载配置:

$ curl -X POST localhost:31425/-/reload

三、配置Prometheus采集数据

1、ICMP 测试（主机探活）

可以通过 ping(icmp) 检测服务器的存活,在 blackbox.yml 使用的配置是icmp 模块：

modules:icmp:prober: icmp

添加Prometheus配置：

   - job_name: "check_hosts"metrics_path: /probeparams:modelus: [icmp]static_configs:- targets:- 10.60.38.208relabel_configs:- source_labels: [__address__]target_label: __param_target- source_labels: [__param_target]target_label: instance- target_label: __address__replacement: blackbox-exporter:9115

2、TCP 测试（监控主机端口存活状态）

检测端口是否存活,在blackbox.yml 配置文件中使用的配置是tcp_connect模块：

modules:tcp_connect:prober: tcp

添加pronetheus配置

   - job_name: "check_ports"metrics_path: /probeparams:module: [tcp_connect]static_configs:- targets:- 10.60.38.208:8080relabel_configs:- source_labels: [__address__]target_label: __param_target- source_labels: [__param_target]target_label: instance- target_label: __address__replacement: blackbox-exporter:9115

3、检测SSL证书过期时间

检测SSL证书,在blackbox.yml 配置文件中使用的配置是模块：

http_2xx:prober: httptimeout: 10shttp:preferred_ip_protocol: "ipv4"valid_http_versions: ["HTTP/1.1", "HTTP/2"]valid_status_codes: [200,301,302,303]method: GET

添加pronetheus配置

   - job_name: "blackbox_ssl"metrics_path: /probeparams:module: [http_2xx]static_configs:- targets:- https://www.baidu.comrelabel_configs:- source_labels: [__address__]target_label: instance- source_labels: [__address__]target_label: __param_target- target_label: __address__replacement: blackbox-exporter:9115

告警规则

端口监测失败 （正则匹配8080端口）
sum by(instance) (probe_success{instance=~".*:8080$",job="check_ports"}) == 0主机ping不通
sum by (instance)(probe_success{job="check_hosts"}) == 0证书还有30天过期
probe_ssl_earliest_cert_expiry{job="blackbox_ssl"} - time() < 86400 * 30

grafana id： 9965 13230