一、介绍

Prometheus 启动的时候，可以加载运行参数 -config.file 指定配置文件，默认为 prometheus.yml。

Prometheus的配置文件是YAML格式。Prometheus的解压包里自带了一个默认的配置文件prometheus.yml。让我们来看一下：

global:scrape_interval:     15s # Set the scrape interval to every 15 seconds. Default is every 1 minute.evaluation_interval: 15s # Evaluate rules every 15 seconds. The default is every 1 minute.# scrape_timeout is set to the global default (10s).# Alertmanager configuration
alerting:alertmanagers:- static_configs:- targets:# - alertmanager:9093# Load rules once and periodically evaluate them according to the global 'evaluation_interval'.
rule_files:# - "first_rules.yml"# - "second_rules.yml"# A scrape configuration containing exactly one endpoint to scrape:
# Here it's Prometheus itself.
scrape_configs:# The job name is added as a label `job=<job_name>` to any timeseries scraped from this config.- job_name: 'prometheus'# metrics_path defaults to '/metrics'# scheme defaults to 'http'.static_configs:- targets: ['localhost:9090']

在这个缺省的配置文件里定义了4个单元：global、alerting、rule_files和scrape_configs。

在配置文件中我们可以指定 global, alerting, rule_files, scrape_configs, remote_write, remote_read 等属性。

global：配置全局的信息，如抓取监控数据的间隔，抓取业务数据接口的超时时间，告警规则执行周期等

alerting：配置告警发送到的alermanager的地址

rule_files：告警规则文件，数据聚合配置

scrape_configs：配置抓取业务监控数据的相关信息，如url，拉取时间间隔，拉取的超时时间等

remote_write：将数据投递到远程地址，如聚合数据投递到hubble-adapter

remote_read：

下面介绍下每个单元。

二、global

global 属于全局的默认配置，它主要包含 4 个属性，

• scrape_interval: 拉取 targets 的默认时间间隔，即拉取业务监控数据的间隔时间。
• scrape_timeout: 拉取一个 target 的超时时间，即拉取业务监控数据接口的超时时间。
• evaluation_interval: 执行 rules 的时间间隔。即多久遍历一次告警规则列表，判断每个规则是否触发告警。和rule_files的加载没关系
• external_labels: 额外的属性，会添加到拉取的数据并存到数据库中。

配置文件结构大概为：

global:scrape_interval:     15s # By default, scrape targets every 15 seconds.evaluation_interval: 15s # By default, scrape targets every 15 seconds.scrape_timeout: 10s # is set to the global default (10s).# Attach these labels to any time series or alerts when communicating with# external systems (federation, remote storage, Alertmanager).external_labels:monitor: 'codelab-monitor'

三、alerting

通常我们可以使用运行参数 -alertmanager.xxx 来配置 Alertmanager， 但是这样不够灵活，没有办法做到动态更新加载，以及动态定义告警属性。

所以 alerting 配置主要用来解决这个问题，它能够更好的管理 Alertmanager, 主要包含 2 个参数：alert_relabel_configs 和 alertmanagers

1、alertmanagers

用于动态发现 Alertmanager 的地址。

如下配置：alertmanager.prom-alert.svc:9093，通过K8S自动发现机制找到本集群内的alertmanager的地址并将告警发送过去。

所以，每个prometheus集群一定会部署一个alertmanager组件。

如下，系统中指定了Alertmanager路径，因为最终需要投递告警到这个服务，如下图：

2、alert_relabel_configs

作用：在告警发生时，动态修改标签内容，一般作用是在告警产生时修改标签，如保留哪些标签（labelkeep），删除哪些标签（labeldrop）。具体的有哪些属性，请参考：Configuration | Prometheus

下面着重说明两个属性：action和regex

action

基于正则表达式匹配执行的操作。包括移除标签，保留标签等，具体可参考：Configuration | Prometheus

action枚举：

replace: 将正则表达式与串联的source_labels匹配。然后，将target_label设置为replace，用替换中的匹配组引用(${1}, ${2}, ...)替换为其值。 如果正则表达式不匹配，则不会进行替换。
keep: 删除其正则表达式与串联的source_labels不匹配的目标。
drop: 删除其正则表达式与串联的source_labels匹配的目标。
hashmod: 将target_label设置为串联的source_labels的哈希的模数。
labelmap: 将正则表达式与所有标签名称匹配。 然后，将匹配标签的值复制到通过替换为它们的值替换的匹配组引用(${1}, ${2}, ...)给出的标签名称。
labeldrop: 将正则表达式与所有标签名称匹配。 任何匹配的标签将从标签集中删除。
labelkeep: 将正则表达式与所有标签名称匹配。 任何不匹配的标签将从标签集中删除。

regex

作用是匹配标签的正则表达式。

案例

下面的案例中，action为labeldrop，就是字面意思，需要移除key为prometheus_replica的标签。

prometheus_replica是自定义的标签，告警的时候就会带上这个标签，由于我们prometheus有两个节点pod0和pod1，但是告警产生的时候我们只需要报出来一条就行了，因此把pod的标签去掉后，两个节点产生的告警就完全一样了，就能控制只产生一条。

source_labels

__开头的是保留label

source_labels也可以定义自定义的label

以下为QKE上的案例：下面写错了一句话，应该是将两个标签合并替换为node，而不是Node

四、rule_files

作用：获取所有规则文件中的规则，包括告警规则和record规则。告警规则好理解，record规则其实就是数据处理的规则，如下：

我们可以单独定义数据聚合规则文件，也可以和告警规则文件放一起，但是一般分开放好理解

注意：一定是规则文件，不包含配置文件，如果指定的文件中包含配置文件内容，则会报错。

按照配置的目录，找了下rancher上对应武汉集群的prometheus项目下的prometheus服务，进入控制台：

我们的配置文件是这样的：

rule_files:
- /etc/prometheus/rules/*rules.yaml

所以进入此目录下发现只有一个alert-rules.yaml，恰好就是我们的告警的配置。

关于rule_files的修改：

1、rule_files文件可以在rancher上修改；

2、通过Prometheus Operator提供的CRD修改。Prometheus Operator会去创建Prometheus、PodMonitor、ServiceMonitor、AlertManager以及PrometheusRule这5个CRD资源对象，所以，可以直接调用K8S的API去修改PrometheusRule，从而达到修改rule以及其他配置的效果，如增加record配置等。

五、scrape_configs

参考：Configuration | Prometheus

scrape_configs 主要用于配置拉取数据节点，每一个拉取配置主要包含以下参数：

• job_name：任务名称
• honor_labels： 用于解决拉取数据标签有冲突，当设置为 true, 以拉取数据为准，否则以服务配置为准
• params：数据拉取访问时带的请求参数
• scrape_interval： 拉取时间间隔
• scrape_timeout: 拉取超时时间
• metrics_path： 拉取节点的 metric 路径
• static_configs：配置访问路径前缀，如ip+port，或者域名地址，或者通过服务发现，类似alertmanager.prom-alert.svc:9093
• scheme： 拉取数据访问协议，如http
• sample_limit： 存储的数据标签个数限制，如果超过限制，该数据将被忽略，不入存储；默认值为0，表示没有限制
• relabel_configs： 拉取数据重置标签配置
• metric_relabel_configs：metric 重置标签配置

六、remote_write

remote_write 主要用于可写远程存储配置，主要包含以下参数：

• url: 访问地址
• remote_timeout: 请求超时时间
• write_relabel_configs: 标签重置配置, 拉取到的数据，经过重置处理后，发送给远程存储

案例：

remote_write:
- url: http://xxx:9988/prom2hubble/push?group=xxxremote_timeout: 30swrite_relabel_configs:- source_labels: [__name__]separator: ;regex: obser:(.*)replacement: $1action: keep- separator: ;regex: (.*)target_label: hubble_endpointreplacement: hubble_qpaas_obseraction: replace- separator: ;regex: (.*)target_label: groupreplacement: hubbleaction: replace- separator: ;regex: (.*)target_label: hubble_stepreplacement: "60"action: replace- separator: ;regex: label_qke_cloud_qiyi_domain_(.*)replacement: $1action: labelmap- separator: ;regex: (job|endpoint|service|pod|instance|namespace|prometheus.*|label_qke_cloud_qiyi_domain_.*)replacement: $1action: labeldropqueue_config:capacity: 2500max_shards: 200min_shards: 1max_samples_per_send: 500batch_send_deadline: 5smin_backoff: 30msmax_backoff: 100msmetadata_config:send: truesend_interval: 1m

七、remote_read

remote_read 主要用于可读远程存储配置，主要包含以下参数：

• url: 访问地址
• remote_timeout: 请求超时时间

八、服务发现

1、介绍

ServiceDiscoveryConfig 主要用于 target 发现，大体分为两类，静态配置和动态发现。

在 Prometheus 的配置中，一个最重要的概念就是数据源 target，而数据源的配置主要分为静态配置和动态发现, 大致为以下几类：

• static_configs: 静态服务发现
• eureka_sd_config：eureka服务发现，发现真实的实例节点的ip+port，参考：Configuration | Prometheus
  • 可参考案例：prometheus/prometheus-eureka.yml at release-2.36 · prometheus/prometheus · GitHub
• dns_sd_configs: DNS 服务发现
• file_sd_configs: 文件服务发现
• consul_sd_configs: Consul 服务发现
• serverset_sd_configs: Serverset 服务发现
• nerve_sd_configs: Nerve 服务发现
• marathon_sd_configs: Marathon 服务发现
• kubernetes_sd_configs: Kubernetes 服务发现
• gce_sd_configs: GCE 服务发现
• ec2_sd_configs: EC2 服务发现
• openstack_sd_configs: OpenStack 服务发现
• azure_sd_configs: Azure 服务发现
• triton_sd_configs: Triton 服务发现

它们具体使用以及配置模板，请参考服务发现配置模板。

它们中最重要的，也是使用最广泛的应该是 static_configs, 其实那些动态类型都可以看成是某些通用业务使用静态服务封装的结果。

2、eureka_sd_configs案例介绍

代码地址：

prometheus/prometheus-eureka.yml at release-2.36 · prometheus/prometheus · GitHub

3、kubernetes_sd_configs

我们的微服务本质上是采用的kubernetes_sd_configs。

但是我们是通过Prometheus Operator提供的ServiceMonitor间接创建了kubernetes_sd_configs

如下关注点：

1、kubernetes_sd_configs中的api_server：抓取指标的地址前缀

2、metrics_path：抓取指标的具体路径

- job_name: qke-generic-hubble-manager/hubble-alarm-agg-condition-sm/0honor_timestamps: truescrape_interval: 30sscrape_timeout: 10smetrics_path: /metrics/prometheusscheme: httpbearer_token_file: /var/run/secrets/kubernetes.io/serviceaccount/tokenrelabel_configs:- source_labels: [__meta_kubernetes_service_label_app]separator: ;regex: m-agg-conditionreplacement: $1action: keep- source_labels: [__meta_kubernetes_endpoint_port_name]separator: ;regex: metricsreplacement: $1action: keep- source_labels: [__meta_kubernetes_endpoint_address_target_kind, __meta_kubernetes_endpoint_address_target_name]separator: ;regex: Node;(.*)target_label: nodereplacement: ${1}action: replace- source_labels: [__meta_kubernetes_endpoint_address_target_kind, __meta_kubernetes_endpoint_address_target_name]separator: ;regex: Pod;(.*)target_label: podreplacement: ${1}action: replace- source_labels: [__meta_kubernetes_namespace]separator: ;regex: (.*)target_label: namespacereplacement: $1action: replace- source_labels: [__meta_kubernetes_service_name]separator: ;regex: (.*)target_label: servicereplacement: $1action: replace- source_labels: [__meta_kubernetes_pod_name]separator: ;regex: (.*)target_label: podreplacement: $1action: replace- source_labels: [__meta_kubernetes_service_name]separator: ;regex: (.*)target_label: jobreplacement: ${1}action: replace- separator: ;regex: (.*)target_label: endpointreplacement: metricsaction: replacekubernetes_sd_configs:- api_server: https://kube-master-bjzyx-public-staging02.cloud.qiyi.domain:6443role: endpointsbearer_token_file: /var/k8s-auth/tokentls_config:ca_file: /var/k8s-auth/ca.crtinsecure_skip_verify: truenamespaces:names:- qke-generic-hubble-manager

十、关于relabel_configs

relabel_configs分为<metric_relabel_configs>和<alert_relabel_configs>两类。

指标：

告警：

七、AlertManager配置

案例：

为什么是webhook_configs？不支持remote_write？

global:
resolve_timeout: 10mroute:
group_by: ['alertname']
group_wait: 10s
group_interval: 10s
repeat_interval: 24h   #重复报警的时间间隔为24h
receiver: hubblereceivers:
- name: 'hubble'
webhook_configs:
- url: 'http://hubble.adapter.qiyi.domain:9988/prom2hubble/alert'

八、告警规则配置

案例：

主要包含几部分：

groups:
- name: PrometheusRule                 #报警规则组的名字，可以类比为hubble的策略模板rules:                               #策略列表- expr: up{job="alertmanager"} == 0  #表达式alert: alertmanagerInstanceDown    #告警的triggernamefor: 2m                            #2分钟比较一次，和连续几个点类似annotations:                       #告警信息必要的信息，labels是告警消息的tag信息alertlevel: "P2"hubblegroup: "hubble-prometheus-k8s"alertvalue: "{{ $value }}"summary: "[prometheus-cluster-wh] alertmanager is down"- expr: increase(alertmanager_notifications_failed_total{job="alertmanager"}[5m])/increase(alertmanager_notifications_total{job="alertmanager"}[5m]) > 0.3alert: alertmanagerSendOutFailfor: 5mannotations:alertlevel: "P2"hubblegroup: "hubble-prometheus-k8s"alertvalue: "{{ $value }}"            #value 就是数据的当前值summary: "[prometheus-cluster-wh] failed to sendout alerts >30%"description: "应用名： {{ $labels.job }}  实例名： {{ $labels.instance }}  ， 环境： {{ $labels.env }} , 当前值为 ： {{ $value }}"   # labels其实就是数据中的tag，如job，instance等

八、配置文件案例

QKE配置文件案例：

global:scrape_interval: 30sscrape_timeout: 10sevaluation_interval: 30sexternal_labels:prometheus: 58-hubble/k8sprometheus_replica: prometheus-k8s-0
alerting:alert_relabel_configs:- separator: ;regex: prometheus_replicareplacement: $1action: labeldropalertmanagers:- scheme: httppath_prefix: /timeout: 10sapi_version: v1relabel_configs:- source_labels: [__meta_kubernetes_service_name]separator: ;regex: alertmanagerreplacement: $1action: keep- source_labels: [__meta_kubernetes_endpoint_port_name]separator: ;regex: httpreplacement: $1action: keepkubernetes_sd_configs:- role: endpointsnamespaces:names:- default
rule_files:
- /etc/prometheus/rules/prometheus-k8s-rulefiles-0/*.yaml
scrape_configs:
- job_name: qke-generic-hubble-grafana-dashboard/hubble-grafana-dashboard-servicemonitor/0honor_timestamps: truescrape_interval: 30sscrape_timeout: 10smetrics_path: /metricsscheme: httprelabel_configs:- source_labels: [__meta_kubernetes_service_label_app]separator: ;regex: hubble-grafana-dashboardreplacement: $1action: keep- source_labels: [__meta_kubernetes_endpoint_port_name]separator: ;regex: httpreplacement: $1action: keep- source_labels: [__meta_kubernetes_endpoint_address_target_kind, __meta_kubernetes_endpoint_address_target_name]separator: ;regex: Node;(.*)target_label: nodereplacement: ${1}action: replace- source_labels: [__meta_kubernetes_endpoint_address_target_kind, __meta_kubernetes_endpoint_address_target_name]separator: ;regex: Pod;(.*)target_label: podreplacement: ${1}action: replace- source_labels: [__meta_kubernetes_namespace]separator: ;regex: (.*)target_label: namespacereplacement: $1action: replace- source_labels: [__meta_kubernetes_service_name]separator: ;regex: (.*)target_label: servicereplacement: $1action: replace- source_labels: [__meta_kubernetes_pod_name]separator: ;regex: (.*)target_label: podreplacement: $1action: replace- source_labels: [__meta_kubernetes_service_name]separator: ;regex: (.*)target_label: jobreplacement: ${1}action: replace- separator: ;regex: (.*)target_label: endpointreplacement: httpaction: replace- source_labels: [__meta_kubernetes_pod_label_app]separator: ;regex: hubble-grafana-dashboardreplacement: $1action: keepmetric_relabel_configs:- source_labels: [__name__]separator: ;regex: (grafana_stat_.*|grafana_.*_response_status_total|process_.*)replacement: $1action: keepkubernetes_sd_configs:- api_server: https://kube-master-bjzyx-public-staging02.cloud.qiyi.domain:6443role: endpointsbearer_token_file: /var/k8s-auth/tokentls_config:ca_file: /var/k8s-auth/ca.crtinsecure_skip_verify: truenamespaces:names:- qke-generic-hubble-grafana-dashboard
- job_name: qke-generic-hubble-manager/hubble-api-open-sm/0honor_timestamps: truescrape_interval: 30sscrape_timeout: 10smetrics_path: /prometheusscheme: httpbearer_token_file: /var/run/secrets/kubernetes.io/serviceaccount/tokenrelabel_configs:- source_labels: [__meta_kubernetes_service_label_app]separator: ;regex: s-api-openreplacement: $1action: keep- source_labels: [__meta_kubernetes_endpoint_port_name]separator: ;regex: metricsreplacement: $1action: keep- source_labels: [__meta_kubernetes_endpoint_address_target_kind, __meta_kubernetes_endpoint_address_target_name]separator: ;regex: Node;(.*)target_label: nodereplacement: ${1}action: replace- source_labels: [__meta_kubernetes_endpoint_address_target_kind, __meta_kubernetes_endpoint_address_target_name]separator: ;regex: Pod;(.*)target_label: podreplacement: ${1}action: replace- source_labels: [__meta_kubernetes_namespace]separator: ;regex: (.*)target_label: namespacereplacement: $1action: replace- source_labels: [__meta_kubernetes_service_name]separator: ;regex: (.*)target_label: servicereplacement: $1action: replace- source_labels: [__meta_kubernetes_pod_name]separator: ;regex: (.*)target_label: podreplacement: $1action: replace- source_labels: [__meta_kubernetes_service_name]separator: ;regex: (.*)target_label: jobreplacement: ${1}action: replace- separator: ;regex: (.*)target_label: endpointreplacement: metricsaction: replacekubernetes_sd_configs:- api_server: https://kube-master-bjzyx-public-staging02.cloud.qiyi.domain:6443role: endpointsbearer_token_file: /var/k8s-auth/tokentls_config:ca_file: /var/k8s-auth/ca.crtinsecure_skip_verify: truenamespaces:names:- qke-generic-hubble-manager
- job_name: qke-generic-hubble-manager/hubble-biz-aiops-sm/0honor_timestamps: truescrape_interval: 30sscrape_timeout: 10smetrics_path: /prometheusscheme: httpbearer_token_file: /var/run/secrets/kubernetes.io/serviceaccount/tokenrelabel_configs:- source_labels: [__meta_kubernetes_service_label_app]separator: ;regex: s-biz-aiopsreplacement: $1action: keep- source_labels: [__meta_kubernetes_endpoint_port_name]separator: ;regex: metricsreplacement: $1action: keep- source_labels: [__meta_kubernetes_endpoint_address_target_kind, __meta_kubernetes_endpoint_address_target_name]separator: ;regex: Node;(.*)target_label: nodereplacement: ${1}action: replace- source_labels: [__meta_kubernetes_endpoint_address_target_kind, __meta_kubernetes_endpoint_address_target_name]separator: ;regex: Pod;(.*)target_label: podreplacement: ${1}action: replace- source_labels: [__meta_kubernetes_namespace]separator: ;regex: (.*)target_label: namespacereplacement: $1action: replace- source_labels: [__meta_kubernetes_service_name]separator: ;regex: (.*)target_label: servicereplacement: $1action: replace- source_labels: [__meta_kubernetes_pod_name]separator: ;regex: (.*)target_label: podreplacement: $1action: replace- source_labels: [__meta_kubernetes_service_name]separator: ;regex: (.*)target_label: jobreplacement: ${1}action: replace- separator: ;regex: (.*)target_label: endpointreplacement: metricsaction: replacekubernetes_sd_configs:- api_server: https://kube-master-bjzyx-public-staging02.cloud.qiyi.domain:6443role: endpointsbearer_token_file: /var/k8s-auth/tokentls_config:ca_file: /var/k8s-auth/ca.crtinsecure_skip_verify: truenamespaces:names:- qke-generic-hubble-manager
- job_name: qke-generic-hubble-manager/hubble-biz-cm-sm/0honor_timestamps: truescrape_interval: 30sscrape_timeout: 10smetrics_path: /prometheusscheme: httpbearer_token_file: /var/run/secrets/kubernetes.io/serviceaccount/tokenrelabel_configs:- source_labels: [__meta_kubernetes_service_label_app]separator: ;regex: s-biz-cmreplacement: $1action: keep- source_labels: [__meta_kubernetes_endpoint_port_name]separator: ;regex: metricsreplacement: $1action: keep- source_labels: [__meta_kubernetes_endpoint_address_target_kind, __meta_kubernetes_endpoint_address_target_name]separator: ;regex: Node;(.*)target_label: nodereplacement: ${1}action: replace- source_labels: [__meta_kubernetes_endpoint_address_target_kind, __meta_kubernetes_endpoint_address_target_name]separator: ;regex: Pod;(.*)target_label: podreplacement: ${1}action: replace- source_labels: [__meta_kubernetes_namespace]separator: ;regex: (.*)target_label: namespacereplacement: $1action: replace- source_labels: [__meta_kubernetes_service_name]separator: ;regex: (.*)target_label: servicereplacement: $1action: replace- source_labels: [__meta_kubernetes_pod_name]separator: ;regex: (.*)target_label: podreplacement: $1action: replace- source_labels: [__meta_kubernetes_service_name]separator: ;regex: (.*)target_label: jobreplacement: ${1}action: replace- separator: ;regex: (.*)target_label: endpointreplacement: metricsaction: replacekubernetes_sd_configs:- api_server: https://kube-master-bjzyx-public-staging02.cloud.qiyi.domain:6443role: endpointsbearer_token_file: /var/k8s-auth/tokentls_config:ca_file: /var/k8s-auth/ca.crtinsecure_skip_verify: truenamespaces:names:- qke-generic-hubble-manager
- job_name: qke-generic-hubble-manager/hubble-biz-stat-sm/0honor_timestamps: truescrape_interval: 30sscrape_timeout: 10smetrics_path: /prometheusscheme: httpbearer_token_file: /var/run/secrets/kubernetes.io/serviceaccount/tokenrelabel_configs:- source_labels: [__meta_kubernetes_service_label_app]separator: ;regex: s-biz-statreplacement: $1action: keep- source_labels: [__meta_kubernetes_endpoint_port_name]separator: ;regex: metricsreplacement: $1action: keep- source_labels: [__meta_kubernetes_endpoint_address_target_kind, __meta_kubernetes_endpoint_address_target_name]separator: ;regex: Node;(.*)target_label: nodereplacement: ${1}action: replace- source_labels: [__meta_kubernetes_endpoint_address_target_kind, __meta_kubernetes_endpoint_address_target_name]separator: ;regex: Pod;(.*)target_label: podreplacement: ${1}action: replace- source_labels: [__meta_kubernetes_namespace]separator: ;regex: (.*)target_label: namespacereplacement: $1action: replace- source_labels: [__meta_kubernetes_service_name]separator: ;regex: (.*)target_label: servicereplacement: $1action: replace- source_labels: [__meta_kubernetes_pod_name]separator: ;regex: (.*)target_label: podreplacement: $1action: replace- source_labels: [__meta_kubernetes_service_name]separator: ;regex: (.*)target_label: jobreplacement: ${1}action: replace- separator: ;regex: (.*)target_label: endpointreplacement: metricsaction: replacekubernetes_sd_configs:- api_server: https://kube-master-bjzyx-public-staging02.cloud.qiyi.domain:6443role: endpointsbearer_token_file: /var/k8s-auth/tokentls_config:ca_file: /var/k8s-auth/ca.crtinsecure_skip_verify: truenamespaces:names:- qke-generic-hubble-manager
- job_name: qke-generic-hubble-manager/hubble-biz-third-sm/0honor_timestamps: truescrape_interval: 30sscrape_timeout: 10smetrics_path: /prometheusscheme: httpbearer_token_file: /var/run/secrets/kubernetes.io/serviceaccount/tokenrelabel_configs:- source_labels: [__meta_kubernetes_service_label_app]separator: ;regex: m-biz-thirdreplacement: $1action: keep- source_labels: [__meta_kubernetes_endpoint_port_name]separator: ;regex: metricsreplacement: $1action: keep- source_labels: [__meta_kubernetes_endpoint_address_target_kind, __meta_kubernetes_endpoint_address_target_name]separator: ;regex: Node;(.*)target_label: nodereplacement: ${1}action: replace- source_labels: [__meta_kubernetes_endpoint_address_target_kind, __meta_kubernetes_endpoint_address_target_name]separator: ;regex: Pod;(.*)target_label: podreplacement: ${1}action: replace- source_labels: [__meta_kubernetes_namespace]separator: ;regex: (.*)target_label: namespacereplacement: $1action: replace- source_labels: [__meta_kubernetes_service_name]separator: ;regex: (.*)target_label: servicereplacement: $1action: replace- source_labels: [__meta_kubernetes_pod_name]separator: ;regex: (.*)target_label: podreplacement: $1action: replace- source_labels: [__meta_kubernetes_service_name]separator: ;regex: (.*)target_label: jobreplacement: ${1}action: replace- separator: ;regex: (.*)target_label: endpointreplacement: metricsaction: replacekubernetes_sd_configs:- api_server: https://kube-master-bjzyx-public-staging02.cloud.qiyi.domain:6443role: endpointsbearer_token_file: /var/k8s-auth/tokentls_config:ca_file: /var/k8s-auth/ca.crtinsecure_skip_verify: truenamespaces:names:- qke-generic-hubble-manager
- job_name: qke-generic-hubble-manager/hubble-task-sm/0honor_timestamps: truescrape_interval: 30sscrape_timeout: 10smetrics_path: /prometheusscheme: httpbearer_token_file: /var/run/secrets/kubernetes.io/serviceaccount/tokenrelabel_configs:- source_labels: [__meta_kubernetes_service_label_app]separator: ;regex: s-hubble-taskreplacement: $1action: keep- source_labels: [__meta_kubernetes_endpoint_port_name]separator: ;regex: metricsreplacement: $1action: keep- source_labels: [__meta_kubernetes_endpoint_address_target_kind, __meta_kubernetes_endpoint_address_target_name]separator: ;regex: Node;(.*)target_label: nodereplacement: ${1}action: replace- source_labels: [__meta_kubernetes_endpoint_address_target_kind, __meta_kubernetes_endpoint_address_target_name]separator: ;regex: Pod;(.*)target_label: podreplacement: ${1}action: replace- source_labels: [__meta_kubernetes_namespace]separator: ;regex: (.*)target_label: namespacereplacement: $1action: replace- source_labels: [__meta_kubernetes_service_name]separator: ;regex: (.*)target_label: servicereplacement: $1action: replace- source_labels: [__meta_kubernetes_pod_name]separator: ;regex: (.*)target_label: podreplacement: $1action: replace- source_labels: [__meta_kubernetes_service_name]separator: ;regex: (.*)target_label: jobreplacement: ${1}action: replace- separator: ;regex: (.*)target_label: endpointreplacement: metricsaction: replacekubernetes_sd_configs:- api_server: https://kube-master-bjzyx-public-staging02.cloud.qiyi.domain:6443role: endpointsbearer_token_file: /var/k8s-auth/tokentls_config:ca_file: /var/k8s-auth/ca.crtinsecure_skip_verify: truenamespaces:names:- qke-generic-hubble-manager
- job_name: qke-generic-hubble-manager/hubble-transfer-sm/0honor_timestamps: truescrape_interval: 30sscrape_timeout: 10smetrics_path: /prometheusscheme: httpbearer_token_file: /var/run/secrets/kubernetes.io/serviceaccount/tokenrelabel_configs:- source_labels: [__meta_kubernetes_service_label_app]separator: ;regex: s-hubble-transferreplacement: $1action: keep- source_labels: [__meta_kubernetes_endpoint_port_name]separator: ;regex: m-hubble-transferreplacement: $1action: keep- source_labels: [__meta_kubernetes_endpoint_address_target_kind, __meta_kubernetes_endpoint_address_target_name]separator: ;regex: Node;(.*)target_label: nodereplacement: ${1}action: replace- source_labels: [__meta_kubernetes_endpoint_address_target_kind, __meta_kubernetes_endpoint_address_target_name]separator: ;regex: Pod;(.*)target_label: podreplacement: ${1}action: replace- source_labels: [__meta_kubernetes_namespace]separator: ;regex: (.*)target_label: namespacereplacement: $1action: replace- source_labels: [__meta_kubernetes_service_name]separator: ;regex: (.*)target_label: servicereplacement: $1action: replace- source_labels: [__meta_kubernetes_pod_name]separator: ;regex: (.*)target_label: podreplacement: $1action: replace- source_labels: [__meta_kubernetes_service_name]separator: ;regex: (.*)target_label: jobreplacement: ${1}action: replace- separator: ;regex: (.*)target_label: endpointreplacement: m-hubble-transferaction: replacekubernetes_sd_configs:- api_server: https://kube-master-bjzyx-public-staging02.cloud.qiyi.domain:6443role: endpointsbearer_token_file: /var/k8s-auth/tokentls_config:ca_file: /var/k8s-auth/ca.crtinsecure_skip_verify: truenamespaces:names:- qke-generic-hubble-manager
- job_name: qke-generic-hubble-manager/network-screen-sm/0honor_timestamps: truescrape_interval: 30sscrape_timeout: 10smetrics_path: /prometheusscheme: httpbearer_token_file: /var/run/secrets/kubernetes.io/serviceaccount/tokenrelabel_configs:- source_labels: [__meta_kubernetes_service_label_app]separator: ;regex: s-network-screenreplacement: $1action: keep- source_labels: [__meta_kubernetes_endpoint_port_name]separator: ;regex: metricsreplacement: $1action: keep- source_labels: [__meta_kubernetes_endpoint_address_target_kind, __meta_kubernetes_endpoint_address_target_name]separator: ;regex: Node;(.*)target_label: nodereplacement: ${1}action: replace- source_labels: [__meta_kubernetes_endpoint_address_target_kind, __meta_kubernetes_endpoint_address_target_name]separator: ;regex: Pod;(.*)target_label: podreplacement: ${1}action: replace- source_labels: [__meta_kubernetes_namespace]separator: ;regex: (.*)target_label: namespacereplacement: $1action: replace- source_labels: [__meta_kubernetes_service_name]separator: ;regex: (.*)target_label: servicereplacement: $1action: replace- source_labels: [__meta_kubernetes_pod_name]separator: ;regex: (.*)target_label: podreplacement: $1action: replace- source_labels: [__meta_kubernetes_service_name]separator: ;regex: (.*)target_label: jobreplacement: ${1}action: replace- separator: ;regex: (.*)target_label: endpointreplacement: metricsaction: replacekubernetes_sd_configs:- api_server: https://kube-master-bjzyx-public-staging02.cloud.qiyi.domain:6443role: endpointsbearer_token_file: /var/k8s-auth/tokentls_config:ca_file: /var/k8s-auth/ca.crtinsecure_skip_verify: truenamespaces:names:- qke-generic-hubble-manager
- job_name: qke-generic-hubble-p-hbs/hubble-p-hbs-svcm/0honor_timestamps: truescrape_interval: 10sscrape_timeout: 10smetrics_path: /json2metricsscheme: httprelabel_configs:- source_labels: [__meta_kubernetes_service_label_app]separator: ;regex: hubble-p-hbsreplacement: $1action: keep- source_labels: [__meta_kubernetes_endpoint_port_name]separator: ;regex: exporter-metricsreplacement: $1action: keep- source_labels: [__meta_kubernetes_endpoint_address_target_kind, __meta_kubernetes_endpoint_address_target_name]separator: ;regex: Node;(.*)target_label: nodereplacement: ${1}action: replace- source_labels: [__meta_kubernetes_endpoint_address_target_kind, __meta_kubernetes_endpoint_address_target_name]separator: ;regex: Pod;(.*)target_label: podreplacement: ${1}action: replace- source_labels: [__meta_kubernetes_namespace]separator: ;regex: (.*)target_label: namespacereplacement: $1action: replace- source_labels: [__meta_kubernetes_service_name]separator: ;regex: (.*)target_label: servicereplacement: $1action: replace- source_labels: [__meta_kubernetes_pod_name]separator: ;regex: (.*)target_label: podreplacement: $1action: replace- source_labels: [__meta_kubernetes_service_name]separator: ;regex: (.*)target_label: jobreplacement: ${1}action: replace- separator: ;regex: (.*)target_label: endpointreplacement: exporter-metricsaction: replacemetric_relabel_configs:- source_labels: [__name__]separator: ;regex: hubble_p_hbs_(.*)replacement: $1action: keepkubernetes_sd_configs:- api_server: https://kube-master-bjzyx-public-staging02.cloud.qiyi.domain:6443role: endpointsbearer_token_file: /var/k8s-auth/tokentls_config:ca_file: /var/k8s-auth/ca.crtinsecure_skip_verify: truenamespaces:names:- qke-generic-hubble-p-hbs
- job_name: qke-generic-hubble-p-query/hubble-p-query-svcm/0honor_timestamps: truescrape_interval: 10sscrape_timeout: 10smetrics_path: /json2metricsscheme: httprelabel_configs:- source_labels: [__meta_kubernetes_service_label_app]separator: ;regex: hubble-p-queryreplacement: $1action: keep- source_labels: [__meta_kubernetes_endpoint_port_name]separator: ;regex: exporter-metricsreplacement: $1action: keep- source_labels: [__meta_kubernetes_endpoint_address_target_kind, __meta_kubernetes_endpoint_address_target_name]separator: ;regex: Node;(.*)target_label: nodereplacement: ${1}action: replace- source_labels: [__meta_kubernetes_endpoint_address_target_kind, __meta_kubernetes_endpoint_address_target_name]separator: ;regex: Pod;(.*)target_label: podreplacement: ${1}action: replace- source_labels: [__meta_kubernetes_namespace]separator: ;regex: (.*)target_label: namespacereplacement: $1action: replace- source_labels: [__meta_kubernetes_service_name]separator: ;regex: (.*)target_label: servicereplacement: $1action: replace- source_labels: [__meta_kubernetes_pod_name]separator: ;regex: (.*)target_label: podreplacement: $1action: replace- source_labels: [__meta_kubernetes_service_name]separator: ;regex: (.*)target_label: jobreplacement: ${1}action: replace- separator: ;regex: (.*)target_label: endpointreplacement: exporter-metricsaction: replacemetric_relabel_configs:- source_labels: [__name__]separator: ;regex: hubble_p_query_(.*)replacement: $1action: keepkubernetes_sd_configs:- api_server: https://kube-master-bjzyx-public-staging02.cloud.qiyi.domain:6443role: endpointsbearer_token_file: /var/k8s-auth/tokentls_config:ca_file: /var/k8s-auth/ca.crtinsecure_skip_verify: truenamespaces:names:- qke-generic-hubble-p-query
- job_name: qke-generic-hubble-p-transfer/hubble-p-transfer-svcm/0honor_timestamps: truescrape_interval: 10sscrape_timeout: 10smetrics_path: /json2metricsscheme: httprelabel_configs:- source_labels: [__meta_kubernetes_service_label_app]separator: ;regex: hubble-p-transferreplacement: $1action: keep- source_labels: [__meta_kubernetes_endpoint_port_name]separator: ;regex: exporter-metricsreplacement: $1action: keep- source_labels: [__meta_kubernetes_endpoint_address_target_kind, __meta_kubernetes_endpoint_address_target_name]separator: ;regex: Node;(.*)target_label: nodereplacement: ${1}action: replace- source_labels: [__meta_kubernetes_endpoint_address_target_kind, __meta_kubernetes_endpoint_address_target_name]separator: ;regex: Pod;(.*)target_label: podreplacement: ${1}action: replace- source_labels: [__meta_kubernetes_namespace]separator: ;regex: (.*)target_label: namespacereplacement: $1action: replace- source_labels: [__meta_kubernetes_service_name]separator: ;regex: (.*)target_label: servicereplacement: $1action: replace- source_labels: [__meta_kubernetes_pod_name]separator: ;regex: (.*)target_label: podreplacement: $1action: replace- source_labels: [__meta_kubernetes_service_name]separator: ;regex: (.*)target_label: jobreplacement: ${1}action: replace- separator: ;regex: (.*)target_label: endpointreplacement: exporter-metricsaction: replacemetric_relabel_configs:- source_labels: [__name__]separator: ;regex: hubble_p_transfer_(.*)replacement: $1action: keepkubernetes_sd_configs:- api_server: https://kube-master-bjzyx-public-staging02.cloud.qiyi.domain:6443role: endpointsbearer_token_file: /var/k8s-auth/tokentls_config:ca_file: /var/k8s-auth/ca.crtinsecure_skip_verify: truenamespaces:names:- qke-generic-hubble-p-transfer
- job_name: qke-generic-hubble-platform/kube-state-metrics/0honor_labels: truehonor_timestamps: truescrape_interval: 30sscrape_timeout: 30smetrics_path: /metricsscheme: httpsbearer_token_file: /var/k8s-auth/tokentls_config:insecure_skip_verify: truerelabel_configs:- source_labels: [__meta_kubernetes_service_label_k8s_app]separator: ;regex: kube-state-metricsreplacement: $1action: keep- source_labels: [__meta_kubernetes_endpoint_port_name]separator: ;regex: https-mainreplacement: $1action: keep- source_labels: [__meta_kubernetes_endpoint_address_target_kind, __meta_kubernetes_endpoint_address_target_name]separator: ;regex: Node;(.*)target_label: nodereplacement: ${1}action: replace- source_labels: [__meta_kubernetes_endpoint_address_target_kind, __meta_kubernetes_endpoint_address_target_name]separator: ;regex: Pod;(.*)target_label: podreplacement: ${1}action: replace- source_labels: [__meta_kubernetes_namespace]separator: ;regex: (.*)target_label: namespacereplacement: $1action: replace- source_labels: [__meta_kubernetes_service_name]separator: ;regex: (.*)target_label: servicereplacement: $1action: replace- source_labels: [__meta_kubernetes_pod_name]separator: ;regex: (.*)target_label: podreplacement: $1action: replace- source_labels: [__meta_kubernetes_service_name]separator: ;regex: (.*)target_label: jobreplacement: ${1}action: replace- source_labels: [__meta_kubernetes_service_label_k8s_app]separator: ;regex: (.+)target_label: jobreplacement: ${1}action: replace- separator: ;regex: (.*)target_label: endpointreplacement: https-mainaction: replace- separator: ;regex: (pod|service|endpoint|namespace)replacement: $1action: labeldropmetric_relabel_configs:- source_labels: [namespace]separator: ;regex: (qke-generic-hubble-platform|qke-generic-hubble-p-updater-server|qke-generic-hubble-grafana-dashboard|qke-generic-hubble-p-transfer|qke-generic-hubble-grafana-api|qke-generic-hubble-aiops|qke-generic-hubble-p-hbs|qke-generic-hubble-p-query|qke-generic-hubble-manager|qke-generic-hubble-self-monitor)replacement: $1action: keepkubernetes_sd_configs:- api_server: https://kube-master-bjzyx-public-staging02.cloud.qiyi.domain:6443role: endpointsbearer_token_file: /var/k8s-auth/tokentls_config:ca_file: /var/k8s-auth/ca.crtinsecure_skip_verify: truenamespaces:names:- monitoring
- job_name: qke-generic-hubble-platform/kubelet/0honor_labels: truehonor_timestamps: truescrape_interval: 30sscrape_timeout: 30smetrics_path: /metrics/cadvisorscheme: httpsbearer_token_file: /var/k8s-auth/tokentls_config:insecure_skip_verify: truerelabel_configs:- source_labels: [__meta_kubernetes_service_label_k8s_app]separator: ;regex: kubeletreplacement: $1action: keep- source_labels: [__meta_kubernetes_endpoint_port_name]separator: ;regex: https-metricsreplacement: $1action: keep- source_labels: [__meta_kubernetes_endpoint_address_target_kind, __meta_kubernetes_endpoint_address_target_name]separator: ;regex: Node;(.*)target_label: nodereplacement: ${1}action: replace- source_labels: [__meta_kubernetes_endpoint_address_target_kind, __meta_kubernetes_endpoint_address_target_name]separator: ;regex: Pod;(.*)target_label: podreplacement: ${1}action: replace- source_labels: [__meta_kubernetes_namespace]separator: ;regex: (.*)target_label: namespacereplacement: $1action: replace- source_labels: [__meta_kubernetes_service_name]separator: ;regex: (.*)target_label: servicereplacement: $1action: replace- source_labels: [__meta_kubernetes_pod_name]separator: ;regex: (.*)target_label: podreplacement: $1action: replace- source_labels: [__meta_kubernetes_service_name]separator: ;regex: (.*)target_label: jobreplacement: ${1}action: replace- source_labels: [__meta_kubernetes_service_label_k8s_app]separator: ;regex: (.+)target_label: jobreplacement: ${1}action: replace- separator: ;regex: (.*)target_label: endpointreplacement: https-metricsaction: replace- source_labels: [__metrics_path__]separator: ;regex: (.*)target_label: metrics_pathreplacement: $1action: replacemetric_relabel_configs:- source_labels: [namespace]separator: ;regex: (qke-generic-hubble-platform|qke-generic-hubble-p-updater-server|qke-generic-hubble-grafana-dashboard|qke-generic-hubble-p-transfer|qke-generic-hubble-grafana-api|qke-generic-hubble-aiops|qke-generic-hubble-p-hbs|qke-generic-hubble-p-query|qke-generic-hubble-manager|qke-generic-hubble-self-monitor)replacement: $1action: keepkubernetes_sd_configs:- api_server: https://kube-master-bjzyx-public-staging02.cloud.qiyi.domain:6443role: endpointsbearer_token_file: /var/k8s-auth/tokentls_config:ca_file: /var/k8s-auth/ca.crtinsecure_skip_verify: truenamespaces:names:- kube-system
- job_name: qke-generic-hubble-self-monitor/hubble-p-transfer-svcm/0honor_timestamps: truescrape_interval: 10sscrape_timeout: 10smetrics_path: /json2metricsscheme: httprelabel_configs:- source_labels: [__meta_kubernetes_service_label_app]separator: ;regex: hubble-p-transferreplacement: $1action: keep- source_labels: [__meta_kubernetes_endpoint_port_name]separator: ;regex: exporter-metricsreplacement: $1action: keep- source_labels: [__meta_kubernetes_endpoint_address_target_kind, __meta_kubernetes_endpoint_address_target_name]separator: ;regex: Node;(.*)target_label: nodereplacement: ${1}action: replace- source_labels: [__meta_kubernetes_endpoint_address_target_kind, __meta_kubernetes_endpoint_address_target_name]separator: ;regex: Pod;(.*)target_label: podreplacement: ${1}action: replace- source_labels: [__meta_kubernetes_namespace]separator: ;regex: (.*)target_label: namespacereplacement: $1action: replace- source_labels: [__meta_kubernetes_service_name]separator: ;regex: (.*)target_label: servicereplacement: $1action: replace- source_labels: [__meta_kubernetes_pod_name]separator: ;regex: (.*)target_label: podreplacement: $1action: replace- source_labels: [__meta_kubernetes_service_name]separator: ;regex: (.*)target_label: jobreplacement: ${1}action: replace- separator: ;regex: (.*)target_label: endpointreplacement: exporter-metricsaction: replacemetric_relabel_configs:- source_labels: [__name__]separator: ;regex: hubble_p_transfer_(.*)replacement: $1action: keepkubernetes_sd_configs:- api_server: https://kube-master-bjzyx-public-staging02.cloud.qiyi.domain:6443role: endpointsbearer_token_file: /var/k8s-auth/tokentls_config:ca_file: /var/k8s-auth/ca.crtinsecure_skip_verify: truenamespaces:names:- qke-generic-hubble-self-monitor
- job_name: qke-generic-hubble-platform/knative-activator/0honor_labels: truehonor_timestamps: truescrape_interval: 30sscrape_timeout: 10smetrics_path: /metricsscheme: httprelabel_configs:- source_labels: [__meta_kubernetes_pod_label_app]separator: ;regex: activatorreplacement: $1action: keep- source_labels: [__meta_kubernetes_namespace]separator: ;regex: (.*)target_label: namespacereplacement: $1action: replace- source_labels: [__meta_kubernetes_pod_container_name]separator: ;regex: (.*)target_label: containerreplacement: $1action: replace- source_labels: [__meta_kubernetes_pod_name]separator: ;regex: (.*)target_label: podreplacement: $1action: replace- separator: ;regex: (.*)target_label: jobreplacement: qke-generic-hubble-platform/knative-activatoraction: replace- source_labels: [__meta_kubernetes_pod_label_knative_activator]separator: ;regex: (.+)target_label: jobreplacement: ${1}action: replace- source_labels: [__meta_kubernetes_namespace, __meta_kubernetes_pod_label_app, __meta_kubernetes_pod_container_port_name]separator: ;regex: knative-serving;activator;metricsreplacement: $1action: keep- source_labels: [__meta_kubernetes_namespace]separator: ;regex: (.*)target_label: namespacereplacement: $1action: replace- source_labels: [__meta_kubernetes_pod_name]separator: ;regex: (.*)target_label: podreplacement: $1action: replace- source_labels: [__meta_kubernetes_service_name]separator: ;regex: (.*)target_label: servicereplacement: $1action: replacekubernetes_sd_configs:- api_server: https://kube-master-bjzyx-public-staging02.cloud.qiyi.domain:6443role: podbearer_token_file: /var/k8s-auth/tokentls_config:ca_file: /var/k8s-auth/ca.crtinsecure_skip_verify: truenamespaces:names:- knative-serving
- job_name: qke-generic-hubble-platform/knative-autoscaler/0honor_labels: truehonor_timestamps: truescrape_interval: 30sscrape_timeout: 10smetrics_path: /metricsscheme: httprelabel_configs:- source_labels: [__meta_kubernetes_pod_label_app]separator: ;regex: autoscalerreplacement: $1action: keep- source_labels: [__meta_kubernetes_namespace]separator: ;regex: (.*)target_label: namespacereplacement: $1action: replace- source_labels: [__meta_kubernetes_pod_container_name]separator: ;regex: (.*)target_label: containerreplacement: $1action: replace- source_labels: [__meta_kubernetes_pod_name]separator: ;regex: (.*)target_label: podreplacement: $1action: replace- separator: ;regex: (.*)target_label: jobreplacement: qke-generic-hubble-platform/knative-autoscaleraction: replace- source_labels: [__meta_kubernetes_pod_label_knative_autoscaler]separator: ;regex: (.+)target_label: jobreplacement: ${1}action: replace- source_labels: [__meta_kubernetes_namespace, __meta_kubernetes_pod_label_app, __meta_kubernetes_pod_container_port_name]separator: ;regex: knative-serving;autoscaler;metricsreplacement: $1action: keep- source_labels: [__meta_kubernetes_namespace]separator: ;regex: (.*)target_label: namespacereplacement: $1action: replace- source_labels: [__meta_kubernetes_pod_name]separator: ;regex: (.*)target_label: podreplacement: $1action: replace- source_labels: [__meta_kubernetes_service_name]separator: ;regex: (.*)target_label: servicereplacement: $1action: replacekubernetes_sd_configs:- api_server: https://kube-master-bjzyx-public-staging02.cloud.qiyi.domain:6443role: podbearer_token_file: /var/k8s-auth/tokentls_config:ca_file: /var/k8s-auth/ca.crtinsecure_skip_verify: truenamespaces:names:- knative-serving
- job_name: qke-generic-hubble-platform/knative-controller/0honor_labels: truehonor_timestamps: truescrape_interval: 30sscrape_timeout: 10smetrics_path: /metricsscheme: httprelabel_configs:- source_labels: [__meta_kubernetes_pod_label_app]separator: ;regex: controllerreplacement: $1action: keep- source_labels: [__meta_kubernetes_namespace]separator: ;regex: (.*)target_label: namespacereplacement: $1action: replace- source_labels: [__meta_kubernetes_pod_container_name]separator: ;regex: (.*)target_label: containerreplacement: $1action: replace- source_labels: [__meta_kubernetes_pod_name]separator: ;regex: (.*)target_label: podreplacement: $1action: replace- separator: ;regex: (.*)target_label: jobreplacement: qke-generic-hubble-platform/knative-controlleraction: replace- source_labels: [__meta_kubernetes_pod_label_knative_controller_app]separator: ;regex: (.+)target_label: jobreplacement: ${1}action: replace- source_labels: [__meta_kubernetes_namespace, __meta_kubernetes_pod_label_app, __meta_kubernetes_pod_container_port_name]separator: ;regex: knative-serving;controller;metricsreplacement: $1action: keep- source_labels: [__meta_kubernetes_namespace]separator: ;regex: (.*)target_label: namespacereplacement: $1action: replace- source_labels: [__meta_kubernetes_pod_name]separator: ;regex: (.*)target_label: podreplacement: $1action: replace- source_labels: [__meta_kubernetes_service_name]separator: ;regex: (.*)target_label: servicereplacement: $1action: replacekubernetes_sd_configs:- api_server: https://kube-master-bjzyx-public-staging02.cloud.qiyi.domain:6443role: podbearer_token_file: /var/k8s-auth/tokentls_config:ca_file: /var/k8s-auth/ca.crtinsecure_skip_verify: truenamespaces:names:- knative-serving
- job_name: qke-generic-hubble-platform/knative-queue-proxy/0honor_labels: truehonor_timestamps: truescrape_interval: 30sscrape_timeout: 10smetrics_path: /metricsscheme: httprelabel_configs:- source_labels: [__meta_kubernetes_pod_label_qke_cloud_qiyi_domain_lite]separator: ;regex: "true"replacement: $1action: keep- source_labels: [__meta_kubernetes_namespace]separator: ;regex: (.*)target_label: namespacereplacement: $1action: replace- source_labels: [__meta_kubernetes_pod_container_name]separator: ;regex: (.*)target_label: containerreplacement: $1action: replace- source_labels: [__meta_kubernetes_pod_name]separator: ;regex: (.*)target_label: podreplacement: $1action: replace- separator: ;regex: (.*)target_label: jobreplacement: qke-generic-hubble-platform/knative-queue-proxyaction: replace- source_labels: [__meta_kubernetes_pod_label_knative_queue_proxy]separator: ;regex: (.+)target_label: jobreplacement: ${1}action: replace- source_labels: [__meta_kubernetes_pod_label_serving_knative_dev_revision, __meta_kubernetes_pod_container_port_name]separator: ;regex: .+;http-usermetricreplacement: $1action: keep- source_labels: [__meta_kubernetes_namespace]separator: ;regex: (.*)target_label: namespacereplacement: $1action: replace- source_labels: [__meta_kubernetes_pod_name]separator: ;regex: (.*)target_label: podreplacement: $1action: replace- source_labels: [__meta_kubernetes_service_name]separator: ;regex: (.*)target_label: servicereplacement: $1action: replacekubernetes_sd_configs:- api_server: https://kube-master-bjzyx-public-staging02.cloud.qiyi.domain:6443role: podbearer_token_file: /var/k8s-auth/tokentls_config:ca_file: /var/k8s-auth/ca.crtinsecure_skip_verify: truenamespaces:names:- qke-generic-hubble-platform- qke-generic-hubble-p-updater-server- qke-generic-hubble-grafana-dashboard- qke-generic-hubble-p-transfer- qke-generic-hubble-grafana-api- qke-generic-hubble-aiops- qke-generic-hubble-p-hbs- qke-generic-hubble-p-query- qke-generic-hubble-manager- qke-generic-hubble-self-monitor
- job_name: qke-generic-hubble-platform/knative-queue-proxy/1honor_labels: truehonor_timestamps: truescrape_interval: 30sscrape_timeout: 10smetrics_path: /metricsscheme: httprelabel_configs:- source_labels: [__meta_kubernetes_pod_label_qke_cloud_qiyi_domain_lite]separator: ;regex: "true"replacement: $1action: keep- source_labels: [__meta_kubernetes_namespace]separator: ;regex: (.*)target_label: namespacereplacement: $1action: replace- source_labels: [__meta_kubernetes_pod_container_name]separator: ;regex: (.*)target_label: containerreplacement: $1action: replace- source_labels: [__meta_kubernetes_pod_name]separator: ;regex: (.*)target_label: podreplacement: $1action: replace- separator: ;regex: (.*)target_label: jobreplacement: qke-generic-hubble-platform/knative-queue-proxyaction: replace- source_labels: [__meta_kubernetes_pod_label_knative_queue_proxy]separator: ;regex: (.+)target_label: jobreplacement: ${1}action: replace- source_labels: [__meta_kubernetes_pod_label_serving_knative_dev_revision, __meta_kubernetes_pod_container_port_name]separator: ;regex: .+;http-autometricreplacement: $1action: keep- source_labels: [__meta_kubernetes_namespace]separator: ;regex: (.*)target_label: namespacereplacement: $1action: replace- source_labels: [__meta_kubernetes_pod_name]separator: ;regex: (.*)target_label: podreplacement: $1action: replace- source_labels: [__meta_kubernetes_service_name]separator: ;regex: (.*)target_label: servicereplacement: $1action: replacekubernetes_sd_configs:- api_server: https://kube-master-bjzyx-public-staging02.cloud.qiyi.domain:6443role: podbearer_token_file: /var/k8s-auth/tokentls_config:ca_file: /var/k8s-auth/ca.crtinsecure_skip_verify: truenamespaces:names:- qke-generic-hubble-platform- qke-generic-hubble-p-updater-server- qke-generic-hubble-grafana-dashboard- qke-generic-hubble-p-transfer- qke-generic-hubble-grafana-api- qke-generic-hubble-aiops- qke-generic-hubble-p-hbs- qke-generic-hubble-p-query- qke-generic-hubble-manager- qke-generic-hubble-self-monitor
- job_name: qke-generic-hubble-platform/knative-webhook/0honor_labels: truehonor_timestamps: truescrape_interval: 30sscrape_timeout: 10smetrics_path: /metricsscheme: httprelabel_configs:- source_labels: [__meta_kubernetes_pod_label_app]separator: ;regex: webhookreplacement: $1action: keep- source_labels: [__meta_kubernetes_namespace]separator: ;regex: (.*)target_label: namespacereplacement: $1action: replace- source_labels: [__meta_kubernetes_pod_container_name]separator: ;regex: (.*)target_label: containerreplacement: $1action: replace- source_labels: [__meta_kubernetes_pod_name]separator: ;regex: (.*)target_label: podreplacement: $1action: replace- separator: ;regex: (.*)target_label: jobreplacement: qke-generic-hubble-platform/knative-webhookaction: replace- source_labels: [__meta_kubernetes_pod_label_knative_webhook]separator: ;regex: (.+)target_label: jobreplacement: ${1}action: replace- source_labels: [__meta_kubernetes_namespace, __meta_kubernetes_pod_label_app, __meta_kubernetes_pod_container_port_name]separator: ;regex: knative-serving;webhook;metricsreplacement: $1action: keep- source_labels: [__meta_kubernetes_namespace]separator: ;regex: (.*)target_label: namespacereplacement: $1action: replace- source_labels: [__meta_kubernetes_pod_name]separator: ;regex: (.*)target_label: podreplacement: $1action: replace- source_labels: [__meta_kubernetes_service_name]separator: ;regex: (.*)target_label: servicereplacement: $1action: replacekubernetes_sd_configs:- api_server: https://kube-master-bjzyx-public-staging02.cloud.qiyi.domain:6443role: podbearer_token_file: /var/k8s-auth/tokentls_config:ca_file: /var/k8s-auth/ca.crtinsecure_skip_verify: truenamespaces:names:- knative-serving

全链路配置案例：

global:
evaluation_interval: 60s
external_labels:
prometheus_replica: $(POD_NAME)
scrape_interval: 60s
alerting:
alert_relabel_configs:
- action: labeldrop
regex: prometheus_replica
alertmanagers:
- scheme: http
static_configs:
- targets:
- alertmanager.prom-alert.svc:9093
rule_files:
- /etc/prometheus/rules/*rules.yaml
scrape_configs:
- job_name: 48e5a419-4721-5921-899c-86aa7122dfb6-nacos-adapter
honor_timestamps: true
scrape_interval: 30s
scrape_timeout: 30s
metrics_path: /prometheus
consul_sd_configs:
- server: http://laputa.prometheus-nacos-adapter.online.qiyi.qae
services:
- 21c7187d-c748-4fc9-916e-0c270f0509ee@@hubble@@hubble-query
relabel_configs:
- source_labels: [__meta_consul_service_id]
target_label: instance
- regex: __meta_consul_service_metadata_(.+)
action: labelmap
- job_name: 5eef17ab-f6b6-5d79-942b-ac41d35ba870-nacos-adapter
honor_timestamps: true
scrape_interval: 30s
scrape_timeout: 30s
metrics_path: /prometheus
consul_sd_configs:
- server: http://laputa.prometheus-nacos-adapter.online.qiyi.qae
services:
- 21c7187d-c748-4fc9-916e-0c270f0509ee@@hubble@@hubble-notice
relabel_configs:
- source_labels: [__meta_consul_service_id]
target_label: instance
- regex: __meta_consul_service_metadata_(.+)
action: labelmap
- job_name: 5f476f71-29c9-527e-b765-58b30c425751-nacos-adapter
honor_timestamps: true
scrape_interval: 30s
scrape_timeout: 30s
metrics_path: /prometheus
consul_sd_configs:
- server: http://laputa.prometheus-nacos-adapter.online.qiyi.qae
services:
- 21c7187d-c748-4fc9-916e-0c270f0509ee@@hubble@@hubble-biz-bq-alarm-event
relabel_configs:
- source_labels: [__meta_consul_service_id]
target_label: instance
- regex: __meta_consul_service_metadata_(.+)
action: labelmap
- job_name: 79f80b60-0ced-55b5-a439-504b15a620ce-nacos-adapter
honor_timestamps: true
scrape_interval: 30s
scrape_timeout: 30s
metrics_path: /prometheus
consul_sd_configs:
- server: http://laputa.prometheus-nacos-adapter.online.qiyi.qae
services:
- 21c7187d-c748-4fc9-916e-0c270f0509ee@@hubble@@hubble-biz-alarm-storm
relabel_configs:
- source_labels: [__meta_consul_service_id]
target_label: instance
- regex: __meta_consul_service_metadata_(.+)
action: labelmap
- job_name: 441b1a98-6c0c-5d8c-970d-425d2c1d412e-nacos-adapter
honor_timestamps: true
scrape_interval: 30s
scrape_timeout: 30s
metrics_path: /prometheus
consul_sd_configs:
- server: http://laputa.prometheus-nacos-adapter.online.qiyi.qae
services:
- 21c7187d-c748-4fc9-916e-0c270f0509ee@@hubble@@hubble-transfer
relabel_configs:
- source_labels: [__meta_consul_service_id]
target_label: instance
- regex: __meta_consul_service_metadata_(.+)
action: labelmap
- job_name: 5b5d4de9-2c2e-579b-8854-8b4942b09e5e-nacos-adapter
honor_timestamps: true
scrape_interval: 30s
scrape_timeout: 30s
metrics_path: /prometheus
consul_sd_configs:
- server: http://laputa.prometheus-nacos-adapter.online.qiyi.qae
services:
- 21c7187d-c748-4fc9-916e-0c270f0509ee@@hubble@@hubble-biz-bq-alarm-query
relabel_configs:
- source_labels: [__meta_consul_service_id]
target_label: instance
- regex: __meta_consul_service_metadata_(.+)
action: labelmap
- job_name: dfbfbb26-9632-5200-9b91-7e989c43969d-nacos-adapter
honor_timestamps: true
scrape_interval: 30s
scrape_timeout: 30s
metrics_path: /prometheus
consul_sd_configs:
- server: http://laputa.prometheus-nacos-adapter.online.qiyi.qae
services:
- 21c7187d-c748-4fc9-916e-0c270f0509ee@@hubble@@hubble-biz-alarm-query
relabel_configs:
- source_labels: [__meta_consul_service_id]
target_label: instance
- regex: __meta_consul_service_metadata_(.+)
action: labelmap
- job_name: f6dc47d3-7e74-5ab4-8502-4b2fe9cb8123-nacos-adapter
honor_timestamps: true
scrape_interval: 30s
scrape_timeout: 30s
metrics_path: /prometheus
consul_sd_configs:
- server: http://laputa.prometheus-nacos-adapter.online.qiyi.qae
services:
- 21c7187d-c748-4fc9-916e-0c270f0509ee@@hubble@@hubble-biz-cm
relabel_configs:
- source_labels: [__meta_consul_service_id]
target_label: instance
- regex: __meta_consul_service_metadata_(.+)
action: labelmap
- job_name: 24e69042-7557-5860-913e-9c7eeab76660-nacos-adapter
honor_timestamps: true
scrape_interval: 30s
scrape_timeout: 30s
metrics_path: /prometheus
consul_sd_configs:
- server: http://laputa.prometheus-nacos-adapter.online.qiyi.qae
services:
- 21c7187d-c748-4fc9-916e-0c270f0509ee@@hubble@@hubble-network-screen
relabel_configs:
- source_labels: [__meta_consul_service_id]
target_label: instance
- regex: __meta_consul_service_metadata_(.+)
action: labelmap
remote_write:
- url: http://hubble.adapter.qiyi.domain:9988/prom2hubble/push?group=hubble-test
write_relabel_configs:
- source_labels: [__name__]
regex: trace_(.*)
action: keep
- source_labels: [project, app]
separator: ':'
target_label: hubble_endpoint
replacement: prometheus_$1
- source_labels: [project]
target_label: hubble_group
- regex: (app|project|prometheus_replica)
action: labeldrop
- target_label: hubble_step
replacement: "60"
name: trace

九、告警规则案例

QKE告警规则案例：

全链路告警规则案例：

 groups:
- name: qytrace-agg.rules
rules:
- expr: |
sum(irate(http_server_requests_duration_seconds_count{env="prod"}[1m])) by (project, app, span, zone, status_code)
record: trace_span_requests_zone_code
- expr: |
sum(irate(http_server_requests_duration_seconds_count{env="prod"}[1m])) by (project, app, zone, status_code)
record: trace_service_requests_zone_code
- expr: |
avg by (project, app, zone, status_code) (sum by(instance, project, app, zone, status_code) (irate(http_server_requests_duration_seconds_count{env="prod"}[1m])))
record: trace_service_avg_requests_zone_code
- expr: |
sum(irate(http_server_requests_duration_seconds_count{env="prod",success="true"}[1m])) by (project, app, span, zone)
record: trace_span_success_requests_zone
- expr: |
sum(irate(http_server_requests_duration_seconds_count{env="prod",success="true"}[1m])) by (project, app, zone)
record: trace_service_success_requests_zone- expr: |
sum(irate(http_server_requests_duration_seconds_count{env="prod",success="true"}[1m])) by (project, app, span, zone)
/ sum(irate(http_server_requests_duration_seconds_count{env="prod",}[1m])) by (project, app, span, zone)
record: trace_span_success_rate_zone
- expr: |
sum(irate(http_server_requests_duration_seconds_count{env="prod",success="true"}[1m])) by (project, app, zone)
/ sum(irate(http_server_requests_duration_seconds_count{env="prod",}[1m])) by (project, app, zone)
record: trace_service_success_rate_zone- expr: |
sum(irate(http_server_requests_duration_seconds_sum{env="prod",success="true"}[1m])) by (project, app, span, zone)
/ sum(irate(http_server_requests_duration_seconds_count{env="prod",}[1m])) by (project, app, span, zone)
record: trace_span_avg_latency_zone
- expr: |
sum(irate(http_server_requests_duration_seconds_sum{env="prod",success="true"}[1m])) by (project, app, zone)
/ sum(irate(http_server_requests_duration_seconds_count{env="prod",}[1m])) by (project, app, zone)
record: trace_service_avg_latency_zone- expr: |
sum(trace_span_requests_zone_code) by (project, app, span, zone)
record: trace_span_requests_zone
- expr: |
sum(trace_service_requests_zone_code) by (project, app, zone)
record: trace_service_requests_zone- expr: sum(jvm_gc_pause_seconds_count{env="prod"} - jvm_gc_pause_seconds_count{env="prod"} offset 1m) by (project, app, instance, zone)
record: trace_service_jvm_gc_cnt_zone
- expr: sum(jvm_gc_pause_seconds_sum{env="prod"} - jvm_gc_pause_seconds_sum{env="prod"} offset 1m) by (project, app, instance, zone)
record: trace_service_jvm_gc_elapsed_zone- expr: |
sum(trace_service_requests_zone_code) by (app)
record: trace_service_requests_app
- expr: |
sum(irate(http_server_requests_duration_seconds_count{env="prod",success="true"}[1m])) by (app)
/ sum(irate(http_server_requests_duration_seconds_count{env="prod",}[1m])) by (app)
record: trace_service_success_rate_app
- expr: |
sum(irate(http_server_requests_duration_seconds_sum{env="prod",success="true"}[1m])) by (app)
/ sum(irate(http_server_requests_duration_seconds_count{env="prod",}[1m])) by (app)
record: trace_service_avg_latency_app
samples-scraped-rules.yaml      groups:
- name: samples-monitoring
rules:
- alert: SamplesScrapedTotal
expr: sum(scrape_samples_scraped{}) > 3000000
for: 2m
labels:
prometheus: hubble-prod
annotations:
alertlevel: "P3"
hubblegroup: "hubble-prometheus-k8s"
alertvalue: "{{$value}}"
summary: "total samples in hubble-prod > 300w"
- alert: SamplesScrapedByJob
expr: sum by (job) (scrape_samples_scraped{}) > 1000000
for: 2m
labels:
prometheus: hubble-prod
annotations:
alertlevel: "P3"
hubblegroup: "hubble-prometheus-k8s"
alertvalue: "{{$value}}"
summary: "samples from {{$labels.job}} in hubble-prod > 100w"

参考：

Prometheus配置文件

Configuration | Prometheus

总结：Promethus配置文件相关推荐

1. Centos7安装Promethus(普罗米修斯）监控系统完整版

   相关博文: 1.Centos7安装Promethus(普罗米修斯)监控系统完整版 2.Promethus(普罗米修斯)监控Mysql数据库 3.Promethus(普罗米修斯)安装Grafana可视化 ...

2. Promethus===》普罗米修斯简介、时序数据库、监控系统的基本使用

   一.Promethus(普罗米修斯)监控系统 能够安装prometheus服务器 能够通过安装node_exporter监控远程linux 能够通过安装mysqld_exporter监控远程mysql ...

3. 配置文件详解+AlertManager微信邮件告警配置

   文章目录 前言 AlertManager告警简单部署 一.AlertManager告警简介 1.简介 2.告警规则组成 1)告警名称 2)告警规则 3.Alertmanager特性 1)分组 2)抑制 ...

4. Promethus(普罗米修斯）监控随笔

   Promethus(普罗米修斯)监控随笔 promethus安装 安装consul用于普罗米修斯监控端注册服务统一监控 安装node_exporter 修改普罗米修配置文件新增监控项 安装grafan ...

5. Promethus的Grafana图形显示MySQL监控数据

   相关博文: 1.Centos7安装Promethus(普罗米修斯)监控系统完整版 2.Promethus(普罗米修斯)监控Mysql数据库 3.Promethus(普罗米修斯)安装Grafana可视化 ...

6. centos7安装promethus(普罗米修斯)

   前言 Prometheus(go语言开发)是一套开源的监控&报警&时间序列数 据库的组合.适合监控docker容器.因为kubernetes(俗称k8s)的流行带动 了promethe ...

7. golang通过RSA算法生成token，go从配置文件中注入密钥文件，go从文件中读取密钥文件，go RSA算法下token生成与解析；go java token共用

   RSA算法 token生成与解析 本文演示两种方式,一种是把密钥文件放在配置文件中,一种是把密钥文件本身放入项目或者容器中. 下面两种的区别在于私钥公钥的初始化, init方法,需要哪种取哪种. 通过 ...

8. Dockerfile springboot项目拿走即用，将yml配置文件从外部挂入容器

   Dockerfile 将springboot项目jar包打成镜像,并将yml配置文件外挂. # 以一个镜像为基础,在其上进行定制.就像我们之前运行了一个 nginx 镜像的容器,再进行修改一样,基础镜 ...

9. 在kotlin companion object中读取spring boot配置文件，静态类使用@Value注解配置

   在kotlin companion object中读取配置文件 静态类使用@Value注解配置 class Config {@Value("\${name}")fun setNam ...

最新文章

1. tomcat外网映射工具
2. 整个宇宙可能是个巨大的神经网络？看科学家们是这样解释的
3. [100]第三波常用命令
4. 再见，Postman...
5. 【Linux】一步一步学Linux——mv命令(30)
6. 内容可编辑_让PDF像WORD一样自由编辑，好用的PDF编辑工具推荐
7. dwr 写的小程序，配置
8. Nginx 其他模块
9. 亚马逊最大无人售货超市开张，云端结账随拿随走
10. 视图单行子查询返回mysql,Oracle命令整理 - osc_sj1kgo4z的个人空间 - OSCHINA - 中文开源技术交流社区...
11. 【爬虫】使用八爪鱼爬行百度地图美食店数据
12. 世界杯花样营销：争夺32亿人眼球中看到三大趋势
13. Java曲线之削峰填谷,科学网—Lorenz曲线之削峰填谷 - 李宁的博文
14. CSBJ综述：微生物组数据挖掘方法的挑战与机遇
15. otg usb 定位_教你简单认识OTG与OTG线
16. 设计模式（四）责任链模式——责任链模式结构
17. 教育培训机构学生管理系统
18. [附源码]java毕业设计网络身份认证技术及方法
19. 华为面试Android岗；群面+技术面+综合面+英语面
20. 1 个方法提升 3 倍执行力

热门文章

1. Stata: VAR (向量自回归) 模型
2. 2018 年，WEB前端开发人员应该关注哪些新晋技术？
3. 五分钟教你Android-Kotlin项目编写
4. 两个整数相乘的java实现
5. 6-4 学生成绩链表处理 (20分) 本题要求实现两个函数，一个将输入的学生成绩组织成单向链表；另一个将成绩低于某分数线的学生结点从链表中删除。 函数接口定义： ```cpp struct stu
6. 菜鸟了解点“调查取证套路”刀在你我手
7. 成人计算机考试操作题视频教程,成人计算机考试操作题模拟.doc
8. Python爬虫新手入门教学（十七）：爬取yy全站小视频
9. 局域网网络流量监控_18个监控网络带宽的Linux命令行工具
10. 北京理工大学 计算机学院男女比例,39所985高校男女比例排名，看看哪些学校比例严重失调！...