像类似ELK(beat -> kafka -> logstash -> es -> kibana)技术一样，k8s监控也是个chain。

node_exporter -> prometheus -> alertmanager -> webhook

node_exporter -> prometheus -> grafana

Prometheus标签可以在多个地方进行增删改

1. 在Prometheus配置文件中重写、删除label

如relabel_configs进行重写，将__meta_consul_service名字替换为application：

- job_name: consul_eureka
  honor_timestamps: true
  scrape_interval: 15s
  scrape_timeout: 10s
  metrics_path: /actuator/prometheus
  scheme: http
  consul_sd_configs:
  - server: cloud-eureka-server-0.eureka-headless.default.svc.cluster.local:8080
    tag_separator: ','
    scheme: http
    username: test
    password: test
    allow_stale: true
    refresh_interval: 30s
  relabel_configs:
    - source_labels: [__meta_consul_service]
    separator: ;
    regex: (.*)
    target_label: application
    replacement: $1
    action: replace

官方链接： https://prometheus.io/docs/prometheus/latest/configuration/configuration/#relabel_config

2. 在Prometheus告警规则中新增label

告警消息中新增label：‘application: RABBITMQ’

groups:
- name: server-group
  rules:
  - alert: 生产环境_rabbitmq_queue_high
    expr: rabbitmq_queue_messages > 2000
    for: 20s
    labels:
      application: RABBITMQ
    annotations:
      summary: '生产环境: rabbitmq队列消息较多, 当前值: {{ $value }}'
3. 在webhook模板中过滤

如钉钉的webhook配置文件中，可适当过滤掉某些label

1
2
3

**详情:**
{{ range .Labels.SortedPairs }}{{ if and (ne (.Name) "alertname") (ne (.Name) "job") (ne (.Name) "exported_application") }}> - {{ .Name }}: {{ .Value | markdown | html }}
{{ end }}{{ end }}

参考链接： https://www.kancloud.cn/cserli/golang/531904