robusta icon indicating copy to clipboard operation
robusta copied to clipboard

Explain TargetDown alerts related to coreDns and kubeDns

Open Sheeproid opened this issue 3 years ago • 0 comments

Is your feature request related to a problem? kube-prometheus-stack that is being used by Robusta, monitors coreDns service by default. (coreDns is a service that handles dns on the cluster.)

However, on some Kubernetes distributions (e.g GKE) kubeDns is used intstead of coreDns. In that case, Robusta will fire "TargetDown" alerts from Prometheus which are not relevant, because the target coreDns should not exist in the first place.

The way for Robusta users to solve this is to define the following in their Robusta generated_values.yaml file:

kube-prometheus-stack:
  coreDns:
    enabled: false
  kubeDns:
    enabled: true

Describe the solution you'd like Create an enricher (action) explains the situation so the Robusta users won't worry, and also how to fix it. The enricher should first check if kubeDns actually exists on the cluster.

This should be its message: " This alert happens because Prometheus is trying to scrape the coreDns service but that service does not exist in your cluster. Robusta detects that kubeDns exists on your cluster, so this is safe to ignore. To disable this alert, add the following YAML to your Robusta configuration:

kube-prometheus-stack:
  coreDns:
    enabled: false
  kubeDns:
    enabled: true

"

This actions should use the following trigger:

- triggers:
  - on_prometheus_alert:
      alert_name: TargetDown
  actions:
  - core_dns_target_down_enricher

Sheeproid avatar Sep 19 '22 13:09 Sheeproid