Alerting labels

[naveensrinivasan]

We are trying to use the alerting labels. This is port of the existing Prometheus alert

  - alert: KubernetesNodeReady
    expr: kube_node_status_condition{condition="Ready",status="true"} == 0
    for: 10m
    labels:
      severity: critical
    annotations:
      summary: Kubernetes Node ready (instance {{ $labels.instance }})
      description: "Node {{ $labels.node }} has been unready for a long time\n  VALUE = {{ $value }}\n  LABELS = {{ $labels }}"

We ran into this error Error: UPGRADE FAILED: parse error at (gmp/templates/alerts/k8s.yaml:18): undefined variable "$labels"

How do we access labels?

[naveensrinivasan]

cc @lyanco

[lyanco]

We're not sure - we think this should work. Try the suggestion here perhaps? https://stackoverflow.com/questions/64693812/how-to-fix-the-error-undefined-variable-labelsin-prometheus/64694592#64694592

[naveensrinivasan]

We're not sure - we think this should work. Try the suggestion here perhaps? https://stackoverflow.com/questions/64693812/how-to-fix-the-error-undefined-variable-labelsin-prometheus/64694592#64694592

Could you please show an example that uses label that should work?

[maxamins]

  - record: job_up_sum
    expr: sum by(job, project_id) (up)
  - alert: TooFewReplicas
    expr: job_up_sum < 3
    for: 1m
    annotations:
      description: "Not enough replicas ({{ $value }}) for job {{ $labels.job }}"

I also added your rules and it worked. I think something might be to your templating.

[StianOvrevage]

If what you are trying to do is pass a label from the query triggering the alert to the alert being passed to alertmanager, try this:

kind: PrometheusRule
.......
  labels:
      instance: '{{`{{`}} $labels.instance{{`}}`}}'

This only worked after adding both single quotes and the rendering of curly brackets in helm.