`openmetrics`: `openmetrics_endpoint` doesn't support Kubernetes' bearer token but `prometheus_url` does

[pmalek]

Additional environment details (Operating System, Cloud provider, etc):

Kubernetes, kind

Problem statement

https://docs.datadoghq.com/integrations/guide/prometheus-host-collection/#parameters-available mentions the usage of bearer_token_path, bearer_token_auth and openmetrics_endpoint but no prometheus_url, in order to make the agent attach ServiceAccount's token to requests.

After some testing and several hours of debugging I came to https://github.com/DataDog/integrations-core/blob/14f280a0f339e53a7703c86d687a8ca32aa683ed/datadog_checks_base/datadog_checks/base/checks/openmetrics/base_check.py#L15 and several other lines in this file which got be thinking that I should use prometheus_url.

That's exactly what was missing. I switched openmetrics_endpoint to prometheus_url and now my Pods are correctly attaching the token to requests.

Possibly related

https://github.com/DataDog/datadog-agent/issues/2667

[tbavelier]

Hello @pmalek , https://docs.datadoghq.com/containers/kubernetes/prometheus/?tab=kubernetesadv2 is the more up-to-date guide to follow for Prometheus/Openmetrics collection in Kubernetes environments.

The bearer_token_auth is indeed exclusive to the prometheus_url implementation / class of the Openmetrics check. Nonetheless, you can replicate the same behaviour with the auth_token configuration parameter : https://github.com/DataDog/integrations-core/blob/master/openmetrics/datadog_checks/openmetrics/data/conf.yaml.example#L424-L461. More specifically, your configuration with openmetrics_endpoint would be :

          auth_token:
            reader:
              type: file
              path: /var/run/secrets/kubernetes.io/serviceaccount/token
            writer:
              type: header
              name: Authorization
              value: Bearer <TOKEN>

This would allow you to use openmetrics_endpoint implementation which is more performant than the legacy prometheus_url implementation which was created from https://docs.datadoghq.com/integrations/prometheus/ while using the Kubernetes service account scope.

[pmalek]

Thanks for the reponse @tbavelier.

Sadly https://docs.datadoghq.com/containers/kubernetes/prometheus/?tab=kubernetesadv2 doesn't mention anything about the auth tokens 😅 But it's good to know that we can use that. I'll check this next time I'm around DD configuration.