High cardinality of Prometheus metrics in self-scrape scenarios (only self-deploy).

Open bwplotka opened this issue 2 years ago • 1 comments

The way we configure scrape jobs causes bit high cardinality of net_conntrack_dialer_conn_failed_total. It would be nice to work with the upstream to provide an option to disable it, or improve our example PodMonitoring for self-scrape to explictly ignore this metric.

Dec 19 '23 14:12 bwplotka

All metrics with scrape_job has the same issue e.g. prometheus_target_sync_length_seconds but they might have some value for some advanced debugging cases.

https://github.com/prometheus/prometheus/blob/bfbb13cf369da4cd1b29ee52c396c902723febfb/scrape/metrics.go#L138

Jan 18 '24 13:01 bwplotka