prometheus
Metric vs Log level
Hi Team,
we have a situation , where we have 8 to 15 million head series in each Prometheus and we have 7 instance of them (federated). Our prometheus are in a constant flooded situation handling the incoming metrics and back end recording rules.
One thought which came to was - do we have something similar to log level for prometheus metrics ? If its there then... we can benefit from it .... by configuring to run all targets in error level in production and in debug/info level in development... This will help control flooding of metrics.
Say, If we write a wrapper on top of prometheus java client API, its going to be messy - hence wanted to check if this request makes sense or is there any other way out ?
Let me know your thoughts how this can be achieved ....
Many thanks. Muthuveerappan
Hi, sure, it makes sense not to expose so many metrics. There are many ways how you could implement this. One idea is to use different registries, and make it configurable which registry you want to expose. Another idea is to add a level label to each metric, and then use relabel rules in the Prometheus server to drop metrics with level debug and info.
