FusionAuth Cloud: Prevent metrics collisions from multiple instances

[mmanes]

Problem

When scraping metrics from /api/prometheus/metrics or /api/status requests round-robin across all instances, making it difficult to gather accurate health information. Furthermore, the metrics names collide, which makes a lot of them useless.

Solution

Allow scraping of individual instances. Possible implementations may include:

• Add endpoint and/or query parameter to get metrics from all instances with one api call. Add labels to metrics to differentiate them. E.g., {instance="hostname-1}
• Supporting a query parameter, like instanceId=1 to selectively scrape each instance.
• Exposing a new hostname via DNS which maps directly to an instance.

Related Issues

• https://github.com/FusionAuth/fusionauth-issues/issues/2809

Additional Context

When we fix this, should remove limitations mentioned in this PR: https://github.com/FusionAuth/fusionauth-site/pull/3294

Community guidelines

All issues filed in this repository must abide by the FusionAuth community guidelines.

How to vote

Please give us a thumbs up or thumbs down as a reaction to help us prioritize this feature. Feel free to comment if you have a particular need or comment on how this feature should work.

[bhalsey]

Internal TDD: https://docs.google.com/document/d/1YZ4b8gqpvzeg3cqkrs4pnr3eyfDyhEntg38AqOHDt_g/edit