Add retry on prometheus list rules

[emanlodovice]

Is your feature request related to a problem? Please describe. Currently the PrometheusRules api endpoint, at the worst case, will list all the rules from all rules. For a cluster with hundreds of rulers one api call will trigger hundreds of GRPC calls, one for each ruler, and if one of these GRPC calls fail we return 5xx. So as the number of rulers increase, the chance of one grpc call failing causing the entire list rules call to fail also increases.

Describe the solution you'd like There should be a retry mechanism on the grpc call to make the endpoint more robust. https://github.com/cortexproject/cortex/blob/master/pkg/ruler/ruler.go#L907

[emanlodovice]

https://pkg.go.dev/google.golang.org/grpc/examples/features/retry#section-readme

[emanlodovice]

Hmm i guess backoff_on_ratelimits and backoff_config on the ruler_client config can be used to tune the retry.