Make MAX_RETRIES configurable

[JosiahParry]

I noticed that the maximum number of retries is set to 16. https://github.com/cloudflare/pingora/blob/bdb13a7e40b19280d703ad1f64abc29ee5baf480/pingora-proxy/src/lib.rs#L68 With my proxy a user may be sent to an application that needs to perform a cold start. That time may be a few seconds or longer.

I have set the retry to wait 100ms before retrying.

    fn fail_to_connect(
        &self,
        _session: &mut Session,
        _peer: &HttpPeer,
        _ctx: &mut Self::CTX,
        mut e: Box<Error>,
    ) -> Box<Error> {
        e.set_retry(true);
        std::thread::sleep(std::time::Duration::from_millis(100));
        e
    }

Ideally, I'd like to be able to configure it to poll every x ms for a maximum Duration before finally giving up instead of a fixed number of retries.

[gumpt]

Envoy's retry policies are worth looking at here for inspiration/a good set of tunables. https://www.envoyproxy.io/docs/envoy/latest/api-v3/config/route/v3/route_components.proto#envoy-v3-api-msg-config-route-v3-retrypolicy

[xlfish233]

Is this issue still open? I'd like to work on implementing a more configurable retry strategy.

[JosiahParry]

@xlfish233 not that i am aware of! This would be an awesome feature

[andrewhavck]

In the near term we've added the ability to configure the hardcoded upper bound.

[Propfend]

@andrewhavck and this option seems to be removed.

[andrewhavck]

@Propfend the option is there?

https://github.com/cloudflare/pingora/blob/main/pingora-core/src/server/configuration/mod.rs#L104