[rate limit filter] Make failure_mode_deny a runtime parameter

[amongil]

Title: Migrating failure_mode_deny parameter from the http rate limit filter from a boolean to a runtime parameter

Description: Currently the failure mode for the rate limit filter is defined via a boolean parameter failure_mode_deny. Changing the value requires a restart of Envoy with different configuration. For incident response, we would like to have this parameter be a runtime parameter instead (similar to filter_enabled) so that we can rollback the failure mode while still performing rate limiting to the best of the external service ability.

The change would break backwards compatibility if made on the existing parameter, so I think a new parameter will have to be added on v3 of the filter, while marking the existing as deprecated, before moving to a v4 version of the filter which would only have failure_mode_deny defined as a runtime parameter, but I would need more guidance on how to do this version change (if this is a feature you are open to introducing to the filter)

I have bandwidth to work on the issue and have a working version in a fork.

[yanavlasov]

I think you can just a runtime override, similar to http_filter_enabled. Why would it break backward compatibility?

[amongil]

I think you can just a runtime override, similar to http_filter_enabled. Why would it break backward compatibility?

That sounds good if it is fine adding a parameter that overrides the existing failure_mode_deny boolean value, if the runtime flag is specified. I will post a PR in the coming days with a proposal.