using a cache for Rate Limiting

[kostas-theo]

@Jeffail,

We have created our own custom processor which can use a redis cluster for rate limiting purposes and would like to contribute the code as new functionality in benthos. Before getting started though I am wondering where and how it should be added in terms of structure. I see a number of different approaches:

• Create a new cache or redis rate limits resource effectively duplicating the existing cache resource and define new args for this functionality under the existing rate_limit processor
• Allow the rate_limit processor to use cache resources and define new args for this functionality
• Create a new rate_limit_cache or rate_limit_redis processor and disregard the rate_limits and cache resource types

Any preferences between these or other ways of structuring this new functionality??

[Jeffail]

Hey @kostas-theo, sounds like a great addition, I'd love a distributed rate limit option.

Ideally this would be implemented as a rate_limit resource component even if it involves a small amount of duplication. That would allow it to be used by the rate_limit processor as well as any HTTP components, etc.

Don't hesitate to ask for guidance or anything, I'm happy to review draft PRs if it helps.

[lucasoares]

@kostas-theo did you had any progress in this?

[lucasoares]

Just submitted a PR for this but I have a few questions there. @Jeffail