fluent
[Feature Request] provide option to throw_exception/block before buffer queue is full
Is your feature request related to a problem? Please describe.
Due to some issues I have have been experiencing, I believe it would make sense to provide options to throw_exception/block before the buffer queue is full.
When preforming tests with file buffering I have noted at least these issues: https://github.com/fluent/fluent-plugin-opensearch/issues/35 https://github.com/fluent/fluent-plugin-opensearch/issues/34
Even with these issues resolved, I believe providing options to throw_exception/block before the buffer queue is full makes sense.
Describe the solution you'd like
I would suggest adding to the buffering and/or flushing parameters: https://docs.fluentd.org/configuration/buffer-section#buffering-parameters https://docs.fluentd.org/configuration/buffer-section#flushing-parameters
Options to set a high-water mark percentage, similar to chunk_full_threshold, and an action, similar to the overflow_action would be required.
For example: chunk_highwater_action - throw_exception/block chunk_highwater_threshold - percentage of chunk size threshold for executing chunk_highwater_action
Describe alternatives you've considered
N/A
Additional context
N/A
Do we need to implement back pressure feature?
e.g) https://github.com/fluent/fluentd/blob/master/lib/fluent/plugin/buffer.rb#L295
Do we need to implement back pressure feature?
I would very much welcome a full featured back pressure feature. A full featured back pressure mechanism requires harmony between output plugins, buffers, and input plugins. I do believe, though, my above request would be helpful in the interim, without impacting much of the plumbing.
Yeah, makes sense but implementing it on Fluentd core is very challenging task for now.
@cosmo0920 - might these two be easier to implement?
[Feature Request] implement RPC endpoints to stop and start sources #3692 [Feature Request] supply monitor_agent stats via RPC or update without_source to skip type monitor_agent #3694
If so - we could use external scripts to monitor and control the up/down state of the sources :)