feat(new transform): Add buffered gate transform

[ilinas]

See issue: #15263

A simple implementation of ring buffer / backtrace event handling that I ended up naming the gate transform. Keeps events in a buffer until a trigger is encountered and the buffer is flushed. When the buffer is full, the oldest events are being dropped, and it works pretty much like the filter transform. The code is essentially a simple VecDeque.

Example configuration:

transforms:
  app_gate:
    type: gate
    inputs:
      - app_logs
    pass_when: '"info" == .level'
    open_when: '"error" == .level'
    auto_close: true
    tail_events: 20
    max_events: 200

Submitting as a draft for now.

[bits-bot]

All committers have signed the CLA.

[ilinas]

Hi @jszwedko, no worries. I am glad that you think this is useful.

I am new to both Rust and Vector code, so this implementation is based on my observations of how other similar transforms were implemented. It's highly possible that I missed something completely obvious, so some sanity check would be highly appreciated.

I know I am missing the docs, but first I wanted to make sure that the general functionality and the config parameters are sensible?

[ilinas]

Thank you so much for your comments @jszwedko.

You are right that some of the option combinations are a bit confusing.

I tried to cover two different use cases here:

1. Traditional backtrace: when something important happens, e.g. '"error" == .level', flush the buffer and close the gate.
2. Manual gate: when you want to manually control the flow, e.g. open_when: '"session started" == .message' and close_when: '"session ended" == .message'.

Maybe it would be more logical to limit the transform to just the first use case? Especially because in the second use case you don't really benefit from having a buffer.

Could rename the transform to something like buffer or backtrace and just have flush_when instead of open_when, and completely ditch close_when and auto_close?

transforms:
  app_buffer:
    type: buffer
    inputs:
      - app_logs
    pass_when: '"info" == .level'
    flush_when: '"error" == .level'
    tail_events: 20
    max_events: 200

[jszwedko]

Thanks for the additional thoughts! I see what you are saying. Maybe it would be easier to just focus on one use-case at a time. It might end up being a better model to have two separate transforms to support the two use-cases. Would you want to refocus this PR just on one of them and then we can have a second PR focused on the other? It sounds like you'd like to target the "backtrace" use-case first?