feat(transforms): Introduce aggregate configuration

[tomers]

If aggregate=true reduce transformation behaves like aggregator that aggregates records and flushes them periodically

type=feat

Closes issue #8747

[netlify[bot]]

✔️ Deploy Preview for vector-project canceled.

🔨 Explore the source changes: c0530ca2fc183c53087ec2e886e5baba6c26c506

🔍 Inspect the deploy log: https://app.netlify.com/sites/vector-project/deploys/611b7b55d238040007135040

[spencergilbert]

Thanks for the PR! It's a little unclear to me what behavior we're trying to introduce here - it seems to me like the expire_after and flush_period options would provide similar today?

[kvitali]

Using the original reduce transform in case similar events are keep coming and coming, the state are not being flushed because stale_since member is being updated after each event. For example:

1. We send keep-alive event each 5 seconds for period of 5 minutes
2. We want to have flush period of 60 seconds
3. This means that with original reduce transform only 1 event will passthru after 6 mins

We want a behavior that during those 5 minutes we will passthru 1 event each 60 seconds, 5 event during the period of 5 minutes. This is the reason we altered the reduce transform code.

I saw that you have new aggregate transform, but it is for metrics only, is it possible that the aggregate transform will support logs also?

[aantn]

I also have a need for something like this. I'd like to reduce logs like:

WARNING LOG1
INFO LOG2
WARNING LOG3

To something like:

["WARNING LOG1", "WARNING LOG3"]
["INFO LOG2"]

The current implementation doesn't support this because it only flushes when the start/end condition are encountered.

@tomers I looked at the PR and I think you're missing something. You can have a dangling event which is never flushed if no events come after it. Is that correct?

[xdatcloud]

I also have a need for something like this. I'd like to reduce logs like:

WARNING LOG1
INFO LOG2
WARNING LOG3

To something like:

["WARNING LOG1", "WARNING LOG3"]
["INFO LOG2"]

The current implementation doesn't support this because it only flushes when the start/end condition are encountered.

@tomers I looked at the PR and I think you're missing something. You can have a dangling event which is never flushed if no events come after it. Is that correct?

Event will be inserted into corresponding ReduceState, and ReduceState has its own stale_since state, which would estimate with expire_after_ms during each flush period, and flush all events under the ReduceState if state is expired.

Thus there won't be dangling events that is never flushed.

[bits-bot]

All committers have signed the CLA.

[netlify[bot]]

✅ Deploy Preview for vrl-playground canceled.

Name	Link
🔨 Latest commit	715d53082158c5d1925c82dae6057b20c6920497
🔍 Latest deploy log	https://app.netlify.com/sites/vrl-playground/deploys/63d292f1aae89c0008ad8281

[netlify[bot]]

✅ Deploy Preview for vector-project ready!

Name	Link
🔨 Latest commit	715d53082158c5d1925c82dae6057b20c6920497
🔍 Latest deploy log	https://app.netlify.com/sites/vector-project/deploys/63d292f1a7263a0009894864
😎 Deploy Preview	https://deploy-preview-8749--vector-project.netlify.app
📱 Preview on mobile	Toggle QR Code... Use your smartphone camera to open QR code link.

---

To edit notification comments on pull requests, go to your Netlify site settings.

[tomers]

I think the proper solution for issue #8747 is https://github.com/vectordotdev/vector/pull/16146. Closing this for now.