zed icon indicating copy to clipboard operation
zed copied to clipboard

aggregation outputs should limit the size of records

Open mccanne opened this issue 5 years ago • 1 comments

The design strategy for zng.Record is that it always fits in memory. Yet, union and collect aggregators can create very large records that exceed limits. As long as records fit in memory, things work fine with spilling and everything. Thus, we should put a configurable limit on the record size. This is a little tricky because the reducers and groupby aggregator need to coordinate on this.

This is a follow-on to the bug fix for #1803

mccanne avatar Dec 12 '20 08:12 mccanne

I think this is fixed... moving into backlog to verify.

There is a bigger project to globally coordinate the memory footprint across all large consumers of memory (principally group-by and sort). (#4025)

mccanne avatar Dec 14 '21 13:12 mccanne