zed
zed copied to clipboard
aggregation outputs should limit the size of records
The design strategy for zng.Record is that it always fits in memory. Yet, union and collect aggregators can create very large records that exceed limits. As long as records fit in memory, things work fine with spilling and everything. Thus, we should put a configurable limit on the record size. This is a little tricky because the reducers and groupby aggregator need to coordinate on this.
This is a follow-on to the bug fix for #1803
I think this is fixed... moving into backlog to verify.
There is a bigger project to globally coordinate the memory footprint across all large consumers of memory (principally group-by and sort). (#4025)