database-stream-processor icon indicating copy to clipboard operation
database-stream-processor copied to clipboard

Lazily consolidate streams

Open Kixiron opened this issue 3 years ago • 1 comments

Currently pretty much every operator expects a sorted & consolidated collection and produces a sorted & consolidated collection, but this work is very often redundant. Ephemeral operators like .filter() or .map() don't care about the order or consolidated-ness of their inputs or outputs, so there's no need to always marshal their inputs/outputs into OrdZSet and the like, they can consume arbitrary inputs and produce unordered batches of values

Kixiron avatar Jul 31 '22 15:07 Kixiron

Part of the reason for the current design is wanting to keep the API simple by reducing the number of different container types and conversions between them. I agree though that this shouldn't come at the cost of performance, plus we can probably hide most of the additional complexity from the user.

ryzhyk avatar Jul 31 '22 17:07 ryzhyk