dozer icon indicating copy to clipboard operation
dozer copied to clipboard

Filtering optimizations

Open snork-alt opened this issue 2 years ago • 0 comments

Currently filtering records is always executed after a product. The reason for this is because the filter condition might be applied using a function requiring inputs from multiple joined tables. However this is is not always the case.

Thus, we should split the filtering into two phases:

  • First phase is executed at the sources, filtering data based on original fields conditions. These filters should have the option to support filter pushdowns to the original source during the snapshotting phase.

  • Second phase is only reserved for filters that require multiple joined tables. These filters shall be applied after a product

snork-alt avatar Sep 03 '23 05:09 snork-alt