kstreamplify icon indicating copy to clipboard operation
kstreamplify copied to clipboard

Stream-stream left and outer join utility method to avoid 3.X grace period behavior

Open clvacher opened this issue 2 years ago • 0 comments

Problem

Join behavior has changed since 3.X, as described in the following links: https://issues.apache.org/jira/browse/KAFKA-13813 https://confluentinc.atlassian.net/browse/KSTREAMS-5249

Suggestion

Custom utility methods need to be implemented, in a similar logic to the one for DedupUtils : the method needs to take a callback function to describe what to do with the join once it is found.

For left joins, implementation can involve a window store on the right side, or even a timestampedKeyValueStore with a scheduled task for purging. We could use polymorphism to make multiple levels of details:

  • default implementation with window store and Pair as the output
  • level 1 advanced implementation with timestampedKeyValueStore, default purge frequency and join behavior as a function parameter
  • level 2 with custom purge frequency and purge behavior as a function parameter

clvacher avatar Jul 05 '23 09:07 clvacher