kafka
kafka copied to clipboard
Optimize self-join
POC for self-join optimization. KIP can be found here: https://cwiki.apache.org/confluence/display/KAFKA/KIP-862%3A+Self-join
It only applies to Stream-Stream joins and not n-way self-joins.
This is an inner-join topology (without the optimization)
Topologies:
Sub-topology: 0
Source: KSTREAM-SOURCE-0000000000 (topics: [topic2])
--> KSTREAM-WINDOWED-0000000001, KSTREAM-WINDOWED-0000000002
Processor: KSTREAM-WINDOWED-0000000001 (stores: [KSTREAM-JOINTHIS-0000000003-store])
--> KSTREAM-JOINTHIS-0000000003
<-- KSTREAM-SOURCE-0000000000
Processor: KSTREAM-WINDOWED-0000000002 (stores: [KSTREAM-JOINOTHER-0000000004-store])
--> KSTREAM-JOINOTHER-0000000004
<-- KSTREAM-SOURCE-0000000000
Processor: KSTREAM-JOINOTHER-0000000004 (stores: [KSTREAM-JOINTHIS-0000000003-store])
--> KSTREAM-MERGE-0000000005
<-- KSTREAM-WINDOWED-0000000002
Processor: KSTREAM-JOINTHIS-0000000003 (stores: [KSTREAM-JOINOTHER-0000000004-store])
--> KSTREAM-MERGE-0000000005
<-- KSTREAM-WINDOWED-0000000001
Processor: KSTREAM-MERGE-0000000005 (stores: [])
--> KSTREAM-PROCESSOR-0000000006
<-- KSTREAM-JOINTHIS-0000000003, KSTREAM-JOINOTHER-0000000004
Processor: KSTREAM-PROCESSOR-0000000006 (stores: [])
--> none
<-- KSTREAM-MERGE-0000000005

and this is the optimized self-join topology
Topologies:
Sub-topology: 0
Source: KSTREAM-SOURCE-0000000000 (topics: [topic1])
--> KSTREAM-WINDOWED-0000000001
Processor: KSTREAM-WINDOWED-0000000001 (stores: [KSTREAM-JOINTHIS-0000000003-store])
--> KSTREAM-MERGE-0000000005
<-- KSTREAM-SOURCE-0000000000
Processor: KSTREAM-MERGE-0000000005 (stores: [KSTREAM-JOINTHIS-0000000003-store])
--> KSTREAM-PROCESSOR-0000000006
<-- KSTREAM-WINDOWED-0000000001
Processor: KSTREAM-PROCESSOR-0000000006 (stores: [])
--> none
<-- KSTREAM-MERGE-0000000005

Summary of testing strategy (including rationale) for the feature or bug fix. Unit and/or integration tests are expected for any behaviour change and system tests should be considered for larger changes.
Committer Checklist (excluded from commit message)
- [ ] Verify design and implementation
- [ ] Verify test coverage and CI build status
- [ ] Verify documentation (including upgrade notes)