databend icon indicating copy to clipboard operation
databend copied to clipboard

refactor(query): refactor hash join spill

Open Dousir9 opened this issue 1 year ago โ€ข 2 comments

I hereby agree to the terms of the CLA available at: https://docs.databend.com/dev/policies/cla/

Summary

Code readability and maintainability

  1. New step transition mechanism: this PR divides hash join probe/build step into three types: SyncStep, AsyncStep, and Finish, ensuring that step transition occurs in the event and this PR redesigns step transition to make the step transition logic clearer and improve code readability and maintainability.
  2. Unify ProbeSpillHandler and BuildSpillHandler to HashJoinSpiller to improve code maintainability.
  3. Unify the spill logic of cross join and other types.
  4. Introduce HashTableType to reduce code.

Bug Fixed

  1. Fixed a corner case of left-related join when spill happens.
  2. Left Anti join can't probe first round when spill happens.

Performance

Improved the performance of partition.

Tests

  • [ ] TPC-H spill test
  • [ ] TPC-DS spill test
  • [ ] Wizard join spill test

Tests

  • [ ] Unit Test
  • [x] Logic Test
  • [ ] Benchmark Test
  • [ ] No Test - Explain why

Type of change

  • [x] Bug Fix (non-breaking change which fixes an issue)
  • [ ] New Feature (non-breaking change which adds functionality)
  • [ ] Breaking Change (fix or feature that could cause existing functionality not to work as expected)
  • [ ] Documentation Update
  • [x] Refactoring
  • [x] Performance Improvement
  • [x] Other (please describe): code maintainability

This change isโ€‚Reviewable

Dousir9 avatar Jun 06 '24 12:06 Dousir9

Docker Image for PR

  • tag: pr-15746-9107967-1717765135

note: this image tag is only available for internal use, please check the internal doc for more details.

github-actions[bot] avatar Jun 07 '24 13:06 github-actions[bot]

Docker Image for PR

  • tag: pr-15746-61aa91b-1718103773

note: this image tag is only available for internal use, please check the internal doc for more details.

github-actions[bot] avatar Jun 11 '24 11:06 github-actions[bot]

Docker Image for PR

  • tag: pr-15746-d5f0732-1723784071

note: this image tag is only available for internal use, please check the internal doc for more details.

github-actions[bot] avatar Aug 16 '24 04:08 github-actions[bot]

Docker Image for PR

  • tag: pr-15746-6e26223-1723829677

note: this image tag is only available for internal use, please check the internal doc for more details.

github-actions[bot] avatar Aug 16 '24 17:08 github-actions[bot]

Docker Image for PR

  • tag: pr-15746-6e26223-1723831273

note: this image tag is only available for internal use, please check the internal doc for more details.

github-actions[bot] avatar Aug 16 '24 18:08 github-actions[bot]

Docker Image for PR

  • tag: pr-15746-550a9dc-1724066510

note: this image tag is only available for internal use, please check the internal doc for more details.

github-actions[bot] avatar Aug 19 '24 11:08 github-actions[bot]

Docker Image for PR

  • tag: pr-15746-7335d28-1724069360

note: this image tag is only available for internal use, please check the internal doc for more details.

github-actions[bot] avatar Aug 19 '24 12:08 github-actions[bot]