paimon icon indicating copy to clipboard operation
paimon copied to clipboard

[Feature] Support Flink read / write data branch

Open sunxiaojian opened this issue 1 year ago • 6 comments

Purpose

Linked issue: close (https://github.com/apache/incubator-paimon/issues/2861)

When writing and reading branch data for Flink, it was found that if the "TableCommitImpl newcommit (String commitUser, String branchName)" interface is added like this The parameter "branchName" has a wide range of impacts, and when adding features in the future, we will also consider whether to support the branch, which is a challenge for developers. Therefore, in order to make developers not need to worry about the branch function and not need to consider branch writing and reading when adding interfaces, I have made the following refactoring and optimization

  1. Refactoring TagManager, SnapshotManager, and SchemeManager to support specifying default branch parameters,

The implementation logic of 'SnapshotManager snapshotManager()', 'TagManager tagManager()', and 'BranchManager branchManager()' directly specifies the default written branch, so that we no longer need to specify the 'branch' parameter when declaring new interfaces for subsequent interfaces.

  1. Add necessary unit test tests, such as CDC testing, to support Flink CDC writing to branches

  2. After this PR, remove interfaces with the branchName parameter such as' TableCommitImpl newcommit (String commitUser, String branchName) '

Tests

API and Format

Documentation

sunxiaojian avatar Mar 16 '24 15:03 sunxiaojian

@FangYongs @schnappi17 PTAL cc @JingsongLi

sunxiaojian avatar Mar 16 '24 16:03 sunxiaojian

@FangYongs @schnappi17 When you have time, please take a look at this PR

sunxiaojian avatar Mar 27 '24 06:03 sunxiaojian

@FangYongs @schnappi17 Do we need to send an email to discuss the part of this optimization ?

sunxiaojian avatar Apr 07 '24 12:04 sunxiaojian

@JingsongLi Please help to take a look at this, thanks a lot!

schnappi17 avatar Apr 24 '24 10:04 schnappi17

@FangYongs Please also help to take a look at~Thanks!

schnappi17 avatar Apr 25 '24 06:04 schnappi17

@JingsongLi PTAL

sunxiaojian avatar May 14 '24 15:05 sunxiaojian