flink-cdc icon indicating copy to clipboard operation
flink-cdc copied to clipboard
verified publisher icon apache

Add Hudi sink connector support

Open voonhous opened this issue 3 months ago • 8 comments

This PR adds Hudi CDC sink support to Flink CDC.

As of now the following features are supported:

  1. Simple bucket index
  2. Non-partitioned tables
  3. MOR tables
  4. Compaction plan generation (Compaction execution will require a separate process as of now)

Future improvements will be made to bring along more future support other native hudi features gradually/iteratively as we are trying to keep the PR small and manageable for reviews.

voonhous avatar Oct 24 '25 08:10 voonhous

@danny0405 @cshuo FYI

voonhous avatar Oct 24 '25 08:10 voonhous

Changes here will require Hudi 1.1.0 to be released first.

voonhous avatar Oct 27 '25 07:10 voonhous

@lvyanquan @yuxiqian Bumped the version, but am not sure why the CI is failing, the E2E tests are passing locally:

image

Can you please advise and help to review? Thank you!

voonhous avatar Nov 21 '25 10:11 voonhous

We've re-triggered the CI tests, and a checkstyle issue was reported. You can fix it and trigger a new test run once it's resolved.

lvyanquan avatar Nov 24 '25 06:11 lvyanquan

@lvyanquan style issue is fixed, thks for helping.

cshuo avatar Nov 24 '25 06:11 cshuo

@lvyanquan the ci is green. Could you help with the final review, thks~

cshuo avatar Nov 26 '25 07:11 cshuo

The failed test is org.apache.flink.cdc.pipeline.tests.MaxComputeE2eITCase.testSingleSplitSingleTable, should be irrelevant to this pr.

cshuo avatar Dec 01 '25 06:12 cshuo

@lvyanquan @yuxiqian gentle reminder for review, thks. cc @leonardBang

cshuo avatar Dec 04 '25 07:12 cshuo

Thanks @voonhous and @cshuo for the great work, the changes looks good to me, but currently our test coverage is pretty low, cloud you add more necessary tests ?

@leonardBang thks for reviewing, will add more tests later.

cshuo avatar Dec 16 '25 03:12 cshuo