[FLINK-35313][mysql-cdc] Add upsert changelog mode to avoid UPDATE_BEFORE records …

[yeezychao]

This PR implements #1898 and exposing support in configuration just for mysql module.

[yuxiqian]

Hi @yeezychao, thanks for your great contribution! Before this PR could be merged, could you please rebase it with latest master branch?

[yeezychao]

Hi @yuxiqian, thanks for your reply, PTAL.

[yuxiqian]

Hi @yeezychao, seems MySQL CI job is failing, could you please take a look? https://github.com/apache/flink-cdc/actions/runs/8844257194/job/24756979040?pr=1907

(The markdown lint check issue is irrelevant to this PR and should be fixed with a git rebase master.)

[yeezychao]

Hi @yeezychao, seems MySQL CI job is failing, could you please take a look? https://github.com/apache/flink-cdc/actions/runs/8844257194/job/24756979040?pr=1907

(The markdown lint check issue is irrelevant to this PR and should be fixed with a git rebase master.)

@yuxiqian Fixed

[yuxiqian]

Hi @yeezychao, you may run mvn spotless:apply to fix style violations, and use mvn verify to ensure your code is following the codestyle rules.

[yeezychao]

@yuxiqian Turning on upsert mode still fails to filter -u data. I am very confused as to why the same PR application failed to test in cdc 3.2 (flink 1.18) version, but it still works in version 2.2 (flink 1.15). Unfortunately, I have not found the reason yet.

[yuxiqian]

@yuxiqian Turning on upsert mode still fails to filter -u data. I am very confused as to why the same PR application failed to test in cdc 3.2 (flink 1.18) version, but it still works in version 2.2 (flink 1.15). Unfortunately, I have not found the reason yet.

@yeezychao Maybe check the output log and confirm if MySQL source actually sends any -U events to downstream? IIRC Flink will automatically append a ChangelogNormalize node to backfill missing update before events if source doesn't provide it.

[yeezychao]

@yuxiqian Turning on upsert mode still fails to filter -u data. I am very confused as to why the same PR application failed to test in cdc 3.2 (flink 1.18) version, but it still works in version 2.2 (flink 1.15). Unfortunately, I have not found the reason yet.

@yeezychao Maybe check the output log and confirm if MySQL source actually sends any -U events to downstream? IIRC Flink will automatically append a ChangelogNormalize node to backfill missing update before events if source doesn't provide it.

@yuxiqian You are right！The ChangelogNormalize node is indeed added under the Flink 1.18 version，but the Flink 1.15 version haven't.Is there a way to disable ChangelogNormalize?

[github-actions[bot]]

This pull request has been automatically marked as stale because it has not had recent activity for 60 days. It will be closed in 30 days if no further activity occurs.