flink-cdc icon indicating copy to clipboard operation
flink-cdc copied to clipboard

[mysql]update primary key,Rowkind does not match the actual operation

Open yanghuaiGit opened this issue 3 years ago • 2 comments

Describe the bug(Please use English) A clear and concise description of what the bug is.

Environment :

  • Flink version :
  • Flink CDC version: master
  • Database and version:

To Reproduce Steps to reproduce the behavior:

  1. Thes test data :
  2. The test code :
  3. The error :

Additional Description If applicable, add screenshots to help explain your problem.

mysql create table t2 ( id int not null primary key, name varchar(255) null );

flinksql

CREATE TABLE debezium_source ( id int NOT NULL, name STRING, primary key (id) not enforced ) WITH ( 'connector' = 'mysql-cdc', 'hostname' = '127.0.0.1', 'port' = '3306', 'username' = 'root', 'password' = 'rootroot', 'database-name' = 'd2', 'table-name' = 't2')

表里有初始值数据 image

如果更新的字段是非主键,则flinkcdc接收数据为updateBefore和updateAfter 2条数据,但是如果主键更新了,则flinkcdc接收的数据为delete和insert2条数据

步骤: 1 UPDATE dujie.t2 t SET t.name = '3' WHERE t.id = 6 2 UPDATE dujie.t2 t SET t.id = 7 WHERE t.id = 6

image

yanghuaiGit avatar Jun 06 '22 04:06 yanghuaiGit

为什么更新主键的数据是 先delete然后insert呢 感觉这种设计是不是导致rowkind类型和数据库里的操作不一致 我将mysql里的binlog数据拿出来发现mysql里的sql都是update 语句

image

我发现是debeziun里的源码判断了是否是主键更新导致的,这儿是否要修改源码进行调整呢 image

yanghuaiGit avatar Jun 06 '22 04:06 yanghuaiGit

I think this behavior is reasonable.

If we get update type changes when the primary key is changed.We will get changes like this:

-U[6,3]
+U[7,3]

For downstream operators, we often need to send changes with the same primary key to the same operator, which will keep the binlog order for the record with the same primary key. Returning the data you said may lead to send the -U[6,3] to an operator, and send +U[7,3] to another, which is a wrong behavior. And we can not keep the order in the binlog.

ruanhang1993 avatar Jul 27 '22 09:07 ruanhang1993