flink-cdc icon indicating copy to clipboard operation
flink-cdc copied to clipboard

about the last one MySqlSnapshotSplit

Open jiangjin-f opened this issue 3 years ago • 1 comments

Describe the bug(Please use English) A clear and concise description of what the bug is.

Environment :

  • Flink version : 1.13.5
  • Flink CDC version: 2.2.1
  • Database and version: mysql

To Reproduce Steps to reproduce the behavior:

  1. Thes test data :
  2. The test code :
  3. The error :

Additional Description If applicable, add screenshots to help explain your problem.

When the last split is read, if the full amount of data is large and a lot of data is written per second, the last split directly uses the primary key > =?, The data volume is too large and the task fails

719fad9b-bcbd-465e-acc8-4d6e35e6a822

c0903bba-5a03-454e-bd2e-5cb100ff2ed8

jiangjin-f avatar Jul 14 '22 09:07 jiangjin-f

I also encountered such problem, the task manager OOM, looking for a solution

JustinLeesin avatar Aug 17 '22 08:08 JustinLeesin

I also encountered the same problem,When table is very large, it's oom

ByteInvoker avatar Apr 25 '23 08:04 ByteInvoker

Closing this issue because it was created before version 2.3.0 (2022-11-10). Please try the latest version of Flink CDC to see if the issue has been resolved. If the issue is still valid, kindly report it on Apache Jira under project Flink with component tag Flink CDC. Thank you!

PatrickRen avatar Feb 28 '24 15:02 PatrickRen