iceberg实际写入数量会比numWrite数量少

[382035656]

Search before asking

• [X] I had searched in the issues and found no similar issues.

What happened

MySQL读取200W数据写入iceberg累加器numRead和numWrite都为200w，iceberg实际查询只有60W

What you expected to happen

MySQL读取200W数据写入iceberg累加器numRead和numWrite都为200w，iceberg实际查询只有60W

How to reproduce

{ "job": { "content": [{ "reader": { "name": "mysqlreader", "parameter": { "column": [{ "name": "id", "type": "bigint" }, { "name": "update_time", "type": "datetime" } ], "connection": [{ "jdbcUrl": ["jdbc:mysql://192.168.20.233:3306/test_sink"], "schema": "test_sink", "table": ["test_sink"] } ], "customSql": "SELECT id,update_time FROM test_sink.test_sink", "password": "root", "username": "root" } }, "writer": { "name": "icebergwriter", "parameter": { "column": [{ "name": "ID", "type": "VARCHAR" }, { "name": "value", "type": "VARCHAR" } ], "defaultFS": "hdfs://192.168.42.131:9000", "hadoopConfig": { "dfs.client.datanode-restart.timeout": "30", "dfs.namenode.decommission.interval": "30", "fs.hdfs.impl.disable.cache": "true" }, "path": "hdfs://192.168.42.131:9000//flink/warehouse/20221230//icedb/TB_B_JOB_INFO2/", "writeMode": "overwrite" } } } ], "setting": { "speed": { "bytes": 0, "channel": 1 } } } }

Anything else

No response

Version

1.12_release

Are you willing to submit PR?

• [ ] Yes I am willing to submit a PR!

Code of Conduct

• [X] I agree to follow this project's Code of Conduct

[mggger]

@382035656 iceberg最近有更新过， 可以试试新版本，重新配置下参数吗？

https://github.com/DTStack/chunjun/blob/master/docs_zh/ChunJun%E8%BF%9E%E6%8E%A5%E5%99%A8/iceberg/iceberg-sink.md

[382035656]

@382035656 iceberg最近有更新过， 可以试试新版本，重新配置下参数吗？

https://github.com/DTStack/chunjun/blob/master/docs_zh/ChunJun%E8%BF%9E%E6%8E%A5%E5%99%A8/iceberg/iceberg-sink.md

新版本同样存在此问题，且新发现bug，iceberg库表不存在的时候也会执行成功,numWrite有数量，iceberg无数据文件

[382035656]

@382035656 iceberg最近有更新过， 可以试试新版本，重新配置下参数吗？

https://github.com/DTStack/chunjun/blob/master/docs_zh/ChunJun%E8%BF%9E%E6%8E%A5%E5%99%A8/iceberg/iceberg-sink.md

新版本同样存在此问题，且新发现bug，iceberg库表不存在的时候也会执行成功,numWrite有数量，iceberg无数据文件

[sqlboylove]

你好，现在问题解决了吗