hudi icon indicating copy to clipboard operation
hudi copied to clipboard
verified publisher icon apache

[SUPPORT]HoodieCleanMetadata and HoodieRestoreMetadata are missing

Open bithw1 opened this issue 1 year ago • 1 comments

Hi,

I download the Hudi source code from master branch,

Then I run the maven build mvn clean package -DskipTests, it looks that the build succeeds, but the model classes like HoodieCleanMetadata ,HoodieRestoreMetadata are not generated, but some are generated, like HoodieCommitMetadata.

I would ask how to generate these classes, thanks

[INFO] Reactor Summary for Hudi 1.0.0-SNAPSHOT:
[INFO]
[INFO] Hudi ............................................... SUCCESS [ 43.732 s]
[INFO] hudi-tests-common .................................. SUCCESS [ 34.005 s]
[INFO] hudi-io ............................................ SUCCESS [ 27.663 s]
[INFO] hudi-common ........................................ SUCCESS [01:52 min]
[INFO] hudi-hadoop-common ................................. SUCCESS [ 18.850 s]
[INFO] hudi-hadoop-mr ..................................... SUCCESS [ 45.268 s]
[INFO] hudi-sync-common ................................... SUCCESS [ 10.296 s]
[INFO] hudi-hive-sync ..................................... SUCCESS [ 46.931 s]
[INFO] hudi-aws ........................................... SUCCESS [ 28.202 s]
[INFO] hudi-timeline-service .............................. SUCCESS [ 13.416 s]
[INFO] hudi-client ........................................ SUCCESS [  0.153 s]
[INFO] hudi-client-common ................................. SUCCESS [ 47.937 s]
[INFO] hudi-spark-client .................................. SUCCESS [01:31 min]
[INFO] hudi-spark-datasource .............................. SUCCESS [  0.129 s]
[INFO] hudi-spark-common_2.12 ............................. SUCCESS [01:24 min]
[INFO] hudi-spark3-common ................................. SUCCESS [ 32.243 s]
[INFO] hudi-spark3.5.x_2.12 ............................... SUCCESS [01:11 min]
[INFO] hudi-java-client ................................... SUCCESS [ 31.958 s]
[INFO] hudi-spark_2.12 .................................... SUCCESS [01:49 min]
[INFO] hudi-gcp ........................................... SUCCESS [ 19.208 s]
[INFO] hudi-utilities_2.12 ................................ SUCCESS [01:57 min]
[INFO] hudi-utilities-bundle_2.12 ......................... SUCCESS [03:35 min]
[INFO] hudi-cli ........................................... SUCCESS [01:02 min]
[INFO] hudi-flink-client .................................. SUCCESS [ 49.350 s]
[INFO] hudi-datahub-sync .................................. SUCCESS [ 17.559 s]
[INFO] hudi-adb-sync ...................................... SUCCESS [ 13.450 s]
[INFO] hudi-sync .......................................... SUCCESS [  0.123 s]
[INFO] hudi-hadoop-mr-bundle .............................. SUCCESS [ 52.128 s]
[INFO] hudi-datahub-sync-bundle ........................... SUCCESS [ 46.528 s]
[INFO] hudi-hive-sync-bundle .............................. SUCCESS [ 36.086 s]
[INFO] hudi-aws-bundle .................................... SUCCESS [ 44.298 s]
[INFO] hudi-gcp-bundle .................................... SUCCESS [ 34.753 s]
[INFO] hudi-spark3.5-bundle_2.12 .......................... SUCCESS [01:18 min]
[INFO] hudi-presto-bundle ................................. SUCCESS [ 50.886 s]
[INFO] hudi-utilities-slim-bundle_2.12 .................... SUCCESS [01:28 min]
[INFO] hudi-timeline-server-bundle ........................ SUCCESS [01:04 min]
[INFO] hudi-trino-bundle .................................. SUCCESS [ 47.248 s]
[INFO] hudi-examples ...................................... SUCCESS [  0.116 s]
[INFO] hudi-examples-common ............................... SUCCESS [  5.957 s]
[INFO] hudi-examples-spark ................................ SUCCESS [ 39.172 s]
[INFO] hudi-flink-datasource .............................. SUCCESS [  0.138 s]
[INFO] hudi-flink1.19.x ................................... SUCCESS [ 18.550 s]
[INFO] hudi-flink ......................................... SUCCESS [ 45.245 s]
[INFO] hudi-examples-flink ................................ SUCCESS [ 15.046 s]
[INFO] hudi-examples-java ................................. SUCCESS [  8.294 s]
[INFO] hudi-flink1.19-bundle .............................. SUCCESS [01:02 min]
[INFO] hudi-examples-k8s .................................. SUCCESS [02:31 min]
[INFO] hudi-flink1.14.x ................................... SUCCESS [ 28.333 s]
[INFO] hudi-flink1.15.x ................................... SUCCESS [ 20.795 s]
[INFO] hudi-flink1.16.x ................................... SUCCESS [ 17.113 s]
[INFO] hudi-flink1.17.x ................................... SUCCESS [ 21.134 s]
[INFO] hudi-flink1.18.x ................................... SUCCESS [ 31.372 s]
[INFO] hudi-kafka-connect ................................. SUCCESS [ 37.436 s]
[INFO] hudi-kafka-connect-bundle .......................... SUCCESS [02:10 min]
[INFO] hudi-cli-bundle_2.12 ............................... SUCCESS [ 44.114 s]
[INFO] ------------------------------------------------------------------------
[INFO] BUILD SUCCESS
[INFO] ------------------------------------------------------------------------
[INFO] Total time:  43:04 min
[INFO] Finished at: 2024-09-28T14:46:06+08:00
[INFO] ------------------------------------------------------------------------

bithw1 avatar Sep 29 '24 12:09 bithw1

Hi @bithw1

I have confirmed that the HoodieCleanMetadata classes are present in both Hudi 0.15.0 and Hudi 1.0.0 jars. Could you please retry with following steps mentioned for 1.0.0 release.

Hudi 0.15.0:

jar tf hudi-common/target/hudi-common-0.15.0.jar | grep HoodieCleanMetadata
org/apache/hudi/avro/model/HoodieCleanMetadata.class
org/apache/hudi/avro/model/HoodieCleanMetadata$Builder.class
org/apache/hudi/avro/model/HoodieCleanMetadata$1.class

Hudi 1.0.0:

git checkout release-1.0.0-beta1
mvn clean install -DskipTests -U 

jar tf hudi-common/target/hudi-common-1.0.0-beta1.jar | grep HoodieCleanMetadata
org/apache/hudi/avro/model/HoodieCleanMetadata.class
org/apache/hudi/avro/model/HoodieCleanMetadata$Builder.class
org/apache/hudi/avro/model/HoodieCleanMetadata$1.class

rangareddy avatar Oct 16 '24 15:10 rangareddy

Hi @bithw1

I tested with Hudi master branch code (1.0.0) as well. I can see the classes are present.

jar tf hudi-common/target/hudi-common-1.0.0-SNAPSHOT.jar | grep HoodieCleanMetadata
org/apache/hudi/avro/model/HoodieCleanMetadata.class
org/apache/hudi/avro/model/HoodieCleanMetadata$Builder.class
org/apache/hudi/avro/model/HoodieCleanMetadata$1.class

rangareddy avatar Oct 21 '24 10:10 rangareddy

@bithw1 As @rangareddy confirmed, we should be good on this. so closing out. Feel free to reopen in case you still have a issue.

ad1happy2go avatar Oct 23 '24 16:10 ad1happy2go