Hans Eschbaum
Hans Eschbaum
@alexeykudinkin Hey Alexey, I'm also still getting the same error after updating to 0.12.1. Hudi: 0.12.1-amzn-0-SNAPSHOT Spark: 3.3.0 EMR: 6.9.0 `spark-submit --master yarn --deploy-mode cluster --conf spark.serializer=org.apache.spark.serializer.KryoSerializer,spark.sql.parquet.datetimeRebaseModeInRead=CORRECTED,spark.sql.parquet.datetimeRebaseModeInWrite=CORRECTED,spark.sql.avro.datetimeRebaseModeInWrite=CORRECTED,spark.sql.avro.datetimeRebaseModeInRead=CORRECTED,spark.sql.legacy.parquet.datetimeRebaseModeInRead=CORRECTED,spark.sql.legacy.parquet.datetimeRebaseModeInWrite=CORRECTED,spark.sql.legacy.parquet.int96RebaseModeInRead=CORRECTED,spark.sql.legacy.parquet.int96RebaseModeInWrite=CORRECTED --class org.apache.hudi.utilities.deltastreamer.HoodieDeltaStreamer /usr/lib/hudi/hudi-utilities-bundle.jar...
Never mind, I got it working. I specified the --conf wrong, having the options comma separated instead of separate --conf statements and it needs to have both spark.sql.avro and spark.sql.parquet...
Can you post your exact spark submit? Do you know why it's failing, what is the data type and value in the column?
Hi lucabem, I haven't run into that, I will have to test that out, maybe it'll get to that tomorrow and I can let you know my results, but I...