BIGTOP-4084. Upgrade Tez to 0.10.4.
https://issues.apache.org/jira/browse/BIGTOP-4084
This PR bumps the verison of Tez to 0.10.3. We can remove most of the patch needed for Tez 0.10.2.
- TEZ-4492, TEZ-4493, TEZ-4520 are contained in Tez 0.10.3.
- We do not need TEZ-4319 since snappy-java-1.1.8.2 (supporting aarch64) is pulled as transitive dependency of hadoop now.
I got protobuf related error on smoke-tests of Tez. I'm looking into the cause.
java.lang.IllegalAccessError: tried to access field com.google.protobuf.AbstractMessage.memoizedSize from class org.apache.tez.dag.api.records.DAGProtos$ConfigurationProto,
While hadoop uses shaded protobuf now, Hadoop 3.3.6 retains dependency on protobuf-java-2.5.0 for compatibility and the protobuf-java-2.5.0.jar is in the classpath. Since TEZ-4363 upgraded the protobuf to 3.21., Tez must use its own protobuf-java-3.21.1.jar.
Since init-hcfs.groovy is contained in hadoop package, smoke-tests need hadoop-pkg with the patch applied. You can manually put the tez.tar.gz on HDFS otherwise.
$ ./docker-hadoop.sh --create 1 ... --stack hdfs,yarn,mapreduce,tez
$ ./docker-hadoop.sh --exec 1 sudo -u hdfs hdfs dfs -put /usr/lib/tez/lib/tez.tar.gz /apps/tez/lib/
$ ./docker-hadoop.sh --smoke-tests tez
+1, tested with Debian 11 and Rocky 9 on x86_64. Thanks @iwasakims.