bigtop icon indicating copy to clipboard operation
bigtop copied to clipboard

BIGTOP-4084. Upgrade Tez to 0.10.4.

Open iwasakims opened this issue 1 year ago • 4 comments

https://issues.apache.org/jira/browse/BIGTOP-4084

This PR bumps the verison of Tez to 0.10.3. We can remove most of the patch needed for Tez 0.10.2.

  • TEZ-4492, TEZ-4493, TEZ-4520 are contained in Tez 0.10.3.
  • We do not need TEZ-4319 since snappy-java-1.1.8.2 (supporting aarch64) is pulled as transitive dependency of hadoop now.

iwasakims avatar Apr 04 '24 13:04 iwasakims

I got protobuf related error on smoke-tests of Tez. I'm looking into the cause.

java.lang.IllegalAccessError: tried to access field com.google.protobuf.AbstractMessage.memoizedSize from class org.apache.tez.dag.api.records.DAGProtos$ConfigurationProto,

iwasakims avatar Apr 04 '24 13:04 iwasakims

While hadoop uses shaded protobuf now, Hadoop 3.3.6 retains dependency on protobuf-java-2.5.0 for compatibility and the protobuf-java-2.5.0.jar is in the classpath. Since TEZ-4363 upgraded the protobuf to 3.21., Tez must use its own protobuf-java-3.21.1.jar.

iwasakims avatar Apr 05 '24 15:04 iwasakims

Since init-hcfs.groovy is contained in hadoop package, smoke-tests need hadoop-pkg with the patch applied. You can manually put the tez.tar.gz on HDFS otherwise.

$ ./docker-hadoop.sh --create 1 ... --stack hdfs,yarn,mapreduce,tez
$ ./docker-hadoop.sh --exec 1 sudo -u hdfs hdfs dfs -put /usr/lib/tez/lib/tez.tar.gz /apps/tez/lib/
$ ./docker-hadoop.sh --smoke-tests tez

iwasakims avatar Oct 18 '24 01:10 iwasakims

+1, tested with Debian 11 and Rocky 9 on x86_64. Thanks @iwasakims.

sekikn avatar Oct 26 '24 04:10 sekikn