hadoop
hadoop copied to clipboard
apache
HADOOP-19091: Add support for Tez to MagicS3GuardCommitter
This commit adds a MRv1 wrapper for the MagicS3GuardCommitter so applications like Hive can use it.
How was this patch tested?
The ITest suite was run with the changes, using the ProfileAWSCredentialsProvider from HADOOP-19384 (will rerun all the tests once that PR is merged in its final form to be sure) using the us-east-1 region. Two tests were added to test the code to handle Tez-generated UUIDs, and to check that the output path generated by the JobContext codepath matches the prefix of the one generated by the TaskAttemptContext codepath.
:confetti_ball: +1 overall
| Vote | Subsystem | Runtime | Logfile | Comment |
|---|---|---|---|---|
| +0 :ok: | reexec | 0m 31s | Docker mode activated. | |
| _ Prechecks _ | ||||
| +1 :green_heart: | dupname | 0m 0s | No case conflicting files found. | |
| +0 :ok: | codespell | 0m 0s | codespell was not available. | |
| +0 :ok: | detsecrets | 0m 0s | detect-secrets was not available. | |
| +0 :ok: | xmllint | 0m 0s | xmllint was not available. | |
| +1 :green_heart: | @author | 0m 0s | The patch does not contain any @author tags. | |
| +1 :green_heart: | test4tests | 0m 0s | The patch appears to include 2 new or modified test files. | |
| _ trunk Compile Tests _ | ||||
| +0 :ok: | mvndep | 7m 6s | Maven dependency ordering for branch | |
| +1 :green_heart: | mvninstall | 31m 0s | trunk passed | |
| +1 :green_heart: | compile | 15m 35s | trunk passed with JDK Ubuntu-11.0.26+4-post-Ubuntu-1ubuntu120.04 | |
| +1 :green_heart: | compile | 13m 33s | trunk passed with JDK Private Build-1.8.0_442-8u442-b06~us1-0ubuntu1~20.04-b06 | |
| +1 :green_heart: | checkstyle | 4m 12s | trunk passed | |
| +1 :green_heart: | mvnsite | 1m 57s | trunk passed | |
| +1 :green_heart: | javadoc | 1m 40s | trunk passed with JDK Ubuntu-11.0.26+4-post-Ubuntu-1ubuntu120.04 | |
| +1 :green_heart: | javadoc | 1m 34s | trunk passed with JDK Private Build-1.8.0_442-8u442-b06~us1-0ubuntu1~20.04-b06 | |
| +1 :green_heart: | spotbugs | 2m 56s | trunk passed | |
| +1 :green_heart: | shadedclient | 35m 0s | branch has no errors when building and testing our client artifacts. | |
| _ Patch Compile Tests _ | ||||
| +0 :ok: | mvndep | 0m 33s | Maven dependency ordering for patch | |
| +1 :green_heart: | mvninstall | 1m 3s | the patch passed | |
| +1 :green_heart: | compile | 14m 52s | the patch passed with JDK Ubuntu-11.0.26+4-post-Ubuntu-1ubuntu120.04 | |
| +1 :green_heart: | javac | 14m 52s | the patch passed | |
| +1 :green_heart: | compile | 13m 39s | the patch passed with JDK Private Build-1.8.0_442-8u442-b06~us1-0ubuntu1~20.04-b06 | |
| +1 :green_heart: | javac | 13m 39s | the patch passed | |
| +1 :green_heart: | blanks | 0m 0s | The patch has no blanks issues. | |
| -0 :warning: | checkstyle | 4m 49s | /results-checkstyle-root.txt | root: The patch generated 8 new + 2 unchanged - 0 fixed = 10 total (was 2) |
| +1 :green_heart: | mvnsite | 1m 56s | the patch passed | |
| +1 :green_heart: | javadoc | 1m 36s | the patch passed with JDK Ubuntu-11.0.26+4-post-Ubuntu-1ubuntu120.04 | |
| +1 :green_heart: | javadoc | 1m 32s | the patch passed with JDK Private Build-1.8.0_442-8u442-b06~us1-0ubuntu1~20.04-b06 | |
| +1 :green_heart: | spotbugs | 3m 14s | the patch passed | |
| +1 :green_heart: | shadedclient | 34m 44s | patch has no errors when building and testing our client artifacts. | |
| _ Other Tests _ | ||||
| +1 :green_heart: | unit | 9m 2s | hadoop-mapreduce-client-core in the patch passed. | |
| +1 :green_heart: | unit | 3m 29s | hadoop-aws in the patch passed. | |
| +1 :green_heart: | asflicense | 1m 2s | The patch does not generate ASF License warnings. | |
| 210m 36s |
| Subsystem | Report/Notes |
|---|---|
| Docker | ClientAPI=1.48 ServerAPI=1.48 base: https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-7441/1/artifact/out/Dockerfile |
| GITHUB PR | https://github.com/apache/hadoop/pull/7441 |
| JIRA Issue | HADOOP-19091 |
| Optional Tests | dupname asflicense compile javac javadoc mvninstall mvnsite unit shadedclient spotbugs checkstyle codespell detsecrets xmllint |
| uname | Linux d9e794223f3c 5.15.0-131-generic #141-Ubuntu SMP Fri Jan 10 21:18:28 UTC 2025 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality | dev-support/bin/hadoop.sh |
| git revision | trunk / dcd41150cd64fc81ed2a3d43430ce8cfeb5d12bf |
| Default Java | Private Build-1.8.0_442-8u442-b06~us1-0ubuntu1~20.04-b06 |
| Multi-JDK versions | /usr/lib/jvm/java-11-openjdk-amd64:Ubuntu-11.0.26+4-post-Ubuntu-1ubuntu120.04 /usr/lib/jvm/java-8-openjdk-amd64:Private Build-1.8.0_442-8u442-b06~us1-0ubuntu1~20.04-b06 |
| Test Results | https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-7441/1/testReport/ |
| Max. process+thread count | 1434 (vs. ulimit of 5500) |
| modules | C: hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core hadoop-tools/hadoop-aws U: . |
| Console output | https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-7441/1/console |
| versions | git=2.25.1 maven=3.6.3 spotbugs=4.2.2 |
| Powered by | Apache Yetus 0.14.0 https://yetus.apache.org |
This message was automatically generated.
:confetti_ball: +1 overall
| Vote | Subsystem | Runtime | Logfile | Comment |
|---|---|---|---|---|
| +0 :ok: | reexec | 0m 37s | Docker mode activated. | |
| _ Prechecks _ | ||||
| +1 :green_heart: | dupname | 0m 0s | No case conflicting files found. | |
| +0 :ok: | codespell | 0m 1s | codespell was not available. | |
| +0 :ok: | detsecrets | 0m 1s | detect-secrets was not available. | |
| +0 :ok: | xmllint | 0m 1s | xmllint was not available. | |
| +1 :green_heart: | @author | 0m 0s | The patch does not contain any @author tags. | |
| +1 :green_heart: | test4tests | 0m 0s | The patch appears to include 2 new or modified test files. | |
| _ trunk Compile Tests _ | ||||
| +0 :ok: | mvndep | 6m 17s | Maven dependency ordering for branch | |
| +1 :green_heart: | mvninstall | 33m 46s | trunk passed | |
| +1 :green_heart: | compile | 15m 39s | trunk passed with JDK Ubuntu-11.0.26+4-post-Ubuntu-1ubuntu120.04 | |
| +1 :green_heart: | compile | 13m 21s | trunk passed with JDK Private Build-1.8.0_442-8u442-b06~us1-0ubuntu1~20.04-b06 | |
| +1 :green_heart: | checkstyle | 4m 16s | trunk passed | |
| +1 :green_heart: | mvnsite | 1m 55s | trunk passed | |
| +1 :green_heart: | javadoc | 1m 42s | trunk passed with JDK Ubuntu-11.0.26+4-post-Ubuntu-1ubuntu120.04 | |
| +1 :green_heart: | javadoc | 1m 35s | trunk passed with JDK Private Build-1.8.0_442-8u442-b06~us1-0ubuntu1~20.04-b06 | |
| +1 :green_heart: | spotbugs | 2m 57s | trunk passed | |
| +1 :green_heart: | shadedclient | 34m 40s | branch has no errors when building and testing our client artifacts. | |
| _ Patch Compile Tests _ | ||||
| +0 :ok: | mvndep | 0m 34s | Maven dependency ordering for patch | |
| +1 :green_heart: | mvninstall | 1m 2s | the patch passed | |
| +1 :green_heart: | compile | 14m 59s | the patch passed with JDK Ubuntu-11.0.26+4-post-Ubuntu-1ubuntu120.04 | |
| +1 :green_heart: | javac | 14m 59s | the patch passed | |
| +1 :green_heart: | compile | 13m 41s | the patch passed with JDK Private Build-1.8.0_442-8u442-b06~us1-0ubuntu1~20.04-b06 | |
| +1 :green_heart: | javac | 13m 41s | the patch passed | |
| +1 :green_heart: | blanks | 0m 0s | The patch has no blanks issues. | |
| -0 :warning: | checkstyle | 4m 13s | /results-checkstyle-root.txt | root: The patch generated 1 new + 2 unchanged - 0 fixed = 3 total (was 2) |
| +1 :green_heart: | mvnsite | 1m 58s | the patch passed | |
| +1 :green_heart: | javadoc | 1m 32s | the patch passed with JDK Ubuntu-11.0.26+4-post-Ubuntu-1ubuntu120.04 | |
| +1 :green_heart: | javadoc | 1m 35s | the patch passed with JDK Private Build-1.8.0_442-8u442-b06~us1-0ubuntu1~20.04-b06 | |
| +1 :green_heart: | spotbugs | 3m 16s | the patch passed | |
| +1 :green_heart: | shadedclient | 35m 22s | patch has no errors when building and testing our client artifacts. | |
| _ Other Tests _ | ||||
| +1 :green_heart: | unit | 8m 54s | hadoop-mapreduce-client-core in the patch passed. | |
| +1 :green_heart: | unit | 3m 30s | hadoop-aws in the patch passed. | |
| +1 :green_heart: | asflicense | 1m 5s | The patch does not generate ASF License warnings. | |
| 212m 35s |
| Subsystem | Report/Notes |
|---|---|
| Docker | ClientAPI=1.48 ServerAPI=1.48 base: https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-7441/2/artifact/out/Dockerfile |
| GITHUB PR | https://github.com/apache/hadoop/pull/7441 |
| JIRA Issue | HADOOP-19091 |
| Optional Tests | dupname asflicense compile javac javadoc mvninstall mvnsite unit shadedclient spotbugs checkstyle codespell detsecrets xmllint |
| uname | Linux 7e4163db0a39 5.15.0-131-generic #141-Ubuntu SMP Fri Jan 10 21:18:28 UTC 2025 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality | dev-support/bin/hadoop.sh |
| git revision | trunk / 1a574675122b899cb521cc308bbfba79cdef2100 |
| Default Java | Private Build-1.8.0_442-8u442-b06~us1-0ubuntu1~20.04-b06 |
| Multi-JDK versions | /usr/lib/jvm/java-11-openjdk-amd64:Ubuntu-11.0.26+4-post-Ubuntu-1ubuntu120.04 /usr/lib/jvm/java-8-openjdk-amd64:Private Build-1.8.0_442-8u442-b06~us1-0ubuntu1~20.04-b06 |
| Test Results | https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-7441/2/testReport/ |
| Max. process+thread count | 1570 (vs. ulimit of 5500) |
| modules | C: hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core hadoop-tools/hadoop-aws U: . |
| Console output | https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-7441/2/console |
| versions | git=2.25.1 maven=3.6.3 spotbugs=4.2.2 |
| Powered by | Apache Yetus 0.14.0 https://yetus.apache.org |
This message was automatically generated.
:confetti_ball: +1 overall
| Vote | Subsystem | Runtime | Logfile | Comment |
|---|---|---|---|---|
| +0 :ok: | reexec | 0m 32s | Docker mode activated. | |
| _ Prechecks _ | ||||
| +1 :green_heart: | dupname | 0m 0s | No case conflicting files found. | |
| +0 :ok: | codespell | 0m 1s | codespell was not available. | |
| +0 :ok: | detsecrets | 0m 1s | detect-secrets was not available. | |
| +0 :ok: | xmllint | 0m 1s | xmllint was not available. | |
| +1 :green_heart: | @author | 0m 0s | The patch does not contain any @author tags. | |
| +1 :green_heart: | test4tests | 0m 0s | The patch appears to include 2 new or modified test files. | |
| _ trunk Compile Tests _ | ||||
| +0 :ok: | mvndep | 6m 43s | Maven dependency ordering for branch | |
| +1 :green_heart: | mvninstall | 31m 45s | trunk passed | |
| +1 :green_heart: | compile | 15m 32s | trunk passed with JDK Ubuntu-11.0.26+4-post-Ubuntu-1ubuntu120.04 | |
| +1 :green_heart: | compile | 13m 53s | trunk passed with JDK Private Build-1.8.0_442-8u442-b06~us1-0ubuntu1~20.04-b06 | |
| +1 :green_heart: | checkstyle | 4m 12s | trunk passed | |
| +1 :green_heart: | mvnsite | 1m 54s | trunk passed | |
| +1 :green_heart: | javadoc | 1m 42s | trunk passed with JDK Ubuntu-11.0.26+4-post-Ubuntu-1ubuntu120.04 | |
| +1 :green_heart: | javadoc | 1m 33s | trunk passed with JDK Private Build-1.8.0_442-8u442-b06~us1-0ubuntu1~20.04-b06 | |
| +1 :green_heart: | spotbugs | 2m 56s | trunk passed | |
| +1 :green_heart: | shadedclient | 34m 53s | branch has no errors when building and testing our client artifacts. | |
| _ Patch Compile Tests _ | ||||
| +0 :ok: | mvndep | 0m 32s | Maven dependency ordering for patch | |
| +1 :green_heart: | mvninstall | 1m 1s | the patch passed | |
| +1 :green_heart: | compile | 14m 46s | the patch passed with JDK Ubuntu-11.0.26+4-post-Ubuntu-1ubuntu120.04 | |
| +1 :green_heart: | javac | 14m 46s | the patch passed | |
| +1 :green_heart: | compile | 13m 34s | the patch passed with JDK Private Build-1.8.0_442-8u442-b06~us1-0ubuntu1~20.04-b06 | |
| +1 :green_heart: | javac | 13m 34s | the patch passed | |
| +1 :green_heart: | blanks | 0m 0s | The patch has no blanks issues. | |
| +1 :green_heart: | checkstyle | 4m 13s | the patch passed | |
| +1 :green_heart: | mvnsite | 1m 53s | the patch passed | |
| +1 :green_heart: | javadoc | 1m 38s | the patch passed with JDK Ubuntu-11.0.26+4-post-Ubuntu-1ubuntu120.04 | |
| +1 :green_heart: | javadoc | 1m 35s | the patch passed with JDK Private Build-1.8.0_442-8u442-b06~us1-0ubuntu1~20.04-b06 | |
| +1 :green_heart: | spotbugs | 3m 17s | the patch passed | |
| +1 :green_heart: | shadedclient | 35m 6s | patch has no errors when building and testing our client artifacts. | |
| _ Other Tests _ | ||||
| +1 :green_heart: | unit | 9m 1s | hadoop-mapreduce-client-core in the patch passed. | |
| +1 :green_heart: | unit | 3m 40s | hadoop-aws in the patch passed. | |
| +1 :green_heart: | asflicense | 1m 6s | The patch does not generate ASF License warnings. | |
| 210m 57s |
| Subsystem | Report/Notes |
|---|---|
| Docker | ClientAPI=1.48 ServerAPI=1.48 base: https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-7441/3/artifact/out/Dockerfile |
| GITHUB PR | https://github.com/apache/hadoop/pull/7441 |
| JIRA Issue | HADOOP-19091 |
| Optional Tests | dupname asflicense compile javac javadoc mvninstall mvnsite unit shadedclient spotbugs checkstyle codespell detsecrets xmllint |
| uname | Linux 01cf0f523040 5.15.0-131-generic #141-Ubuntu SMP Fri Jan 10 21:18:28 UTC 2025 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality | dev-support/bin/hadoop.sh |
| git revision | trunk / 37cc2d326e6c62d37b5377490ed2518754931d1a |
| Default Java | Private Build-1.8.0_442-8u442-b06~us1-0ubuntu1~20.04-b06 |
| Multi-JDK versions | /usr/lib/jvm/java-11-openjdk-amd64:Ubuntu-11.0.26+4-post-Ubuntu-1ubuntu120.04 /usr/lib/jvm/java-8-openjdk-amd64:Private Build-1.8.0_442-8u442-b06~us1-0ubuntu1~20.04-b06 |
| Test Results | https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-7441/3/testReport/ |
| Max. process+thread count | 1570 (vs. ulimit of 5500) |
| modules | C: hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core hadoop-tools/hadoop-aws U: . |
| Console output | https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-7441/3/console |
| versions | git=2.25.1 maven=3.6.3 spotbugs=4.2.2 |
| Powered by | Apache Yetus 0.14.0 https://yetus.apache.org |
This message was automatically generated.
Any update? Can we continue to advance this PR so that Hive can utilize this feature?
Thanks.
We're closing this stale PR because it has been open for 100 days with no activity. This isn't a judgement on the merit of the PR in any way. It's just a way of keeping the PR queue manageable. If you feel like this was a mistake, or you would like to continue working on it, please feel free to re-open it and ask for a committer to remove the stale tag and review again. Thanks all for your contribution.
![github-actions[bot] avatar](https://avatars.githubusercontent.com/in/15368?v=4 "Published by a pub.dev verified publisher"){kind=small}
:broken_heart: -1 overall
| Vote | Subsystem | Runtime | Logfile | Comment |
|---|---|---|---|---|
| +0 :ok: | reexec | 0m 52s | Docker mode activated. | |
| _ Prechecks _ | ||||
| +1 :green_heart: | dupname | 0m 0s | No case conflicting files found. | |
| +0 :ok: | codespell | 0m 1s | codespell was not available. | |
| +0 :ok: | detsecrets | 0m 1s | detect-secrets was not available. | |
| +0 :ok: | xmllint | 0m 1s | xmllint was not available. | |
| +1 :green_heart: | @author | 0m 0s | The patch does not contain any @author tags. | |
| +1 :green_heart: | test4tests | 0m 0s | The patch appears to include 2 new or modified test files. | |
| _ trunk Compile Tests _ | ||||
| +0 :ok: | mvndep | 7m 53s | Maven dependency ordering for branch | |
| +1 :green_heart: | mvninstall | 36m 12s | trunk passed | |
| +1 :green_heart: | compile | 17m 33s | trunk passed with JDK Ubuntu-21.0.7+6-Ubuntu-0ubuntu120.04 | |
| +1 :green_heart: | compile | 17m 54s | trunk passed with JDK Ubuntu-17.0.15+6-Ubuntu-0ubuntu120.04 | |
| +1 :green_heart: | checkstyle | 3m 12s | trunk passed | |
| +1 :green_heart: | mvnsite | 2m 11s | trunk passed | |
| +1 :green_heart: | javadoc | 1m 38s | trunk passed with JDK Ubuntu-21.0.7+6-Ubuntu-0ubuntu120.04 | |
| +1 :green_heart: | javadoc | 1m 34s | trunk passed with JDK Ubuntu-17.0.15+6-Ubuntu-0ubuntu120.04 | |
| -1 :x: | spotbugs | 1m 58s | /branch-spotbugs-hadoop-mapreduce-project_hadoop-mapreduce-client_hadoop-mapreduce-client-core-warnings.html | hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core in trunk has 178 extant spotbugs warnings. |
| -1 :x: | spotbugs | 1m 39s | /branch-spotbugs-hadoop-tools_hadoop-aws-warnings.html | hadoop-tools/hadoop-aws in trunk has 188 extant spotbugs warnings. |
| +1 :green_heart: | shadedclient | 31m 55s | branch has no errors when building and testing our client artifacts. | |
| _ Patch Compile Tests _ | ||||
| +0 :ok: | mvndep | 0m 30s | Maven dependency ordering for patch | |
| -1 :x: | mvninstall | 0m 37s | /patch-mvninstall-hadoop-tools_hadoop-aws.txt | hadoop-aws in the patch failed. |
| -1 :x: | compile | 15m 46s | /patch-compile-root-jdkUbuntu-21.0.7+6-Ubuntu-0ubuntu120.04.txt | root in the patch failed with JDK Ubuntu-21.0.7+6-Ubuntu-0ubuntu120.04. |
| -1 :x: | javac | 15m 46s | /patch-compile-root-jdkUbuntu-21.0.7+6-Ubuntu-0ubuntu120.04.txt | root in the patch failed with JDK Ubuntu-21.0.7+6-Ubuntu-0ubuntu120.04. |
| -1 :x: | compile | 17m 6s | /patch-compile-root-jdkUbuntu-17.0.15+6-Ubuntu-0ubuntu120.04.txt | root in the patch failed with JDK Ubuntu-17.0.15+6-Ubuntu-0ubuntu120.04. |
| -1 :x: | javac | 17m 6s | /patch-compile-root-jdkUbuntu-17.0.15+6-Ubuntu-0ubuntu120.04.txt | root in the patch failed with JDK Ubuntu-17.0.15+6-Ubuntu-0ubuntu120.04. |
| +1 :green_heart: | blanks | 0m 0s | The patch has no blanks issues. | |
| +1 :green_heart: | checkstyle | 3m 9s | the patch passed | |
| -1 :x: | mvnsite | 1m 0s | /patch-mvnsite-hadoop-tools_hadoop-aws.txt | hadoop-aws in the patch failed. |
| -1 :x: | javadoc | 0m 49s | /results-javadoc-javadoc-hadoop-tools_hadoop-aws-jdkUbuntu-21.0.7+6-Ubuntu-0ubuntu120.04.txt | hadoop-tools_hadoop-aws-jdkUbuntu-21.0.7+6-Ubuntu-0ubuntu120.04 with JDK Ubuntu-21.0.7+6-Ubuntu-0ubuntu120.04 generated 8 new + 926 unchanged - 0 fixed = 934 total (was 926) |
| -1 :x: | javadoc | 0m 50s | /results-javadoc-javadoc-hadoop-tools_hadoop-aws-jdkUbuntu-17.0.15+6-Ubuntu-0ubuntu120.04.txt | hadoop-tools_hadoop-aws-jdkUbuntu-17.0.15+6-Ubuntu-0ubuntu120.04 with JDK Ubuntu-17.0.15+6-Ubuntu-0ubuntu120.04 generated 7 new + 848 unchanged - 0 fixed = 855 total (was 848) |
| -1 :x: | spotbugs | 0m 58s | /patch-spotbugs-hadoop-tools_hadoop-aws.txt | hadoop-aws in the patch failed. |
| +1 :green_heart: | shadedclient | 33m 19s | patch has no errors when building and testing our client artifacts. | |
| _ Other Tests _ | ||||
| +1 :green_heart: | unit | 8m 53s | hadoop-mapreduce-client-core in the patch passed. | |
| -1 :x: | unit | 0m 59s | /patch-unit-hadoop-tools_hadoop-aws.txt | hadoop-aws in the patch failed. |
| +1 :green_heart: | asflicense | 1m 1s | The patch does not generate ASF License warnings. | |
| 218m 13s |
| Subsystem | Report/Notes |
|---|---|
| Docker | ClientAPI=1.52 ServerAPI=1.52 base: https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-7441/4/artifact/out/Dockerfile |
| GITHUB PR | https://github.com/apache/hadoop/pull/7441 |
| JIRA Issue | HADOOP-19091 |
| Optional Tests | dupname asflicense compile javac javadoc mvninstall mvnsite unit shadedclient spotbugs checkstyle codespell detsecrets xmllint |
| uname | Linux bd47a9aa172f 5.15.0-157-generic #167-Ubuntu SMP Wed Sep 17 21:35:53 UTC 2025 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality | dev-support/bin/hadoop.sh |
| git revision | trunk / 17ee8e4c99b02a76b0a53701aa8863b77d583cfe |
| Default Java | Ubuntu-17.0.15+6-Ubuntu-0ubuntu120.04 |
| Multi-JDK versions | /usr/lib/jvm/java-21-openjdk-amd64:Ubuntu-21.0.7+6-Ubuntu-0ubuntu120.04 /usr/lib/jvm/java-17-openjdk-amd64:Ubuntu-17.0.15+6-Ubuntu-0ubuntu120.04 |
| Test Results | https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-7441/4/testReport/ |
| Max. process+thread count | 1087 (vs. ulimit of 5500) |
| modules | C: hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core hadoop-tools/hadoop-aws U: . |
| Console output | https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-7441/4/console |
| versions | git=2.25.1 maven=3.9.11 spotbugs=4.9.7 |
| Powered by | Apache Yetus 0.14.0 https://yetus.apache.org |
This message was automatically generated.
:broken_heart: -1 overall
| Vote | Subsystem | Runtime | Logfile | Comment |
|---|---|---|---|---|
| +0 :ok: | reexec | 0m 54s | Docker mode activated. | |
| _ Prechecks _ | ||||
| +1 :green_heart: | dupname | 0m 1s | No case conflicting files found. | |
| +0 :ok: | codespell | 0m 0s | codespell was not available. | |
| +0 :ok: | detsecrets | 0m 0s | detect-secrets was not available. | |
| +0 :ok: | xmllint | 0m 0s | xmllint was not available. | |
| +1 :green_heart: | @author | 0m 0s | The patch does not contain any @author tags. | |
| +1 :green_heart: | test4tests | 0m 0s | The patch appears to include 2 new or modified test files. | |
| _ trunk Compile Tests _ | ||||
| +0 :ok: | mvndep | 10m 54s | Maven dependency ordering for branch | |
| +1 :green_heart: | mvninstall | 36m 8s | trunk passed | |
| +1 :green_heart: | compile | 17m 56s | trunk passed with JDK Ubuntu-21.0.7+6-Ubuntu-0ubuntu120.04 | |
| +1 :green_heart: | compile | 17m 57s | trunk passed with JDK Ubuntu-17.0.15+6-Ubuntu-0ubuntu120.04 | |
| +1 :green_heart: | checkstyle | 3m 39s | trunk passed | |
| +1 :green_heart: | mvnsite | 2m 11s | trunk passed | |
| +1 :green_heart: | javadoc | 1m 40s | trunk passed with JDK Ubuntu-21.0.7+6-Ubuntu-0ubuntu120.04 | |
| +1 :green_heart: | javadoc | 1m 36s | trunk passed with JDK Ubuntu-17.0.15+6-Ubuntu-0ubuntu120.04 | |
| -1 :x: | spotbugs | 2m 0s | /branch-spotbugs-hadoop-mapreduce-project_hadoop-mapreduce-client_hadoop-mapreduce-client-core-warnings.html | hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core in trunk has 178 extant spotbugs warnings. |
| -1 :x: | spotbugs | 1m 38s | /branch-spotbugs-hadoop-tools_hadoop-aws-warnings.html | hadoop-tools/hadoop-aws in trunk has 188 extant spotbugs warnings. |
| +1 :green_heart: | shadedclient | 32m 14s | branch has no errors when building and testing our client artifacts. | |
| _ Patch Compile Tests _ | ||||
| +0 :ok: | mvndep | 0m 32s | Maven dependency ordering for patch | |
| +1 :green_heart: | mvninstall | 1m 18s | the patch passed | |
| +1 :green_heart: | compile | 17m 1s | the patch passed with JDK Ubuntu-21.0.7+6-Ubuntu-0ubuntu120.04 | |
| +1 :green_heart: | javac | 17m 1s | the patch passed | |
| +1 :green_heart: | compile | 17m 57s | the patch passed with JDK Ubuntu-17.0.15+6-Ubuntu-0ubuntu120.04 | |
| +1 :green_heart: | javac | 17m 57s | the patch passed | |
| +1 :green_heart: | blanks | 0m 0s | The patch has no blanks issues. | |
| +1 :green_heart: | checkstyle | 3m 37s | the patch passed | |
| +1 :green_heart: | mvnsite | 2m 8s | the patch passed | |
| -1 :x: | javadoc | 0m 52s | /results-javadoc-javadoc-hadoop-tools_hadoop-aws-jdkUbuntu-21.0.7+6-Ubuntu-0ubuntu120.04.txt | hadoop-tools_hadoop-aws-jdkUbuntu-21.0.7+6-Ubuntu-0ubuntu120.04 with JDK Ubuntu-21.0.7+6-Ubuntu-0ubuntu120.04 generated 8 new + 926 unchanged - 0 fixed = 934 total (was 926) |
| -1 :x: | javadoc | 0m 51s | /results-javadoc-javadoc-hadoop-tools_hadoop-aws-jdkUbuntu-17.0.15+6-Ubuntu-0ubuntu120.04.txt | hadoop-tools_hadoop-aws-jdkUbuntu-17.0.15+6-Ubuntu-0ubuntu120.04 with JDK Ubuntu-17.0.15+6-Ubuntu-0ubuntu120.04 generated 7 new + 848 unchanged - 0 fixed = 855 total (was 848) |
| -1 :x: | spotbugs | 1m 55s | /new-spotbugs-hadoop-tools_hadoop-aws.html | hadoop-tools/hadoop-aws generated 3 new + 188 unchanged - 0 fixed = 191 total (was 188) |
| +1 :green_heart: | shadedclient | 33m 30s | patch has no errors when building and testing our client artifacts. | |
| _ Other Tests _ | ||||
| +1 :green_heart: | unit | 8m 45s | hadoop-mapreduce-client-core in the patch passed. | |
| +1 :green_heart: | unit | 3m 37s | hadoop-aws in the patch passed. | |
| +1 :green_heart: | asflicense | 1m 3s | The patch does not generate ASF License warnings. | |
| 229m 36s |
| Reason | Tests |
|---|---|
| SpotBugs | module:hadoop-tools/hadoop-aws |
| Unknown bug pattern CT_CONSTRUCTOR_THROW in new org.apache.hadoop.fs.s3a.commit.AbstractS3ACommitter(Path, JobContext) At AbstractS3ACommitter.java:new org.apache.hadoop.fs.s3a.commit.AbstractS3ACommitter(Path, JobContext) At AbstractS3ACommitter.java:[line 262] | |
| Unknown bug pattern CT_CONSTRUCTOR_THROW in new org.apache.hadoop.fs.s3a.commit.magic.MagicS3GuardCommitter(Path, JobContext) At MagicS3GuardCommitter.java:new org.apache.hadoop.fs.s3a.commit.magic.MagicS3GuardCommitter(Path, JobContext) At MagicS3GuardCommitter.java:[line 98] | |
| Unknown bug pattern CT_CONSTRUCTOR_THROW in new org.apache.hadoop.fs.s3a.commit.staging.StagingCommitter(Path, JobContext) At StagingCommitter.java:new org.apache.hadoop.fs.s3a.commit.staging.StagingCommitter(Path, JobContext) At StagingCommitter.java:[line 153] |
| Subsystem | Report/Notes |
|---|---|
| Docker | ClientAPI=1.52 ServerAPI=1.52 base: https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-7441/5/artifact/out/Dockerfile |
| GITHUB PR | https://github.com/apache/hadoop/pull/7441 |
| JIRA Issue | HADOOP-19091 |
| Optional Tests | dupname asflicense compile javac javadoc mvninstall mvnsite unit shadedclient spotbugs checkstyle codespell detsecrets xmllint |
| uname | Linux 572e0c06e3b2 5.15.0-157-generic #167-Ubuntu SMP Wed Sep 17 21:35:53 UTC 2025 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality | dev-support/bin/hadoop.sh |
| git revision | trunk / 92624bbf4d053211b5af1104d0cbd6d3a0cb6ab4 |
| Default Java | Ubuntu-17.0.15+6-Ubuntu-0ubuntu120.04 |
| Multi-JDK versions | /usr/lib/jvm/java-21-openjdk-amd64:Ubuntu-21.0.7+6-Ubuntu-0ubuntu120.04 /usr/lib/jvm/java-17-openjdk-amd64:Ubuntu-17.0.15+6-Ubuntu-0ubuntu120.04 |
| Test Results | https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-7441/5/testReport/ |
| Max. process+thread count | 1086 (vs. ulimit of 5500) |
| modules | C: hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core hadoop-tools/hadoop-aws U: . |
| Console output | https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-7441/5/console |
| versions | git=2.25.1 maven=3.9.11 spotbugs=4.9.7 |
| Powered by | Apache Yetus 0.14.0 https://yetus.apache.org |
This message was automatically generated.
@steveloughran Most of my test failures were fixed once I rebased to current trunk. I ran the hadoop-aws ITests on an AWS EMR cluster (in us-east-1) against S3 us-east-1 , and the only test failure is an object upload timeout:
ERROR] org.apache.hadoop.fs.s3a.impl.ITestConnectionTimeouts.testObjectUploadTimeouts -- Time elapsed: 18.71 s <<< FAILURE! java.lang.AssertionError: [Duration of write] Expecting: <PT16.212S> to be less than: <PT10S> at org.apache.hadoop.fs.s3a.impl.ITestConnectionTimeouts.testObjectUploadTimeouts(ITestConnectionTimeouts.java:247) at java.lang.reflect.Method.invoke(Method.java:498) at java.util.ArrayList.forEach(ArrayList.java:1259) at java.util.ArrayList.forEach(ArrayList.java:1259)
Yetus is reporting some javadoc warnings, will go through and see if any other javadoc comments can be added that would be pertinent. This PR can be marked as not stale, let me know if anything in particular needs fixing. checkstyle seems to be passing.
:broken_heart: -1 overall
| Vote | Subsystem | Runtime | Logfile | Comment |
|---|---|---|---|---|
| +0 :ok: | reexec | 0m 50s | Docker mode activated. | |
| _ Prechecks _ | ||||
| +1 :green_heart: | dupname | 0m 0s | No case conflicting files found. | |
| +0 :ok: | codespell | 0m 1s | codespell was not available. | |
| +0 :ok: | detsecrets | 0m 1s | detect-secrets was not available. | |
| +0 :ok: | xmllint | 0m 1s | xmllint was not available. | |
| +1 :green_heart: | @author | 0m 0s | The patch does not contain any @author tags. | |
| +1 :green_heart: | test4tests | 0m 0s | The patch appears to include 2 new or modified test files. | |
| _ trunk Compile Tests _ | ||||
| +0 :ok: | mvndep | 7m 50s | Maven dependency ordering for branch | |
| +1 :green_heart: | mvninstall | 35m 1s | trunk passed | |
| +1 :green_heart: | compile | 17m 22s | trunk passed with JDK Ubuntu-21.0.7+6-Ubuntu-0ubuntu120.04 | |
| +1 :green_heart: | compile | 17m 48s | trunk passed with JDK Ubuntu-17.0.15+6-Ubuntu-0ubuntu120.04 | |
| +1 :green_heart: | checkstyle | 3m 39s | trunk passed | |
| +1 :green_heart: | mvnsite | 2m 12s | trunk passed | |
| +1 :green_heart: | javadoc | 1m 39s | trunk passed with JDK Ubuntu-21.0.7+6-Ubuntu-0ubuntu120.04 | |
| +1 :green_heart: | javadoc | 1m 34s | trunk passed with JDK Ubuntu-17.0.15+6-Ubuntu-0ubuntu120.04 | |
| -1 :x: | spotbugs | 1m 59s | /branch-spotbugs-hadoop-mapreduce-project_hadoop-mapreduce-client_hadoop-mapreduce-client-core-warnings.html | hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core in trunk has 178 extant spotbugs warnings. |
| -1 :x: | spotbugs | 1m 34s | /branch-spotbugs-hadoop-tools_hadoop-aws-warnings.html | hadoop-tools/hadoop-aws in trunk has 188 extant spotbugs warnings. |
| +1 :green_heart: | shadedclient | 32m 17s | branch has no errors when building and testing our client artifacts. | |
| _ Patch Compile Tests _ | ||||
| +0 :ok: | mvndep | 0m 31s | Maven dependency ordering for patch | |
| +1 :green_heart: | mvninstall | 1m 19s | the patch passed | |
| +1 :green_heart: | compile | 16m 37s | the patch passed with JDK Ubuntu-21.0.7+6-Ubuntu-0ubuntu120.04 | |
| +1 :green_heart: | javac | 16m 37s | the patch passed | |
| +1 :green_heart: | compile | 17m 46s | the patch passed with JDK Ubuntu-17.0.15+6-Ubuntu-0ubuntu120.04 | |
| +1 :green_heart: | javac | 17m 46s | the patch passed | |
| +1 :green_heart: | blanks | 0m 0s | The patch has no blanks issues. | |
| +1 :green_heart: | checkstyle | 3m 37s | the patch passed | |
| +1 :green_heart: | mvnsite | 2m 8s | the patch passed | |
| -1 :x: | javadoc | 0m 51s | /results-javadoc-javadoc-hadoop-tools_hadoop-aws-jdkUbuntu-21.0.7+6-Ubuntu-0ubuntu120.04.txt | hadoop-tools_hadoop-aws-jdkUbuntu-21.0.7+6-Ubuntu-0ubuntu120.04 with JDK Ubuntu-21.0.7+6-Ubuntu-0ubuntu120.04 generated 7 new + 926 unchanged - 0 fixed = 933 total (was 926) |
| -1 :x: | javadoc | 0m 51s | /results-javadoc-javadoc-hadoop-tools_hadoop-aws-jdkUbuntu-17.0.15+6-Ubuntu-0ubuntu120.04.txt | hadoop-tools_hadoop-aws-jdkUbuntu-17.0.15+6-Ubuntu-0ubuntu120.04 with JDK Ubuntu-17.0.15+6-Ubuntu-0ubuntu120.04 generated 6 new + 848 unchanged - 0 fixed = 854 total (was 848) |
| -1 :x: | spotbugs | 1m 54s | /new-spotbugs-hadoop-tools_hadoop-aws.html | hadoop-tools/hadoop-aws generated 3 new + 188 unchanged - 0 fixed = 191 total (was 188) |
| +1 :green_heart: | shadedclient | 32m 46s | patch has no errors when building and testing our client artifacts. | |
| _ Other Tests _ | ||||
| +1 :green_heart: | unit | 8m 49s | hadoop-mapreduce-client-core in the patch passed. | |
| +1 :green_heart: | unit | 3m 40s | hadoop-aws in the patch passed. | |
| +1 :green_heart: | asflicense | 1m 4s | The patch does not generate ASF License warnings. | |
| 223m 36s |
| Reason | Tests |
|---|---|
| SpotBugs | module:hadoop-tools/hadoop-aws |
| Unknown bug pattern CT_CONSTRUCTOR_THROW in new org.apache.hadoop.fs.s3a.commit.AbstractS3ACommitter(Path, JobContext) At AbstractS3ACommitter.java:new org.apache.hadoop.fs.s3a.commit.AbstractS3ACommitter(Path, JobContext) At AbstractS3ACommitter.java:[line 262] | |
| Unknown bug pattern CT_CONSTRUCTOR_THROW in new org.apache.hadoop.fs.s3a.commit.magic.MagicS3GuardCommitter(Path, JobContext) At MagicS3GuardCommitter.java:new org.apache.hadoop.fs.s3a.commit.magic.MagicS3GuardCommitter(Path, JobContext) At MagicS3GuardCommitter.java:[line 98] | |
| Unknown bug pattern CT_CONSTRUCTOR_THROW in new org.apache.hadoop.fs.s3a.commit.staging.StagingCommitter(Path, JobContext) At StagingCommitter.java:new org.apache.hadoop.fs.s3a.commit.staging.StagingCommitter(Path, JobContext) At StagingCommitter.java:[line 153] |
| Subsystem | Report/Notes |
|---|---|
| Docker | ClientAPI=1.52 ServerAPI=1.52 base: https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-7441/6/artifact/out/Dockerfile |
| GITHUB PR | https://github.com/apache/hadoop/pull/7441 |
| JIRA Issue | HADOOP-19091 |
| Optional Tests | dupname asflicense compile javac javadoc mvninstall mvnsite unit shadedclient spotbugs checkstyle codespell detsecrets xmllint |
| uname | Linux e1e75b2e9cd2 5.15.0-157-generic #167-Ubuntu SMP Wed Sep 17 21:35:53 UTC 2025 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality | dev-support/bin/hadoop.sh |
| git revision | trunk / 7e923c56f2aa790aadc4fffd6242736d84acbae4 |
| Default Java | Ubuntu-17.0.15+6-Ubuntu-0ubuntu120.04 |
| Multi-JDK versions | /usr/lib/jvm/java-21-openjdk-amd64:Ubuntu-21.0.7+6-Ubuntu-0ubuntu120.04 /usr/lib/jvm/java-17-openjdk-amd64:Ubuntu-17.0.15+6-Ubuntu-0ubuntu120.04 |
| Test Results | https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-7441/6/testReport/ |
| Max. process+thread count | 1073 (vs. ulimit of 5500) |
| modules | C: hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core hadoop-tools/hadoop-aws U: . |
| Console output | https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-7441/6/console |
| versions | git=2.25.1 maven=3.9.11 spotbugs=4.9.7 |
| Powered by | Apache Yetus 0.14.0 https://yetus.apache.org |
This message was automatically generated.