hadoop
hadoop copied to clipboard
apache
HADOOP-18967: Allow no-downtime migration of HDFS clusters into secure mode
Description of PR
This PR adds two new boolean settings to DataNodes:
-
dfs.datanode.block.access.token.unsafe.allowed-not-required -
dfs.datanode.unsafe.sasl.allowed-not-required
With these two new settings, it is now possible for a Hadoop cluster with HDFS to enable secure mode without downtime. Please see the included documentation changes for more detailed discussion, and usage recommendations.
How was this patch tested?
My employer (HubSpot) used a version of this patch on our internal fork of Hadoop to transition all of our HBase clusters into secure mode, without major incident. The company's SaaS product is running on secure-mode HBase clusters now. There are also unit tests.
For code changes:
- [x] Does the title or this PR starts with the corresponding JIRA issue id (e.g. 'HADOOP-17799. Your PR title ...')?
- [ ] Object storage: have the integration tests been executed and the endpoint declared according to the connector-specific documentation?
- [ ] If adding new dependencies to the code, are these dependencies licensed in a way that is compatible for inclusion under ASF 2.0?
- [ ] If applicable, have you updated the
LICENSE,LICENSE-binary,NOTICE-binaryfiles?
:broken_heart: -1 overall
| Vote | Subsystem | Runtime | Logfile | Comment |
|---|---|---|---|---|
| +0 :ok: | reexec | 0m 31s | Docker mode activated. | |
| _ Prechecks _ | ||||
| +1 :green_heart: | dupname | 0m 0s | No case conflicting files found. | |
| +0 :ok: | codespell | 0m 0s | codespell was not available. | |
| +0 :ok: | detsecrets | 0m 0s | detect-secrets was not available. | |
| +0 :ok: | markdownlint | 0m 0s | markdownlint was not available. | |
| +1 :green_heart: | @author | 0m 0s | The patch does not contain any @author tags. | |
| +1 :green_heart: | test4tests | 0m 0s | The patch appears to include 4 new or modified test files. | |
| _ trunk Compile Tests _ | ||||
| +0 :ok: | mvndep | 14m 45s | Maven dependency ordering for branch | |
| +1 :green_heart: | mvninstall | 31m 20s | trunk passed | |
| +1 :green_heart: | compile | 18m 45s | trunk passed with JDK Ubuntu-11.0.20.1+1-post-Ubuntu-0ubuntu120.04 | |
| +1 :green_heart: | compile | 17m 24s | trunk passed with JDK Private Build-1.8.0_382-8u382-ga-1~20.04.1-b05 | |
| +1 :green_heart: | checkstyle | 4m 28s | trunk passed | |
| +1 :green_heart: | mvnsite | 4m 21s | trunk passed | |
| +1 :green_heart: | javadoc | 3m 8s | trunk passed with JDK Ubuntu-11.0.20.1+1-post-Ubuntu-0ubuntu120.04 | |
| +1 :green_heart: | javadoc | 3m 25s | trunk passed with JDK Private Build-1.8.0_382-8u382-ga-1~20.04.1-b05 | |
| +1 :green_heart: | spotbugs | 9m 0s | trunk passed | |
| +1 :green_heart: | shadedclient | 35m 40s | branch has no errors when building and testing our client artifacts. | |
| _ Patch Compile Tests _ | ||||
| +0 :ok: | mvndep | 0m 30s | Maven dependency ordering for patch | |
| +1 :green_heart: | mvninstall | 3m 9s | the patch passed | |
| +1 :green_heart: | compile | 17m 40s | the patch passed with JDK Ubuntu-11.0.20.1+1-post-Ubuntu-0ubuntu120.04 | |
| +1 :green_heart: | javac | 17m 40s | the patch passed | |
| +1 :green_heart: | compile | 16m 57s | the patch passed with JDK Private Build-1.8.0_382-8u382-ga-1~20.04.1-b05 | |
| +1 :green_heart: | javac | 16m 57s | the patch passed | |
| -1 :x: | blanks | 0m 0s | /blanks-eol.txt | The patch has 1 line(s) that end in blanks. Use git apply --whitespace=fix <<patch_file>>. Refer https://git-scm.com/docs/git-apply |
| -0 :warning: | checkstyle | 4m 24s | /results-checkstyle-root.txt | root: The patch generated 10 new + 519 unchanged - 2 fixed = 529 total (was 521) |
| +1 :green_heart: | mvnsite | 4m 19s | the patch passed | |
| +1 :green_heart: | javadoc | 3m 3s | the patch passed with JDK Ubuntu-11.0.20.1+1-post-Ubuntu-0ubuntu120.04 | |
| +1 :green_heart: | javadoc | 3m 20s | the patch passed with JDK Private Build-1.8.0_382-8u382-ga-1~20.04.1-b05 | |
| +1 :green_heart: | spotbugs | 9m 38s | the patch passed | |
| +1 :green_heart: | shadedclient | 35m 50s | patch has no errors when building and testing our client artifacts. | |
| _ Other Tests _ | ||||
| +1 :green_heart: | unit | 19m 59s | hadoop-common in the patch passed. | |
| +1 :green_heart: | unit | 2m 42s | hadoop-hdfs-client in the patch passed. | |
| -1 :x: | unit | 224m 43s | /patch-unit-hadoop-hdfs-project_hadoop-hdfs.txt | hadoop-hdfs in the patch passed. |
| +1 :green_heart: | asflicense | 1m 2s | The patch does not generate ASF License warnings. | |
| 491m 16s |
| Reason | Tests |
|---|---|
| Failed junit tests | hadoop.hdfs.server.blockmanagement.TestBlockTokenWithDFS |
| hadoop.hdfs.TestErasureCodingPolicies | |
| hadoop.hdfs.server.blockmanagement.TestBlockTokenWithDFSStriped | |
| hadoop.hdfs.TestDecommissionWithStriped | |
| hadoop.tools.TestHdfsConfigFields | |
| hadoop.hdfs.TestReconstructStripedFileWithValidator |
| Subsystem | Report/Notes |
|---|---|
| Docker | ClientAPI=1.43 ServerAPI=1.43 base: https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-6293/2/artifact/out/Dockerfile |
| GITHUB PR | https://github.com/apache/hadoop/pull/6293 |
| Optional Tests | dupname asflicense compile javac javadoc mvninstall mvnsite unit shadedclient spotbugs checkstyle codespell detsecrets markdownlint |
| uname | Linux 67acca6c8924 5.15.0-88-generic #98-Ubuntu SMP Mon Oct 2 15:18:56 UTC 2023 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality | dev-support/bin/hadoop.sh |
| git revision | trunk / 95aecfebccf3f6280f8ac89b3557da7811280803 |
| Default Java | Private Build-1.8.0_382-8u382-ga-1~20.04.1-b05 |
| Multi-JDK versions | /usr/lib/jvm/java-11-openjdk-amd64:Ubuntu-11.0.20.1+1-post-Ubuntu-0ubuntu120.04 /usr/lib/jvm/java-8-openjdk-amd64:Private Build-1.8.0_382-8u382-ga-1~20.04.1-b05 |
| Test Results | https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-6293/2/testReport/ |
| Max. process+thread count | 4066 (vs. ulimit of 5500) |
| modules | C: hadoop-common-project/hadoop-common hadoop-hdfs-project/hadoop-hdfs-client hadoop-hdfs-project/hadoop-hdfs U: . |
| Console output | https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-6293/2/console |
| versions | git=2.25.1 maven=3.6.3 spotbugs=4.2.2 |
| Powered by | Apache Yetus 0.14.0 https://yetus.apache.org |
This message was automatically generated.
:broken_heart: -1 overall
| Vote | Subsystem | Runtime | Logfile | Comment |
|---|---|---|---|---|
| +0 :ok: | reexec | 0m 40s | Docker mode activated. | |
| _ Prechecks _ | ||||
| +1 :green_heart: | dupname | 0m 0s | No case conflicting files found. | |
| +0 :ok: | codespell | 0m 0s | codespell was not available. | |
| +0 :ok: | detsecrets | 0m 0s | detect-secrets was not available. | |
| +0 :ok: | markdownlint | 0m 0s | markdownlint was not available. | |
| +1 :green_heart: | @author | 0m 0s | The patch does not contain any @author tags. | |
| +1 :green_heart: | test4tests | 0m 0s | The patch appears to include 4 new or modified test files. | |
| _ trunk Compile Tests _ | ||||
| +0 :ok: | mvndep | 14m 54s | Maven dependency ordering for branch | |
| +1 :green_heart: | mvninstall | 31m 12s | trunk passed | |
| +1 :green_heart: | compile | 19m 3s | trunk passed with JDK Ubuntu-11.0.20.1+1-post-Ubuntu-0ubuntu120.04 | |
| +1 :green_heart: | compile | 17m 5s | trunk passed with JDK Private Build-1.8.0_382-8u382-ga-1~20.04.1-b05 | |
| +1 :green_heart: | checkstyle | 4m 25s | trunk passed | |
| +1 :green_heart: | mvnsite | 4m 24s | trunk passed | |
| +1 :green_heart: | javadoc | 3m 9s | trunk passed with JDK Ubuntu-11.0.20.1+1-post-Ubuntu-0ubuntu120.04 | |
| +1 :green_heart: | javadoc | 3m 26s | trunk passed with JDK Private Build-1.8.0_382-8u382-ga-1~20.04.1-b05 | |
| +1 :green_heart: | spotbugs | 9m 5s | trunk passed | |
| +1 :green_heart: | shadedclient | 35m 56s | branch has no errors when building and testing our client artifacts. | |
| _ Patch Compile Tests _ | ||||
| +0 :ok: | mvndep | 0m 30s | Maven dependency ordering for patch | |
| +1 :green_heart: | mvninstall | 3m 4s | the patch passed | |
| +1 :green_heart: | compile | 17m 43s | the patch passed with JDK Ubuntu-11.0.20.1+1-post-Ubuntu-0ubuntu120.04 | |
| +1 :green_heart: | javac | 17m 43s | the patch passed | |
| +1 :green_heart: | compile | 16m 55s | the patch passed with JDK Private Build-1.8.0_382-8u382-ga-1~20.04.1-b05 | |
| +1 :green_heart: | javac | 16m 55s | the patch passed | |
| -1 :x: | blanks | 0m 0s | /blanks-eol.txt | The patch has 1 line(s) that end in blanks. Use git apply --whitespace=fix <<patch_file>>. Refer https://git-scm.com/docs/git-apply |
| -0 :warning: | checkstyle | 4m 13s | /results-checkstyle-root.txt | root: The patch generated 10 new + 519 unchanged - 2 fixed = 529 total (was 521) |
| +1 :green_heart: | mvnsite | 4m 17s | the patch passed | |
| +1 :green_heart: | javadoc | 3m 4s | the patch passed with JDK Ubuntu-11.0.20.1+1-post-Ubuntu-0ubuntu120.04 | |
| +1 :green_heart: | javadoc | 3m 20s | the patch passed with JDK Private Build-1.8.0_382-8u382-ga-1~20.04.1-b05 | |
| +1 :green_heart: | spotbugs | 9m 43s | the patch passed | |
| +1 :green_heart: | shadedclient | 35m 42s | patch has no errors when building and testing our client artifacts. | |
| _ Other Tests _ | ||||
| +1 :green_heart: | unit | 20m 21s | hadoop-common in the patch passed. | |
| +1 :green_heart: | unit | 2m 41s | hadoop-hdfs-client in the patch passed. | |
| -1 :x: | unit | 232m 25s | /patch-unit-hadoop-hdfs-project_hadoop-hdfs.txt | hadoop-hdfs in the patch passed. |
| +1 :green_heart: | asflicense | 1m 5s | The patch does not generate ASF License warnings. | |
| 499m 39s |
| Reason | Tests |
|---|---|
| Failed junit tests | hadoop.hdfs.server.blockmanagement.TestBlockTokenWithDFS |
| hadoop.hdfs.TestReconstructStripedFileWithRandomECPolicy | |
| hadoop.hdfs.TestErasureCodingPolicies | |
| hadoop.hdfs.server.blockmanagement.TestBlockTokenWithDFSStriped | |
| hadoop.hdfs.TestDecommissionWithStriped | |
| hadoop.tools.TestHdfsConfigFields | |
| hadoop.hdfs.server.namenode.TestNameNodeMXBean |
| Subsystem | Report/Notes |
|---|---|
| Docker | ClientAPI=1.43 ServerAPI=1.43 base: https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-6293/1/artifact/out/Dockerfile |
| GITHUB PR | https://github.com/apache/hadoop/pull/6293 |
| Optional Tests | dupname asflicense compile javac javadoc mvninstall mvnsite unit shadedclient spotbugs checkstyle codespell detsecrets markdownlint |
| uname | Linux 93ca3685c718 5.15.0-88-generic #98-Ubuntu SMP Mon Oct 2 15:18:56 UTC 2023 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality | dev-support/bin/hadoop.sh |
| git revision | trunk / 95aecfebccf3f6280f8ac89b3557da7811280803 |
| Default Java | Private Build-1.8.0_382-8u382-ga-1~20.04.1-b05 |
| Multi-JDK versions | /usr/lib/jvm/java-11-openjdk-amd64:Ubuntu-11.0.20.1+1-post-Ubuntu-0ubuntu120.04 /usr/lib/jvm/java-8-openjdk-amd64:Private Build-1.8.0_382-8u382-ga-1~20.04.1-b05 |
| Test Results | https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-6293/1/testReport/ |
| Max. process+thread count | 3947 (vs. ulimit of 5500) |
| modules | C: hadoop-common-project/hadoop-common hadoop-hdfs-project/hadoop-hdfs-client hadoop-hdfs-project/hadoop-hdfs U: . |
| Console output | https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-6293/1/console |
| versions | git=2.25.1 maven=3.6.3 spotbugs=4.2.2 |
| Powered by | Apache Yetus 0.14.0 https://yetus.apache.org |
This message was automatically generated.
:broken_heart: -1 overall
| Vote | Subsystem | Runtime | Logfile | Comment |
|---|---|---|---|---|
| +0 :ok: | reexec | 0m 32s | Docker mode activated. | |
| _ Prechecks _ | ||||
| +1 :green_heart: | dupname | 0m 0s | No case conflicting files found. | |
| +0 :ok: | codespell | 0m 0s | codespell was not available. | |
| +0 :ok: | detsecrets | 0m 0s | detect-secrets was not available. | |
| +0 :ok: | markdownlint | 0m 0s | markdownlint was not available. | |
| +0 :ok: | xmllint | 0m 0s | xmllint was not available. | |
| +1 :green_heart: | @author | 0m 0s | The patch does not contain any @author tags. | |
| +1 :green_heart: | test4tests | 0m 0s | The patch appears to include 4 new or modified test files. | |
| _ trunk Compile Tests _ | ||||
| +0 :ok: | mvndep | 14m 41s | Maven dependency ordering for branch | |
| +1 :green_heart: | mvninstall | 31m 19s | trunk passed | |
| +1 :green_heart: | compile | 17m 38s | trunk passed with JDK Ubuntu-11.0.20.1+1-post-Ubuntu-0ubuntu120.04 | |
| +1 :green_heart: | compile | 16m 25s | trunk passed with JDK Private Build-1.8.0_382-8u382-ga-1~20.04.1-b05 | |
| +1 :green_heart: | checkstyle | 4m 45s | trunk passed | |
| +1 :green_heart: | mvnsite | 4m 20s | trunk passed | |
| +1 :green_heart: | javadoc | 3m 10s | trunk passed with JDK Ubuntu-11.0.20.1+1-post-Ubuntu-0ubuntu120.04 | |
| +1 :green_heart: | javadoc | 3m 22s | trunk passed with JDK Private Build-1.8.0_382-8u382-ga-1~20.04.1-b05 | |
| +1 :green_heart: | spotbugs | 8m 34s | trunk passed | |
| +1 :green_heart: | shadedclient | 37m 15s | branch has no errors when building and testing our client artifacts. | |
| _ Patch Compile Tests _ | ||||
| +0 :ok: | mvndep | 0m 31s | Maven dependency ordering for patch | |
| +1 :green_heart: | mvninstall | 3m 8s | the patch passed | |
| +1 :green_heart: | compile | 17m 32s | the patch passed with JDK Ubuntu-11.0.20.1+1-post-Ubuntu-0ubuntu120.04 | |
| +1 :green_heart: | javac | 17m 32s | the patch passed | |
| +1 :green_heart: | compile | 15m 47s | the patch passed with JDK Private Build-1.8.0_382-8u382-ga-1~20.04.1-b05 | |
| +1 :green_heart: | javac | 15m 47s | the patch passed | |
| +1 :green_heart: | blanks | 0m 0s | The patch has no blanks issues. | |
| +1 :green_heart: | checkstyle | 4m 14s | root: The patch generated 0 new + 519 unchanged - 2 fixed = 519 total (was 521) | |
| +1 :green_heart: | mvnsite | 4m 14s | the patch passed | |
| +1 :green_heart: | javadoc | 3m 6s | the patch passed with JDK Ubuntu-11.0.20.1+1-post-Ubuntu-0ubuntu120.04 | |
| +1 :green_heart: | javadoc | 3m 16s | the patch passed with JDK Private Build-1.8.0_382-8u382-ga-1~20.04.1-b05 | |
| +1 :green_heart: | spotbugs | 9m 37s | the patch passed | |
| +1 :green_heart: | shadedclient | 38m 53s | patch has no errors when building and testing our client artifacts. | |
| _ Other Tests _ | ||||
| +1 :green_heart: | unit | 19m 51s | hadoop-common in the patch passed. | |
| +1 :green_heart: | unit | 2m 43s | hadoop-hdfs-client in the patch passed. | |
| -1 :x: | unit | 216m 24s | /patch-unit-hadoop-hdfs-project_hadoop-hdfs.txt | hadoop-hdfs in the patch passed. |
| +1 :green_heart: | asflicense | 1m 5s | The patch does not generate ASF License warnings. | |
| 483m 45s |
| Reason | Tests |
|---|---|
| Failed junit tests | hadoop.hdfs.protocol.datatransfer.sasl.TestSaslDataTransfer |
| hadoop.hdfs.server.blockmanagement.TestBlockTokenWithDFSStriped | |
| hadoop.hdfs.server.datanode.TestDirectoryScanner |
| Subsystem | Report/Notes |
|---|---|
| Docker | ClientAPI=1.43 ServerAPI=1.43 base: https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-6293/3/artifact/out/Dockerfile |
| GITHUB PR | https://github.com/apache/hadoop/pull/6293 |
| Optional Tests | dupname asflicense compile javac javadoc mvninstall mvnsite unit shadedclient spotbugs checkstyle codespell detsecrets markdownlint xmllint |
| uname | Linux 7b299c91b0bd 5.15.0-88-generic #98-Ubuntu SMP Mon Oct 2 15:18:56 UTC 2023 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality | dev-support/bin/hadoop.sh |
| git revision | trunk / 3cedb8e3428b933c681fc9c08f8c4a2cc0e391fb |
| Default Java | Private Build-1.8.0_382-8u382-ga-1~20.04.1-b05 |
| Multi-JDK versions | /usr/lib/jvm/java-11-openjdk-amd64:Ubuntu-11.0.20.1+1-post-Ubuntu-0ubuntu120.04 /usr/lib/jvm/java-8-openjdk-amd64:Private Build-1.8.0_382-8u382-ga-1~20.04.1-b05 |
| Test Results | https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-6293/3/testReport/ |
| Max. process+thread count | 3925 (vs. ulimit of 5500) |
| modules | C: hadoop-common-project/hadoop-common hadoop-hdfs-project/hadoop-hdfs-client hadoop-hdfs-project/hadoop-hdfs U: . |
| Console output | https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-6293/3/console |
| versions | git=2.25.1 maven=3.6.3 spotbugs=4.2.2 |
| Powered by | Apache Yetus 0.14.0 https://yetus.apache.org |
This message was automatically generated.
@charlesconnell Thanks for your great work here. I am sure it will be helpful for some end users migrate to secure mode cluster. Not review carefully, one concerns, just notice that there are some improvement at IPC layer, any affect to YARN sub-project? another thing, some configuration item named 'dfs.datanode.*' but appear at common layer, do you mind to change to one common item which could be used for both HDFS and YARN(maybe support later?). Thanks again.
cc @goiri @ayushtkn @zhangshuyan0 @slfan1989 would you mind to take review?
@Hexiaoqiao Thanks for having a look. This PR should only impact HDFS. There are changes in the IPC layer code, but I do not expect this will impact other components in practice. I should have been more clear from the start that this only changes HDFS, because other Hadoop components already allow no-downtime migration into secure mode. I've updated the ticket and the documentation in this PR to be more clear about this.
:broken_heart: -1 overall
| Vote | Subsystem | Runtime | Logfile | Comment |
|---|---|---|---|---|
| +0 :ok: | reexec | 17m 24s | Docker mode activated. | |
| _ Prechecks _ | ||||
| +1 :green_heart: | dupname | 0m 1s | No case conflicting files found. | |
| +0 :ok: | codespell | 0m 0s | codespell was not available. | |
| +0 :ok: | detsecrets | 0m 0s | detect-secrets was not available. | |
| +0 :ok: | markdownlint | 0m 0s | markdownlint was not available. | |
| +0 :ok: | xmllint | 0m 0s | xmllint was not available. | |
| +1 :green_heart: | @author | 0m 0s | The patch does not contain any @author tags. | |
| +1 :green_heart: | test4tests | 0m 0s | The patch appears to include 5 new or modified test files. | |
| _ trunk Compile Tests _ | ||||
| +0 :ok: | mvndep | 79m 24s | Maven dependency ordering for branch | |
| +1 :green_heart: | mvninstall | 35m 9s | trunk passed | |
| +1 :green_heart: | compile | 18m 5s | trunk passed with JDK Ubuntu-11.0.20.1+1-post-Ubuntu-0ubuntu120.04 | |
| +1 :green_heart: | compile | 16m 39s | trunk passed with JDK Private Build-1.8.0_382-8u382-ga-1~20.04.1-b05 | |
| +1 :green_heart: | checkstyle | 4m 45s | trunk passed | |
| +1 :green_heart: | mvnsite | 4m 24s | trunk passed | |
| +1 :green_heart: | javadoc | 3m 24s | trunk passed with JDK Ubuntu-11.0.20.1+1-post-Ubuntu-0ubuntu120.04 | |
| +1 :green_heart: | javadoc | 3m 31s | trunk passed with JDK Private Build-1.8.0_382-8u382-ga-1~20.04.1-b05 | |
| +1 :green_heart: | spotbugs | 8m 49s | trunk passed | |
| +1 :green_heart: | shadedclient | 40m 19s | branch has no errors when building and testing our client artifacts. | |
| _ Patch Compile Tests _ | ||||
| +0 :ok: | mvndep | 0m 31s | Maven dependency ordering for patch | |
| +1 :green_heart: | mvninstall | 2m 55s | the patch passed | |
| +1 :green_heart: | compile | 17m 24s | the patch passed with JDK Ubuntu-11.0.20.1+1-post-Ubuntu-0ubuntu120.04 | |
| +1 :green_heart: | javac | 17m 24s | the patch passed | |
| +1 :green_heart: | compile | 16m 19s | the patch passed with JDK Private Build-1.8.0_382-8u382-ga-1~20.04.1-b05 | |
| +1 :green_heart: | javac | 16m 19s | the patch passed | |
| +1 :green_heart: | blanks | 0m 0s | The patch has no blanks issues. | |
| +1 :green_heart: | checkstyle | 4m 35s | root: The patch generated 0 new + 526 unchanged - 2 fixed = 526 total (was 528) | |
| +1 :green_heart: | mvnsite | 4m 22s | the patch passed | |
| +1 :green_heart: | javadoc | 3m 18s | the patch passed with JDK Ubuntu-11.0.20.1+1-post-Ubuntu-0ubuntu120.04 | |
| +1 :green_heart: | javadoc | 3m 24s | the patch passed with JDK Private Build-1.8.0_382-8u382-ga-1~20.04.1-b05 | |
| +1 :green_heart: | spotbugs | 9m 25s | the patch passed | |
| +1 :green_heart: | shadedclient | 42m 5s | patch has no errors when building and testing our client artifacts. | |
| _ Other Tests _ | ||||
| +1 :green_heart: | unit | 19m 15s | hadoop-common in the patch passed. | |
| +1 :green_heart: | unit | 2m 41s | hadoop-hdfs-client in the patch passed. | |
| -1 :x: | unit | 246m 16s | /patch-unit-hadoop-hdfs-project_hadoop-hdfs.txt | hadoop-hdfs in the patch passed. |
| +1 :green_heart: | asflicense | 1m 6s | The patch does not generate ASF License warnings. | |
| 607m 2s |
| Reason | Tests |
|---|---|
| Failed junit tests | hadoop.hdfs.server.datanode.TestDirectoryScanner |
| Subsystem | Report/Notes |
|---|---|
| Docker | ClientAPI=1.43 ServerAPI=1.43 base: https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-6293/4/artifact/out/Dockerfile |
| GITHUB PR | https://github.com/apache/hadoop/pull/6293 |
| Optional Tests | dupname asflicense compile javac javadoc mvninstall mvnsite unit shadedclient spotbugs checkstyle codespell detsecrets markdownlint xmllint |
| uname | Linux 0c0621e896c9 5.15.0-88-generic #98-Ubuntu SMP Mon Oct 2 15:18:56 UTC 2023 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality | dev-support/bin/hadoop.sh |
| git revision | trunk / b280d6d438f4f65051b04bbe64c29c4f334ba749 |
| Default Java | Private Build-1.8.0_382-8u382-ga-1~20.04.1-b05 |
| Multi-JDK versions | /usr/lib/jvm/java-11-openjdk-amd64:Ubuntu-11.0.20.1+1-post-Ubuntu-0ubuntu120.04 /usr/lib/jvm/java-8-openjdk-amd64:Private Build-1.8.0_382-8u382-ga-1~20.04.1-b05 |
| Test Results | https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-6293/4/testReport/ |
| Max. process+thread count | 2190 (vs. ulimit of 5500) |
| modules | C: hadoop-common-project/hadoop-common hadoop-hdfs-project/hadoop-hdfs-client hadoop-hdfs-project/hadoop-hdfs U: . |
| Console output | https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-6293/4/console |
| versions | git=2.25.1 maven=3.6.3 spotbugs=4.2.2 |
| Powered by | Apache Yetus 0.14.0 https://yetus.apache.org |
This message was automatically generated.
:broken_heart: -1 overall
| Vote | Subsystem | Runtime | Logfile | Comment |
|---|---|---|---|---|
| +0 :ok: | reexec | 0m 45s | Docker mode activated. | |
| _ Prechecks _ | ||||
| +1 :green_heart: | dupname | 0m 0s | No case conflicting files found. | |
| +0 :ok: | codespell | 0m 0s | codespell was not available. | |
| +0 :ok: | detsecrets | 0m 0s | detect-secrets was not available. | |
| +0 :ok: | markdownlint | 0m 0s | markdownlint was not available. | |
| +0 :ok: | xmllint | 0m 0s | xmllint was not available. | |
| +1 :green_heart: | @author | 0m 0s | The patch does not contain any @author tags. | |
| +1 :green_heart: | test4tests | 0m 0s | The patch appears to include 5 new or modified test files. | |
| _ trunk Compile Tests _ | ||||
| +0 :ok: | mvndep | 53m 16s | Maven dependency ordering for branch | |
| +1 :green_heart: | mvninstall | 35m 22s | trunk passed | |
| +1 :green_heart: | compile | 18m 11s | trunk passed with JDK Ubuntu-11.0.20.1+1-post-Ubuntu-0ubuntu120.04 | |
| +1 :green_heart: | compile | 16m 31s | trunk passed with JDK Private Build-1.8.0_382-8u382-ga-1~20.04.1-b05 | |
| +1 :green_heart: | checkstyle | 4m 38s | trunk passed | |
| +1 :green_heart: | mvnsite | 4m 25s | trunk passed | |
| +1 :green_heart: | javadoc | 3m 22s | trunk passed with JDK Ubuntu-11.0.20.1+1-post-Ubuntu-0ubuntu120.04 | |
| +1 :green_heart: | javadoc | 3m 30s | trunk passed with JDK Private Build-1.8.0_382-8u382-ga-1~20.04.1-b05 | |
| +1 :green_heart: | spotbugs | 8m 49s | trunk passed | |
| +1 :green_heart: | shadedclient | 40m 23s | branch has no errors when building and testing our client artifacts. | |
| _ Patch Compile Tests _ | ||||
| +0 :ok: | mvndep | 0m 29s | Maven dependency ordering for patch | |
| +1 :green_heart: | mvninstall | 2m 54s | the patch passed | |
| +1 :green_heart: | compile | 17m 31s | the patch passed with JDK Ubuntu-11.0.20.1+1-post-Ubuntu-0ubuntu120.04 | |
| +1 :green_heart: | javac | 17m 31s | the patch passed | |
| +1 :green_heart: | compile | 16m 23s | the patch passed with JDK Private Build-1.8.0_382-8u382-ga-1~20.04.1-b05 | |
| +1 :green_heart: | javac | 16m 23s | the patch passed | |
| +1 :green_heart: | blanks | 0m 0s | The patch has no blanks issues. | |
| +1 :green_heart: | checkstyle | 4m 38s | root: The patch generated 0 new + 524 unchanged - 2 fixed = 524 total (was 526) | |
| +1 :green_heart: | mvnsite | 4m 21s | the patch passed | |
| +1 :green_heart: | javadoc | 3m 23s | the patch passed with JDK Ubuntu-11.0.20.1+1-post-Ubuntu-0ubuntu120.04 | |
| +1 :green_heart: | javadoc | 4m 12s | the patch passed with JDK Private Build-1.8.0_382-8u382-ga-1~20.04.1-b05 | |
| +1 :green_heart: | spotbugs | 9m 30s | the patch passed | |
| +1 :green_heart: | shadedclient | 41m 35s | patch has no errors when building and testing our client artifacts. | |
| _ Other Tests _ | ||||
| +1 :green_heart: | unit | 19m 42s | hadoop-common in the patch passed. | |
| +1 :green_heart: | unit | 2m 40s | hadoop-hdfs-client in the patch passed. | |
| -1 :x: | unit | 247m 53s | /patch-unit-hadoop-hdfs-project_hadoop-hdfs.txt | hadoop-hdfs in the patch passed. |
| +1 :green_heart: | asflicense | 1m 6s | The patch does not generate ASF License warnings. | |
| 566m 56s |
| Reason | Tests |
|---|---|
| Failed junit tests | hadoop.hdfs.server.datanode.TestDirectoryScanner |
| Subsystem | Report/Notes |
|---|---|
| Docker | ClientAPI=1.43 ServerAPI=1.43 base: https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-6293/7/artifact/out/Dockerfile |
| GITHUB PR | https://github.com/apache/hadoop/pull/6293 |
| Optional Tests | dupname asflicense compile javac javadoc mvninstall mvnsite unit shadedclient spotbugs checkstyle codespell detsecrets markdownlint xmllint |
| uname | Linux 5204554e05c9 5.15.0-88-generic #98-Ubuntu SMP Mon Oct 2 15:18:56 UTC 2023 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality | dev-support/bin/hadoop.sh |
| git revision | trunk / 6c11f58ccc52d8ca48d5d5c89bf5cca78d2f5e5a |
| Default Java | Private Build-1.8.0_382-8u382-ga-1~20.04.1-b05 |
| Multi-JDK versions | /usr/lib/jvm/java-11-openjdk-amd64:Ubuntu-11.0.20.1+1-post-Ubuntu-0ubuntu120.04 /usr/lib/jvm/java-8-openjdk-amd64:Private Build-1.8.0_382-8u382-ga-1~20.04.1-b05 |
| Test Results | https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-6293/7/testReport/ |
| Max. process+thread count | 2562 (vs. ulimit of 5500) |
| modules | C: hadoop-common-project/hadoop-common hadoop-hdfs-project/hadoop-hdfs-client hadoop-hdfs-project/hadoop-hdfs U: . |
| Console output | https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-6293/7/console |
| versions | git=2.25.1 maven=3.6.3 spotbugs=4.2.2 |
| Powered by | Apache Yetus 0.14.0 https://yetus.apache.org |
This message was automatically generated.
:confetti_ball: +1 overall
| Vote | Subsystem | Runtime | Logfile | Comment |
|---|---|---|---|---|
| +0 :ok: | reexec | 18m 8s | Docker mode activated. | |
| _ Prechecks _ | ||||
| +1 :green_heart: | dupname | 0m 0s | No case conflicting files found. | |
| +0 :ok: | codespell | 0m 0s | codespell was not available. | |
| +0 :ok: | detsecrets | 0m 0s | detect-secrets was not available. | |
| +0 :ok: | markdownlint | 0m 0s | markdownlint was not available. | |
| +0 :ok: | xmllint | 0m 0s | xmllint was not available. | |
| +1 :green_heart: | @author | 0m 0s | The patch does not contain any @author tags. | |
| +1 :green_heart: | test4tests | 0m 0s | The patch appears to include 5 new or modified test files. | |
| _ trunk Compile Tests _ | ||||
| +0 :ok: | mvndep | 77m 20s | Maven dependency ordering for branch | |
| +1 :green_heart: | mvninstall | 35m 49s | trunk passed | |
| +1 :green_heart: | compile | 20m 38s | trunk passed with JDK Ubuntu-11.0.20.1+1-post-Ubuntu-0ubuntu120.04 | |
| +1 :green_heart: | compile | 19m 19s | trunk passed with JDK Private Build-1.8.0_382-8u382-ga-1~20.04.1-b05 | |
| +1 :green_heart: | checkstyle | 5m 12s | trunk passed | |
| +1 :green_heart: | mvnsite | 4m 15s | trunk passed | |
| +1 :green_heart: | javadoc | 3m 23s | trunk passed with JDK Ubuntu-11.0.20.1+1-post-Ubuntu-0ubuntu120.04 | |
| +1 :green_heart: | javadoc | 3m 25s | trunk passed with JDK Private Build-1.8.0_382-8u382-ga-1~20.04.1-b05 | |
| +1 :green_heart: | spotbugs | 9m 34s | trunk passed | |
| +1 :green_heart: | shadedclient | 43m 1s | branch has no errors when building and testing our client artifacts. | |
| _ Patch Compile Tests _ | ||||
| +0 :ok: | mvndep | 0m 29s | Maven dependency ordering for patch | |
| +1 :green_heart: | mvninstall | 3m 3s | the patch passed | |
| +1 :green_heart: | compile | 19m 43s | the patch passed with JDK Ubuntu-11.0.20.1+1-post-Ubuntu-0ubuntu120.04 | |
| +1 :green_heart: | javac | 19m 43s | the patch passed | |
| +1 :green_heart: | compile | 18m 40s | the patch passed with JDK Private Build-1.8.0_382-8u382-ga-1~20.04.1-b05 | |
| +1 :green_heart: | javac | 18m 40s | the patch passed | |
| +1 :green_heart: | blanks | 0m 0s | The patch has no blanks issues. | |
| +1 :green_heart: | checkstyle | 4m 52s | root: The patch generated 0 new + 525 unchanged - 2 fixed = 525 total (was 527) | |
| +1 :green_heart: | mvnsite | 4m 20s | the patch passed | |
| +1 :green_heart: | javadoc | 3m 17s | the patch passed with JDK Ubuntu-11.0.20.1+1-post-Ubuntu-0ubuntu120.04 | |
| +1 :green_heart: | javadoc | 3m 23s | the patch passed with JDK Private Build-1.8.0_382-8u382-ga-1~20.04.1-b05 | |
| +1 :green_heart: | spotbugs | 10m 3s | the patch passed | |
| +1 :green_heart: | shadedclient | 41m 36s | patch has no errors when building and testing our client artifacts. | |
| _ Other Tests _ | ||||
| +1 :green_heart: | unit | 19m 48s | hadoop-common in the patch passed. | |
| +1 :green_heart: | unit | 2m 38s | hadoop-hdfs-client in the patch passed. | |
| +1 :green_heart: | unit | 263m 56s | hadoop-hdfs in the patch passed. | |
| +1 :green_heart: | asflicense | 1m 14s | The patch does not generate ASF License warnings. | |
| 637m 48s |
| Subsystem | Report/Notes |
|---|---|
| Docker | ClientAPI=1.43 ServerAPI=1.43 base: https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-6293/5/artifact/out/Dockerfile |
| GITHUB PR | https://github.com/apache/hadoop/pull/6293 |
| Optional Tests | dupname asflicense compile javac javadoc mvninstall mvnsite unit shadedclient spotbugs checkstyle codespell detsecrets markdownlint xmllint |
| uname | Linux 16a77c198417 5.15.0-88-generic #98-Ubuntu SMP Mon Oct 2 15:18:56 UTC 2023 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality | dev-support/bin/hadoop.sh |
| git revision | trunk / b280d6d438f4f65051b04bbe64c29c4f334ba749 |
| Default Java | Private Build-1.8.0_382-8u382-ga-1~20.04.1-b05 |
| Multi-JDK versions | /usr/lib/jvm/java-11-openjdk-amd64:Ubuntu-11.0.20.1+1-post-Ubuntu-0ubuntu120.04 /usr/lib/jvm/java-8-openjdk-amd64:Private Build-1.8.0_382-8u382-ga-1~20.04.1-b05 |
| Test Results | https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-6293/5/testReport/ |
| Max. process+thread count | 3121 (vs. ulimit of 5500) |
| modules | C: hadoop-common-project/hadoop-common hadoop-hdfs-project/hadoop-hdfs-client hadoop-hdfs-project/hadoop-hdfs U: . |
| Console output | https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-6293/5/console |
| versions | git=2.25.1 maven=3.6.3 spotbugs=4.2.2 |
| Powered by | Apache Yetus 0.14.0 https://yetus.apache.org |
This message was automatically generated.
:confetti_ball: +1 overall
| Vote | Subsystem | Runtime | Logfile | Comment |
|---|---|---|---|---|
| +0 :ok: | reexec | 0m 46s | Docker mode activated. | |
| _ Prechecks _ | ||||
| +1 :green_heart: | dupname | 0m 0s | No case conflicting files found. | |
| +0 :ok: | codespell | 0m 1s | codespell was not available. | |
| +0 :ok: | detsecrets | 0m 1s | detect-secrets was not available. | |
| +0 :ok: | markdownlint | 0m 1s | markdownlint was not available. | |
| +0 :ok: | xmllint | 0m 1s | xmllint was not available. | |
| +1 :green_heart: | @author | 0m 0s | The patch does not contain any @author tags. | |
| +1 :green_heart: | test4tests | 0m 0s | The patch appears to include 5 new or modified test files. | |
| _ trunk Compile Tests _ | ||||
| +0 :ok: | mvndep | 61m 22s | Maven dependency ordering for branch | |
| +1 :green_heart: | mvninstall | 37m 20s | trunk passed | |
| +1 :green_heart: | compile | 20m 52s | trunk passed with JDK Ubuntu-11.0.20.1+1-post-Ubuntu-0ubuntu120.04 | |
| +1 :green_heart: | compile | 19m 53s | trunk passed with JDK Private Build-1.8.0_382-8u382-ga-1~20.04.1-b05 | |
| +1 :green_heart: | checkstyle | 5m 1s | trunk passed | |
| +1 :green_heart: | mvnsite | 4m 31s | trunk passed | |
| +1 :green_heart: | javadoc | 3m 28s | trunk passed with JDK Ubuntu-11.0.20.1+1-post-Ubuntu-0ubuntu120.04 | |
| +1 :green_heart: | javadoc | 3m 26s | trunk passed with JDK Private Build-1.8.0_382-8u382-ga-1~20.04.1-b05 | |
| +1 :green_heart: | spotbugs | 9m 14s | trunk passed | |
| +1 :green_heart: | shadedclient | 40m 52s | branch has no errors when building and testing our client artifacts. | |
| _ Patch Compile Tests _ | ||||
| +0 :ok: | mvndep | 0m 29s | Maven dependency ordering for patch | |
| +1 :green_heart: | mvninstall | 3m 10s | the patch passed | |
| +1 :green_heart: | compile | 19m 49s | the patch passed with JDK Ubuntu-11.0.20.1+1-post-Ubuntu-0ubuntu120.04 | |
| +1 :green_heart: | javac | 19m 49s | the patch passed | |
| +1 :green_heart: | compile | 18m 56s | the patch passed with JDK Private Build-1.8.0_382-8u382-ga-1~20.04.1-b05 | |
| +1 :green_heart: | javac | 18m 56s | the patch passed | |
| +1 :green_heart: | blanks | 0m 0s | The patch has no blanks issues. | |
| +1 :green_heart: | checkstyle | 4m 49s | root: The patch generated 0 new + 525 unchanged - 2 fixed = 525 total (was 527) | |
| +1 :green_heart: | mvnsite | 4m 28s | the patch passed | |
| +1 :green_heart: | javadoc | 3m 17s | the patch passed with JDK Ubuntu-11.0.20.1+1-post-Ubuntu-0ubuntu120.04 | |
| +1 :green_heart: | javadoc | 3m 27s | the patch passed with JDK Private Build-1.8.0_382-8u382-ga-1~20.04.1-b05 | |
| +1 :green_heart: | spotbugs | 10m 0s | the patch passed | |
| +1 :green_heart: | shadedclient | 42m 13s | patch has no errors when building and testing our client artifacts. | |
| _ Other Tests _ | ||||
| +1 :green_heart: | unit | 19m 51s | hadoop-common in the patch passed. | |
| +1 :green_heart: | unit | 2m 45s | hadoop-hdfs-client in the patch passed. | |
| +1 :green_heart: | unit | 262m 42s | hadoop-hdfs in the patch passed. | |
| +1 :green_heart: | asflicense | 1m 10s | The patch does not generate ASF License warnings. | |
| 604m 48s |
| Subsystem | Report/Notes |
|---|---|
| Docker | ClientAPI=1.43 ServerAPI=1.43 base: https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-6293/6/artifact/out/Dockerfile |
| GITHUB PR | https://github.com/apache/hadoop/pull/6293 |
| Optional Tests | dupname asflicense compile javac javadoc mvninstall mvnsite unit shadedclient spotbugs checkstyle codespell detsecrets markdownlint xmllint |
| uname | Linux 3a4e0ac2fcc9 5.15.0-88-generic #98-Ubuntu SMP Mon Oct 2 15:18:56 UTC 2023 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality | dev-support/bin/hadoop.sh |
| git revision | trunk / e54bbf89084a68528f7bc6ba3c1e56c522ad1d7f |
| Default Java | Private Build-1.8.0_382-8u382-ga-1~20.04.1-b05 |
| Multi-JDK versions | /usr/lib/jvm/java-11-openjdk-amd64:Ubuntu-11.0.20.1+1-post-Ubuntu-0ubuntu120.04 /usr/lib/jvm/java-8-openjdk-amd64:Private Build-1.8.0_382-8u382-ga-1~20.04.1-b05 |
| Test Results | https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-6293/6/testReport/ |
| Max. process+thread count | 3135 (vs. ulimit of 5500) |
| modules | C: hadoop-common-project/hadoop-common hadoop-hdfs-project/hadoop-hdfs-client hadoop-hdfs-project/hadoop-hdfs U: . |
| Console output | https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-6293/6/console |
| versions | git=2.25.1 maven=3.6.3 spotbugs=4.2.2 |
| Powered by | Apache Yetus 0.14.0 https://yetus.apache.org |
This message was automatically generated.
We're closing this stale PR because it has been open for 100 days with no activity. This isn't a judgement on the merit of the PR in any way. It's just a way of keeping the PR queue manageable. If you feel like this was a mistake, or you would like to continue working on it, please feel free to re-open it and ask for a committer to remove the stale tag and review again. Thanks all for your contribution.
![github-actions[bot] avatar](https://avatars.githubusercontent.com/in/15368?v=4 "Published by a pub.dev verified publisher"){kind=small}