hadoop
hadoop copied to clipboard
apache
HDFS-16446. Consider ioutils of disk when choosing volume
JIRA: HDFS-16446.
Consider ioutils of disk when choosing volume to avoid busy disks.
Document: https://docs.google.com/document/d/1Ko1J7shz8hVLnNACT6PKVQ_leIHf_YaIFA2s3yJMZHQ/edit?usp=sharing
Architecture is as follows:
:broken_heart: -1 overall
| Vote | Subsystem | Runtime | Logfile | Comment |
|---|---|---|---|---|
| +0 :ok: | reexec | 44m 21s | Docker mode activated. | |
| _ Prechecks _ | ||||
| +1 :green_heart: | dupname | 0m 1s | No case conflicting files found. | |
| +0 :ok: | codespell | 0m 0s | codespell was not available. | |
| +1 :green_heart: | @author | 0m 0s | The patch does not contain any @author tags. | |
| +1 :green_heart: | test4tests | 0m 0s | The patch appears to include 4 new or modified test files. | |
| _ trunk Compile Tests _ | ||||
| +0 :ok: | mvndep | 12m 40s | Maven dependency ordering for branch | |
| +1 :green_heart: | mvninstall | 25m 12s | trunk passed | |
| +1 :green_heart: | compile | 20m 57s | trunk passed | |
| +1 :green_heart: | checkstyle | 3m 57s | trunk passed | |
| +1 :green_heart: | mvnsite | 3m 18s | trunk passed | |
| +1 :green_heart: | javadoc | 3m 20s | trunk passed | |
| +1 :green_heart: | spotbugs | 6m 8s | trunk passed | |
| +1 :green_heart: | shadedclient | 25m 56s | branch has no errors when building and testing our client artifacts. | |
| _ Patch Compile Tests _ | ||||
| +0 :ok: | mvndep | 0m 28s | Maven dependency ordering for patch | |
| +1 :green_heart: | mvninstall | 2m 27s | the patch passed | |
| +1 :green_heart: | compile | 20m 6s | the patch passed | |
| -1 :x: | cc | 20m 6s | /results-compile-cc-root.txt | root generated 18 new + 189 unchanged - 16 fixed = 207 total (was 205) |
| +1 :green_heart: | golang | 20m 6s | the patch passed | |
| +1 :green_heart: | javac | 20m 6s | the patch passed | |
| +1 :green_heart: | blanks | 0m 0s | The patch has no blanks issues. | |
| -0 :warning: | checkstyle | 3m 50s | /results-checkstyle-root.txt | root: The patch generated 10 new + 519 unchanged - 0 fixed = 529 total (was 519) |
| +1 :green_heart: | mvnsite | 3m 17s | the patch passed | |
| +1 :green_heart: | xml | 0m 1s | The patch has no ill-formed XML file. | |
| +1 :green_heart: | javadoc | 3m 20s | the patch passed | |
| -1 :x: | spotbugs | 3m 50s | /new-spotbugs-hadoop-hdfs-project_hadoop-hdfs.html | hadoop-hdfs-project/hadoop-hdfs generated 7 new + 0 unchanged - 0 fixed = 7 total (was 0) |
| -1 :x: | shadedclient | 26m 38s | patch has errors when building and testing our client artifacts. | |
| _ Other Tests _ | ||||
| -1 :x: | unit | 17m 31s | /patch-unit-hadoop-common-project_hadoop-common.txt | hadoop-common in the patch passed. |
| -1 :x: | unit | 367m 35s | /patch-unit-hadoop-hdfs-project_hadoop-hdfs.txt | hadoop-hdfs in the patch passed. |
| -1 :x: | asflicense | 1m 19s | /results-asflicense.txt | The patch generated 1 ASF License warnings. |
| 595m 52s |
| Reason | Tests |
|---|---|
| SpotBugs | module:hadoop-hdfs-project/hadoop-hdfs |
| org.apache.hadoop.hdfs.server.datanode.DiskIOUtilManager.getDiskIoUtils() may fail to close stream At DiskIOUtilManager.java:stream At DiskIOUtilManager.java:[line 203] | |
| Possible null pointer dereference in new org.apache.hadoop.hdfs.server.datanode.DiskIOUtilManager$DiskLocation(DiskIOUtilManager, StorageLocation) due to return value of called method Dereferenced at DiskIOUtilManager.java:new org.apache.hadoop.hdfs.server.datanode.DiskIOUtilManager$DiskLocation(DiskIOUtilManager, StorageLocation) due to return value of called method Dereferenced at DiskIOUtilManager.java:[line 56] | |
| Possible null pointer dereference in new org.apache.hadoop.hdfs.server.datanode.DiskIOUtilManager$DiskLocation(DiskIOUtilManager, StorageLocation) due to return value of called method Dereferenced at DiskIOUtilManager.java:new org.apache.hadoop.hdfs.server.datanode.DiskIOUtilManager$DiskLocation(DiskIOUtilManager, StorageLocation) due to return value of called method Dereferenced at DiskIOUtilManager.java:[line 61] | |
| Should org.apache.hadoop.hdfs.server.datanode.DiskIOUtilManager$DiskLocation be a static inner class? At DiskIOUtilManager.java:inner class? At DiskIOUtilManager.java:[lines 43-71] | |
| Should org.apache.hadoop.hdfs.server.datanode.DiskIOUtilManager$IOStat be a static inner class? At DiskIOUtilManager.java:inner class? At DiskIOUtilManager.java:[lines 85-110] | |
| Unread field:DiskIOUtilManager.java:[line 86] | |
| org.apache.hadoop.hdfs.server.datanode.fsdataset.AvailableSpaceVolumeChoosingPolicy$IOUtilVolumePair defines compareTo(AvailableSpaceVolumeChoosingPolicy$IOUtilVolumePair) and uses Object.equals() At AvailableSpaceVolumeChoosingPolicy.java:Object.equals() At AvailableSpaceVolumeChoosingPolicy.java:[lines 442-450] | |
| Failed junit tests | hadoop.util.TestNativeLibraryChecker |
| hadoop.service.launcher.TestServiceInterruptHandling | |
| hadoop.metrics2.sink.TestFileSink | |
| hadoop.crypto.TestCryptoCodec | |
| hadoop.crypto.TestCryptoStreamsWithOpensslSm4CtrCryptoCodec | |
| hadoop.tools.TestHdfsConfigFields | |
| hadoop.hdfs.server.datanode.fsdataset.TestAvailableSpaceVolumeChoosingPolicy |
| Subsystem | Report/Notes |
|---|---|
| Docker | ClientAPI=1.41 ServerAPI=1.41 base: https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-3960/1/artifact/out/Dockerfile |
| GITHUB PR | https://github.com/apache/hadoop/pull/3960 |
| Optional Tests | dupname asflicense compile javac javadoc mvninstall mvnsite unit shadedclient spotbugs checkstyle codespell cc golang xml |
| uname | Linux ffdbe3525f75 4.15.0-162-generic #170-Ubuntu SMP Mon Oct 18 11:38:05 UTC 2021 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality | dev-support/bin/hadoop.sh |
| git revision | trunk / 24e98c8ab8d1596f102c7de7d28553d866bae9ca |
| Default Java | Red Hat, Inc.-1.8.0_322-b06 |
| Test Results | https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-3960/1/testReport/ |
| Max. process+thread count | 2137 (vs. ulimit of 5500) |
| modules | C: hadoop-common-project/hadoop-common hadoop-hdfs-project/hadoop-hdfs U: . |
| Console output | https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-3960/1/console |
| versions | git=2.9.5 maven=3.6.3 spotbugs=4.2.2 |
| Powered by | Apache Yetus 0.14.0-SNAPSHOT https://yetus.apache.org |
This message was automatically generated.
Some quick comments. I'm still trying to understand how the disk io stats get considered. Some documentation would be helpful.
Instead of being part of AvailableSpaceVolumeChoosingPolicy, would it make sense to become a new volume choosing policy?
Thanks @jojochuang for your comments, I will add documentation and update the code.
When doing the POC testing, we define a new policy, named IOUtilVolumeChoosingPolicy, only considering the ioutils without considering the disk space usage, it will cause uneven disk utilization.
So we change AvailableSpaceVolumeChoosingPolicy, under the same disk usage, preferred ioutils smaller disks, filtering the busy disks.
What you said about offering a new policy is a good idea.
:broken_heart: -1 overall
| Vote | Subsystem | Runtime | Logfile | Comment |
|---|---|---|---|---|
| +0 :ok: | reexec | 0m 59s | Docker mode activated. | |
| _ Prechecks _ | ||||
| +1 :green_heart: | dupname | 0m 0s | No case conflicting files found. | |
| +0 :ok: | codespell | 0m 0s | codespell was not available. | |
| +1 :green_heart: | @author | 0m 0s | The patch does not contain any @author tags. | |
| +1 :green_heart: | test4tests | 0m 0s | The patch appears to include 4 new or modified test files. | |
| _ trunk Compile Tests _ | ||||
| +0 :ok: | mvndep | 25m 6s | Maven dependency ordering for branch | |
| +1 :green_heart: | mvninstall | 29m 19s | trunk passed | |
| +1 :green_heart: | compile | 25m 35s | trunk passed | |
| +1 :green_heart: | checkstyle | 4m 24s | trunk passed | |
| +1 :green_heart: | mvnsite | 3m 52s | trunk passed | |
| +1 :green_heart: | javadoc | 3m 48s | trunk passed | |
| +1 :green_heart: | spotbugs | 7m 7s | trunk passed | |
| +1 :green_heart: | shadedclient | 27m 26s | branch has no errors when building and testing our client artifacts. | |
| _ Patch Compile Tests _ | ||||
| +0 :ok: | mvndep | 0m 38s | Maven dependency ordering for patch | |
| +1 :green_heart: | mvninstall | 2m 44s | the patch passed | |
| +1 :green_heart: | compile | 23m 52s | the patch passed | |
| -1 :x: | cc | 23m 52s | /results-compile-cc-root.txt | root generated 22 new + 185 unchanged - 20 fixed = 207 total (was 205) |
| +1 :green_heart: | golang | 23m 52s | the patch passed | |
| +1 :green_heart: | javac | 23m 52s | the patch passed | |
| +1 :green_heart: | blanks | 0m 0s | The patch has no blanks issues. | |
| -0 :warning: | checkstyle | 4m 6s | /results-checkstyle-root.txt | root: The patch generated 10 new + 519 unchanged - 0 fixed = 529 total (was 519) |
| +1 :green_heart: | mvnsite | 3m 46s | the patch passed | |
| +1 :green_heart: | xml | 0m 1s | The patch has no ill-formed XML file. | |
| +1 :green_heart: | javadoc | 3m 43s | the patch passed | |
| -1 :x: | spotbugs | 4m 24s | /new-spotbugs-hadoop-hdfs-project_hadoop-hdfs.html | hadoop-hdfs-project/hadoop-hdfs generated 7 new + 0 unchanged - 0 fixed = 7 total (was 0) |
| -1 :x: | shadedclient | 27m 37s | patch has errors when building and testing our client artifacts. | |
| _ Other Tests _ | ||||
| -1 :x: | unit | 19m 19s | /patch-unit-hadoop-common-project_hadoop-common.txt | hadoop-common in the patch passed. |
| -1 :x: | unit | 375m 36s | /patch-unit-hadoop-hdfs-project_hadoop-hdfs.txt | hadoop-hdfs in the patch passed. |
| -1 :x: | asflicense | 1m 12s | /results-asflicense.txt | The patch generated 1 ASF License warnings. |
| 594m 10s |
| Reason | Tests |
|---|---|
| SpotBugs | module:hadoop-hdfs-project/hadoop-hdfs |
| org.apache.hadoop.hdfs.server.datanode.DiskIOUtilManager.getDiskIoUtils() may fail to close stream At DiskIOUtilManager.java:stream At DiskIOUtilManager.java:[line 203] | |
| Possible null pointer dereference in new org.apache.hadoop.hdfs.server.datanode.DiskIOUtilManager$DiskLocation(DiskIOUtilManager, StorageLocation) due to return value of called method Dereferenced at DiskIOUtilManager.java:new org.apache.hadoop.hdfs.server.datanode.DiskIOUtilManager$DiskLocation(DiskIOUtilManager, StorageLocation) due to return value of called method Dereferenced at DiskIOUtilManager.java:[line 56] | |
| Possible null pointer dereference in new org.apache.hadoop.hdfs.server.datanode.DiskIOUtilManager$DiskLocation(DiskIOUtilManager, StorageLocation) due to return value of called method Dereferenced at DiskIOUtilManager.java:new org.apache.hadoop.hdfs.server.datanode.DiskIOUtilManager$DiskLocation(DiskIOUtilManager, StorageLocation) due to return value of called method Dereferenced at DiskIOUtilManager.java:[line 61] | |
| Should org.apache.hadoop.hdfs.server.datanode.DiskIOUtilManager$DiskLocation be a static inner class? At DiskIOUtilManager.java:inner class? At DiskIOUtilManager.java:[lines 43-71] | |
| Should org.apache.hadoop.hdfs.server.datanode.DiskIOUtilManager$IOStat be a static inner class? At DiskIOUtilManager.java:inner class? At DiskIOUtilManager.java:[lines 85-110] | |
| Unread field:DiskIOUtilManager.java:[line 86] | |
| org.apache.hadoop.hdfs.server.datanode.fsdataset.AvailableSpaceVolumeChoosingPolicy$IOUtilVolumePair defines compareTo(AvailableSpaceVolumeChoosingPolicy$IOUtilVolumePair) and uses Object.equals() At AvailableSpaceVolumeChoosingPolicy.java:Object.equals() At AvailableSpaceVolumeChoosingPolicy.java:[lines 442-450] | |
| Failed junit tests | hadoop.util.TestNativeLibraryChecker |
| hadoop.service.launcher.TestServiceInterruptHandling | |
| hadoop.crypto.TestCryptoCodec | |
| hadoop.crypto.TestCryptoStreamsWithOpensslSm4CtrCryptoCodec | |
| hadoop.tools.TestHdfsConfigFields | |
| hadoop.hdfs.server.datanode.fsdataset.TestAvailableSpaceVolumeChoosingPolicy |
| Subsystem | Report/Notes |
|---|---|
| Docker | ClientAPI=1.41 ServerAPI=1.41 base: https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-3960/2/artifact/out/Dockerfile |
| GITHUB PR | https://github.com/apache/hadoop/pull/3960 |
| Optional Tests | dupname asflicense compile javac javadoc mvninstall mvnsite unit shadedclient spotbugs checkstyle codespell cc golang xml |
| uname | Linux 011c688e0d67 4.15.0-162-generic #170-Ubuntu SMP Mon Oct 18 11:38:05 UTC 2021 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality | dev-support/bin/hadoop.sh |
| git revision | trunk / 24e98c8ab8d1596f102c7de7d28553d866bae9ca |
| Default Java | Red Hat, Inc.-1.8.0_322-b06 |
| Test Results | https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-3960/2/testReport/ |
| Max. process+thread count | 3137 (vs. ulimit of 5500) |
| modules | C: hadoop-common-project/hadoop-common hadoop-hdfs-project/hadoop-hdfs U: . |
| Console output | https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-3960/2/console |
| versions | git=2.9.5 maven=3.6.3 spotbugs=4.2.2 |
| Powered by | Apache Yetus 0.14.0-SNAPSHOT https://yetus.apache.org |
This message was automatically generated.
:broken_heart: -1 overall
| Vote | Subsystem | Runtime | Logfile | Comment |
|---|---|---|---|---|
| +0 :ok: | reexec | 0m 0s | Docker mode activated. | |
| -1 :x: | docker | 19m 36s | Docker failed to build yetus/hadoop:13467f45240. |
| Subsystem | Report/Notes |
|---|---|
| GITHUB PR | https://github.com/apache/hadoop/pull/3960 |
| Console output | https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-3960/3/console |
| versions | git=2.17.1 |
| Powered by | Apache Yetus 0.14.0-SNAPSHOT https://yetus.apache.org |
This message was automatically generated.
:broken_heart: -1 overall
| Vote | Subsystem | Runtime | Logfile | Comment |
|---|---|---|---|---|
| +0 :ok: | reexec | 0m 0s | Docker mode activated. | |
| -1 :x: | docker | 14m 54s | Docker failed to build yetus/hadoop:13467f45240. |
| Subsystem | Report/Notes |
|---|---|
| GITHUB PR | https://github.com/apache/hadoop/pull/3960 |
| Console output | https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-3960/4/console |
| versions | git=2.17.1 |
| Powered by | Apache Yetus 0.14.0-SNAPSHOT https://yetus.apache.org |
This message was automatically generated.
:broken_heart: -1 overall
| Vote | Subsystem | Runtime | Logfile | Comment |
|---|---|---|---|---|
| +0 :ok: | reexec | 26m 18s | Docker mode activated. | |
| _ Prechecks _ | ||||
| +1 :green_heart: | dupname | 0m 0s | No case conflicting files found. | |
| +0 :ok: | codespell | 0m 0s | codespell was not available. | |
| +1 :green_heart: | @author | 0m 0s | The patch does not contain any @author tags. | |
| +1 :green_heart: | test4tests | 0m 0s | The patch appears to include 4 new or modified test files. | |
| _ trunk Compile Tests _ | ||||
| +0 :ok: | mvndep | 12m 15s | Maven dependency ordering for branch | |
| +1 :green_heart: | mvninstall | 26m 40s | trunk passed | |
| +1 :green_heart: | compile | 21m 39s | trunk passed | |
| +1 :green_heart: | checkstyle | 3m 58s | trunk passed | |
| +1 :green_heart: | mvnsite | 3m 55s | trunk passed | |
| +1 :green_heart: | javadoc | 3m 56s | trunk passed | |
| +0 :ok: | spotbugs | 0m 32s | branch/hadoop-project no spotbugs output file (spotbugsXml.xml) | |
| +1 :green_heart: | shadedclient | 25m 55s | branch has no errors when building and testing our client artifacts. | |
| _ Patch Compile Tests _ | ||||
| +0 :ok: | mvndep | 0m 23s | Maven dependency ordering for patch | |
| +1 :green_heart: | mvninstall | 2m 34s | the patch passed | |
| +1 :green_heart: | compile | 20m 28s | the patch passed | |
| -1 :x: | cc | 20m 28s | /results-compile-cc-root.txt | root generated 30 new + 177 unchanged - 28 fixed = 207 total (was 205) |
| +1 :green_heart: | golang | 20m 28s | the patch passed | |
| +1 :green_heart: | javac | 20m 28s | the patch passed | |
| +1 :green_heart: | blanks | 0m 0s | The patch has no blanks issues. | |
| +1 :green_heart: | checkstyle | 3m 53s | the patch passed | |
| +1 :green_heart: | mvnsite | 3m 47s | the patch passed | |
| +1 :green_heart: | xml | 0m 3s | The patch has no ill-formed XML file. | |
| +1 :green_heart: | javadoc | 3m 49s | the patch passed | |
| +0 :ok: | spotbugs | 0m 29s | hadoop-project has no data from spotbugs | |
| -1 :x: | spotbugs | 3m 50s | /new-spotbugs-hadoop-hdfs-project_hadoop-hdfs.html | hadoop-hdfs-project/hadoop-hdfs generated 7 new + 0 unchanged - 0 fixed = 7 total (was 0) |
| +1 :green_heart: | shadedclient | 26m 19s | patch has no errors when building and testing our client artifacts. | |
| _ Other Tests _ | ||||
| +1 :green_heart: | unit | 0m 28s | hadoop-project in the patch passed. | |
| -1 :x: | unit | 17m 51s | /patch-unit-hadoop-common-project_hadoop-common.txt | hadoop-common in the patch passed. |
| -1 :x: | unit | 353m 8s | /patch-unit-hadoop-hdfs-project_hadoop-hdfs.txt | hadoop-hdfs in the patch passed. |
| -1 :x: | asflicense | 1m 1s | /results-asflicense.txt | The patch generated 1 ASF License warnings. |
| 569m 16s |
| Reason | Tests |
|---|---|
| SpotBugs | module:hadoop-hdfs-project/hadoop-hdfs |
| Inconsistent synchronization of org.apache.hadoop.hdfs.server.datanode.DataNode.diskIOUtilManager; locked 50% of time Unsynchronized access at DataNode.java:50% of time Unsynchronized access at DataNode.java:[line 2473] | |
| org.apache.hadoop.hdfs.server.datanode.DiskIOUtilManager.getDiskIoUtils() may fail to close stream At DiskIOUtilManager.java:stream At DiskIOUtilManager.java:[line 209] | |
| Possible null pointer dereference in new org.apache.hadoop.hdfs.server.datanode.DiskIOUtilManager$DiskLocation(StorageLocation) due to return value of called method Dereferenced at DiskIOUtilManager.java:new org.apache.hadoop.hdfs.server.datanode.DiskIOUtilManager$DiskLocation(StorageLocation) due to return value of called method Dereferenced at DiskIOUtilManager.java:[line 54] | |
| Possible null pointer dereference in new org.apache.hadoop.hdfs.server.datanode.DiskIOUtilManager$DiskLocation(StorageLocation) due to return value of called method Dereferenced at DiskIOUtilManager.java:new org.apache.hadoop.hdfs.server.datanode.DiskIOUtilManager$DiskLocation(StorageLocation) due to return value of called method Dereferenced at DiskIOUtilManager.java:[line 59] | |
| Should org.apache.hadoop.hdfs.server.datanode.DiskIOUtilManager$IOStat be a static inner class? At DiskIOUtilManager.java:inner class? At DiskIOUtilManager.java:[lines 87-112] | |
| Unread field:DiskIOUtilManager.java:[line 88] | |
| org.apache.hadoop.hdfs.server.datanode.fsdataset.AvailableSpaceVolumeChoosingPolicy$IOUtilVolumePair defines compareTo(AvailableSpaceVolumeChoosingPolicy$IOUtilVolumePair) and uses Object.equals() At AvailableSpaceVolumeChoosingPolicy.java:Object.equals() At AvailableSpaceVolumeChoosingPolicy.java:[lines 443-451] | |
| Failed junit tests | hadoop.service.launcher.TestServiceInterruptHandling |
| hadoop.crypto.TestCryptoCodec | |
| hadoop.crypto.TestCryptoStreamsWithOpensslSm4CtrCryptoCodec | |
| hadoop.tools.TestHdfsConfigFields | |
| hadoop.hdfs.server.datanode.fsdataset.TestAvailableSpaceVolumeChoosingPolicy |
| Subsystem | Report/Notes |
|---|---|
| Docker | ClientAPI=1.41 ServerAPI=1.41 base: https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-3960/5/artifact/out/Dockerfile |
| GITHUB PR | https://github.com/apache/hadoop/pull/3960 |
| Optional Tests | dupname asflicense compile javac javadoc mvninstall mvnsite unit shadedclient spotbugs checkstyle codespell cc golang xml |
| uname | Linux b0c15dd70f74 4.15.0-162-generic #170-Ubuntu SMP Mon Oct 18 11:38:05 UTC 2021 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality | dev-support/bin/hadoop.sh |
| git revision | trunk / c59e97d3bc9b63a578ebfed3021eb18170dbba91 |
| Default Java | Red Hat, Inc.-1.8.0_322-b06 |
| Test Results | https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-3960/5/testReport/ |
| Max. process+thread count | 2425 (vs. ulimit of 5500) |
| modules | C: hadoop-project hadoop-common-project/hadoop-common hadoop-hdfs-project/hadoop-hdfs U: . |
| Console output | https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-3960/5/console |
| versions | git=2.9.5 maven=3.6.3 spotbugs=4.2.2 |
| Powered by | Apache Yetus 0.14.0-SNAPSHOT https://yetus.apache.org |
This message was automatically generated.
:broken_heart: -1 overall
| Vote | Subsystem | Runtime | Logfile | Comment |
|---|---|---|---|---|
| +0 :ok: | reexec | 44m 58s | Docker mode activated. | |
| _ Prechecks _ | ||||
| +1 :green_heart: | dupname | 0m 0s | No case conflicting files found. | |
| +0 :ok: | codespell | 0m 0s | codespell was not available. | |
| +1 :green_heart: | @author | 0m 0s | The patch does not contain any @author tags. | |
| +1 :green_heart: | test4tests | 0m 0s | The patch appears to include 4 new or modified test files. | |
| _ trunk Compile Tests _ | ||||
| +0 :ok: | mvndep | 12m 26s | Maven dependency ordering for branch | |
| +1 :green_heart: | mvninstall | 27m 12s | trunk passed | |
| +1 :green_heart: | compile | 21m 11s | trunk passed | |
| +1 :green_heart: | checkstyle | 3m 56s | trunk passed | |
| +1 :green_heart: | mvnsite | 3m 53s | trunk passed | |
| +1 :green_heart: | javadoc | 3m 59s | trunk passed | |
| +0 :ok: | spotbugs | 0m 32s | branch/hadoop-project no spotbugs output file (spotbugsXml.xml) | |
| +1 :green_heart: | shadedclient | 26m 24s | branch has no errors when building and testing our client artifacts. | |
| _ Patch Compile Tests _ | ||||
| +0 :ok: | mvndep | 0m 23s | Maven dependency ordering for patch | |
| +1 :green_heart: | mvninstall | 2m 35s | the patch passed | |
| +1 :green_heart: | compile | 20m 37s | the patch passed | |
| -1 :x: | cc | 20m 37s | /results-compile-cc-root.txt | root generated 34 new + 173 unchanged - 32 fixed = 207 total (was 205) |
| +1 :green_heart: | golang | 20m 37s | the patch passed | |
| -1 :x: | javac | 20m 37s | /results-compile-javac-root.txt | root generated 1 new + 1677 unchanged - 1 fixed = 1678 total (was 1678) |
| +1 :green_heart: | blanks | 0m 0s | The patch has no blanks issues. | |
| -0 :warning: | checkstyle | 3m 51s | /results-checkstyle-root.txt | root: The patch generated 3 new + 515 unchanged - 0 fixed = 518 total (was 515) |
| +1 :green_heart: | mvnsite | 3m 48s | the patch passed | |
| +1 :green_heart: | xml | 0m 3s | The patch has no ill-formed XML file. | |
| +1 :green_heart: | javadoc | 3m 56s | the patch passed | |
| +0 :ok: | spotbugs | 0m 29s | hadoop-project has no data from spotbugs | |
| -1 :x: | spotbugs | 3m 53s | /new-spotbugs-hadoop-hdfs-project_hadoop-hdfs.html | hadoop-hdfs-project/hadoop-hdfs generated 2 new + 0 unchanged - 0 fixed = 2 total (was 0) |
| +1 :green_heart: | shadedclient | 26m 5s | patch has no errors when building and testing our client artifacts. | |
| _ Other Tests _ | ||||
| +1 :green_heart: | unit | 0m 29s | hadoop-project in the patch passed. | |
| -1 :x: | unit | 18m 2s | /patch-unit-hadoop-common-project_hadoop-common.txt | hadoop-common in the patch passed. |
| -1 :x: | unit | 201m 2s | /patch-unit-hadoop-hdfs-project_hadoop-hdfs.txt | hadoop-hdfs in the patch passed. |
| +0 :ok: | asflicense | 1m 10s | ASF License check generated no output? | |
| 436m 42s |
| Reason | Tests |
|---|---|
| SpotBugs | module:hadoop-hdfs-project/hadoop-hdfs |
| Possible null pointer dereference in new org.apache.hadoop.hdfs.server.datanode.DiskIOUtilManager$DiskLocation(StorageLocation) due to return value of called method Dereferenced at DiskIOUtilManager.java:new org.apache.hadoop.hdfs.server.datanode.DiskIOUtilManager$DiskLocation(StorageLocation) due to return value of called method Dereferenced at DiskIOUtilManager.java:[line 49] | |
| org.apache.hadoop.hdfs.server.datanode.fsdataset.AvailableSpaceVolumeChoosingPolicy$IOUtilVolumePair defines equals and uses Object.hashCode() At AvailableSpaceVolumeChoosingPolicy.java:Object.hashCode() At AvailableSpaceVolumeChoosingPolicy.java:[lines 459-463] | |
| Failed junit tests | hadoop.service.launcher.TestServiceInterruptHandling |
| hadoop.crypto.TestCryptoStreamsWithOpensslSm4CtrCryptoCodec | |
| hadoop.crypto.TestCryptoCodec | |
| hadoop.hdfs.server.datanode.TestDataNodeTcpNoDelay | |
| hadoop.hdfs.server.datanode.TestDataNodeFaultInjector | |
| hadoop.hdfs.server.namenode.ha.TestRetryCacheWithHA | |
| hadoop.hdfs.server.namenode.ha.TestHASafeMode | |
| hadoop.hdfs.server.datanode.TestDataNodeReconfiguration | |
| hadoop.tools.TestHdfsConfigFields |
| Subsystem | Report/Notes |
|---|---|
| Docker | ClientAPI=1.41 ServerAPI=1.41 base: https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-3960/6/artifact/out/Dockerfile |
| GITHUB PR | https://github.com/apache/hadoop/pull/3960 |
| Optional Tests | dupname asflicense compile javac javadoc mvninstall mvnsite unit shadedclient spotbugs checkstyle codespell cc golang xml |
| uname | Linux d603716e9e49 4.15.0-153-generic #160-Ubuntu SMP Thu Jul 29 06:54:29 UTC 2021 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality | dev-support/bin/hadoop.sh |
| git revision | trunk / d1c38ea9554ffc26820b022e30ad6aefc4fa3575 |
| Default Java | Red Hat, Inc.-1.8.0_322-b06 |
| Test Results | https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-3960/6/testReport/ |
| Max. process+thread count | 3098 (vs. ulimit of 5500) |
| modules | C: hadoop-project hadoop-common-project/hadoop-common hadoop-hdfs-project/hadoop-hdfs U: . |
| Console output | https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-3960/6/console |
| versions | git=2.9.5 maven=3.6.3 spotbugs=4.2.2 |
| Powered by | Apache Yetus 0.14.0-SNAPSHOT https://yetus.apache.org |
This message was automatically generated.
:broken_heart: -1 overall
| Vote | Subsystem | Runtime | Logfile | Comment |
|---|---|---|---|---|
| +0 :ok: | reexec | 0m 54s | Docker mode activated. | |
| _ Prechecks _ | ||||
| +1 :green_heart: | dupname | 0m 0s | No case conflicting files found. | |
| +0 :ok: | codespell | 0m 0s | codespell was not available. | |
| +1 :green_heart: | @author | 0m 1s | The patch does not contain any @author tags. | |
| +1 :green_heart: | test4tests | 0m 0s | The patch appears to include 4 new or modified test files. | |
| _ trunk Compile Tests _ | ||||
| +0 :ok: | mvndep | 12m 13s | Maven dependency ordering for branch | |
| +1 :green_heart: | mvninstall | 27m 6s | trunk passed | |
| +1 :green_heart: | compile | 21m 20s | trunk passed | |
| +1 :green_heart: | checkstyle | 3m 56s | trunk passed | |
| +1 :green_heart: | mvnsite | 3m 50s | trunk passed | |
| +1 :green_heart: | javadoc | 3m 56s | trunk passed | |
| +0 :ok: | spotbugs | 0m 32s | branch/hadoop-project no spotbugs output file (spotbugsXml.xml) | |
| +1 :green_heart: | shadedclient | 26m 18s | branch has no errors when building and testing our client artifacts. | |
| _ Patch Compile Tests _ | ||||
| +0 :ok: | mvndep | 0m 23s | Maven dependency ordering for patch | |
| +1 :green_heart: | mvninstall | 2m 36s | the patch passed | |
| +1 :green_heart: | compile | 20m 36s | the patch passed | |
| -1 :x: | cc | 20m 36s | /results-compile-cc-root.txt | root generated 6 new + 201 unchanged - 4 fixed = 207 total (was 205) |
| +1 :green_heart: | golang | 20m 36s | the patch passed | |
| +1 :green_heart: | javac | 20m 36s | the patch passed | |
| +1 :green_heart: | blanks | 0m 0s | The patch has no blanks issues. | |
| -0 :warning: | checkstyle | 3m 56s | /results-checkstyle-root.txt | root: The patch generated 1 new + 516 unchanged - 0 fixed = 517 total (was 516) |
| +1 :green_heart: | mvnsite | 3m 48s | the patch passed | |
| +1 :green_heart: | xml | 0m 2s | The patch has no ill-formed XML file. | |
| +1 :green_heart: | javadoc | 3m 58s | the patch passed | |
| +0 :ok: | spotbugs | 0m 29s | hadoop-project has no data from spotbugs | |
| -1 :x: | spotbugs | 3m 49s | /new-spotbugs-hadoop-hdfs-project_hadoop-hdfs.html | hadoop-hdfs-project/hadoop-hdfs generated 2 new + 0 unchanged - 0 fixed = 2 total (was 0) |
| +1 :green_heart: | shadedclient | 26m 11s | patch has no errors when building and testing our client artifacts. | |
| _ Other Tests _ | ||||
| +1 :green_heart: | unit | 0m 28s | hadoop-project in the patch passed. | |
| -1 :x: | unit | 17m 50s | /patch-unit-hadoop-common-project_hadoop-common.txt | hadoop-common in the patch passed. |
| -1 :x: | unit | 347m 43s | /patch-unit-hadoop-hdfs-project_hadoop-hdfs.txt | hadoop-hdfs in the patch passed. |
| -1 :x: | asflicense | 1m 0s | /results-asflicense.txt | The patch generated 1 ASF License warnings. |
| 538m 45s |
| Reason | Tests |
|---|---|
| SpotBugs | module:hadoop-hdfs-project/hadoop-hdfs |
| Possible null pointer dereference in new org.apache.hadoop.hdfs.server.datanode.DiskIOUtilManager$DiskLocation(StorageLocation) due to return value of called method Dereferenced at DiskIOUtilManager.java:new org.apache.hadoop.hdfs.server.datanode.DiskIOUtilManager$DiskLocation(StorageLocation) due to return value of called method Dereferenced at DiskIOUtilManager.java:[line 58] | |
| Possible null pointer dereference in new org.apache.hadoop.hdfs.server.datanode.DiskIOUtilManager$DiskLocation(StorageLocation) due to return value of called method Dereferenced at DiskIOUtilManager.java:new org.apache.hadoop.hdfs.server.datanode.DiskIOUtilManager$DiskLocation(StorageLocation) due to return value of called method Dereferenced at DiskIOUtilManager.java:[line 63] | |
| Failed junit tests | hadoop.service.launcher.TestServiceInterruptHandling |
| hadoop.crypto.TestCryptoCodec | |
| hadoop.crypto.TestCryptoStreamsWithOpensslSm4CtrCryptoCodec | |
| hadoop.tools.TestHdfsConfigFields | |
| hadoop.hdfs.server.datanode.fsdataset.TestAvailableSpaceVolumeChoosingPolicy |
| Subsystem | Report/Notes |
|---|---|
| Docker | ClientAPI=1.41 ServerAPI=1.41 base: https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-3960/7/artifact/out/Dockerfile |
| GITHUB PR | https://github.com/apache/hadoop/pull/3960 |
| Optional Tests | dupname asflicense compile javac javadoc mvninstall mvnsite unit shadedclient spotbugs checkstyle codespell cc golang xml |
| uname | Linux 4666557d64a5 4.15.0-162-generic #170-Ubuntu SMP Mon Oct 18 11:38:05 UTC 2021 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality | dev-support/bin/hadoop.sh |
| git revision | trunk / e6565d20d31e73c3b9475eed790b30e802134de2 |
| Default Java | Red Hat, Inc.-1.8.0_322-b06 |
| Test Results | https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-3960/7/testReport/ |
| Max. process+thread count | 1899 (vs. ulimit of 5500) |
| modules | C: hadoop-project hadoop-common-project/hadoop-common hadoop-hdfs-project/hadoop-hdfs U: . |
| Console output | https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-3960/7/console |
| versions | git=2.9.5 maven=3.6.3 spotbugs=4.2.2 |
| Powered by | Apache Yetus 0.14.0-SNAPSHOT https://yetus.apache.org |
This message was automatically generated.
:broken_heart: -1 overall
| Vote | Subsystem | Runtime | Logfile | Comment |
|---|---|---|---|---|
| +0 :ok: | reexec | 0m 55s | Docker mode activated. | |
| _ Prechecks _ | ||||
| +1 :green_heart: | dupname | 0m 1s | No case conflicting files found. | |
| +0 :ok: | codespell | 0m 0s | codespell was not available. | |
| +1 :green_heart: | @author | 0m 0s | The patch does not contain any @author tags. | |
| +1 :green_heart: | test4tests | 0m 0s | The patch appears to include 4 new or modified test files. | |
| _ trunk Compile Tests _ | ||||
| +0 :ok: | mvndep | 12m 19s | Maven dependency ordering for branch | |
| +1 :green_heart: | mvninstall | 26m 44s | trunk passed | |
| +1 :green_heart: | compile | 21m 19s | trunk passed | |
| +1 :green_heart: | checkstyle | 4m 0s | trunk passed | |
| +1 :green_heart: | mvnsite | 3m 51s | trunk passed | |
| +1 :green_heart: | javadoc | 4m 2s | trunk passed | |
| +0 :ok: | spotbugs | 0m 32s | branch/hadoop-project no spotbugs output file (spotbugsXml.xml) | |
| +1 :green_heart: | shadedclient | 25m 58s | branch has no errors when building and testing our client artifacts. | |
| _ Patch Compile Tests _ | ||||
| +0 :ok: | mvndep | 0m 23s | Maven dependency ordering for patch | |
| +1 :green_heart: | mvninstall | 2m 35s | the patch passed | |
| +1 :green_heart: | compile | 20m 33s | the patch passed | |
| -1 :x: | cc | 20m 33s | /results-compile-cc-root.txt | root generated 20 new + 187 unchanged - 18 fixed = 207 total (was 205) |
| +1 :green_heart: | golang | 20m 33s | the patch passed | |
| +1 :green_heart: | javac | 20m 33s | the patch passed | |
| +1 :green_heart: | blanks | 0m 0s | The patch has no blanks issues. | |
| -0 :warning: | checkstyle | 3m 54s | /results-checkstyle-root.txt | root: The patch generated 1 new + 516 unchanged - 0 fixed = 517 total (was 516) |
| +1 :green_heart: | mvnsite | 3m 50s | the patch passed | |
| +1 :green_heart: | xml | 0m 3s | The patch has no ill-formed XML file. | |
| +1 :green_heart: | javadoc | 3m 49s | the patch passed | |
| +0 :ok: | spotbugs | 0m 30s | hadoop-project has no data from spotbugs | |
| +1 :green_heart: | shadedclient | 26m 25s | patch has no errors when building and testing our client artifacts. | |
| _ Other Tests _ | ||||
| +1 :green_heart: | unit | 0m 29s | hadoop-project in the patch passed. | |
| -1 :x: | unit | 17m 53s | /patch-unit-hadoop-common-project_hadoop-common.txt | hadoop-common in the patch passed. |
| -1 :x: | unit | 337m 26s | /patch-unit-hadoop-hdfs-project_hadoop-hdfs.txt | hadoop-hdfs in the patch passed. |
| +1 :green_heart: | asflicense | 1m 2s | The patch does not generate ASF License warnings. | |
| 528m 12s |
| Reason | Tests |
|---|---|
| Failed junit tests | hadoop.service.launcher.TestServiceInterruptHandling |
| hadoop.crypto.TestCryptoCodec | |
| hadoop.crypto.TestCryptoStreamsWithOpensslSm4CtrCryptoCodec | |
| hadoop.tools.TestHdfsConfigFields |
| Subsystem | Report/Notes |
|---|---|
| Docker | ClientAPI=1.41 ServerAPI=1.41 base: https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-3960/8/artifact/out/Dockerfile |
| GITHUB PR | https://github.com/apache/hadoop/pull/3960 |
| Optional Tests | dupname asflicense compile javac javadoc mvninstall mvnsite unit shadedclient spotbugs checkstyle codespell cc golang xml |
| uname | Linux c600376df19b 4.15.0-162-generic #170-Ubuntu SMP Mon Oct 18 11:38:05 UTC 2021 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality | dev-support/bin/hadoop.sh |
| git revision | trunk / c7882a443a8e7c002a2489d980602f4da36cf310 |
| Default Java | Red Hat, Inc.-1.8.0_322-b06 |
| Test Results | https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-3960/8/testReport/ |
| Max. process+thread count | 3137 (vs. ulimit of 5500) |
| modules | C: hadoop-project hadoop-common-project/hadoop-common hadoop-hdfs-project/hadoop-hdfs U: . |
| Console output | https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-3960/8/console |
| versions | git=2.9.5 maven=3.6.3 spotbugs=4.2.2 |
| Powered by | Apache Yetus 0.14.0-SNAPSHOT https://yetus.apache.org |
This message was automatically generated.
:broken_heart: -1 overall
| Vote | Subsystem | Runtime | Logfile | Comment |
|---|---|---|---|---|
| +0 :ok: | reexec | 0m 55s | Docker mode activated. | |
| _ Prechecks _ | ||||
| +1 :green_heart: | dupname | 0m 0s | No case conflicting files found. | |
| +0 :ok: | codespell | 0m 0s | codespell was not available. | |
| +1 :green_heart: | @author | 0m 0s | The patch does not contain any @author tags. | |
| +1 :green_heart: | test4tests | 0m 0s | The patch appears to include 4 new or modified test files. | |
| _ trunk Compile Tests _ | ||||
| +0 :ok: | mvndep | 12m 17s | Maven dependency ordering for branch | |
| +1 :green_heart: | mvninstall | 26m 59s | trunk passed | |
| +1 :green_heart: | compile | 21m 22s | trunk passed | |
| +1 :green_heart: | checkstyle | 4m 2s | trunk passed | |
| +1 :green_heart: | mvnsite | 3m 50s | trunk passed | |
| +1 :green_heart: | javadoc | 3m 58s | trunk passed | |
| +0 :ok: | spotbugs | 0m 33s | branch/hadoop-project no spotbugs output file (spotbugsXml.xml) | |
| +1 :green_heart: | shadedclient | 26m 9s | branch has no errors when building and testing our client artifacts. | |
| _ Patch Compile Tests _ | ||||
| +0 :ok: | mvndep | 0m 23s | Maven dependency ordering for patch | |
| +1 :green_heart: | mvninstall | 2m 33s | the patch passed | |
| +1 :green_heart: | compile | 20m 32s | the patch passed | |
| -1 :x: | cc | 20m 32s | /results-compile-cc-root.txt | root generated 22 new + 185 unchanged - 20 fixed = 207 total (was 205) |
| +1 :green_heart: | golang | 20m 32s | the patch passed | |
| +1 :green_heart: | javac | 20m 32s | the patch passed | |
| +1 :green_heart: | blanks | 0m 0s | The patch has no blanks issues. | |
| -0 :warning: | checkstyle | 3m 54s | /results-checkstyle-root.txt | root: The patch generated 1 new + 516 unchanged - 0 fixed = 517 total (was 516) |
| +1 :green_heart: | mvnsite | 3m 49s | the patch passed | |
| +1 :green_heart: | xml | 0m 2s | The patch has no ill-formed XML file. | |
| +1 :green_heart: | javadoc | 3m 55s | the patch passed | |
| +0 :ok: | spotbugs | 0m 30s | hadoop-project has no data from spotbugs | |
| +1 :green_heart: | shadedclient | 26m 5s | patch has no errors when building and testing our client artifacts. | |
| _ Other Tests _ | ||||
| +1 :green_heart: | unit | 0m 28s | hadoop-project in the patch passed. | |
| -1 :x: | unit | 17m 51s | /patch-unit-hadoop-common-project_hadoop-common.txt | hadoop-common in the patch passed. |
| -1 :x: | unit | 352m 32s | /patch-unit-hadoop-hdfs-project_hadoop-hdfs.txt | hadoop-hdfs in the patch passed. |
| +1 :green_heart: | asflicense | 1m 0s | The patch does not generate ASF License warnings. | |
| 543m 40s |
| Reason | Tests |
|---|---|
| Failed junit tests | hadoop.service.launcher.TestServiceInterruptHandling |
| hadoop.crypto.TestCryptoCodec | |
| hadoop.crypto.TestCryptoStreamsWithOpensslSm4CtrCryptoCodec | |
| hadoop.hdfs.server.datanode.TestDataXceiverBackwardsCompat |
| Subsystem | Report/Notes |
|---|---|
| Docker | ClientAPI=1.41 ServerAPI=1.41 base: https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-3960/9/artifact/out/Dockerfile |
| GITHUB PR | https://github.com/apache/hadoop/pull/3960 |
| Optional Tests | dupname asflicense compile javac javadoc mvninstall mvnsite unit shadedclient spotbugs checkstyle codespell cc golang xml |
| uname | Linux 0a731bb5fd2a 4.15.0-162-generic #170-Ubuntu SMP Mon Oct 18 11:38:05 UTC 2021 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality | dev-support/bin/hadoop.sh |
| git revision | trunk / 512a5ba75cde11a93fa66e3b4d40e086803414cc |
| Default Java | Red Hat, Inc.-1.8.0_322-b06 |
| Test Results | https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-3960/9/testReport/ |
| Max. process+thread count | 2394 (vs. ulimit of 5500) |
| modules | C: hadoop-project hadoop-common-project/hadoop-common hadoop-hdfs-project/hadoop-hdfs U: . |
| Console output | https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-3960/9/console |
| versions | git=2.9.5 maven=3.6.3 spotbugs=4.2.2 |
| Powered by | Apache Yetus 0.14.0-SNAPSHOT https://yetus.apache.org |
This message was automatically generated.
:broken_heart: -1 overall
| Vote | Subsystem | Runtime | Logfile | Comment |
|---|---|---|---|---|
| +0 :ok: | reexec | 0m 55s | Docker mode activated. | |
| _ Prechecks _ | ||||
| +1 :green_heart: | dupname | 0m 0s | No case conflicting files found. | |
| +0 :ok: | codespell | 0m 0s | codespell was not available. | |
| +1 :green_heart: | @author | 0m 0s | The patch does not contain any @author tags. | |
| +1 :green_heart: | test4tests | 0m 0s | The patch appears to include 4 new or modified test files. | |
| _ trunk Compile Tests _ | ||||
| +0 :ok: | mvndep | 12m 29s | Maven dependency ordering for branch | |
| +1 :green_heart: | mvninstall | 30m 22s | trunk passed | |
| +1 :green_heart: | compile | 21m 16s | trunk passed | |
| +1 :green_heart: | checkstyle | 4m 5s | trunk passed | |
| +1 :green_heart: | mvnsite | 3m 21s | trunk passed | |
| +1 :green_heart: | javadoc | 3m 31s | trunk passed | |
| +1 :green_heart: | spotbugs | 6m 9s | trunk passed | |
| +1 :green_heart: | shadedclient | 26m 19s | branch has no errors when building and testing our client artifacts. | |
| _ Patch Compile Tests _ | ||||
| +0 :ok: | mvndep | 0m 23s | Maven dependency ordering for patch | |
| +1 :green_heart: | mvninstall | 2m 24s | the patch passed | |
| +1 :green_heart: | compile | 20m 37s | the patch passed | |
| -1 :x: | cc | 20m 37s | /results-compile-cc-root.txt | root generated 9 new + 198 unchanged - 7 fixed = 207 total (was 205) |
| +1 :green_heart: | golang | 20m 37s | the patch passed | |
| +1 :green_heart: | javac | 20m 37s | the patch passed | |
| +1 :green_heart: | blanks | 0m 0s | The patch has no blanks issues. | |
| -0 :warning: | checkstyle | 3m 56s | /results-checkstyle-root.txt | root: The patch generated 1 new + 516 unchanged - 0 fixed = 517 total (was 516) |
| +1 :green_heart: | mvnsite | 3m 20s | the patch passed | |
| +1 :green_heart: | xml | 0m 2s | The patch has no ill-formed XML file. | |
| +1 :green_heart: | javadoc | 3m 24s | the patch passed | |
| +1 :green_heart: | spotbugs | 6m 31s | the patch passed | |
| +1 :green_heart: | shadedclient | 26m 17s | patch has no errors when building and testing our client artifacts. | |
| _ Other Tests _ | ||||
| -1 :x: | unit | 17m 53s | /patch-unit-hadoop-common-project_hadoop-common.txt | hadoop-common in the patch passed. |
| -1 :x: | unit | 360m 26s | /patch-unit-hadoop-hdfs-project_hadoop-hdfs.txt | hadoop-hdfs in the patch passed. |
| +1 :green_heart: | asflicense | 1m 13s | The patch does not generate ASF License warnings. | |
| 552m 14s |
| Reason | Tests |
|---|---|
| Failed junit tests | hadoop.service.launcher.TestServiceInterruptHandling |
| hadoop.crypto.TestCryptoCodec | |
| hadoop.crypto.TestCryptoStreamsWithOpensslSm4CtrCryptoCodec | |
| hadoop.hdfs.server.namenode.TestFileTruncate |
| Subsystem | Report/Notes |
|---|---|
| Docker | ClientAPI=1.41 ServerAPI=1.41 base: https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-3960/10/artifact/out/Dockerfile |
| GITHUB PR | https://github.com/apache/hadoop/pull/3960 |
| Optional Tests | dupname asflicense compile javac javadoc mvninstall mvnsite unit shadedclient spotbugs checkstyle codespell cc golang xml |
| uname | Linux f8b5ed4ec11f 4.15.0-162-generic #170-Ubuntu SMP Mon Oct 18 11:38:05 UTC 2021 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality | dev-support/bin/hadoop.sh |
| git revision | trunk / 41cfc189c7216473f4454fbe82012bb8dda52de5 |
| Default Java | Red Hat, Inc.-1.8.0_322-b06 |
| Test Results | https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-3960/10/testReport/ |
| Max. process+thread count | 1911 (vs. ulimit of 5500) |
| modules | C: hadoop-common-project/hadoop-common hadoop-hdfs-project/hadoop-hdfs U: . |
| Console output | https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-3960/10/console |
| versions | git=2.9.5 maven=3.6.3 spotbugs=4.2.2 |
| Powered by | Apache Yetus 0.14.0-SNAPSHOT https://yetus.apache.org |
This message was automatically generated.
Hi @jojochuang @ayushtkn @tasanuma , I added a document for this. Could you please take a look at this? Thanks a lot.
The failed unit tests seem unrelated to this change.
Hi, @tomscut , we also face slow nodes caused by io, but in most cases it is slow to read data because the frequency of reading in the cluster far exceeds that of writing. We have counted many cases, when reading slowly, the iowait is basically high, but only one or a few disks are full of ioutil (12 disks per dn). Therefore, I have the following points to discuss.
- Is there any further consideration for slow reading?
- Have you considered reporting iowait to nn, like XmitsInProgress, ActiveTransferThreadCount, etc, so that it is taken into account when choosing a DN?
@whbing
Hi, @tomscut , we also face slow nodes caused by io, but in most cases it is slow to read data because the frequency of reading in the cluster far exceeds that of writing. We have counted many cases, when reading slowly, the iowait is basically high, but only one or a few disks are full of ioutil (12 disks per dn). Therefore, I have the following points to discuss.
- Is there any further consideration for slow reading?
- Have you considered reporting iowait to nn, like XmitsInProgress, ActiveTransferThreadCount, etc, so that it is taken into account when choosing a DN?
Hi, @whbing. Your first suggestion is good. But sorry I've some other opinions, VolumeChoosingPolicy is only relevant during the write operation, determining which disk on a DataNode should store the data block. When reading files from HDFS, the focus is on selecting the appropriate DataNode that contains the required data blocks, not on selecting specific disks within a DataNode. Sorry if I'm wrong.
We're closing this stale PR because it has been open for 100 days with no activity. This isn't a judgement on the merit of the PR in any way. It's just a way of keeping the PR queue manageable. If you feel like this was a mistake, or you would like to continue working on it, please feel free to re-open it and ask for a committer to remove the stale tag and review again. Thanks all for your contribution.
![github-actions[bot] avatar](https://avatars.githubusercontent.com/in/15368?v=4 "Published by a pub.dev verified publisher"){kind=small}