hadoop icon indicating copy to clipboard operation
hadoop copied to clipboard
verified publisher icon apache

HDFS-17352. Add configuration to control whether DN delete this replica from disk when client requests a missing block

Open haiyang1987 opened this issue 1 year ago • 8 comments

Description of PR

https://issues.apache.org/jira/browse/HDFS-17352

As discussed at https://github.com/apache/hadoop/pull/6464#issuecomment-1902959898 we should add configuration to control whether DN delete this replica from disk when client requests a missing block.

haiyang1987 avatar Feb 16 '24 13:02 haiyang1987

The result of the CI is here. The failed tests seem not to be related.

https://ci-hadoop.apache.org/blue/organizations/jenkins/hadoop-multibranch/detail/PR-6559/3/tests

haiyang1987 avatar Feb 21 '24 06:02 haiyang1987

Update PR to support dynamically configured. Hi @ZanderXu @tomscut please help review it again, thanks~

haiyang1987 avatar Feb 22 '24 11:02 haiyang1987

:broken_heart: -1 overall

Vote Subsystem Runtime Logfile Comment
+0 :ok: reexec 0m 22s Docker mode activated.
_ Prechecks _
+1 :green_heart: dupname 0m 0s No case conflicting files found.
+0 :ok: codespell 0m 1s codespell was not available.
+0 :ok: detsecrets 0m 1s detect-secrets was not available.
+0 :ok: xmllint 0m 1s xmllint was not available.
+1 :green_heart: @author 0m 0s The patch does not contain any @author tags.
+1 :green_heart: test4tests 0m 0s The patch appears to include 5 new or modified test files.
_ trunk Compile Tests _
+1 :green_heart: mvninstall 50m 31s trunk passed
+1 :green_heart: compile 0m 41s trunk passed with JDK Ubuntu-11.0.21+9-post-Ubuntu-0ubuntu120.04
+1 :green_heart: compile 0m 40s trunk passed with JDK Private Build-1.8.0_392-8u392-ga-1~20.04-b08
+1 :green_heart: checkstyle 0m 40s trunk passed
+1 :green_heart: mvnsite 0m 47s trunk passed
+1 :green_heart: javadoc 0m 45s trunk passed with JDK Ubuntu-11.0.21+9-post-Ubuntu-0ubuntu120.04
+1 :green_heart: javadoc 1m 2s trunk passed with JDK Private Build-1.8.0_392-8u392-ga-1~20.04-b08
+1 :green_heart: spotbugs 1m 42s trunk passed
+1 :green_heart: shadedclient 20m 41s branch has no errors when building and testing our client artifacts.
_ Patch Compile Tests _
+1 :green_heart: mvninstall 0m 37s the patch passed
+1 :green_heart: compile 0m 36s the patch passed with JDK Ubuntu-11.0.21+9-post-Ubuntu-0ubuntu120.04
+1 :green_heart: javac 0m 36s the patch passed
+1 :green_heart: compile 0m 34s the patch passed with JDK Private Build-1.8.0_392-8u392-ga-1~20.04-b08
+1 :green_heart: javac 0m 34s the patch passed
+1 :green_heart: blanks 0m 0s The patch has no blanks issues.
+1 :green_heart: checkstyle 0m 31s the patch passed
+1 :green_heart: mvnsite 0m 37s the patch passed
+1 :green_heart: javadoc 0m 32s the patch passed with JDK Ubuntu-11.0.21+9-post-Ubuntu-0ubuntu120.04
+1 :green_heart: javadoc 0m 59s the patch passed with JDK Private Build-1.8.0_392-8u392-ga-1~20.04-b08
+1 :green_heart: spotbugs 1m 43s the patch passed
+1 :green_heart: shadedclient 20m 30s patch has no errors when building and testing our client artifacts.
_ Other Tests _
-1 :x: unit 204m 3s /patch-unit-hadoop-hdfs-project_hadoop-hdfs.txt hadoop-hdfs in the patch passed.
+1 :green_heart: asflicense 0m 30s The patch does not generate ASF License warnings.
310m 8s
Reason Tests
Failed junit tests hadoop.hdfs.server.datanode.TestLargeBlockReport
hadoop.hdfs.server.datanode.TestDirectoryScanner
hadoop.hdfs.protocol.TestBlockListAsLongs
hadoop.hdfs.tools.TestDFSAdmin
Subsystem Report/Notes
Docker ClientAPI=1.44 ServerAPI=1.44 base: https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-6559/5/artifact/out/Dockerfile
GITHUB PR https://github.com/apache/hadoop/pull/6559
Optional Tests dupname asflicense compile javac javadoc mvninstall mvnsite unit shadedclient spotbugs checkstyle codespell detsecrets xmllint
uname Linux e1573661312f 5.15.0-94-generic #104-Ubuntu SMP Tue Jan 9 15:25:40 UTC 2024 x86_64 x86_64 x86_64 GNU/Linux
Build tool maven
Personality dev-support/bin/hadoop.sh
git revision trunk / cada170bb5603c7921ae6eae957dff26deb76c5d
Default Java Private Build-1.8.0_392-8u392-ga-1~20.04-b08
Multi-JDK versions /usr/lib/jvm/java-11-openjdk-amd64:Ubuntu-11.0.21+9-post-Ubuntu-0ubuntu120.04 /usr/lib/jvm/java-8-openjdk-amd64:Private Build-1.8.0_392-8u392-ga-1~20.04-b08
Test Results https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-6559/5/testReport/
Max. process+thread count 4349 (vs. ulimit of 5500)
modules C: hadoop-hdfs-project/hadoop-hdfs U: hadoop-hdfs-project/hadoop-hdfs
Console output https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-6559/5/console
versions git=2.25.1 maven=3.6.3 spotbugs=4.2.2
Powered by Apache Yetus 0.14.0 https://yetus.apache.org

This message was automatically generated.

hadoop-yetus avatar Feb 24 '24 07:02 hadoop-yetus

The failed tests seem not to be related.

haiyang1987 avatar Feb 24 '24 11:02 haiyang1987

Hi @ZanderXu @tomscut @zhangshuyan0 @tasanuma please help me review again this PR when you are free, thanks ~

haiyang1987 avatar Feb 27 '24 02:02 haiyang1987

Hi @Hexiaoqiao Sir would you mind look it again, thanks~

haiyang1987 avatar Mar 12 '24 10:03 haiyang1987

Thanks involve me here. I think @zhangshuyan0 should be more professional about this improvement. Let's wait her/his feedback.

Hexiaoqiao avatar Mar 13 '24 02:03 Hexiaoqiao

Thanks involve me here. I think @zhangshuyan0 should be more professional about this improvement. Let's wait her/his feedback.

ok, thanks for your comment.

haiyang1987 avatar Mar 13 '24 02:03 haiyang1987

Hi @zhangshuyan0 please help me review again this PR when you are free, thanks ~

haiyang1987 avatar Mar 29 '24 08:03 haiyang1987

:confetti_ball: +1 overall

Vote Subsystem Runtime Logfile Comment
_ Prechecks _
+1 :green_heart: dupname 0m 02s No case conflicting files found.
+0 :ok: spotbugs 0m 00s spotbugs executables are not available.
+0 :ok: codespell 0m 01s codespell was not available.
+0 :ok: detsecrets 0m 01s detect-secrets was not available.
+0 :ok: xmllint 0m 01s xmllint was not available.
+1 :green_heart: @author 0m 00s The patch does not contain any @author tags.
+1 :green_heart: test4tests 0m 00s The patch appears to include 5 new or modified test files.
_ trunk Compile Tests _
+1 :green_heart: mvninstall 97m 05s trunk passed
+1 :green_heart: compile 6m 28s trunk passed
+1 :green_heart: checkstyle 5m 07s trunk passed
+1 :green_heart: mvnsite 7m 08s trunk passed
+1 :green_heart: javadoc 6m 36s trunk passed
+1 :green_heart: shadedclient 153m 07s branch has no errors when building and testing our client artifacts.
_ Patch Compile Tests _
+1 :green_heart: mvninstall 4m 48s the patch passed
+1 :green_heart: compile 3m 42s the patch passed
+1 :green_heart: javac 3m 42s the patch passed
+1 :green_heart: blanks 0m 00s The patch has no blanks issues.
+1 :green_heart: checkstyle 2m 36s the patch passed
+1 :green_heart: mvnsite 4m 29s the patch passed
+1 :green_heart: javadoc 3m 50s the patch passed
+1 :green_heart: shadedclient 167m 08s patch has no errors when building and testing our client artifacts.
_ Other Tests _
+1 :green_heart: asflicense 5m 41s The patch does not generate ASF License warnings.
445m 34s
Subsystem Report/Notes
GITHUB PR https://github.com/apache/hadoop/pull/6559
Optional Tests dupname asflicense compile javac javadoc mvninstall mvnsite unit shadedclient spotbugs checkstyle codespell detsecrets xmllint
uname MINGW64_NT-10.0-17763 d777404a85ec 3.4.10-87d57229.x86_64 2024-02-14 20:17 UTC x86_64 Msys
Build tool maven
Personality /c/hadoop/dev-support/bin/hadoop.sh
git revision trunk / cada170bb5603c7921ae6eae957dff26deb76c5d
Default Java Azul Systems, Inc.-1.8.0_332-b09
Test Results https://ci-hadoop.apache.org/job/hadoop-multibranch-windows-10/job/PR-6559/1/testReport/
modules C: hadoop-hdfs-project/hadoop-hdfs U: hadoop-hdfs-project/hadoop-hdfs
Console output https://ci-hadoop.apache.org/job/hadoop-multibranch-windows-10/job/PR-6559/1/console
versions git=2.44.0.windows.1
Powered by Apache Yetus 0.14.0 https://yetus.apache.org

This message was automatically generated.

hadoop-yetus avatar Apr 25 '24 07:04 hadoop-yetus