HDFS-17556. Avoid adding block to neededReconstruction repeatedly in decommission

[lfxy]

HDFS-17556 In decommission and maintenance process, before added to BlockManager::neededReconstruction block will be check if it has been added. The check contains if block is in BlockManager::neededReconstruction or in PendingReconstructionBlocks::pendingReconstructions as below code. But it also need to check if it is in PendingReconstructionBlocks::timedOutItems. Or else DatanodeAdminDefaultMonitor will add block to BlockManager::neededReconstruction repeatedly if block time out in PendingReconstructionBlocks::pendingReconstructions. if (!blockManager.neededReconstruction.contains(block) && blockManager.pendingReconstruction.getNumReplicas(block) == 0 && blockManager.isPopulatingReplQueues()) { // Process these blocks only when active NN is out of safe mode. blockManager.neededReconstruction.add(block, liveReplicas, num.readOnlyReplicas(), num.outOfServiceReplicas(), blockManager.getExpectedRedundancyNum(block)); }

[hadoop-yetus]

:broken_heart: -1 overall

Vote	Subsystem	Runtime	Logfile	Comment
+0 :ok:	reexec	18m 11s		Docker mode activated.
			_ Prechecks _
+1 :green_heart:	dupname	0m 0s		No case conflicting files found.
+0 :ok:	codespell	0m 0s		codespell was not available.
+0 :ok:	detsecrets	0m 0s		detect-secrets was not available.
+1 :green_heart:	@author	0m 0s		The patch does not contain any @author tags.
+1 :green_heart:	test4tests	0m 0s		The patch appears to include 1 new or modified test files.
			_ trunk Compile Tests _
+1 :green_heart:	mvninstall	44m 49s		trunk passed
+1 :green_heart:	compile	1m 23s		trunk passed with JDK Ubuntu-11.0.23+9-post-Ubuntu-1ubuntu120.04.2
+1 :green_heart:	compile	1m 18s		trunk passed with JDK Private Build-1.8.0_412-8u412-ga-1~20.04.1-b08
+1 :green_heart:	checkstyle	1m 10s		trunk passed
+1 :green_heart:	mvnsite	1m 22s		trunk passed
+1 :green_heart:	javadoc	1m 8s		trunk passed with JDK Ubuntu-11.0.23+9-post-Ubuntu-1ubuntu120.04.2
+1 :green_heart:	javadoc	1m 43s		trunk passed with JDK Private Build-1.8.0_412-8u412-ga-1~20.04.1-b08
+1 :green_heart:	spotbugs	3m 15s		trunk passed
+1 :green_heart:	shadedclient	35m 41s		branch has no errors when building and testing our client artifacts.
			_ Patch Compile Tests _
+1 :green_heart:	mvninstall	1m 11s		the patch passed
+1 :green_heart:	compile	1m 14s		the patch passed with JDK Ubuntu-11.0.23+9-post-Ubuntu-1ubuntu120.04.2
+1 :green_heart:	javac	1m 14s		the patch passed
+1 :green_heart:	compile	1m 10s		the patch passed with JDK Private Build-1.8.0_412-8u412-ga-1~20.04.1-b08
+1 :green_heart:	javac	1m 10s		the patch passed
+1 :green_heart:	blanks	0m 0s		The patch has no blanks issues.
+1 :green_heart:	checkstyle	0m 59s		the patch passed
+1 :green_heart:	mvnsite	1m 17s		the patch passed
+1 :green_heart:	javadoc	0m 53s		the patch passed with JDK Ubuntu-11.0.23+9-post-Ubuntu-1ubuntu120.04.2
+1 :green_heart:	javadoc	1m 42s		the patch passed with JDK Private Build-1.8.0_412-8u412-ga-1~20.04.1-b08
+1 :green_heart:	spotbugs	3m 12s		the patch passed
+1 :green_heart:	shadedclient	35m 59s		patch has no errors when building and testing our client artifacts.
			_ Other Tests _
-1 :x:	unit	302m 16s	/patch-unit-hadoop-hdfs-project_hadoop-hdfs.txt	hadoop-hdfs in the patch passed.
+1 :green_heart:	asflicense	0m 47s		The patch does not generate ASF License warnings.
		458m 59s

Reason	Tests
Failed junit tests	hadoop.hdfs.server.blockmanagement.TestBlocksWithNotEnoughRacks
	hadoop.hdfs.server.blockmanagement.TestDatanodeManager
	hadoop.hdfs.TestMaintenanceWithStriped
	hadoop.hdfs.server.namenode.TestUpgradeDomainBlockPlacementPolicy
	hadoop.hdfs.server.namenode.TestHostsFiles

Subsystem	Report/Notes
Docker	ClientAPI=1.45 ServerAPI=1.45 base: https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-6896/1/artifact/out/Dockerfile
GITHUB PR	https://github.com/apache/hadoop/pull/6896
Optional Tests	dupname asflicense compile javac javadoc mvninstall mvnsite unit shadedclient spotbugs checkstyle codespell detsecrets
uname	Linux fa3b3f697cee 5.15.0-106-generic #116-Ubuntu SMP Wed Apr 17 09:17:56 UTC 2024 x86_64 x86_64 x86_64 GNU/Linux
Build tool	maven
Personality	dev-support/bin/hadoop.sh
git revision	trunk / 4dc4bea54d04c002b6b6304ddb68daacbd5697f8
Default Java	Private Build-1.8.0_412-8u412-ga-1~20.04.1-b08
Multi-JDK versions	/usr/lib/jvm/java-11-openjdk-amd64:Ubuntu-11.0.23+9-post-Ubuntu-1ubuntu120.04.2 /usr/lib/jvm/java-8-openjdk-amd64:Private Build-1.8.0_412-8u412-ga-1~20.04.1-b08
Test Results	https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-6896/1/testReport/
Max. process+thread count	3085 (vs. ulimit of 5500)
modules	C: hadoop-hdfs-project/hadoop-hdfs U: hadoop-hdfs-project/hadoop-hdfs
Console output	https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-6896/1/console
versions	git=2.25.1 maven=3.6.3 spotbugs=4.2.2
Powered by	Apache Yetus 0.14.0 https://yetus.apache.org

This message was automatically generated.

[hadoop-yetus]

:broken_heart: -1 overall

Vote	Subsystem	Runtime	Logfile	Comment
+0 :ok:	reexec	0m 54s		Docker mode activated.
			_ Prechecks _
+1 :green_heart:	dupname	0m 1s		No case conflicting files found.
+0 :ok:	codespell	0m 0s		codespell was not available.
+0 :ok:	detsecrets	0m 0s		detect-secrets was not available.
+1 :green_heart:	@author	0m 0s		The patch does not contain any @author tags.
+1 :green_heart:	test4tests	0m 0s		The patch appears to include 1 new or modified test files.
			_ trunk Compile Tests _
+1 :green_heart:	mvninstall	44m 46s		trunk passed
+1 :green_heart:	compile	1m 22s		trunk passed with JDK Ubuntu-11.0.23+9-post-Ubuntu-1ubuntu120.04.2
+1 :green_heart:	compile	1m 16s		trunk passed with JDK Private Build-1.8.0_412-8u412-ga-1~20.04.1-b08
+1 :green_heart:	checkstyle	1m 12s		trunk passed
+1 :green_heart:	mvnsite	1m 25s		trunk passed
+1 :green_heart:	javadoc	1m 10s		trunk passed with JDK Ubuntu-11.0.23+9-post-Ubuntu-1ubuntu120.04.2
+1 :green_heart:	javadoc	1m 44s		trunk passed with JDK Private Build-1.8.0_412-8u412-ga-1~20.04.1-b08
+1 :green_heart:	spotbugs	3m 14s		trunk passed
+1 :green_heart:	shadedclient	35m 44s		branch has no errors when building and testing our client artifacts.
			_ Patch Compile Tests _
+1 :green_heart:	mvninstall	1m 9s		the patch passed
+1 :green_heart:	compile	1m 12s		the patch passed with JDK Ubuntu-11.0.23+9-post-Ubuntu-1ubuntu120.04.2
+1 :green_heart:	javac	1m 12s		the patch passed
+1 :green_heart:	compile	1m 7s		the patch passed with JDK Private Build-1.8.0_412-8u412-ga-1~20.04.1-b08
+1 :green_heart:	javac	1m 7s		the patch passed
+1 :green_heart:	blanks	0m 0s		The patch has no blanks issues.
+1 :green_heart:	checkstyle	0m 58s		the patch passed
+1 :green_heart:	mvnsite	1m 12s		the patch passed
+1 :green_heart:	javadoc	0m 54s		the patch passed with JDK Ubuntu-11.0.23+9-post-Ubuntu-1ubuntu120.04.2
+1 :green_heart:	javadoc	1m 37s		the patch passed with JDK Private Build-1.8.0_412-8u412-ga-1~20.04.1-b08
+1 :green_heart:	spotbugs	3m 14s		the patch passed
+1 :green_heart:	shadedclient	35m 39s		patch has no errors when building and testing our client artifacts.
			_ Other Tests _
-1 :x:	unit	275m 8s	/patch-unit-hadoop-hdfs-project_hadoop-hdfs.txt	hadoop-hdfs in the patch passed.
+1 :green_heart:	asflicense	0m 46s		The patch does not generate ASF License warnings.
		414m 0s

Reason	Tests
Failed junit tests	hadoop.hdfs.server.blockmanagement.TestDatanodeManager

Subsystem	Report/Notes
Docker	ClientAPI=1.45 ServerAPI=1.45 base: https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-6896/2/artifact/out/Dockerfile
GITHUB PR	https://github.com/apache/hadoop/pull/6896
Optional Tests	dupname asflicense compile javac javadoc mvninstall mvnsite unit shadedclient spotbugs checkstyle codespell detsecrets
uname	Linux 77923fa8bbc0 5.15.0-106-generic #116-Ubuntu SMP Wed Apr 17 09:17:56 UTC 2024 x86_64 x86_64 x86_64 GNU/Linux
Build tool	maven
Personality	dev-support/bin/hadoop.sh
git revision	trunk / 55321d488e80e0f81f65e2c4d962216f418613bc
Default Java	Private Build-1.8.0_412-8u412-ga-1~20.04.1-b08
Multi-JDK versions	/usr/lib/jvm/java-11-openjdk-amd64:Ubuntu-11.0.23+9-post-Ubuntu-1ubuntu120.04.2 /usr/lib/jvm/java-8-openjdk-amd64:Private Build-1.8.0_412-8u412-ga-1~20.04.1-b08
Test Results	https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-6896/2/testReport/
Max. process+thread count	3687 (vs. ulimit of 5500)
modules	C: hadoop-hdfs-project/hadoop-hdfs U: hadoop-hdfs-project/hadoop-hdfs
Console output	https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-6896/2/console
versions	git=2.25.1 maven=3.6.3 spotbugs=4.2.2
Powered by	Apache Yetus 0.14.0 https://yetus.apache.org

This message was automatically generated.

[Hexiaoqiao]

@lfxy Thanks for your report and PR. It looks good to me. Please check the failed unit test and if related to this PR. It's better to add another test to cover this case if possible. Thanks again.

[lfxy]

@Hexiaoqiao Thanks for your review! The failed unit test is org.apache.hadoop.hdfs.server.datanode.TestDirectoryScanner and I think it's not related to this PR. I will add another UT.

[hadoop-yetus]

:broken_heart: -1 overall

Vote	Subsystem	Runtime	Logfile	Comment
+0 :ok:	reexec	0m 56s		Docker mode activated.
			_ Prechecks _
+1 :green_heart:	dupname	0m 0s		No case conflicting files found.
+0 :ok:	codespell	0m 1s		codespell was not available.
+0 :ok:	detsecrets	0m 1s		detect-secrets was not available.
+1 :green_heart:	@author	0m 0s		The patch does not contain any @author tags.
+1 :green_heart:	test4tests	0m 0s		The patch appears to include 2 new or modified test files.
			_ trunk Compile Tests _
+1 :green_heart:	mvninstall	45m 24s		trunk passed
+1 :green_heart:	compile	1m 23s		trunk passed with JDK Ubuntu-11.0.24+8-post-Ubuntu-1ubuntu320.04
+1 :green_heart:	compile	1m 17s		trunk passed with JDK Private Build-1.8.0_422-8u422-b05-1~20.04-b05
+1 :green_heart:	checkstyle	1m 11s		trunk passed
+1 :green_heart:	mvnsite	1m 26s		trunk passed
+1 :green_heart:	javadoc	1m 10s		trunk passed with JDK Ubuntu-11.0.24+8-post-Ubuntu-1ubuntu320.04
+1 :green_heart:	javadoc	1m 49s		trunk passed with JDK Private Build-1.8.0_422-8u422-b05-1~20.04-b05
+1 :green_heart:	spotbugs	3m 21s		trunk passed
+1 :green_heart:	shadedclient	36m 19s		branch has no errors when building and testing our client artifacts.
			_ Patch Compile Tests _
+1 :green_heart:	mvninstall	1m 11s		the patch passed
+1 :green_heart:	compile	1m 15s		the patch passed with JDK Ubuntu-11.0.24+8-post-Ubuntu-1ubuntu320.04
+1 :green_heart:	javac	1m 15s		the patch passed
+1 :green_heart:	compile	1m 8s		the patch passed with JDK Private Build-1.8.0_422-8u422-b05-1~20.04-b05
+1 :green_heart:	javac	1m 8s		the patch passed
+1 :green_heart:	blanks	0m 0s		The patch has no blanks issues.
-0 :warning:	checkstyle	0m 57s	/results-checkstyle-hadoop-hdfs-project_hadoop-hdfs.txt	hadoop-hdfs-project/hadoop-hdfs: The patch generated 2 new + 51 unchanged - 0 fixed = 53 total (was 51)
+1 :green_heart:	mvnsite	1m 17s		the patch passed
+1 :green_heart:	javadoc	0m 53s		the patch passed with JDK Ubuntu-11.0.24+8-post-Ubuntu-1ubuntu320.04
+1 :green_heart:	javadoc	1m 37s		the patch passed with JDK Private Build-1.8.0_422-8u422-b05-1~20.04-b05
+1 :green_heart:	spotbugs	3m 16s		the patch passed
+1 :green_heart:	shadedclient	36m 10s		patch has no errors when building and testing our client artifacts.
			_ Other Tests _
-1 :x:	unit	274m 37s	/patch-unit-hadoop-hdfs-project_hadoop-hdfs.txt	hadoop-hdfs in the patch passed.
+1 :green_heart:	asflicense	0m 45s		The patch does not generate ASF License warnings.
		415m 23s

Reason	Tests
Failed junit tests	hadoop.hdfs.server.blockmanagement.TestBlockManager

Subsystem	Report/Notes
Docker	ClientAPI=1.46 ServerAPI=1.46 base: https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-6896/3/artifact/out/Dockerfile
GITHUB PR	https://github.com/apache/hadoop/pull/6896
Optional Tests	dupname asflicense compile javac javadoc mvninstall mvnsite unit shadedclient spotbugs checkstyle codespell detsecrets
uname	Linux 3ef59b56ac77 5.15.0-117-generic #127-Ubuntu SMP Fri Jul 5 20:13:28 UTC 2024 x86_64 x86_64 x86_64 GNU/Linux
Build tool	maven
Personality	dev-support/bin/hadoop.sh
git revision	trunk / 55262697eebf32132b2b4b5665a4ffb6884396b9
Default Java	Private Build-1.8.0_422-8u422-b05-1~20.04-b05
Multi-JDK versions	/usr/lib/jvm/java-11-openjdk-amd64:Ubuntu-11.0.24+8-post-Ubuntu-1ubuntu320.04 /usr/lib/jvm/java-8-openjdk-amd64:Private Build-1.8.0_422-8u422-b05-1~20.04-b05
Test Results	https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-6896/3/testReport/
Max. process+thread count	3179 (vs. ulimit of 5500)
modules	C: hadoop-hdfs-project/hadoop-hdfs U: hadoop-hdfs-project/hadoop-hdfs
Console output	https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-6896/3/console
versions	git=2.25.1 maven=3.6.3 spotbugs=4.2.2
Powered by	Apache Yetus 0.14.0 https://yetus.apache.org

This message was automatically generated.

[hadoop-yetus]

:confetti_ball: +1 overall

Vote	Subsystem	Runtime	Logfile	Comment
+0 :ok:	reexec	0m 30s		Docker mode activated.
			_ Prechecks _
+1 :green_heart:	dupname	0m 0s		No case conflicting files found.
+0 :ok:	codespell	0m 1s		codespell was not available.
+0 :ok:	detsecrets	0m 1s		detect-secrets was not available.
+1 :green_heart:	@author	0m 0s		The patch does not contain any @author tags.
+1 :green_heart:	test4tests	0m 0s		The patch appears to include 2 new or modified test files.
			_ trunk Compile Tests _
+1 :green_heart:	mvninstall	46m 1s		trunk passed
+1 :green_heart:	compile	1m 24s		trunk passed with JDK Ubuntu-11.0.24+8-post-Ubuntu-1ubuntu320.04
+1 :green_heart:	compile	1m 20s		trunk passed with JDK Private Build-1.8.0_422-8u422-b05-1~20.04-b05
+1 :green_heart:	checkstyle	1m 11s		trunk passed
+1 :green_heart:	mvnsite	1m 24s		trunk passed
+1 :green_heart:	javadoc	1m 7s		trunk passed with JDK Ubuntu-11.0.24+8-post-Ubuntu-1ubuntu320.04
+1 :green_heart:	javadoc	1m 43s		trunk passed with JDK Private Build-1.8.0_422-8u422-b05-1~20.04-b05
+1 :green_heart:	spotbugs	3m 19s		trunk passed
+1 :green_heart:	shadedclient	36m 15s		branch has no errors when building and testing our client artifacts.
			_ Patch Compile Tests _
+1 :green_heart:	mvninstall	1m 13s		the patch passed
+1 :green_heart:	compile	1m 14s		the patch passed with JDK Ubuntu-11.0.24+8-post-Ubuntu-1ubuntu320.04
+1 :green_heart:	javac	1m 14s		the patch passed
+1 :green_heart:	compile	1m 7s		the patch passed with JDK Private Build-1.8.0_422-8u422-b05-1~20.04-b05
+1 :green_heart:	javac	1m 7s		the patch passed
+1 :green_heart:	blanks	0m 0s		The patch has no blanks issues.
+1 :green_heart:	checkstyle	0m 58s		the patch passed
+1 :green_heart:	mvnsite	1m 14s		the patch passed
+1 :green_heart:	javadoc	0m 55s		the patch passed with JDK Ubuntu-11.0.24+8-post-Ubuntu-1ubuntu320.04
+1 :green_heart:	javadoc	1m 38s		the patch passed with JDK Private Build-1.8.0_422-8u422-b05-1~20.04-b05
+1 :green_heart:	spotbugs	3m 18s		the patch passed
+1 :green_heart:	shadedclient	36m 6s		patch has no errors when building and testing our client artifacts.
			_ Other Tests _
+1 :green_heart:	unit	223m 37s		hadoop-hdfs in the patch passed.
+1 :green_heart:	asflicense	0m 48s		The patch does not generate ASF License warnings.
		364m 23s

Subsystem	Report/Notes
Docker	ClientAPI=1.46 ServerAPI=1.46 base: https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-6896/4/artifact/out/Dockerfile
GITHUB PR	https://github.com/apache/hadoop/pull/6896
Optional Tests	dupname asflicense compile javac javadoc mvninstall mvnsite unit shadedclient spotbugs checkstyle codespell detsecrets
uname	Linux aa957c4f5bc8 5.15.0-117-generic #127-Ubuntu SMP Fri Jul 5 20:13:28 UTC 2024 x86_64 x86_64 x86_64 GNU/Linux
Build tool	maven
Personality	dev-support/bin/hadoop.sh
git revision	trunk / 2bd848dd2450cd7d2c8e2d2a98d9dfaba5cfe449
Default Java	Private Build-1.8.0_422-8u422-b05-1~20.04-b05
Multi-JDK versions	/usr/lib/jvm/java-11-openjdk-amd64:Ubuntu-11.0.24+8-post-Ubuntu-1ubuntu320.04 /usr/lib/jvm/java-8-openjdk-amd64:Private Build-1.8.0_422-8u422-b05-1~20.04-b05
Test Results	https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-6896/4/testReport/
Max. process+thread count	3982 (vs. ulimit of 5500)
modules	C: hadoop-hdfs-project/hadoop-hdfs U: hadoop-hdfs-project/hadoop-hdfs
Console output	https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-6896/4/console
versions	git=2.25.1 maven=3.6.3 spotbugs=4.2.2
Powered by	Apache Yetus 0.14.0 https://yetus.apache.org

This message was automatically generated.

[lfxy]

@Hexiaoqiao I have add new UT to cover this case, and please review again. Thanks!

[haiyang1987]

Well work, Leave some comments inline, Thanks~

[lfxy]

@haiyang1987 Thank you for your suggestion. I have update the UT.

[hadoop-yetus]

:confetti_ball: +1 overall

Vote	Subsystem	Runtime	Logfile	Comment
+0 :ok:	reexec	0m 54s		Docker mode activated.
			_ Prechecks _
+1 :green_heart:	dupname	0m 0s		No case conflicting files found.
+0 :ok:	codespell	0m 0s		codespell was not available.
+0 :ok:	detsecrets	0m 0s		detect-secrets was not available.
+1 :green_heart:	@author	0m 0s		The patch does not contain any @author tags.
+1 :green_heart:	test4tests	0m 0s		The patch appears to include 2 new or modified test files.
			_ trunk Compile Tests _
+1 :green_heart:	mvninstall	44m 53s		trunk passed
+1 :green_heart:	compile	1m 25s		trunk passed with JDK Ubuntu-11.0.24+8-post-Ubuntu-1ubuntu320.04
+1 :green_heart:	compile	1m 17s		trunk passed with JDK Private Build-1.8.0_422-8u422-b05-1~20.04-b05
+1 :green_heart:	checkstyle	1m 10s		trunk passed
+1 :green_heart:	mvnsite	1m 25s		trunk passed
+1 :green_heart:	javadoc	1m 7s		trunk passed with JDK Ubuntu-11.0.24+8-post-Ubuntu-1ubuntu320.04
+1 :green_heart:	javadoc	1m 43s		trunk passed with JDK Private Build-1.8.0_422-8u422-b05-1~20.04-b05
+1 :green_heart:	spotbugs	3m 19s		trunk passed
+1 :green_heart:	shadedclient	36m 3s		branch has no errors when building and testing our client artifacts.
			_ Patch Compile Tests _
+1 :green_heart:	mvninstall	1m 11s		the patch passed
+1 :green_heart:	compile	1m 15s		the patch passed with JDK Ubuntu-11.0.24+8-post-Ubuntu-1ubuntu320.04
+1 :green_heart:	javac	1m 15s		the patch passed
+1 :green_heart:	compile	1m 8s		the patch passed with JDK Private Build-1.8.0_422-8u422-b05-1~20.04-b05
+1 :green_heart:	javac	1m 8s		the patch passed
+1 :green_heart:	blanks	0m 0s		The patch has no blanks issues.
-0 :warning:	checkstyle	0m 58s	/results-checkstyle-hadoop-hdfs-project_hadoop-hdfs.txt	hadoop-hdfs-project/hadoop-hdfs: The patch generated 1 new + 51 unchanged - 0 fixed = 52 total (was 51)
+1 :green_heart:	mvnsite	1m 13s		the patch passed
+1 :green_heart:	javadoc	0m 53s		the patch passed with JDK Ubuntu-11.0.24+8-post-Ubuntu-1ubuntu320.04
+1 :green_heart:	javadoc	1m 40s		the patch passed with JDK Private Build-1.8.0_422-8u422-b05-1~20.04-b05
+1 :green_heart:	spotbugs	3m 15s		the patch passed
+1 :green_heart:	shadedclient	35m 59s		patch has no errors when building and testing our client artifacts.
			_ Other Tests _
+1 :green_heart:	unit	272m 18s		hadoop-hdfs in the patch passed.
+1 :green_heart:	asflicense	0m 47s		The patch does not generate ASF License warnings.
		412m 7s

Subsystem	Report/Notes
Docker	ClientAPI=1.46 ServerAPI=1.46 base: https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-6896/5/artifact/out/Dockerfile
GITHUB PR	https://github.com/apache/hadoop/pull/6896
Optional Tests	dupname asflicense compile javac javadoc mvninstall mvnsite unit shadedclient spotbugs checkstyle codespell detsecrets
uname	Linux 87a6c1ccca12 5.15.0-117-generic #127-Ubuntu SMP Fri Jul 5 20:13:28 UTC 2024 x86_64 x86_64 x86_64 GNU/Linux
Build tool	maven
Personality	dev-support/bin/hadoop.sh
git revision	trunk / 01c9c841fe2173c2981c1de332b06cc5618a44d8
Default Java	Private Build-1.8.0_422-8u422-b05-1~20.04-b05
Multi-JDK versions	/usr/lib/jvm/java-11-openjdk-amd64:Ubuntu-11.0.24+8-post-Ubuntu-1ubuntu320.04 /usr/lib/jvm/java-8-openjdk-amd64:Private Build-1.8.0_422-8u422-b05-1~20.04-b05
Test Results	https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-6896/5/testReport/
Max. process+thread count	3103 (vs. ulimit of 5500)
modules	C: hadoop-hdfs-project/hadoop-hdfs U: hadoop-hdfs-project/hadoop-hdfs
Console output	https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-6896/5/console
versions	git=2.25.1 maven=3.6.3 spotbugs=4.2.2
Powered by	Apache Yetus 0.14.0 https://yetus.apache.org

This message was automatically generated.

[hadoop-yetus]

:confetti_ball: +1 overall

Vote	Subsystem	Runtime	Logfile	Comment
+0 :ok:	reexec	0m 29s		Docker mode activated.
			_ Prechecks _
+1 :green_heart:	dupname	0m 0s		No case conflicting files found.
+0 :ok:	codespell	0m 1s		codespell was not available.
+0 :ok:	detsecrets	0m 1s		detect-secrets was not available.
+1 :green_heart:	@author	0m 0s		The patch does not contain any @author tags.
+1 :green_heart:	test4tests	0m 0s		The patch appears to include 2 new or modified test files.
			_ trunk Compile Tests _
+1 :green_heart:	mvninstall	46m 10s		trunk passed
+1 :green_heart:	compile	1m 22s		trunk passed with JDK Ubuntu-11.0.24+8-post-Ubuntu-1ubuntu320.04
+1 :green_heart:	compile	1m 16s		trunk passed with JDK Private Build-1.8.0_422-8u422-b05-1~20.04-b05
+1 :green_heart:	checkstyle	1m 13s		trunk passed
+1 :green_heart:	mvnsite	1m 24s		trunk passed
+1 :green_heart:	javadoc	1m 9s		trunk passed with JDK Ubuntu-11.0.24+8-post-Ubuntu-1ubuntu320.04
+1 :green_heart:	javadoc	1m 44s		trunk passed with JDK Private Build-1.8.0_422-8u422-b05-1~20.04-b05
+1 :green_heart:	spotbugs	3m 16s		trunk passed
+1 :green_heart:	shadedclient	36m 3s		branch has no errors when building and testing our client artifacts.
			_ Patch Compile Tests _
+1 :green_heart:	mvninstall	1m 11s		the patch passed
+1 :green_heart:	compile	1m 16s		the patch passed with JDK Ubuntu-11.0.24+8-post-Ubuntu-1ubuntu320.04
+1 :green_heart:	javac	1m 16s		the patch passed
+1 :green_heart:	compile	1m 11s		the patch passed with JDK Private Build-1.8.0_422-8u422-b05-1~20.04-b05
+1 :green_heart:	javac	1m 11s		the patch passed
+1 :green_heart:	blanks	0m 0s		The patch has no blanks issues.
+1 :green_heart:	checkstyle	0m 58s		the patch passed
+1 :green_heart:	mvnsite	1m 15s		the patch passed
+1 :green_heart:	javadoc	0m 53s		the patch passed with JDK Ubuntu-11.0.24+8-post-Ubuntu-1ubuntu320.04
+1 :green_heart:	javadoc	1m 38s		the patch passed with JDK Private Build-1.8.0_422-8u422-b05-1~20.04-b05
+1 :green_heart:	spotbugs	3m 15s		the patch passed
+1 :green_heart:	shadedclient	35m 55s		patch has no errors when building and testing our client artifacts.
			_ Other Tests _
+1 :green_heart:	unit	223m 4s		hadoop-hdfs in the patch passed.
+1 :green_heart:	asflicense	0m 47s		The patch does not generate ASF License warnings.
		363m 30s

Subsystem	Report/Notes
Docker	ClientAPI=1.46 ServerAPI=1.46 base: https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-6896/6/artifact/out/Dockerfile
GITHUB PR	https://github.com/apache/hadoop/pull/6896
Optional Tests	dupname asflicense compile javac javadoc mvninstall mvnsite unit shadedclient spotbugs checkstyle codespell detsecrets
uname	Linux 1a43d5f27b1f 5.15.0-117-generic #127-Ubuntu SMP Fri Jul 5 20:13:28 UTC 2024 x86_64 x86_64 x86_64 GNU/Linux
Build tool	maven
Personality	dev-support/bin/hadoop.sh
git revision	trunk / d9edab0f319d8cb0ad01f0746ac905d270bd231c
Default Java	Private Build-1.8.0_422-8u422-b05-1~20.04-b05
Multi-JDK versions	/usr/lib/jvm/java-11-openjdk-amd64:Ubuntu-11.0.24+8-post-Ubuntu-1ubuntu320.04 /usr/lib/jvm/java-8-openjdk-amd64:Private Build-1.8.0_422-8u422-b05-1~20.04-b05
Test Results	https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-6896/6/testReport/
Max. process+thread count	3973 (vs. ulimit of 5500)
modules	C: hadoop-hdfs-project/hadoop-hdfs U: hadoop-hdfs-project/hadoop-hdfs
Console output	https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-6896/6/console
versions	git=2.25.1 maven=3.6.3 spotbugs=4.2.2
Powered by	Apache Yetus 0.14.0 https://yetus.apache.org

This message was automatically generated.

[zhengchenyu]

I think you may mean that add block to pendingReconstruction in two place is duplicated.

• (1) add decommission block to pendingReconstruction
• (2) add timed out block to pendingReconstruction

I think in (2), when we add timed out block to pendingReconstruction, we will use isNeededReconstruction to check. Will it really be repeated? I think it is better to check in (2) than in (1), because the number of blocks checked in (1) is often all blocks on a DN, while in (2) all timedout blocks are checked. The latter is usually smaller than the former.

[lfxy]

@zhengchenyu I think it's wrong you described about pendingReconstruction. The block flow in decommission is: (1)DatanodeAdminManager::monitor --> (2)neededReconstruction --> (3)PendingReconstructionBlocks::pendingReconstructions --> (4)PendingReconstructionBlocks::timedOutItems--> (5)neededReconstruction If reconstruction task failed in dn, block will flow from (2) to (5) circularly. When the block is in timedOutItems and we don't check, DatanodeAdminManager will add block to neededReconstruction repeated.

[zhengchenyu]

@lfxy you does not catch me. I said the same thing as you said. My (1) is just your (1) -> (2), My (2) is just your (4) -> (5). I suggest checking in (4)->(5). Although we filter out timed out blocks in (1) -> (2), we still need to add these block in (4) -> (5). We can do the check in (4) -> (5). Generally speaking, the verification cost in (4) -> (5) is lower, because there are fewer blocks to check. And there is no need to change timedOutItems from List to Set.

[lfxy]

@zhengchenyu We should check (1) -> (2), but not (4) -> (5), because (4)->(5) is not only used in decommission flow.

[lfxy]

@Hexiaoqiao @haiyang1987 Hi, Could this commit be merged?

[github-actions[bot]]

We're closing this stale PR because it has been open for 100 days with no activity. This isn't a judgement on the merit of the PR in any way. It's just a way of keeping the PR queue manageable. If you feel like this was a mistake, or you would like to continue working on it, please feel free to re-open it and ask for a committer to remove the stale tag and review again. Thanks all for your contribution.