HDFS-17641 .Psane/backport hdfs 17641 to 3.3

[PrateekSane]

Description of PR

Backporting metrics for Low Redundancy blocks and additional tracked metrics for misreplicated blocks back to 3.3 branch.

Requires cherry picks of the following https://github.com/apache/hadoop/pull/7123 https://github.com/apache/hadoop/commit/275c478330d5c8cae3c15b876cc8128d164e9fa0

How was this patch tested?

mvn clean install -DskipTests

For code changes:

• [x] Does the title or this PR starts with the corresponding JIRA issue id (e.g. 'HADOOP-17799. Your PR title ...')?
• [x] Object storage: have the integration tests been executed and the endpoint declared according to the connector-specific documentation?
• [x] If adding new dependencies to the code, are these dependencies licensed in a way that is compatible for inclusion under ASF 2.0?
• [x] If applicable, have you updated the LICENSE, LICENSE-binary, NOTICE-binary files?

[hadoop-yetus]

:broken_heart: -1 overall

Vote	Subsystem	Runtime	Logfile	Comment
+0 :ok:	reexec	6m 13s		Docker mode activated.
			_ Prechecks _
+1 :green_heart:	dupname	0m 0s		No case conflicting files found.
+0 :ok:	codespell	0m 0s		codespell was not available.
+0 :ok:	detsecrets	0m 0s		detect-secrets was not available.
+0 :ok:	markdownlint	0m 0s		markdownlint was not available.
+0 :ok:	buf	0m 1s		buf was not available.
+0 :ok:	buf	0m 1s		buf was not available.
+1 :green_heart:	@author	0m 0s		The patch does not contain any @author tags.
+1 :green_heart:	test4tests	0m 0s		The patch appears to include 5 new or modified test files.
			_ branch-3.3 Compile Tests _
+0 :ok:	mvndep	14m 11s		Maven dependency ordering for branch
+1 :green_heart:	mvninstall	34m 12s		branch-3.3 passed
+1 :green_heart:	compile	17m 48s		branch-3.3 passed
+1 :green_heart:	checkstyle	2m 46s		branch-3.3 passed
+1 :green_heart:	mvnsite	5m 22s		branch-3.3 passed
+1 :green_heart:	javadoc	4m 44s		branch-3.3 passed
-1 :x:	spotbugs	2m 37s	/branch-spotbugs-hadoop-hdfs-project_hadoop-hdfs-client-warnings.html	hadoop-hdfs-project/hadoop-hdfs-client in branch-3.3 has 2 extant spotbugs warnings.
+1 :green_heart:	shadedclient	34m 33s		branch has no errors when building and testing our client artifacts.
			_ Patch Compile Tests _
+0 :ok:	mvndep	0m 34s		Maven dependency ordering for patch
+1 :green_heart:	mvninstall	3m 40s		the patch passed
+1 :green_heart:	compile	17m 1s		the patch passed
+1 :green_heart:	cc	17m 1s		the patch passed
+1 :green_heart:	javac	17m 1s		the patch passed
+1 :green_heart:	blanks	0m 0s		The patch has no blanks issues.
-0 :warning:	checkstyle	3m 8s	/results-checkstyle-root.txt	root: The patch generated 1 new + 416 unchanged - 1 fixed = 417 total (was 417)
+1 :green_heart:	mvnsite	5m 21s		the patch passed
+1 :green_heart:	javadoc	4m 43s		the patch passed
+1 :green_heart:	spotbugs	10m 42s		the patch passed
+1 :green_heart:	shadedclient	34m 31s		patch has no errors when building and testing our client artifacts.
			_ Other Tests _
+1 :green_heart:	unit	18m 5s		hadoop-common in the patch passed.
+1 :green_heart:	unit	2m 34s		hadoop-hdfs-client in the patch passed.
-1 :x:	unit	202m 0s	/patch-unit-hadoop-hdfs-project_hadoop-hdfs.txt	hadoop-hdfs in the patch passed.
+1 :green_heart:	unit	16m 44s		hadoop-hdfs-rbf in the patch passed.
+1 :green_heart:	asflicense	1m 9s		The patch does not generate ASF License warnings.
		453m 8s

Reason	Tests
Failed junit tests	hadoop.hdfs.protocol.TestBlockListAsLongs
	hadoop.hdfs.server.datanode.TestLargeBlockReport
	hadoop.hdfs.server.namenode.ha.TestEditLogTailer
	hadoop.hdfs.server.balancer.TestBalancerWithHANameNodes
	hadoop.hdfs.TestFileCreation
	hadoop.hdfs.server.datanode.TestDataNodeErasureCodingMetrics
	hadoop.hdfs.server.sps.TestExternalStoragePolicySatisfier

Subsystem	Report/Notes
Docker	ClientAPI=1.47 ServerAPI=1.47 base: https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-7208/1/artifact/out/Dockerfile
GITHUB PR	https://github.com/apache/hadoop/pull/7208
Optional Tests	dupname asflicense mvnsite codespell detsecrets markdownlint compile javac javadoc mvninstall unit shadedclient spotbugs checkstyle cc buflint bufcompat
uname	Linux 632c2fd9f2c8 5.15.0-124-generic #134-Ubuntu SMP Fri Sep 27 20:20:17 UTC 2024 x86_64 x86_64 x86_64 GNU/Linux
Build tool	maven
Personality	dev-support/bin/hadoop.sh
git revision	branch-3.3 / 60bc95a2db7fd90bbb6f0ab81c540d44e96c4459
Default Java	Private Build-1.8.0_362-8u372-ga~us1-0ubuntu1~18.04-b09
Test Results	https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-7208/1/testReport/
Max. process+thread count	3777 (vs. ulimit of 5500)
modules	C: hadoop-common-project/hadoop-common hadoop-hdfs-project/hadoop-hdfs-client hadoop-hdfs-project/hadoop-hdfs hadoop-hdfs-project/hadoop-hdfs-rbf U: .
Console output	https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-7208/1/console
versions	git=2.17.1 maven=3.6.0 spotbugs=4.2.2
Powered by	Apache Yetus 0.14.0 https://yetus.apache.org

This message was automatically generated.

[hadoop-yetus]

:broken_heart: -1 overall

Vote	Subsystem	Runtime	Logfile	Comment
+0 :ok:	reexec	3m 46s		Docker mode activated.
			_ Prechecks _
+1 :green_heart:	dupname	0m 0s		No case conflicting files found.
+0 :ok:	codespell	0m 0s		codespell was not available.
+0 :ok:	detsecrets	0m 0s		detect-secrets was not available.
+0 :ok:	markdownlint	0m 0s		markdownlint was not available.
+0 :ok:	buf	0m 0s		buf was not available.
+0 :ok:	buf	0m 0s		buf was not available.
+1 :green_heart:	@author	0m 0s		The patch does not contain any @author tags.
+1 :green_heart:	test4tests	0m 0s		The patch appears to include 5 new or modified test files.
			_ branch-3.3 Compile Tests _
+0 :ok:	mvndep	14m 9s		Maven dependency ordering for branch
+1 :green_heart:	mvninstall	21m 35s		branch-3.3 passed
+1 :green_heart:	compile	10m 44s		branch-3.3 passed
+1 :green_heart:	checkstyle	1m 26s		branch-3.3 passed
+1 :green_heart:	mvnsite	3m 25s		branch-3.3 passed
+1 :green_heart:	javadoc	3m 10s		branch-3.3 passed
-1 :x:	spotbugs	1m 29s	/branch-spotbugs-hadoop-hdfs-project_hadoop-hdfs-client-warnings.html	hadoop-hdfs-project/hadoop-hdfs-client in branch-3.3 has 2 extant spotbugs warnings.
+1 :green_heart:	shadedclient	20m 27s		branch has no errors when building and testing our client artifacts.
			_ Patch Compile Tests _
+0 :ok:	mvndep	0m 20s		Maven dependency ordering for patch
+1 :green_heart:	mvninstall	2m 17s		the patch passed
+1 :green_heart:	compile	10m 30s		the patch passed
+1 :green_heart:	cc	10m 30s		the patch passed
+1 :green_heart:	javac	10m 30s		the patch passed
+1 :green_heart:	blanks	0m 0s		The patch has no blanks issues.
-0 :warning:	checkstyle	1m 25s	/results-checkstyle-root.txt	root: The patch generated 1 new + 416 unchanged - 1 fixed = 417 total (was 417)
+1 :green_heart:	mvnsite	3m 19s		the patch passed
+1 :green_heart:	javadoc	3m 0s		the patch passed
+1 :green_heart:	spotbugs	6m 14s		the patch passed
+1 :green_heart:	shadedclient	20m 27s		patch has no errors when building and testing our client artifacts.
			_ Other Tests _
+1 :green_heart:	unit	15m 10s		hadoop-common in the patch passed.
+1 :green_heart:	unit	1m 56s		hadoop-hdfs-client in the patch passed.
-1 :x:	unit	170m 41s	/patch-unit-hadoop-hdfs-project_hadoop-hdfs.txt	hadoop-hdfs in the patch passed.
+1 :green_heart:	unit	14m 35s		hadoop-hdfs-rbf in the patch passed.
+1 :green_heart:	asflicense	0m 46s		The patch does not generate ASF License warnings.
		337m 23s

Reason	Tests
Failed junit tests	hadoop.hdfs.protocol.TestBlockListAsLongs
	hadoop.hdfs.server.datanode.TestDataNodeRollingUpgrade
	hadoop.hdfs.tools.TestDFSAdmin
	hadoop.hdfs.server.balancer.TestBalancerWithHANameNodes
	hadoop.hdfs.server.datanode.fsdataset.impl.TestFsDatasetImpl
	hadoop.hdfs.TestReconstructStripedFileWithValidator
	hadoop.hdfs.server.datanode.TestLargeBlockReport

Subsystem	Report/Notes
Docker	ClientAPI=1.47 ServerAPI=1.47 base: https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-7208/2/artifact/out/Dockerfile
GITHUB PR	https://github.com/apache/hadoop/pull/7208
Optional Tests	dupname asflicense mvnsite codespell detsecrets markdownlint compile javac javadoc mvninstall unit shadedclient spotbugs checkstyle cc buflint bufcompat
uname	Linux be5a920b0d05 5.15.0-124-generic #134-Ubuntu SMP Fri Sep 27 20:20:17 UTC 2024 x86_64 x86_64 x86_64 GNU/Linux
Build tool	maven
Personality	dev-support/bin/hadoop.sh
git revision	branch-3.3 / b7a3139e78f420a4c79eae7fe189c1c0da55fee6
Default Java	Private Build-1.8.0_362-8u372-ga~us1-0ubuntu1~18.04-b09
Test Results	https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-7208/2/testReport/
Max. process+thread count	4714 (vs. ulimit of 5500)
modules	C: hadoop-common-project/hadoop-common hadoop-hdfs-project/hadoop-hdfs-client hadoop-hdfs-project/hadoop-hdfs hadoop-hdfs-project/hadoop-hdfs-rbf U: .
Console output	https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-7208/2/console
versions	git=2.17.1 maven=3.6.0 spotbugs=4.2.2
Powered by	Apache Yetus 0.14.0 https://yetus.apache.org

This message was automatically generated.

[hadoop-yetus]

:broken_heart: -1 overall

Vote	Subsystem	Runtime	Logfile	Comment
+0 :ok:	reexec	6m 3s		Docker mode activated.
			_ Prechecks _
+1 :green_heart:	dupname	0m 1s		No case conflicting files found.
+0 :ok:	codespell	0m 1s		codespell was not available.
+0 :ok:	detsecrets	0m 1s		detect-secrets was not available.
+0 :ok:	markdownlint	0m 1s		markdownlint was not available.
+0 :ok:	buf	0m 0s		buf was not available.
+0 :ok:	buf	0m 0s		buf was not available.
+1 :green_heart:	@author	0m 0s		The patch does not contain any @author tags.
+1 :green_heart:	test4tests	0m 0s		The patch appears to include 5 new or modified test files.
			_ branch-3.3 Compile Tests _
+0 :ok:	mvndep	15m 22s		Maven dependency ordering for branch
+1 :green_heart:	mvninstall	33m 58s		branch-3.3 passed
+1 :green_heart:	compile	17m 51s		branch-3.3 passed
+1 :green_heart:	checkstyle	2m 45s		branch-3.3 passed
+1 :green_heart:	mvnsite	5m 18s		branch-3.3 passed
+1 :green_heart:	javadoc	4m 41s		branch-3.3 passed
-1 :x:	spotbugs	2m 34s	/branch-spotbugs-hadoop-hdfs-project_hadoop-hdfs-client-warnings.html	hadoop-hdfs-project/hadoop-hdfs-client in branch-3.3 has 2 extant spotbugs warnings.
+1 :green_heart:	shadedclient	35m 20s		branch has no errors when building and testing our client artifacts.
			_ Patch Compile Tests _
+0 :ok:	mvndep	0m 33s		Maven dependency ordering for patch
+1 :green_heart:	mvninstall	3m 38s		the patch passed
+1 :green_heart:	compile	17m 9s		the patch passed
+1 :green_heart:	cc	17m 9s		the patch passed
+1 :green_heart:	javac	17m 9s		the patch passed
+1 :green_heart:	blanks	0m 0s		The patch has no blanks issues.
-0 :warning:	checkstyle	2m 40s	/results-checkstyle-root.txt	root: The patch generated 1 new + 416 unchanged - 1 fixed = 417 total (was 417)
+1 :green_heart:	mvnsite	5m 22s		the patch passed
+1 :green_heart:	javadoc	4m 42s		the patch passed
+1 :green_heart:	spotbugs	10m 37s		the patch passed
+1 :green_heart:	shadedclient	34m 11s		patch has no errors when building and testing our client artifacts.
			_ Other Tests _
+1 :green_heart:	unit	18m 3s		hadoop-common in the patch passed.
+1 :green_heart:	unit	2m 33s		hadoop-hdfs-client in the patch passed.
-1 :x:	unit	195m 56s	/patch-unit-hadoop-hdfs-project_hadoop-hdfs.txt	hadoop-hdfs in the patch passed.
+1 :green_heart:	unit	16m 50s		hadoop-hdfs-rbf in the patch passed.
+1 :green_heart:	asflicense	1m 13s		The patch does not generate ASF License warnings.
		447m 41s

Reason	Tests
Failed junit tests	hadoop.hdfs.protocol.TestBlockListAsLongs
	hadoop.hdfs.server.datanode.TestLargeBlockReport
	hadoop.hdfs.server.balancer.TestBalancerWithHANameNodes
	hadoop.hdfs.server.datanode.TestDataNodeRollingUpgrade

Subsystem	Report/Notes
Docker	ClientAPI=1.47 ServerAPI=1.47 base: https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-7208/3/artifact/out/Dockerfile
GITHUB PR	https://github.com/apache/hadoop/pull/7208
Optional Tests	dupname asflicense mvnsite codespell detsecrets markdownlint compile javac javadoc mvninstall unit shadedclient spotbugs checkstyle cc buflint bufcompat
uname	Linux 8efe64e74005 5.15.0-124-generic #134-Ubuntu SMP Fri Sep 27 20:20:17 UTC 2024 x86_64 x86_64 x86_64 GNU/Linux
Build tool	maven
Personality	dev-support/bin/hadoop.sh
git revision	branch-3.3 / e68c0ac5ef6d5b91d0178cc4c793895ed395a7cf
Default Java	Private Build-1.8.0_362-8u372-ga~us1-0ubuntu1~18.04-b09
Test Results	https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-7208/3/testReport/
Max. process+thread count	3810 (vs. ulimit of 5500)
modules	C: hadoop-common-project/hadoop-common hadoop-hdfs-project/hadoop-hdfs-client hadoop-hdfs-project/hadoop-hdfs hadoop-hdfs-project/hadoop-hdfs-rbf U: .
Console output	https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-7208/3/console
versions	git=2.17.1 maven=3.6.0 spotbugs=4.2.2
Powered by	Apache Yetus 0.14.0 https://yetus.apache.org

This message was automatically generated.

[ctrezzo]

@PrateekSane this seems to be a backwards incompatible change as we are adding a new required field to ClientNamenodeProtocol.proto. As such, I am not sure we can safely backport this to the 3.x line.

[hadoop-yetus]

:broken_heart: -1 overall

Vote	Subsystem	Runtime	Logfile	Comment
+0 :ok:	reexec	3m 49s		Docker mode activated.
			_ Prechecks _
+1 :green_heart:	dupname	0m 0s		No case conflicting files found.
+0 :ok:	codespell	0m 0s		codespell was not available.
+0 :ok:	detsecrets	0m 0s		detect-secrets was not available.
+0 :ok:	markdownlint	0m 0s		markdownlint was not available.
+0 :ok:	buf	0m 0s		buf was not available.
+0 :ok:	buf	0m 0s		buf was not available.
+1 :green_heart:	@author	0m 0s		The patch does not contain any @author tags.
+1 :green_heart:	test4tests	0m 0s		The patch appears to include 5 new or modified test files.
			_ branch-3.3 Compile Tests _
+0 :ok:	mvndep	13m 51s		Maven dependency ordering for branch
+1 :green_heart:	mvninstall	23m 26s		branch-3.3 passed
+1 :green_heart:	compile	9m 57s		branch-3.3 passed
+1 :green_heart:	checkstyle	1m 19s		branch-3.3 passed
+1 :green_heart:	mvnsite	2m 50s		branch-3.3 passed
+1 :green_heart:	javadoc	2m 44s		branch-3.3 passed
-1 :x:	spotbugs	1m 29s	/branch-spotbugs-hadoop-hdfs-project_hadoop-hdfs-client-warnings.html	hadoop-hdfs-project/hadoop-hdfs-client in branch-3.3 has 2 extant spotbugs warnings.
+1 :green_heart:	shadedclient	22m 33s		branch has no errors when building and testing our client artifacts.
			_ Patch Compile Tests _
+0 :ok:	mvndep	0m 23s		Maven dependency ordering for patch
+1 :green_heart:	mvninstall	2m 3s		the patch passed
+1 :green_heart:	compile	9m 48s		the patch passed
+1 :green_heart:	cc	9m 48s		the patch passed
+1 :green_heart:	javac	9m 48s		the patch passed
+1 :green_heart:	blanks	0m 0s		The patch has no blanks issues.
-0 :warning:	checkstyle	1m 17s	/results-checkstyle-root.txt	root: The patch generated 1 new + 416 unchanged - 1 fixed = 417 total (was 417)
+1 :green_heart:	mvnsite	2m 47s		the patch passed
+1 :green_heart:	javadoc	2m 37s		the patch passed
+1 :green_heart:	spotbugs	6m 27s		the patch passed
+1 :green_heart:	shadedclient	21m 35s		patch has no errors when building and testing our client artifacts.
			_ Other Tests _
+1 :green_heart:	unit	15m 22s		hadoop-common in the patch passed.
+1 :green_heart:	unit	1m 54s		hadoop-hdfs-client in the patch passed.
-1 :x:	unit	168m 22s	/patch-unit-hadoop-hdfs-project_hadoop-hdfs.txt	hadoop-hdfs in the patch passed.
+1 :green_heart:	unit	14m 32s		hadoop-hdfs-rbf in the patch passed.
+1 :green_heart:	asflicense	0m 44s		The patch does not generate ASF License warnings.
		335m 47s

Reason	Tests
Failed junit tests	hadoop.hdfs.protocol.TestBlockListAsLongs
	hadoop.hdfs.TestDFSInotifyEventInputStreamKerberized
	hadoop.hdfs.TestLeaseRecovery2
	hadoop.hdfs.server.datanode.TestLargeBlockReport

Subsystem	Report/Notes
Docker	ClientAPI=1.48 ServerAPI=1.48 base: https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-7208/4/artifact/out/Dockerfile
GITHUB PR	https://github.com/apache/hadoop/pull/7208
Optional Tests	dupname asflicense mvnsite codespell detsecrets markdownlint compile javac javadoc mvninstall unit shadedclient spotbugs checkstyle cc buflint bufcompat
uname	Linux 1fb98b81e191 5.15.0-136-generic #147-Ubuntu SMP Sat Mar 15 15:53:30 UTC 2025 x86_64 x86_64 x86_64 GNU/Linux
Build tool	maven
Personality	dev-support/bin/hadoop.sh
git revision	branch-3.3 / 079f31a2c24a2dc236492dfe5837c3fd6bc228d9
Default Java	Private Build-1.8.0_362-8u372-ga~us1-0ubuntu1~18.04-b09
Test Results	https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-7208/4/testReport/
Max. process+thread count	4777 (vs. ulimit of 5500)
modules	C: hadoop-common-project/hadoop-common hadoop-hdfs-project/hadoop-hdfs-client hadoop-hdfs-project/hadoop-hdfs hadoop-hdfs-project/hadoop-hdfs-rbf U: .
Console output	https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-7208/4/console
versions	git=2.17.1 maven=3.6.0 spotbugs=4.2.2
Powered by	Apache Yetus 0.14.0 https://yetus.apache.org

This message was automatically generated.

[github-actions[bot]]

We're closing this stale PR because it has been open for 100 days with no activity. This isn't a judgement on the merit of the PR in any way. It's just a way of keeping the PR queue manageable. If you feel like this was a mistake, or you would like to continue working on it, please feel free to re-open it and ask for a committer to remove the stale tag and review again. Thanks all for your contribution.