HDDS-13884. Populate block file metadata with helper fields
What changes were proposed in this pull request?
Saving the following fields as block file metadata can help with debugging and improve the visibility into the system.
- volume/bucket/key name
- parentId/objectId
- Original key creation timestamp
Storing such information can help answer questions about any arbitrary block file such as
- For which key was this block file created?
- Regardless of replication/ EC reconstruction/OS filesystem timestamps , when was this block file originally created for the very first time?
Today some of these questions can be answered only if the chain of metadata from OM to Datanodes is intact. Others can't be answered at all.
This PR:
- Saves metadata to block files for both Ratis and EC replicated files.
- The IO clients supply the extra metadata that cannot be retrieved on the datanode side easily.
- The datanode saves the information to the block file metadata table in RocksDB
What is the link to the Apache JIRA
https://issues.apache.org/jira/browse/HDDS-13884
How was this patch tested?
New tests added
CI: https://github.com/ptlrs/ozone/actions/runs/19176103214
rpatel@Mac DS-db9b617d-a5c6-4c90-b1f8-481710227e98 % ~/Github/ozone/hadoop-ozone/dist/target/ozone-2.2.0-SNAPSHOT/bin/ozone debug ldb --db container.db scan --cf block_data
{ "1|115816896921600001": {
"blockID" : {
"containerBlockID" : {
"containerID" : 1,
"localID" : 115816896921600001
},
"blockCommitSequenceId" : 2
},
"metadata" : {
"BUCKET_NAME" : "obs-27a9b2b4-edda-428f-b9f7-828444858936",
"CREATION_TIME" : "2025-11-08T04:51:59.172Z",
"KEY_NAME" : "ca3490e7-04e7-44d9-8a92-096b2e1ee948",
"OBJECT_ID" : "-9223372036854775040",
"PARENT_OBJECT_ID" : "0",
"TYPE" : "KEY",
"VOLUME_NAME" : "b04c6e09-5128-4204-8da6-ca5c9072afe7"
},
"chunkList" : {
"memoizedHashCode" : 0,
"memoizedSize" : -1,
"unknownFields" : {
"fields" : { }
},
"bitField0_" : 15,
"chunkName_" : {
"hash" : 0,
"bytes" : "MTE1ODE2ODk2OTIxNjAwMDAxX2NodW5rXzE="
},
"offset_" : 0,
"len_" : 45,
"metadata_" : [ ],
"checksumData_" : {
"memoizedHashCode" : 0,
"memoizedSize" : -1,
"unknownFields" : {
"fields" : { }
},
"bitField0_" : 3,
"type_" : 2,
"bytesPerChecksum_" : 16384,
"checksums_" : [ {
"hash" : 0,
"bytes" : "5X5F/Q=="
} ],
"memoizedIsInitialized" : 1
},
"stripeChecksum_" : {
"hash" : 0,
"bytes" : ""
},
"memoizedIsInitialized" : 1
},
"size" : 45
}
}
rpatel@Mac DS-db9b617d-a5c6-4c90-b1f8-481710227e98 % pwd
/Users/rpatel/Github/ozone/hadoop-ozone/integration-test/target/test-dir/MiniOzoneClusterImpl-7acd6bfe-c321-40a4-a886-885c46a0b278/ozone-metadata/datanode-7/data-0/hdds/7acd6bfe-c321-40a4-a886-885c46a0b278/DS-db9b617d-a5c6-4c90-b1f8-481710227e98
This PR has been marked as stale due to 21 days of inactivity. Please comment or remove the stale label to keep it open. Otherwise, it will be automatically closed in 7 days.
Hi @swamirishi @errose28, can you take a look at this PR?
This PR has been marked as stale due to 21 days of inactivity. Please comment or remove the stale label to keep it open. Otherwise, it will be automatically closed in 7 days.