ozone icon indicating copy to clipboard operation
ozone copied to clipboard

HDDS-13884. Populate block file metadata with helper fields

Open ptlrs opened this issue 2 months ago • 2 comments

What changes were proposed in this pull request?

Saving the following fields as block file metadata can help with debugging and improve the visibility into the system.

  • volume/bucket/key name
  • parentId/objectId
  • Original key creation timestamp

Storing such information can help answer questions about any arbitrary block file such as

  • For which key was this block file created?
  • Regardless of replication/ EC reconstruction/OS filesystem timestamps , when was this block file originally created for the very first time?

Today some of these questions can be answered only if the chain of metadata from OM to Datanodes is intact. Others can't be answered at all.

This PR:

  1. Saves metadata to block files for both Ratis and EC replicated files.
  2. The IO clients supply the extra metadata that cannot be retrieved on the datanode side easily.
  3. The datanode saves the information to the block file metadata table in RocksDB

What is the link to the Apache JIRA

https://issues.apache.org/jira/browse/HDDS-13884

How was this patch tested?

New tests added

CI: https://github.com/ptlrs/ozone/actions/runs/19176103214

rpatel@Mac DS-db9b617d-a5c6-4c90-b1f8-481710227e98 % ~/Github/ozone/hadoop-ozone/dist/target/ozone-2.2.0-SNAPSHOT/bin/ozone debug ldb --db container.db scan --cf block_data
{ "1|115816896921600001": {
  "blockID" : {
    "containerBlockID" : {
      "containerID" : 1,
      "localID" : 115816896921600001
    },
    "blockCommitSequenceId" : 2
  },
  "metadata" : {
    "BUCKET_NAME" : "obs-27a9b2b4-edda-428f-b9f7-828444858936",
    "CREATION_TIME" : "2025-11-08T04:51:59.172Z",
    "KEY_NAME" : "ca3490e7-04e7-44d9-8a92-096b2e1ee948",
    "OBJECT_ID" : "-9223372036854775040",
    "PARENT_OBJECT_ID" : "0",
    "TYPE" : "KEY",
    "VOLUME_NAME" : "b04c6e09-5128-4204-8da6-ca5c9072afe7"
  },
  "chunkList" : {
    "memoizedHashCode" : 0,
    "memoizedSize" : -1,
    "unknownFields" : {
      "fields" : { }
    },
    "bitField0_" : 15,
    "chunkName_" : {
      "hash" : 0,
      "bytes" : "MTE1ODE2ODk2OTIxNjAwMDAxX2NodW5rXzE="
    },
    "offset_" : 0,
    "len_" : 45,
    "metadata_" : [ ],
    "checksumData_" : {
      "memoizedHashCode" : 0,
      "memoizedSize" : -1,
      "unknownFields" : {
        "fields" : { }
      },
      "bitField0_" : 3,
      "type_" : 2,
      "bytesPerChecksum_" : 16384,
      "checksums_" : [ {
        "hash" : 0,
        "bytes" : "5X5F/Q=="
      } ],
      "memoizedIsInitialized" : 1
    },
    "stripeChecksum_" : {
      "hash" : 0,
      "bytes" : ""
    },
    "memoizedIsInitialized" : 1
  },
  "size" : 45
}
 }
rpatel@Mac DS-db9b617d-a5c6-4c90-b1f8-481710227e98 % pwd
/Users/rpatel/Github/ozone/hadoop-ozone/integration-test/target/test-dir/MiniOzoneClusterImpl-7acd6bfe-c321-40a4-a886-885c46a0b278/ozone-metadata/datanode-7/data-0/hdds/7acd6bfe-c321-40a4-a886-885c46a0b278/DS-db9b617d-a5c6-4c90-b1f8-481710227e98

ptlrs avatar Nov 07 '25 17:11 ptlrs

This PR has been marked as stale due to 21 days of inactivity. Please comment or remove the stale label to keep it open. Otherwise, it will be automatically closed in 7 days.

github-actions[bot] avatar Dec 02 '25 00:12 github-actions[bot]

Hi @swamirishi @errose28, can you take a look at this PR?

ptlrs avatar Dec 03 '25 18:12 ptlrs

This PR has been marked as stale due to 21 days of inactivity. Please comment or remove the stale label to keep it open. Otherwise, it will be automatically closed in 7 days.

github-actions[bot] avatar Dec 25 '25 00:12 github-actions[bot]