Implement a fingerprinting mechanism to track compaction states in a more efficient manner

[capistrant]

Description

Compaction State Fingerprinting

Instead of storing CompactionState as the lastCompactionState field in every compaction segment, generate a fingerprint for a CompactionState and attach that to compacted segments. Add new centralized storage for CompactionState where individual states can be looked up by the aforementioned fingerprint. Since it is common for many segments in a data source to share a single CompactionState, this greatly reduces the metadata storage overhead for storing compaction states.

Metadata Store Changes

druid_segments

Add new column compaction_state_fingerprint that stores the fingerprint representation of the segments current compaction state. It can be null if no compaction has taken place.

druid_compactionStates

New metadata table that stores the full CompactionState associated with a fingerprint. Segments can look up their full compaction state here by using the compaction_state_fingerprint that they are associated with.

CompactionStateManager

The CompactionStateManager is responsible for managing the persistence and lifecycle of compaction states on the Coordinator. It stores unique compaction configurations (identified by fingerprints) in the metadata database and maintains a cache to optimize lookups. The manager tracks which compaction states are actively referenced by segments, marking unreferenced states as unused and periodically cleaning up old unused states. This fingerprinting approach allows Druid to efficiently store and retrieve compaction metadata without duplicating identical compaction configurations across multiple segments, while the cache layer minimizes database queries for frequently accessed compaction states.

OnHeapCompactionStateManager

Meant to serve as a mechanism for testing and simulations where metadata persistence may not be available/needed

Legacy lastCompactionState Roadmap

This PR implements no automatic transition to fingerprints for segments who are compacted and store CompactionState in their lastCompactionState field. Instead this PR aims to continue supporting lastCompactionState in Compaction decision making for segments compacted before fingerprinting. This means that legacy segments will not have to be re-compacted simply because they are not fingerprinted, as long as they have the proper CompactionState as specified by the compaction configuration for the data source in question.

This PR also continues to write both the new fingerprint as well as the legacy lastCompactionState by default. This allows normal rolling upgrade order as well as Druid version rollback without un-needed re-compaction. An operator can disable writing lastCompactionState by updating the cluster compaction config, after the Druid upgrade completes. Eventually, Druid code base will cease writing lastCompactionState at all and instead force using fingerprinting going forward. I think this should be done in the Druid version following the first version that this new feature is seen in. Even at this point, lastCompactionState will need to continue to be supported for already written segments, unless we want to devise an automated migration plan that can run in the background of a cluster to get all compacted segments migrated to fingerprinting.

Release note

coming soon

Upgrade Note

Metadata store changes are required for this upgrade. If you already have druid.metadata.storage.connector.createTables set to true no action is needed. If you have this feature disabled, you will need to alter the segments table and create the compactionStates table. Postgres DDL is provided below as a guide. You will have to adapt the syntax to your metadata store backend as well as use proper table naming depending on your configured table prefix and database.

-- create the compaction states lookup table and associated indices
CREATE TABLE druid_compactionStates (
    id BIGSERIAL NOT NULL,
    created_date VARCHAR(255) NOT NULL,
    datasource VARCHAR(255) NOT NULL,
    fingerprint VARCHAR(255) NOT NULL,
    payload BYTEA NOT NULL,
    used BOOLEAN NOT NULL,
    used_status_last_updated VARCHAR(255) NOT NULL,
    PRIMARY KEY (id),
    UNIQUE (fingerprint)
  );

  CREATE INDEX idx_druid_compactionStates_fingerprint ON druid_compactionStates(fingerprint);
  CREATE INDEX idx_druid_compactionStates_used ON druid_compactionStates(used, used_status_last_updated);

-- modify druid_segments table to have a column for storing compaction state fingerprints
ALTER TABLE druid_segments ADD COLUMN compaction_state_fingerprint VARCHAR(255);

---

Key changed/added classes in this PR

• CompactionStatus
• CompactionConfigBasedJobTemplate
• CompactionState
• SQLMetadataConnector
• CompactionStateManager
• CompactSegments
• KillUnreferencedCompactionState

---

This PR has:

• [x] been self-reviewed.
  • [ ] using the concurrency checklist (Remove this item if the PR doesn't have any relation to concurrency.)
• [x] added documentation for new or modified features or behaviors.
• [ ] a release note entry in the PR description.
• [ ] added Javadocs for most classes and all non-trivial methods. Linked related entities via Javadoc links.
• [x] added or updated version, license, or notice information in licenses.yaml
• [ ] added comments explaining the "why" and the intent of the code wherever would not be obvious for an unfamiliar reader.
• [x] added unit tests or modified existing tests to cover new code paths, ensuring the threshold for code coverage is met.
• [x] added integration tests.
• [x] been tested in a test Druid cluster.