Implement a fingerprinting mechanism to track compaction states in a more efficient manner #18844

capistrant · 2025-12-15T20:16:25Z

Description

Compaction State Fingerprinting

Instead of storing CompactionState as the lastCompactionState field in every compaction segment, generate a fingerprint for a CompactionState and attach that to compacted segments. Add new centralized storage for CompactionState where individual states can be looked up by the aforementioned fingerprint. Since it is common for many segments in a data source to share a single CompactionState, this greatly reduces the metadata storage overhead for storing compaction states.

Metadata Store Changes

`druid_segments`

Add new column compaction_state_fingerprint that stores the fingerprint representation of the segments current compaction state. It can be null if no compaction has taken place.

`druid_compactionStates`

New metadata table that stores the full CompactionState associated with a fingerprint. Segments can look up their full compaction state here by using the compaction_state_fingerprint that they are associated with.

`CompactionStateManager`

The CompactionStateManager is responsible for managing the persistence and lifecycle of compaction states on the Coordinator. It stores unique compaction configurations (identified by fingerprints) in the metadata database and maintains a cache to optimize lookups. The manager tracks which compaction states are actively referenced by segments, marking unreferenced states as unused and periodically cleaning up old unused states. This fingerprinting approach allows Druid to efficiently store and retrieve compaction metadata without duplicating identical compaction configurations across multiple segments, while the cache layer minimizes database queries for frequently accessed compaction states.

`OnHeapCompactionStateManager`

Meant to serve as a mechanism for testing and simulations where metadata persistence may not be available/needed

Legacy `lastCompactionState` Roadmap

This PR implements no automatic transition to fingerprints for segments who are compacted and store CompactionState in their lastCompactionState field. Instead this PR aims to continue supporting lastCompactionState in Compaction decision making for segments compacted before fingerprinting. This means that legacy segments will not have to be re-compacted simply because they are not fingerprinted, as long as they have the proper CompactionState as specified by the compaction configuration for the data source in question.

This PR also continues to write both the new fingerprint as well as the legacy lastCompactionState by default. This allows normal rolling upgrade order as well as Druid version rollback without un-needed re-compaction. An operator can disable writing lastCompactionState by updating the cluster compaction config, after the Druid upgrade completes. Eventually, Druid code base will cease writing lastCompactionState at all and instead force using fingerprinting going forward. I think this should be done in the Druid version following the first version that this new feature is seen in. Even at this point, lastCompactionState will need to continue to be supported for already written segments, unless we want to devise an automated migration plan that can run in the background of a cluster to get all compacted segments migrated to fingerprinting.

Release note

coming soon

Upgrade Note

Metadata store changes are required for this upgrade. If you already have druid.metadata.storage.connector.createTables set to true no action is needed. If you have this feature disabled, you will need to alter the segments table and create the compactionStates table. Postgres DDL is provided below as a guide. You will have to adapt the syntax to your metadata store backend as well as use proper table naming depending on your configured table prefix and database.

-- create the compaction states lookup table and associated indices
CREATE TABLE druid_compactionStates (
    id BIGSERIAL NOT NULL,
    created_date VARCHAR(255) NOT NULL,
    datasource VARCHAR(255) NOT NULL,
    fingerprint VARCHAR(255) NOT NULL,
    payload BYTEA NOT NULL,
    used BOOLEAN NOT NULL,
    used_status_last_updated VARCHAR(255) NOT NULL,
    PRIMARY KEY (id),
    UNIQUE (fingerprint)
  );

  CREATE INDEX idx_druid_compactionStates_fingerprint ON druid_compactionStates(fingerprint);
  CREATE INDEX idx_druid_compactionStates_used ON druid_compactionStates(used, used_status_last_updated);

-- modify druid_segments table to have a column for storing compaction state fingerprints
ALTER TABLE druid_segments ADD COLUMN compaction_state_fingerprint VARCHAR(255);

Key changed/added classes in this PR

CompactionStatus
CompactionConfigBasedJobTemplate
CompactionState
SQLMetadataConnector
CompactionStateManager
CompactSegments
KillUnreferencedCompactionState

This PR has:

… storage configurable

processing/src/main/java/org/apache/druid/timeline/CompactionState.java

+    {
+      DefaultObjectMapper baseMapper = new DefaultObjectMapper();
+      baseMapper.configure(SerializationFeature.ORDER_MAP_ENTRIES_BY_KEYS, true);
+      baseMapper.configure(MapperFeature.SORT_PROPERTIES_ALPHABETICALLY, true);


processing/src/test/java/org/apache/druid/timeline/DataSegmentTest.java

...ervice/src/main/java/org/apache/druid/indexing/compact/CompactionConfigBasedJobTemplate.java

processing/src/main/java/org/apache/druid/timeline/CompactionState.java

server/src/main/java/org/apache/druid/server/coordinator/duty/CompactSegments.java

server/src/main/java/org/apache/druid/server/compaction/CompactionStatus.java

server/src/main/java/org/apache/druid/segment/metadata/CompactionStateManager.java

capistrant · 2025-12-15T22:28:29Z

server/src/main/java/org/apache/druid/segment/metadata/CompactionStateManager.java

+  @LifecycleStop
+  public void stop()
+  {
+    fingerprintCache.invalidateAll();


does this cache object need any other lifecycle cleanup?

server/src/main/java/org/apache/druid/segment/metadata/CompactionStateManager.java

capistrant · 2025-12-15T22:30:45Z

server/src/main/java/org/apache/druid/segment/metadata/CompactionStateManager.java

what about if the operator has create tables disabled and does not properly create the table before upgrading?

server/src/main/java/org/apache/druid/segment/metadata/CompactionStateManager.java

kfaraz

Thanks for the feature, @capistrant !

I have started going through the PR, leaving a partial review here.
I am yet to go through several changes, such as the ones made in CompactionStatus, DatasourceCompactibleSegmentIterator, etc.

kfaraz · 2025-12-16T09:01:14Z

server/src/main/java/org/apache/druid/segment/metadata/HeapMemoryCompactionStateManager.java

+ * <p>
+ * Useful for simulations and unit tests where database persistence is not needed.
+ */
+public class HeapMemoryCompactionStateManager extends CompactionStateManager


Might be cleaner to let CompactionStateManager be an interface, and let both the heap-based and the concrete class implement it.

kfaraz · 2025-12-16T09:01:49Z

server/src/main/java/org/apache/druid/segment/metadata/HeapMemoryCompactionStateManager.java

+ * In-memory implementation of {@link CompactionStateManager} that stores
+ * compaction state fingerprints in heap memory without requiring a database.
+ * <p>
+ * Useful for simulations and unit tests where database persistence is not needed.


If this is used only in tests, we should probably put it in the test source root src/test/java.

That is where I originally put it, but then I tried to use it in a simulation class which is in the app code, not test. Let me review this though, maybe I am mistaken on how it is all working with the simulations

Oh, I see. Are you referring to CoordinatorSimulationBuilder or some other class?

no CompactionRunSimulator, https://github.com/apache/druid/pull/18844/files#diff-b8a4fdf52e09ff26fa6f5610c021d196b9fa99673b83051de794ed07257be13b ... It creates CompactSegments instance, which as of now requires a CompactionStateManager. But I guess if we go the route of not supporting fingerprinting in the coordinator duty led compaction, this may not be a problem and it can be moved to the test space.

kfaraz · 2025-12-16T12:04:44Z

docs/configuration/index.md

+|`druid.manager.compactionState.cacheSize`|The maximum number of compaction state fingerprints to cache in memory on the coordinator and overlord. Compaction state fingerprints are used to track the compaction configuration applied to segments. Consider increasing this value if you have a large number of datasources with compaction configurations.|`100`|
+|`druid.manager.compactionState.prewarmSize`|The number of most recently used compaction state fingerprints to load into cache on Coordinator startup. This pre-warms the cache to improve performance immediately after startup.|`100`|


Both Coordinator and Overlord (with segment metadata caching enabled) already keep all used segments in memory, including the respective (interned) CompactionState objects as well.
I don't think the number of distinct CompactState objects that we keep in memory will increase after this patch.

Do we still need to worry about the cache size of these objects?
Does a cache miss trigger a fetch from metadata store?

kfaraz · 2025-12-16T12:11:34Z

processing/src/main/java/org/apache/druid/timeline/CompactionState.java

 {
+
+  /**
+   * Lazy initialization holder for deterministic ObjectMapper.


I wonder if we shouldn't just inject this mapper annotated with @Sorted or @Deterministic as a lazy singleton. It may be injected into CompactionStateManager and fingerprints will always be created by that class rather than using a static utility method.

processing/src/main/java/org/apache/druid/timeline/DataSegment.java

kfaraz · 2025-12-16T13:51:14Z

...ervice/src/main/java/org/apache/druid/indexing/compact/CompactionConfigBasedJobTemplate.java

+    if (segmentIterator.hasNext()) {
+      // If we are going to create compaction jobs for this compaction state, we need to persist the fingerprint -> state
+      // mapping so compacted segments from these jobs can reference a valid compaction state.
+      params.getCompactionStateManager().persistCompactionState(


The templates should only perform lightweight (i.e. non-IO) read-only operations as createCompactionJobs may be called frequently.
We should not do any persistence here.
Instead, the params can hold some mapping where we can add this compaction state and perform persistence on-demand (perhaps in the CompactionJobQueue).

thank you for the guidance. Will work on how to get this out of hot path

kfaraz · 2025-12-16T14:27:01Z

multi-stage-query/src/main/java/org/apache/druid/msq/exec/ControllerImpl.java

    }
  }

+  private static Function<Set<DataSegment>, Set<DataSegment>> addCompactionStateFingerprintToSegments(String compactionStateFingerprint)


Let's re-use the static function from AbstractTask itself?

sure! I didn't know if it was bad form to reach into that class from MSQ. But I like having just one impl

I think it is fine to use AbstractTask in the MSQ code. Alternatively, you can put the method in IndexTaskUtils too.

kfaraz · 2025-12-16T14:27:28Z

multi-stage-query/src/main/java/org/apache/druid/msq/exec/ControllerImpl.java

              Tasks.DEFAULT_STORE_COMPACTION_STATE
          );

+      String compactionStateFingerprint = querySpec.getContext()


Suggested change

String compactionStateFingerprint = querySpec.getContext()

final String compactionStateFingerprint = querySpec.getContext()

kfaraz · 2025-12-16T14:29:42Z

website/.spelling

 pre-compute
 pre-computed
 pre-computing
+pre-dates


predates need not be hyphenated.

sometimes my inability to spell, compounded by my inability to google how to spell, is embarrassing. this is one of those times. will fix

kfaraz · 2025-12-16T14:38:43Z

server/src/main/java/org/apache/druid/segment/metadata/CompactionStateManager.java

+ * </p>
+ */
+@ManageLifecycle
+public class CompactionStateManager


I don't feel that pre-warming the cache is really necessary. The fingerprint needs to be retrieved only when running the CompactionJobQueue on Overlord or CompactSegments on Coordinator.

Let's always keep all the compaction states in memory. We are already keeping all the used segments in memory. The distinct CompactionState objects will be fairly small in number and size.

The states can be cached in HeapMemorySegmentMetadataCache which already serves as a cache for used segments, pending segments and schemas.

If possible, let's support the fingerprint flow only with compaction supervisors and not the Coordinator-based CompactSegments duty. That would simplify the new flow and be another motivation for users to migrate to using compaction supervisors.

If possible, let's support the fingerprint flow only with compaction supervisors and not the Coordinator-based CompactSegments duty. That would simplify the new flow and be another motivation for users to migrate to using compaction supervisors.

would we want to deprecate CompactSegments compaction on the coordinator in this case? so we aren't forever supporting compaction without fingerprints + compaction with fingerprints?

Yes, the plan was to deprecate CompactSegments once compaction supervisors took off. I don't fully recall if compaction supervisors is already marked GA or not. They would also have to be made the default, if we want to start deprecation of CompactSegments.

But I feel all of this should be out of scope for the current PR.

If supporting the fingerprint logic in CompactSegments is not additional work and does not complicate the flow, we can leave it as is.

My only concern is that there should be just one service that is responsible for persisting new fingerprints. I would prefer that to be the Overlord, so that it always has a consistent cache state. So we either just don't support fingerprints on the Coordinator or we handle persistence by calling an Overlord API.

(I am yet to go through the whole PR to identify all the call sites that may persist a compaction state. I have only found the one in CompactionConfigBasedJobTemplate so far.)

capistrant added 14 commits December 15, 2025 13:50

meatadata store bits part 1

d8490b0

annotate segments with compaction fingerprint before persist

3d2d423

Add ability to generate compaction state fingerprint

48854f4

add fingerprint to task context and make legacy last compaction state…

c6a3367

… storage configurable

update embedded tests for compaction supervisors to flex fingerprints

f3b706e

checkpoint with persisting compaction states

46fb807

add duty to clean up unused compaction states

0fef358

take fingerprints into account in CompactionStatus

edeaf30

Add and improve tests

97daf3f

get rid of some todo comments

dbcdfcf

fix checkstyle

38f6d15

cleanup some more TODO

4cf1197

Add some docs

ba269bd

update web console

f168bc9

github-actions bot added Area - Documentation Area - Batch Ingestion Area - Web Console Area - Ingestion Area - MSQ For multi stage queries - https://github.com/apache/druid/issues/12262 labels Dec 15, 2025

capistrant added the Area - Compaction label Dec 15, 2025

github-advanced-security bot found potential problems Dec 15, 2025

View reviewed changes

capistrant added 3 commits December 15, 2025 15:19

make cache size configurable and fix some spelling

2292b15

fixup use of deprecated builder

74c8ebc

fix checktyle

adac5ec

capistrant commented Dec 15, 2025

View reviewed changes

fix coordinator compactsegments duty and respond to self review comments

4fb3a9c

kfaraz reviewed Dec 16, 2025

View reviewed changes

		\|`druid.manager.compactionState.cacheSize`\|The maximum number of compaction state fingerprints to cache in memory on the coordinator and overlord. Compaction state fingerprints are used to track the compaction configuration applied to segments. Consider increasing this value if you have a large number of datasources with compaction configurations.\|`100`\|
		\|`druid.manager.compactionState.prewarmSize`\|The number of most recently used compaction state fingerprints to load into cache on Coordinator startup. This pre-warms the cache to improve performance immediately after startup.\|`100`\|

	String compactionStateFingerprint = querySpec.getContext()
	final String compactionStateFingerprint = querySpec.getContext()

Implement a fingerprinting mechanism to track compaction states in a more efficient manner #18844

Are you sure you want to change the base?

Implement a fingerprinting mechanism to track compaction states in a more efficient manner #18844

Uh oh!

Conversation

capistrant commented Dec 15, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Description

Compaction State Fingerprinting

Metadata Store Changes

druid_segments

druid_compactionStates

CompactionStateManager

OnHeapCompactionStateManager

Legacy lastCompactionState Roadmap

Release note

Upgrade Note

Key changed/added classes in this PR

Uh oh!

Check notice

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

kfaraz left a comment

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Reviewers

Assignees

Labels

Projects

capistrant commented Dec 15, 2025 •

edited

Loading

`druid_segments`

`druid_compactionStates`

`CompactionStateManager`

`OnHeapCompactionStateManager`

Legacy `lastCompactionState` Roadmap