Skip to content

fix: Fix bug in reindexCascade timeline generation where too many search intervals are skipped#19389

Merged
gianm merged 1 commit intoapache:masterfrom
capistrant:cascading-reindexing-skipOffsets-bugfix
Apr 29, 2026
Merged

fix: Fix bug in reindexCascade timeline generation where too many search intervals are skipped#19389
gianm merged 1 commit intoapache:masterfrom
capistrant:cascading-reindexing-skipOffsets-bugfix

Conversation

@capistrant
Copy link
Copy Markdown
Contributor

@capistrant capistrant commented Apr 28, 2026

Description

The original implementation is not incorrect in its application of skip offsets, but it is overly conservative. The way the original code works is that it:

  1. generates granularity aligned search intervals for all existing rules from the rule provider
  2. iterates these search intervals
    a. If the interval end falls after the calculated skip offset threshold (if skipOffsetFromNow or skipOffsetFromLatest is set), do not create jobs for the search interval

2a. is overly conservative. It is completely valid to take such an interval and truncate its end to granularity.bucketStart(skipOffsetThreshold) where granularity is the segment granularity of the search interval, and skipOffsetThreshold is the calculated threshold that we must skip creating compaction jobs for any data after. If the interval created by the original start and new end is valid, we can create compaction jobs for it.

Release note

Be more precise in applying skipOffsetFromNow and skipOffsetFromLatest when using reindexCascade compaction supervisor. Previously more search intervals than necessary could be skipped when generating compaction jobs if you were using one of the two skip offset styles.


Key changed/added classes in this PR
  • CascadingReindexingTemplate

This PR has:

  • been self-reviewed.
  • added documentation for new or modified features or behaviors.
  • a release note entry in the PR description.
  • added Javadocs for most classes and all non-trivial methods. Linked related entities via Javadoc links.
  • added or updated version, license, or notice information in licenses.yaml
  • added comments explaining the "why" and the intent of the code wherever would not be obvious for an unfamiliar reader.
  • added unit tests or modified existing tests to cover new code paths, ensuring the threshold for code coverage is met.
  • added integration tests.
  • been tested in a test Druid cluster.

Copy link
Copy Markdown
Member

@FrankChen021 FrankChen021 left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I have reviewed the code for correctness, edge cases, concurrency, and integration risks; no issues found.


This is an automated review by Codex GPT-5

@gianm gianm merged commit 516e3af into apache:master Apr 29, 2026
89 of 94 checks passed
@github-actions github-actions Bot added this to the 38.0.0 milestone Apr 29, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants