Skip to content

apache/master->master: b210c99149c6378e585fb76af51b7a6798fc9c6b#1294

Open
kgyrtkirk wants to merge 18 commits into
masterfrom
sync/apache-b210c99149c6378e585fb76af51b7a6798fc9c6b
Open

apache/master->master: b210c99149c6378e585fb76af51b7a6798fc9c6b#1294
kgyrtkirk wants to merge 18 commits into
masterfrom
sync/apache-b210c99149c6378e585fb76af51b7a6798fc9c6b

Conversation

@kgyrtkirk

Copy link
Copy Markdown
Owner

No description provided.

capistrant and others added 18 commits May 22, 2026 10:13
* fix unit tests, bump actions-timeline

CI is failing to startup to run unit tests, complaining about actions-timeline version not being allowed, switched to latest per https://github.com/apache/infrastructure-actions/blob/main/actions.yml

* fix S3InputSourceTest
changes:
* add `PartialSegmentMetadataCacheEntry` a `CacheEntry` that range-reads the V10 header on mount, constructs `PartialSegmentFileMapperV10`, and shrinks its reservation to actual on-disk size
* add `PartialSegmentBundleCacheEntry` and `PartialSegmentBundleCacheEntryIdentifier` are `CacheEntry` associated with each file bundle of a v10 segment that sparse-allocates and evicts its containers as a unit; places holds metadata and transitive parent bundle entries holds via the `StorageLocation` methods (weak reference holds on the parent cache entries) and reference-counted usage references
* add `PartialSegmentCacheBootstrap` a helper that restores partial-format entries from on-disk layout on historical startup (not wired up yet); cleans orphaned bundles
* add `ResizableCacheEntry` interface and `StorageLocation.adjustReservation` (shrink-only) so the metadata entry can tighten its reservation post-mount
* rename `SegmentFileBuilder.startFileGroup` → `startFileBundle`; introduce `ROOT_BUNDLE_NAME` as the default bundle for containers written without an explicit declaration                                                              * rename json field `SegmentFileContainerMetadata.fileGroup` → `bundle`; now non-null via getter, normalizes to `ROOT_BUNDLE_NAME` in the constructor, default value omitted from JSON using a custom `JsonInclude` filter
* Extract shared `DirectoryBackedRangeReader` and `CountingRangeReader` test helpers; consolidate duplicates across processing + server tests
…#19491) (apache#19497)

OrcInputFormat.initialize() — which swaps Thread.currentThread().setContextClassLoader() and calls FileSystem.get(conf) — was invoked on every createReader() call. When a ParallelIndexTask runs multiple ORC subtasks concurrently in the same JVM (as in embedded tests)
* Add default value for thread enabling

* Peon disable thread renaming

* Add benchmark query types

* Add groupby benchmark

* Specify query type

* Docs for thread
* Bump jackson to 2.21.3

Jackson 2.21 (issue apache#1381) changed the default resolution of
@JacksonInject when combined with @JsonProperty on the same parameter:
the injected value now wins over the JSON value, where 2.20 treated
the inject as a fallback used only when JSON did not supply one.

DruidNode's serviceName, port, and tlsPort parameters carry both
annotations, with JSON expected to win when supplied — this is how
DruidNode JSON config files have always worked. Add the explicit
useInput = OptBoolean.TRUE to restore that contract.

A repo-wide audit confirmed DruidNode's three parameters are the only
sites in Druid where @JacksonInject and @JsonProperty annotate the
same parameter; everywhere else the annotations are on distinct
parameters and are unaffected.

Also adds the previously-missing license entry for org.jspecify:jspecify
1.0.0 in extensions-core/kubernetes-extensions, which the
check-licenses dependency report flagged.

* Preserve @JacksonInject metadata in GuiceAnnotationIntrospector

findInjectableValue was returning JacksonInject.Value.forId(id), which
strips useInput and optional from the original annotation. Production
deserialization happens to remain correct under jackson 2.21 because
AnnotationIntrospectorPair.findInjectableValue falls back to the
secondary (default Jackson) introspector and merges the recovered
useInput onto the primary's Value via withUseInput.

That fallback is undocumented as part of the introspector contract and
would silently regress if the pair semantics change, or if this
introspector were ever installed standalone for a special-purpose
mapper. Construct the Value via JacksonInject.Value.from(annotation)
.withId(id) so the introspector returns a complete Value on its own
and no longer relies on the pair to fix it up.

The annotation lookup is hoisted to the top of findInjectableValue so
the non-null contract between it and findGuiceInjectId is explicit —
findGuiceInjectId now documents the precondition and trusts the caller
to verify, eliminating the duplicate getAnnotation call.

Defensive cleanup motivated by FasterXML/jackson-databind#1381; no
observable behavior change.
…#19477)

* resetOffsetsAndBackfill using bounded stream supervisor

* Reject non-positive backfillTaskCount

* Reset supervisor after backfill Supervisor has already been started

* Add helper method specHasConcurrentLocks

* Fix doc reference

* Move validations into helper function

* Add embedded-test for resetSupervisorAndBackfill

* Remove flaky waitUntilPublishedRecordsAreIngested

* Update KafkaBoundedSupervisorTest.java

* Wait for supervisor to be RUNNING

* Use checkpointed offset if > requested reset offset to prevent duplicate ingestion

* Update KafkaBoundedSupervisorTest.java

* Revert "Use checkpointed offset if > requested reset offset to prevent duplicate ingestion"

resetOffsetsForwardOnly does not fully close the race it targets (the write is
still unconditional) and the duplicate scenario it addresses is narrower than
the overlap case, which cannot be solved without suspending the main supervisor.
Accepting the limitation and documenting it is preferable to the added complexity.

This reverts commit 89b5fec.

* Doc update - duplication notice and Kinesis callout

* Rename endpoint from resetOffsetsAndBackfill to resetToLatestAndBackfill

* Update test name to reflect new endpoint

* Address clean up from review comments

* Log out start/end offsets

* Add abstract createBackfillSpec

* Unit test createBackfillSpec

* Fix deprecation notices

* Rename functions to align with new endpoint name

* Add null check and rename for consistency
Caffeine 3 raised the Java baseline to 11, tightened the AsyncCache
surface, and replaced size-LRU eviction with W-TinyLFU with explicit
admission control. The Caffeine APIs Druid uses (Cache, Caffeine
builder, Weigher, CacheStats) are stable across the transition.

Errorprone 2.49.0 is required because caffeine 3.2.4 pulls
error_prone_annotations 2.49.0 transitively, which violates the
requireUpperBoundDeps enforcer rule without the bump.

CaffeineCacheTest.testSizeEviction is rewritten for W-TinyLFU: the old
test pre-read key1 multiple times before putting key2, biasing the
admission policy to keep key1 and reject val2, so the assertion that
key1 was evicted no longer holds. The rewrite avoids the pre-reads
and asserts only that eviction happened and the cache stayed under
bound, mirroring caffeine's own EvictionTest patterns.

Also adds the previously-missing license entry for org.jspecify:jspecify
1.0.0 in extensions-core/kubernetes-extensions, which the
check-licenses dependency report flags. This was missing pre-bump and
is unrelated to caffeine/errorprone, but the CI license check fails
without it, so it is included here to keep the PR green.
* Web console support for resetToLatestAndBackfill

* Make pretty

* Update supervisor-reset-to-latest-dialog.tsx
…with disk utilization (apache#19422)

The existing linear penalization factor is still ineffective in large skew scenarios where the CostBalancerStrategy's cost forces a move/load (even with the utilization-based penalty). This switches the penalty to scale exponentially with the disk utilization, ensuring that near-full historicals are penalized. This is also particularly helpful when the size of segments on the cluster vary wildly.

This also marks the diskNormalized strategy as ready for production use.
…ocessors (apache#19536)

changes:
* `AWSClientConfig` now defaults `maxConnections` to scale with available processors `(max(50, 4 * cores))` to be in sync with virtual storage mode historical download thread pool size
* tests with artificial `RuntimeInfo` to cover the config scaling
@github-actions github-actions Bot added the GHA label Jun 2, 2026
@kgyrtkirk kgyrtkirk enabled auto-merge June 2, 2026 09:09
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Projects

None yet

Development

Successfully merging this pull request may close these issues.

8 participants