Skip to content

[improve][metadata] Add partition key resolver support for Oxia index scans#26148

Open
void-ptr974 wants to merge 1 commit into
apache:masterfrom
void-ptr974:improve/oxia-index-scan-partition-key
Open

[improve][metadata] Add partition key resolver support for Oxia index scans#26148
void-ptr974 wants to merge 1 commit into
apache:masterfrom
void-ptr974:improve/oxia-index-scan-partition-key

Conversation

@void-ptr974

@void-ptr974 void-ptr974 commented Jul 5, 2026

Copy link
Copy Markdown
Contributor

Motivation

Oxia records can be routed with PartitionKey. Native secondary-index scans return primary paths, and some callers can derive the routing key from each returned path. A single scan-level PartitionKey only covers scans scoped to one routing key; it does not cover scans that may span multiple routing keys.

For example, transaction op records are stored under /txn/op/<txnId>-<seq> with PartitionKey(txnId). A scan by segment can return op records for multiple transaction IDs, so each follow-up primary read can use the transaction ID derived from the returned path.

This PR adds an option for passing that per-result routing hint and uses it in transaction metadata index scans.

Modifications

  • Add Option.PartitionKeyResolver to derive a concrete PartitionKey from a primary path discovered during a scan.
  • Resolve a concrete PartitionKey before Oxia loads each primary record returned by scanByIndex.
  • Pass fixed or path-derived routing hints from transaction metadata index scans.
  • Add multi-shard Oxia coverage for partition-key-aware index scans.

Verifying this change

  • Make sure that the change passes the CI checks.

This change added tests and can be verified as follows:

  • Added a multi-shard Oxia test for partition-key-aware index scans.
  • ./gradlew :pulsar-metadata:compileTestJava :pulsar-broker:compileJava :pulsar-metadata:checkstyleMain :pulsar-metadata:checkstyleTest :pulsar-broker:checkstyleMain -x :managed-ledger:compileTestJava
  • ./gradlew :pulsar-metadata:test --tests org.apache.pulsar.metadata.OxiaPartitionKeyTest

Does this pull request potentially affect one of the following parts:

  • Dependencies (add or upgrade a dependency)
  • The public API
  • The schema
  • The default values of configurations
  • The threading model
  • The binary protocol
  • The REST endpoints
  • The admin CLI options
  • The metrics
  • Anything that affects deployment

This adds a new metadata-store Option subtype. It does not change existing option behavior, configuration defaults, wire protocol, or metadata format.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant