Skip to content

Bind Object Store reads to the backing stream by name for mirrored buckets#976

Open
ggiesen wants to merge 2 commits into
nats-io:mainfrom
ggiesen:fix-505-objectstore-mirrored-streams
Open

Bind Object Store reads to the backing stream by name for mirrored buckets#976
ggiesen wants to merge 2 commits into
nats-io:mainfrom
ggiesen:fix-505-objectstore-mirrored-streams

Conversation

@ggiesen

@ggiesen ggiesen commented Jun 12, 2026

Copy link
Copy Markdown

Bind Object Store reads to the backing stream by name for mirrored buckets

Fixes #505

Problem

ObjectStore.get() and ObjectStore.watch() (and therefore list(), which
consumes watch()) located the backing stream by subject lookup
(JetStreamManager.find_stream_name_by_subject). A mirrored bucket's stream
binds no subjects, so both paths raised NotFoundError when reading a
bucket whose backing stream is a mirror - e.g. an Object Store mirrored from
another JetStream domain over a leafnode link, the topology #505 describes.

The bucket name already determines the stream name (OBJ_<bucket>), and the
ObjectStore object already carries it (self._stream) - the lookup was not
just fragile against mirrors, it was unnecessary.

The Key-Value half of #505 was fixed by #807 (included since v2.15.0); the
regression tests added here cover both KV and Object Store so the whole issue
stays fixed.

Fix

Pass stream=self._stream to the two subscriptions that previously resolved
by subject:

  • ObjectStore.get() - the chunk-read ordered consumer
  • ObjectStore.watch() - the meta watcher (also serves list())

This matches the Go reference client, which binds with
nats.OrderedConsumer(), nats.BindStream(streamName) in both places
(jetstream/object.go - Get and Watch; same pattern in the legacy
object.go). Behavior is specified by ADR-20 (JetStream-based Object Stores);
the KV sibling is ADR-8.

Credit: #808 (@JoaoLockem) proposed the same two-site fix; this PR adds the
regression coverage the fix needs to land safely.

Tests

MirroredDomainBucketsTest, on a new HubLeafJetStreamServerTestCase
fixture: a hub server and a leaf server with distinct JetStream domains
connected over a leafnode link, mirrors created with
external: { api: "$JS.hub.API" }.

The cross-domain topology is load-bearing for the repro: a same-server
mirror would let the subject lookup resolve the origin stream and silently
mask the defect. (A same-server origin-deleted variant can also reproduce it,
but the cross-domain shape is the one users actually run and the one #505
reports.)

  • test_object_store_get_from_cross_domain_mirror - multi-chunk get()
    (256 KiB + 17 B across 128 KiB chunks) plus list() against the mirror.
    Fails with NotFoundError (raised from
    manager.find_stream_name_by_subject) without the fix; passes with it.
  • test_kv_get_and_keys_from_cross_domain_mirror - KV get() and keys()
    (watch path) against a KV-shaped mirror (max_msgs_per_subject inherited
    from the origin, as nats kv add --mirror does). Passes before and after
    (regression guard for Avoid $JS.API.STREAM.NAMES call since stream name is known #807).

Conformance against the Go client

Same topology, same mirrored buckets on the leaf node, read with the patched
Python client and natscli 0.4.0 (nats.go):

Read path Go client Patched nats-py Result
object get artifacts tiny (mirror) 307,200 bytes 307,200 bytes SHA-256 identical (c5ec6f90…)
object ls artifacts (mirror) tiny ['tiny'] match
kv get cfg alpha (mirror) value-alpha value-alpha match
kv ls cfg (mirror) alpha, beta ['alpha', 'beta'] match

Unpatched nats-py 2.15.0 fails the first two with NotFoundError where the
Go client succeeds.

Validation

  • uv run pytest nats/tests/test_js.py::MirroredDomainBucketsTest — 2 passed
  • Full nats/tests suite — green (nats-server v2.14.2, CPython 3.13)
  • uv run ruff check / uv run ruff format --check — clean

Note to maintainers: #808 proposed the same two-line fix - happy to rebase this PR to tests-only if you'd prefer to land #808 for the fix itself; the goal is just that #505 ends up fixed with regression coverage either way.

…ckets

ObjectStore.get() and ObjectStore.watch() (and therefore list()) located
the backing stream by subject lookup. A mirrored bucket's stream binds no
subjects, so both paths raised NotFoundError when reading a bucket whose
backing stream is a mirror — e.g. an Object Store mirrored from another
JetStream domain over a leafnode link.

The bucket name already determines the stream name (OBJ_<bucket>), so pass
stream= to both subscriptions, matching the Go client, which binds with
OrderedConsumer()+BindStream(streamName) in both GetObject and Watch
(nats.go object.go). The KV half of this defect was already fixed by nats-io#807;
the new HubLeafJetStreamServerTestCase covers cross-domain mirror reads
for both KV and Object Store so the whole of nats-io#505 stays fixed.

Fixes nats-io#505

Signed-off-by: Gary Giesen <ggiesen+claude@giesen.me>
@ggiesen ggiesen marked this pull request as ready for review June 12, 2026 16:43
@caspervonb caspervonb added the nats-legacy Legacy asyncio NATS client label Jun 13, 2026
@caspervonb caspervonb self-requested a review June 19, 2026 20:12
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

nats-legacy Legacy asyncio NATS client

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Key Value and Object store do not always work with mirrored streams

2 participants