ADR-58: JetStream durable stream sourcing/mirroring #389

MauriceVanVeen · 2025-11-06T11:19:02Z

Stream sourcing/mirroring on WorkQueue or Interest streams is not reliable due to the use of an ephemeral AckNone consumer. This ADR proposes a design for durable stream sourcing/mirroring that works reliably on all stream retention types.

Signed-off-by: Maurice van Veen [email protected]

neilalexander · 2025-11-10T20:32:24Z

@roeschter Please add comments inline using the review function, otherwise it's extremely difficult to thread and track replies to individual comments.

roeschter

Missing specifications?
-Will the sourcing side verify that the consumer has the right AckPolicy of AckFlowControl?
-What will happen to sourcing from a workqueue is NOT consumer is set? WE shouldn't fail (compatibility), but should we at least print a warning

roeschter · 2025-11-10T21:28:08Z

adr/ADR-57.md

+
+Some additional tooling will be required to create the durable consumer with the proper configuration. But through the
+use of new fields/values on the consumer configuration, the server will be able to help enforce the correct
+configuration.


This is too vague. Which fields? What will be enforced? We need to at least reference where the details are specified,

Updated to:

... But through the use of a new AckPolicy=AckFlowControl field, the server will be able to help enforce the correct configuration.

The enforcing it does is then mentioned in the "Consumer configuration" section, specifically:

Requires FlowControl and Heartbeat to be set.

roeschter · 2025-11-10T21:29:24Z

adr/ADR-57.md

+
+The durable consumer used for stream sourcing/mirroring will need to be just as performant as the current ephemeral
+variant. The current ephemeral consumer configuration uses `AckNone` which is problematic for WorkQueue and Interest
+streams. A different `AckPolicy` will need to be used to ensure that messages are not lost.


A new AckPolicy=AckFlowControl will be introduce.

This is just introductory text explaining AckNone can't be used, the mention of the new field is done below:

Uses AckPolicy=AckFlowControl instead of AckNone

adr/ADR-57.md

roeschter · 2025-11-10T21:32:10Z

adr/ADR-57.md

+- `AckPolicy=AckFlowControl` will function like `AckAll` although the server will not use the ack reply on the received
+  messages.
+- The server responds to the flow control messages, including the stream sequence (`Nats-Last-Stream`) and delivery
+  sequence (`Nats-Last-Consumer`) as headers to specify which messages have been successfully stored.


The receiving stream responds with a flow control message, which includes the stream sequence (Nats-Last-Stream) and delivery sequence (Nats-Last-Consumer) as headers to signalling which messages have been successfully stored

Slightly updated to:

The receiving server responds to the flow control messages, which includes the stream sequence (Nats-Last-Stream) and delivery sequence (Nats-Last-Consumer) as headers to signal which messages have been successfully stored.

Specifically the server that's receiving the flow control message responds to it. It doesn't send flow control messages itself.

roeschter · 2025-11-10T21:32:57Z

adr/ADR-57.md

+  messages.
+- The server responds to the flow control messages, including the stream sequence (`Nats-Last-Stream`) and delivery
+  sequence (`Nats-Last-Consumer`) as headers to specify which messages have been successfully stored.
+- The server receiving the flow control response will ack messages based on these stream/delivery sequences.


The server receiving the flow control response will internally acknowledge all prior messages in the consumer according to its retention policy. For work queue and interest based streams this may result in messages deletion.

Have kept the addition of For WorkQueue and Interest streams this may result in messages deletion. But am inclined to not change the former. A response to a flow control message will not mean ALL prior messages are to be acknowledged. Only those messages that are below the stream/delivery sequences are acked.

roeschter · 2025-11-10T21:36:55Z

adr/ADR-57.md

+- The consumer will be reset, resembling the delivery state of creating a new consumer with `opt_start_seq` set to the
+  specified sequence.
+- The pending and redelivered messages will always be reset.
+- The delivered stream/consumer sequences will always be reset.


The delivered consumer sequence will always be reset.

Have updated this to delivered stream and consumer sequence. These are two independent sequences that need to be reset.

adr/ADR-57.md

jnmoyne · 2025-12-01T12:07:49Z

LGTM

MauriceVanVeen · 2025-12-04T17:49:43Z

Opened a draft PR on the server with a working version for this. I'll update this ADR with some refinements soon.

MauriceVanVeen · 2025-12-05T08:44:55Z

This ADR is now also updated to reflect the initial implemention. Requiring to specify MaxAckPending as upper bound for pending messages, unset AckWait/BackOff, and infinite MaxDeliver.

adr/ADR-57.md

ripienaar

LGTM, feels like we need a proper ok from @derekcollison also maybe?

Signed-off-by: Maurice van Veen <[email protected]>

roeschter · 2026-01-06T16:10:47Z

adr/ADR-58.md

+
+The durable consumer used for stream sourcing/mirroring will need to be just as performant as the current ephemeral
+variant. The current ephemeral consumer configuration uses `AckNone` which is problematic for WorkQueue and Interest
+streams. A different `AckPolicy` (`AckFlowControl`) will need to be used to ensure that messages are not lost.


Trivial change:
A new AckPolicy (AckFlowControl) will need to be used to ensure that performance is on par with an OrderedConsumer and the messages are not lost.

roeschter · 2026-01-06T16:15:04Z

adr/ADR-58.md

+- Requires `FlowControl` and `Heartbeat` to be set.
+- Uses `AckPolicy=AckFlowControl` instead of `AckNone`.
+- `AckPolicy=AckFlowControl` will function like `AckAll` although the receiving server will not use the current ack
+  reply format by acknowledging individual messages.


This is hard to understand - Suggestion:
AckPolicy=AckFlowControl will function like AckAll. The flow control messages (see below) will ack message rather than the current ack reply format.

roeschter · 2026-01-06T16:15:51Z

adr/ADR-58.md

+- Uses `AckPolicy=AckFlowControl` instead of `AckNone`.
+- `AckPolicy=AckFlowControl` will function like `AckAll` although the receiving server will not use the current ack
+  reply format by acknowledging individual messages.
+- The receiving server responds to the flow control messages, which includes the stream sequence (`Nats-Last-Stream`)


"With a" rather than "to a"
The receiving server responds with a flow control messages

roeschter · 2026-01-06T16:23:05Z

adr/ADR-58.md

+  and delivery sequence (`Nats-Last-Consumer`) as headers to signal which messages have been successfully stored.
+- The server receiving the flow control response will ack messages based on these stream/delivery sequences. For
+  WorkQueue and Interest streams this may result in messages deletion.
+- Acknowledgements happen based on flow control limits, usually a data window size. But if the stream is idle the


I vaguely remember there is a reasons for moving the ack floor. In the next section we talk about flow control ack being done according to window size. So this one is for very slow moevign streams?

But if the source/mirror is idle (low message rate) the Heartbeat will also trigger a flow control message to move the acknowledgement floor up even if the window size has not been reached.

Indeed, if a stream receives small messages very slowly, it will not trigger the flow control messages often. So, if the stream hasn't received messages for Heartbeat amount of time, then a flow control message will also be sent. This ensures that a WorkQueue/Interest stream can remove messages after ack in such a slow/idle stream case.

roeschter · 2026-01-06T16:25:34Z

adr/ADR-58.md

+  `MaxAckPending` setting. `MaxAckPending` determines the maximum number of pending messages that can be sent before the
+  sourcing pauses. A flow control message will be automatically sent (no need to wait for a `Heartbeat`) so these
+  messages are acknowledged and new messages can be sent as soon as possible.
+- Since acknowledgements happen based on dynamic flow control, it being determined either by data size, `MaxAckPending`


MUST HAVE is fine, but is this being verified? Will ackwait or backoff settings result in an error? a warning? Or result in undefined behavior?

Anything that's not allowed will error. Have added that here.

roeschter · 2026-01-06T16:30:34Z

adr/ADR-58.md

+
+Importantly, the server should also handle the case where a user manually resets the consumer that's used for sourcing.
+The server should handle this gracefully and ensure no messages are lost. However, the user could also reset the
+consumer such that it moves ahead in the stream. The server should also handle this by properly skipping over those


How will the receiving server detect that this is an intentional skip rather than a gap? Please document.

Done, consumer delivery sequence goes back to 1, the gap is the difference between the received stream sequence and the previously received stream sequence.

roeschter · 2026-01-06T16:32:16Z

adr/ADR-58.md

+
+Client should support the `$JS.API.CONSUMER.RESET.<STREAM>.<CONSUMER>` reset API. Clients should not rely on this call
+to be initiated by the client process, but it potentially being called by another process, by the CLI for example.
+Importantly, clients should not fail when the consumer delivery sequence is not monotonic, except when needed for the


Even with a reset the consumer delivery sequence should still be monotonic. BUT the stream sequence may move backwards!

Resetting the consumer also resets the consumer delivery sequence back to 1, in that case it's not monotonic anymore due to the reset. But it's intended to be monotonic after that (until another reset), and a gap/unordered is detected if it's not monotonically increasing.

The stream sequence may move arbitrarily forward or backward and that doesn't matter for the client's ordered consumer implementations, also why it's not mentioned here.

Signed-off-by: Maurice van Veen <[email protected]>

This comment was marked as outdated.

Sign in to view

roeschter suggested changes Nov 10, 2025

View reviewed changes

ripienaar reviewed Nov 11, 2025

View reviewed changes

adr/ADR-57.md Outdated Show resolved Hide resolved

adr/ADR-57.md Outdated Show resolved Hide resolved

MauriceVanVeen requested review from ripienaar and roeschter November 11, 2025 14:55

MauriceVanVeen mentioned this pull request Nov 19, 2025

(2.14) Reset consumer to new starting sequence nats-io/nats-server#7489

Draft

MauriceVanVeen mentioned this pull request Dec 4, 2025

(2.14) Support durable and flow-controled stream sourcing/mirroring nats-io/nats-server#7613

Draft

MauriceVanVeen force-pushed the maurice/durable-sourcing branch from f741e0a to ed4847e Compare December 5, 2025 09:05

ripienaar reviewed Dec 8, 2025

View reviewed changes

adr/ADR-57.md Outdated Show resolved Hide resolved

adr/ADR-57.md Outdated Show resolved Hide resolved

adr/ADR-57.md Outdated Show resolved Hide resolved

MauriceVanVeen force-pushed the maurice/durable-sourcing branch from c381cc7 to 06e8ab3 Compare December 8, 2025 12:34

MauriceVanVeen requested a review from ripienaar December 8, 2025 12:35

ripienaar approved these changes Dec 8, 2025

View reviewed changes

MauriceVanVeen force-pushed the maurice/durable-sourcing branch from 06e8ab3 to 00aef99 Compare January 6, 2026 15:26

MauriceVanVeen changed the title ~~ADR-57: JetStream durable stream sourcing/mirroring~~ ADR-58: JetStream durable stream sourcing/mirroring Jan 6, 2026

ADR-58: JetStream durable stream sourcing/mirroring

e7c3980

Signed-off-by: Maurice van Veen <[email protected]>

MauriceVanVeen force-pushed the maurice/durable-sourcing branch from 00aef99 to e7c3980 Compare January 6, 2026 15:40

roeschter reviewed Jan 6, 2026

View reviewed changes

ADR-58: Feedback

0e238ac

Signed-off-by: Maurice van Veen <[email protected]>

ADR-58: JetStream durable stream sourcing/mirroring #389

Are you sure you want to change the base?

ADR-58: JetStream durable stream sourcing/mirroring #389

Uh oh!

Conversation

MauriceVanVeen commented Nov 6, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

This comment was marked as outdated.

This comment was marked as outdated.

neilalexander commented Nov 10, 2025

Uh oh!

roeschter left a comment

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

jnmoyne commented Dec 1, 2025

Uh oh!

MauriceVanVeen commented Dec 4, 2025

Uh oh!

MauriceVanVeen commented Dec 5, 2025

Uh oh!

Uh oh!

Uh oh!

Uh oh!

ripienaar left a comment

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

MauriceVanVeen commented Nov 6, 2025 •

edited

Loading