Skip to content

Commit 14c26c6

Browse files
committed
docs: correct the watermark exemption contract
In this commit, we fix the architecture doc's false claim that the OutboxPublisher folded path bypasses DurableMailbox.Send (it does not; the exemption is now real and implemented via the outbox-ID context marker), document the control-priority exemption and the boot messages that carry it, describe the hardened probe (single-flight, ambient-tx stripping), and state the known residual honestly: an in-turn Tell into a saturated peer fails the sender's turn and burns the inbound message's delivery attempts, with postpone semantics as the planned structural fix and the dead-letter tooling from #1119 as the interim recovery path.
1 parent 95e773a commit 14c26c6

3 files changed

Lines changed: 74 additions & 25 deletions

File tree

baselib/actor/AGENTS.md

Lines changed: 16 additions & 5 deletions
Original file line numberDiff line numberDiff line change
@@ -67,11 +67,22 @@ crash-safe at-least-once delivery with exactly-once deduplication.
6767
`TrySend`/`TryTell` inherit the check and a refusal needs no cleanup).
6868
Both default to 0 (disabled); consumer sites opt in with
6969
`DefaultSoftHighWatermark` (1000) / `DefaultHardHighWatermark` (10000).
70-
Priority `>= RestartPriority` is always exempt so recovery lands. Depth is
71-
read via a 1s TTL-cached probe plus a local sent-since-probe delta
72-
(one-sided: overshoots, never undershoots), and a probe failure fails
73-
OPEN. The OutboxPublisher's folded delivery path bypasses `Send` and is
74-
deliberately unthrottled.
70+
Two exemptions: priority `>= ControlPriority` (restart plus the
71+
boot-critical restore/resume messages — `RestoreNonTerminalRequest`,
72+
`ResumeUnrollRequest`, `ResumeCreditOpRequest` — whose refusal would be a
73+
fatal-boot crash loop), and outbox-propagated deliveries (detected via
74+
the outbox-ID context marker; the publisher's claim path bumps attempts
75+
in its own tx, so a refusal would dead-letter a committed CDC row rather
76+
than shed load). Depth is read via a 1s TTL-cached, single-flighted probe
77+
(run outside the mutex AND outside the sender's ambient tx via
78+
`WithoutTx` — joining a SERIALIZABLE writer would take whole-partition
79+
predicate locks) plus a local sent-since-probe delta (one-sided for local
80+
traffic; remote sends invisible for up to one window). A probe failure
81+
falls back to the cached estimate, or fails OPEN with no baseline. The
82+
soft episode is evaluated before the hard refusal so the warning fires
83+
even when the first probe already reads saturated. Known residual: an
84+
in-turn Tell into a saturated peer fails the sender's turn and burns the
85+
inbound message's attempts (postpone semantics are the planned fix).
7586
- `MailboxDepthStore` / `MailboxDepthCount` — Narrow, optional read surface
7687
(`MailboxDepth(ctx, mailboxID)`, `MailboxDepths(ctx)`) discovered by type
7788
assertion on the `DeliveryStore`, deliberately NOT embedded in it (test

baselib/actor/CLAUDE.md

Lines changed: 16 additions & 5 deletions
Original file line numberDiff line numberDiff line change
@@ -67,11 +67,22 @@ crash-safe at-least-once delivery with exactly-once deduplication.
6767
`TrySend`/`TryTell` inherit the check and a refusal needs no cleanup).
6868
Both default to 0 (disabled); consumer sites opt in with
6969
`DefaultSoftHighWatermark` (1000) / `DefaultHardHighWatermark` (10000).
70-
Priority `>= RestartPriority` is always exempt so recovery lands. Depth is
71-
read via a 1s TTL-cached probe plus a local sent-since-probe delta
72-
(one-sided: overshoots, never undershoots), and a probe failure fails
73-
OPEN. The OutboxPublisher's folded delivery path bypasses `Send` and is
74-
deliberately unthrottled.
70+
Two exemptions: priority `>= ControlPriority` (restart plus the
71+
boot-critical restore/resume messages — `RestoreNonTerminalRequest`,
72+
`ResumeUnrollRequest`, `ResumeCreditOpRequest` — whose refusal would be a
73+
fatal-boot crash loop), and outbox-propagated deliveries (detected via
74+
the outbox-ID context marker; the publisher's claim path bumps attempts
75+
in its own tx, so a refusal would dead-letter a committed CDC row rather
76+
than shed load). Depth is read via a 1s TTL-cached, single-flighted probe
77+
(run outside the mutex AND outside the sender's ambient tx via
78+
`WithoutTx` — joining a SERIALIZABLE writer would take whole-partition
79+
predicate locks) plus a local sent-since-probe delta (one-sided for local
80+
traffic; remote sends invisible for up to one window). A probe failure
81+
falls back to the cached estimate, or fails OPEN with no baseline. The
82+
soft episode is evaluated before the hard refusal so the warning fires
83+
even when the first probe already reads saturated. Known residual: an
84+
in-turn Tell into a saturated peer fails the sender's turn and burns the
85+
inbound message's attempts (postpone semantics are the planned fix).
7586
- `MailboxDepthStore` / `MailboxDepthCount` — Narrow, optional read surface
7687
(`MailboxDepth(ctx, mailboxID)`, `MailboxDepths(ctx)`) discovered by type
7788
assertion on the `DeliveryStore`, deliberately NOT embedded in it (test

docs/durable_actor_architecture.md

Lines changed: 42 additions & 15 deletions
Original file line numberDiff line numberDiff line change
@@ -299,21 +299,41 @@ on ack, so the count is exactly the undelivered backlog):
299299
drains.
300300

301301
Both default to zero (disabled) on the config structs; the bound is opted
302-
into per actor. Messages with priority `>= RestartPriority` are always
303-
exempt, because the `RestartMessage` that would un-wedge a stuck actor must
304-
not be refused by the very backlog it exists to drain. A store that does not
305-
implement `actor.MailboxDepthStore` runs without watermarks entirely, and a
306-
failed depth probe fails OPEN (the send is admitted): a broken monitoring
307-
read must not become message loss.
302+
into per actor. Two message classes are always exempt:
303+
304+
- **Control-priority messages** (`priority >= actor.ControlPriority`, which
305+
includes `RestartPriority`): the restart, restore, and resume messages
306+
that would un-wedge a stuck actor must not be refused by the very backlog
307+
they exist to drain. Boot-time restores (`RestoreNonTerminalRequest` in
308+
OOR, `ResumeUnrollRequest` in unroll, `ResumeCreditOpRequest` in credit)
309+
carry this priority because the daemon treats their failure as fatal: a
310+
refusal there would turn a backed-up mailbox into a restart crash loop.
311+
- **Outbox-propagated deliveries** (detected via the outbox ID the
312+
publisher stamps into the context): the message was already accepted at
313+
its true producer and durably committed to the outbox, so refusing the
314+
CDC hand-off sheds nothing. Worse, the publisher's claim path bumps
315+
delivery attempts in its own transaction, so repeated refusals would
316+
dead-letter the committed outbox row (and any DurableAsk response it
317+
carries) instead of exerting backpressure.
318+
319+
A store that does not implement `actor.MailboxDepthStore` runs without
320+
watermarks entirely, and a failed depth probe fails OPEN (the send is
321+
admitted): a broken monitoring read must not become message loss.
308322

309323
### The probe
310324

311325
The depth read is TTL-cached (one second) with a local count of sends
312326
accepted since the last probe added on top, so the common send path pays no
313-
extra query. The estimate is deliberately one-sided: local sends push it up
314-
immediately, while acks and remote sends only surface at the next probe.
315-
Overshooting is the safe direction for an admission check, and the
316-
enforcement error is bounded by one probe window.
327+
extra query. The estimate is one-sided for local traffic (local sends push
328+
it up immediately, acks surface at the next probe; overshooting is the safe
329+
direction for an admission check), while sends from other processes stay
330+
invisible for up to one window, so the bound is approximate rather than
331+
exact. The probe is single-flighted (concurrent senders use the cached
332+
estimate rather than stacking behind the query) and runs with the sender's
333+
ambient transaction stripped (`WithoutTx`): joining a SERIALIZABLE writer
334+
would take predicate locks over the whole mailbox partition and manufacture
335+
serialization conflicts with the consumer's acks, precisely when the system
336+
is already contended.
317337

318338
### Who refuses, and what callers do
319339

@@ -326,11 +346,18 @@ enforcement error is bounded by one probe window.
326346
- **Local producers** (RPC handlers, other actors) see the error from
327347
`Tell`/`Ask` and propagate it; at ten thousand parked messages, failing
328348
loudly is the only move that helps.
329-
- **The OutboxPublisher's folded delivery path is deliberately unthrottled**:
330-
it enqueues into the target mailbox inside the publisher's own write
331-
transaction, bypassing `DurableMailbox.Send`, so CDC delivery is never
332-
refused. Throttling it would only move the backlog from the target mailbox
333-
to the outbox table while breaking the claim-expiry retry contract.
349+
- **The OutboxPublisher's folded delivery path is exempt** via the outbox-ID
350+
context marker described above, so CDC delivery is never refused and the
351+
claim-expiry retry contract is untouched.
352+
353+
**Known residual**: an actor turn that Tells into a saturated peer inside
354+
its own commit (e.g. an OOR session's transport send into serverconn
355+
egress) fails the whole turn, which nacks the INBOUND message and burns one
356+
of its finite delivery attempts; a long enough saturation episode
357+
dead-letters it. The dead-letter tooling (#1119) makes those visible and
358+
requeueable, and the planned postpone semantics (re-enqueue without burning
359+
attempts) are the structural fix; until then, saturation-driven turn
360+
failures ride the ordinary retry/dead-letter path.
334361

335362
### Observability
336363

0 commit comments

Comments
 (0)