Skip to content

#17 A dead-lettered command gets a terminal processing_failed first, so Watcher isn't left waiting - #19

Merged
gregoryfoster merged 5 commits into
mainfrom
17-dlq-terminal-fact
Oct 2, 2026
Merged

gregoryfoster merged 5 commits into
mainfrom
17-dlq-terminal-fact

Conversation

@gregoryfoster

Copy link
Copy Markdown
Contributor

Closes #17. Refs #1, CannObserv/watcher#325.

Why. Watcher's reaper now re-issues only while other content.derived facts flow (watcher#325, issuecomment-5958475061). A command Processor dead-letters at the escape cap used to get no fact. It would sit in flight, and Watcher's 'Processor not consuming' signal would fire against a healthy Processor in any quiet period.

What. At the cap, before dead-lettering a decodable command, the consumer publishes processing_failed with terminal=true, reason=extraction_error, and detail = dead-lettered: gave up on attempt 3: … (handler.publish_gave_up).

  • No second fact. Skipped when this entry's fact already went out: a refused ack, or a dead-letter retried after the fact landed. Tracked in Consumer._fact_out and cleared on ack, dead-letter, or a deleted reclaim.
  • A transient refusal leaves the entry pending (uncapped). A non-transient one is logged (failure fact refused) and the entry is dead-lettered anyway.
  • Frames that are not commands get no fact, as before.
  • Docs. Spec §4 (both escape rows, plus a new note) and DEPLOYMENT.md's dead-letter line are amended.

Tests (red first, c2ec537):

  • the escape cap publishes one terminal fact, then dead-letters;
  • a refused ack at the cap adds no failure fact;
  • a transiently refused failure fact leaves the entry pending, then lands once;
  • a retried dead-letter doesn't duplicate the fact;
  • a non-transiently refused failure fact still dead-letters.

Full suite: 235 passed.

Conflicts: #18 also edits spec §4 (a different bullet). Whichever ships second gets rebased.

🤖 Generated with Claude Code

gregoryfoster and others added 5 commits October 2, 2026 21:03
…the DLQ, once, never after its own fact

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…raction_error first

Watcher's reaper now re-issues only while other facts flow (watcher#325), so a
command dead-lettered with no fact would sit in flight, and Watcher's health
would read a quiet Processor as down. At the cap the consumer publishes
processing_failed (terminal, extraction_error, detail 'dead-lettered: …'):
- skipped when this entry's fact already went out (a refused ack, or a retried
  dead-letter);
- a transient refusal leaves the entry pending;
- a non-transient refusal is logged and the entry is dead-lettered anyway.
Spec §4 and DEPLOYMENT.md are amended.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…ed, skipped or refused

An operator triaging the DLQ can now tell from the journal whether Watcher was told.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
… extraction_error, detail capped, refusals raised

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…fact

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
@gregoryfoster
gregoryfoster merged commit 6e518d8 into main Oct 2, 2026
2 checks passed
@gregoryfoster
gregoryfoster deleted the 17-dlq-terminal-fact branch October 2, 2026 21:07
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Dead-lettering a decodable command publishes a terminal processing_failed, so Watcher isn't left waiting

1 participant