Skip to content

Fix durable scheduled ceremony delivery - #1009

Draft
jim80net wants to merge 3 commits into
mainfrom
agent/scheduled-ceremony-delivery
Draft

Fix durable scheduled ceremony delivery#1009
jim80net wants to merge 3 commits into
mainfrom
agent/scheduled-ceremony-delivery

Conversation

@jim80net

@jim80net jim80net commented Aug 8, 2026

Copy link
Copy Markdown
Owner

Summary

  • route daemon schedules through a durable scheduled-work kind so coordinator-to-adjutant aliasing survives busy panes
  • persist independent trigger, queue acceptance, pre-delivery attempt, confirmed delivery, artifact deadline/confirmation, and escalation transitions per occurrence
  • fail closed on ambiguous instruction delivery while retrying direct-owner failure alerts at least once with a stable failure ID
  • allow schedules to declare an expected artifact and bounded production window; escalate missing, empty, stale, or late artifacts
  • fail closed on corrupt lifecycle state and retain independent daily occurrences across restart

Regression coverage

  • exact coordinator -> adjutant alias with a busy first attempt
  • restart requeue before delivery and no duplicate after confirmed delivery
  • ambiguous ErrUnconfirmed and post-paste ErrPanelBlocked behavior for instructions and escalations
  • artifact confirmation plus empty, stale, same-instant, late, and missing-artifact failure cases
  • corrupt state, independent next-day occurrence, phase-correct queue acceptance, and roster validation

Gates

  • go test ./...
  • go test -race ./internal/watch ./internal/roster ./cmd/flotilla
  • go vet ./...
  • git diff --check
  • independent read-only review: PASS, no blocking findings

Operational constraints

Nonblocking follow-ups

  • retry/transition policy for schedule-state persistence failures during busy-result and enqueue acknowledgements
  • completed occurrence retention/pruning policy

Summary by cubic

Make scheduled ceremonies durable and correctly classified. Scheduled work now uses KindScheduled (not KindDetector) so it survives busy panes; artifacts are a second-line check after confirmed delivery.

  • New Features

    • KindScheduled for ceremony instructions with busy deferral and single coordinator→adjutant alias; works even with no artifact config.
    • Durable per-occurrence lifecycle (trigger → enqueue → attempt → delivery → artifact → escalation) in flotilla-schedule-state.json, with pre-attempt receipts and post-accept enqueue timestamps.
    • Expected-artifact + production-window checks; confirm non-empty files written strictly after delivery and before the deadline; escalate missing/empty/stale/late artifacts to the owning coordinator with a stable failure ID and at-least-once retry.
    • Independent daily occurrences; interrupted/ambiguous attempts fail closed (no instruction replay). Example roster adds morning parade, evening walk, recursive retro, and productivity self-audit.
  • Bug Fixes

    • Scheduled instructions alias once to the adjutant; direct-to-owner escalations bypass aliasing.
    • No duplicates after confirmed delivery or across restarts; only queued-without-attempt work is retried. Ambiguous instruction outcomes escalate instead of replaying.
    • Unreadable/corrupt schedule state disables scheduling (no mass replays).
    • Wired scheduler hooks before the injector worker starts; busy results persist deferral; enqueue time is recorded only after acceptance.

Written for commit 1e7d045. Summary will update on new commits.

Review in cubic

@jim80net

jim80net commented Aug 8, 2026

Copy link
Copy Markdown
Owner Author

Independent read-only review: PASS, no blocking findings. The review specifically re-checked ambiguous ErrUnconfirmed / post-paste ErrPanelBlocked handling, stable-ID direct-owner escalation retry, phase-correct enqueue state, strict artifact bounds, corrupt-state fail-closed behavior, and independent daily occurrences.

Local gates:

  • go test ./... — PASS
  • go test -race ./internal/watch ./internal/roster ./cmd/flotilla — PASS
  • go vet ./... — PASS
  • git diff --check — PASS

GitHub gates:

Operational constraints held: draft only; no merge, activation, watch restart, or flotilla-dev recycle. PR #1008's recycle hold remains untouched.

@jim80net

jim80net commented Aug 8, 2026

Copy link
Copy Markdown
Owner Author

Addendum folded into this PR at 124e81db (no separate patch):

  • one exhaustive busy-recipient contract now classifies every known injector job kind;
  • only heartbeat/detector ticks discard as genuinely time-relative;
  • operator relays, interrupts, sends, and scheduled work defer until a terminal outcome, with their producer-owned durable record;
  • unknown future kinds reject loudly until their durability is designed;
  • queued alerts describe retry posture and terminal reporting without promising successful delivery;
  • all four Cos ceremonies now declare dedicated artifact contracts and bounded windows. Evening walk and productivity use dedicated dated receipts so unrelated parade/backlog writes cannot manufacture success;
  • the four-row regression matrix proves busy cos -> cos-adj redirect, retry and confirmed delivery, artifact observation, unrelated-write negative controls, missing-artifact escalation, direct-owner bypass, actual alert delivery to cos, and durable escalation confirmation.

Independent read-only re-review: PASS, no blocking findings.

Gates:

  • go test ./... — PASS
  • go test -race ./internal/watch ./internal/roster ./cmd/flotilla — PASS
  • go vet ./... — PASS
  • git diff --check — PASS

No merge, activation, daemon restart, or flotilla-dev recycle was performed. PR #1008's recycle hold remains untouched.

@jim80net

jim80net commented Aug 8, 2026

Copy link
Copy Markdown
Owner Author

Hosted gates for addendum head 124e81db:

@jim80net

jim80net commented Aug 8, 2026

Copy link
Copy Markdown
Owner Author

Correction applied in 1e7d0452 after source-level review.

The primary defect is ceremony classification, not a general injector delivery-contract defect: the scheduler previously emitted once-daily work as disposable KindDetector. PR #1009 now emits KindScheduled, which follows the existing durable/no-operator-escalation KindSend busy posture. Heartbeat and detector behavior is unchanged.

The generalized busy-policy enum, unknown-kind rejection, and queued-string changes from the prior addendum have been removed. A dedicated regression with no artifact configuration proves that a scheduled ceremony is classified as KindScheduled, defers on a busy recipient, later confirms delivery, and raises no operator escalation.

Expected-artifact lifecycle coverage for all four Cos ceremonies remains as independent second-line detection after confirmed delivery; it does not gate enqueue, retry, or the primary classification fix.

Independent correction review: PASS, no blocking findings.

Fresh gates:

  • go test -count=1 ./internal/watch ./internal/roster
  • go test -race -count=1 ./internal/watch ./internal/roster
  • go test ./...
  • go vet ./internal/watch ./internal/roster
  • git diff --check

No merge, activation, watcher restart, or recycle performed.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant