feat(workflow-engine): add namespace throttle sweep state machine

[danielskovli]

Description

The core of the namespace circuit breaker (#18481, ADR in #19954): a periodic sweep (NamespaceThrottleService) running detect → throttle → probe → release under a Postgres advisory lock — single writer across replicas; a held lock skips the cycle rather than queueing. With Throttling.Enabled=false the loop never starts.

• Detect: one GROUP BY namespace over the new partial index; trip = absolute floor AND ratio, both from EngineSettings.Throttling.
• Park: bulk-stamps throttled_until = now + window (±20% jitter per row), clamped per-stamp to each workflow's current step's retry deadline — the anchor rule is extracted from WorkflowHandler into a shared StepExtensions.ResolveRetryAnchor (behavior-neutral) and combined with the existing GetDeadline; a row at/past its deadline is left unparked so the final attempt is never delayed. Stamps are guarded WHERE status = Requeued and never touch updated_at.
• Canaries: K earliest-backoff_until Requeued workflows, unparked, judged purely by requeue-count comparison against counts recorded at selection (race-free vs. mid-attempt; a vanished canary counts as progressed). Unanimous failure → window ×2 (capped) + canary rotation; any progress → HalfOpen (quorum of one), first cohort releasing in the same sweep.
• Release: oldest-first cohorts doubling each sweep, stamped now + random()·SweepInterval (jittered smear, not a NULL-clear, so a big cohort doesn't hit one poll cycle). Re-trip during recovery is evaluated over the unparked population only (parked rows are still Requeued — raw counts would re-trip instantly) and returns to Open keeping the grown window.
• Closed: the row lingers for a grace period (5 × SweepInterval, named constant) during which stragglers' throttled_until is cleared, then is deleted. A re-trip from Closed starts over at the initial window — the grace period judged the incident over.
• Publication: a read-only IThrottleStateView singleton (refreshed by every replica each cycle, lock holder or not) exposing Open breakers only — HalfOpen is deliberately excluded so cooperative parking (next PR) cannot hide the failures the re-trip signal is judged on.
• Observability: engine.throttle.tripped/extended/released/closed counters tagged namespace, an open-breakers gauge, per-cycle activity, and structured logs for every transition.
• Release cohort size is in-memory by design: losing it on restart/failover restarts cohorts at canary scale and re-doubles — a few extra-cautious sweeps, never incorrect.

Stacked on #19962.

Verification

• Full engine suite: 701 passed, 0 failed (Resilience 57, Models 63, Data 70, Core 197, Repository 166, Integration 148 — real Postgres/Testcontainers).
• 16 new sweep tests drive cycles with explicit clock values (zero sleeps): trip gating, jitter bounds, deadline clamp (near-deadline exact, past-deadline no-stamp), canary judgments, rotation + window cap, quorum-of-one, cohort order/doubling/smear, re-trip window retention, grace cleanup + row deletion, advisory-lock skip, disabled no-op.
• No snapshot or wire-contract changes. The hosted-services registration-order snapshot lives in workflow-engine-app and will legitimately change when its CI runs against this chain.

🤖 Generated with Claude Code

[coderabbitai]

Important

Review skipped

Draft detected.

Please check the settings in the CodeRabbit UI or the .coderabbit.yaml file in this repository. To trigger a single review, invoke the @coderabbitai review command.

⚙️ Run configuration

Configuration used: Path: .coderabbit.yaml

Review profile: CHILL

Plan: Pro Plus

Run ID: 0f1cff30-a50a-4f14-816b-1246c1d75a27

You can disable this status message by setting the reviews.review_status to false in the CodeRabbit configuration file.

Use the checkbox below for a quick retry:

• 🔍 Trigger review

---

Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out.

❤️ Share

• X
• Mastodon
• Reddit
• LinkedIn

_{Comment @coderabbitai help to get the list of available commands.}