docs(webhooks): document retry policy (exponential backoff + jitter + configurable cap)

Updates Delivery and retries to reflect the formula-driven retry
schedule landing in photon-hq/spectrum-webhook — exponential backoff
with ±50% jitter, four operator-tunable knobs (initial delay, growth
factor, per-attempt cap, total attempts), and an explicit honest
statement that there is no DLQ today.

Customer-facing changes:
- Contract bullet now mentions jitter and the ~9.3s worst-case sleep
  budget alongside the existing ~6.2s expected case.
- Retry-policy table grows an "Actual jittered range" column so
  customers know to expect [100ms, 300ms), [500ms, 1500ms), etc.
- New "Why jitter matters" section explains the thundering-herd
  failure mode that motivates the design.
- New "Tunable on our side" section lists the four env knobs as
  operator-only, with a pointer for customers who need different
  retry behaviour (issue / Discord).
- Dedupe-key advice now points explicitly at X-Spectrum-Webhook-Id
  plus payload.message.id rather than naming the variables locally.

PR body intentionally references the webhook-side PR placeholder; the
cross-link gets filled in after both PRs are open.

Co-authored-by: Cursor <cursoragent@cursor.com>

Author: yanxue06

webhooks/delivery.mdx

@@ -7,10 +7,10 @@ You know what arrives ([Events](/webhooks/events)) and how to prove it's real ([
 
 ## The contract at a glance
 
-- **Strong retry behaviour.** Up to 4 attempts per event, with backoff on `5xx`, `408`, `429`, network errors, and worker-side timeouts. The vast majority of deliveries land on attempt 1; the retries are there for the occasional bad minute on your side.
+- **Strong retry behaviour.** Up to 4 attempts per event by default, with exponential backoff plus jitter on `5xx`, `408`, `429`, network errors, and worker-side timeouts. The vast majority of deliveries land on attempt 1; the retries are there for the occasional bad minute on your side.
 - **Fast acknowledgement.** Any `2xx` ends it — the worker stops as soon as your server says ok.
 - **Fast permanent failure.** Other `4xx` codes (`400`/`401`/`404`/etc.) are treated as fatal — we don't waste your retry budget when the request will never succeed.
-- **Bounded budget.** 30-second per-attempt timeout, with ~6.2 seconds of backoff sleeps between attempts. If your server is still down after the final attempt, the event is logged and the worker moves on.
+- **Bounded budget.** 30-second per-attempt timeout, with up to ~9.3 seconds of backoff sleeps between attempts (jittered). If your server is still down after the final attempt, the event is logged and the worker moves on — there is no dead-letter queue today.
 - **At-least-once delivery.** A retry after your server timed out can re-deliver an event you already processed — always dedupe in your handler (see [Be idempotent](#be-idempotent) below).
 - **URL guard, fail-closed.** Before every attempt the worker validates the target URL: it must be `https://`, must resolve to a public address, and must not redirect. A URL that fails the check is dropped immediately — fatal, no retry — see [Where we won't deliver](#where-we-wont-deliver) below.
 
@@ -37,34 +37,57 @@ sequenceDiagram
 
   W->>Y: POST — attempt 1
   Y-->>W: 503
-  W->>W: wait 200ms
+  W->>W: wait ~200ms (±50%)
   W->>Y: POST — attempt 2
   Y-->>W: 503
-  W->>W: wait 1s
+  W->>W: wait ~1s (±50%)
   W->>Y: POST — attempt 3
   Y-->>W: 200 OK
   Note over W: ✓ delivered after retry
 ```
 
-The backoff *sleeps* total ~6.2 seconds (200ms + 1s + 5s). Wall-clock time also includes per-attempt network time, bounded by the 30-second per-attempt timeout: a healthy delivery finishes in milliseconds, while a worst case where every attempt hangs to the timeout can run up to ~2 minutes before the worker gives up. It stops as soon as it gets a 2xx or determines further retries are pointless.
+The backoff *sleeps* sum to ~6.2 seconds in the average case (200ms + 1s + 5s) and ~9.3 seconds in the worst case (jitter ceiling). Wall-clock time also includes per-attempt network time, bounded by the 30-second per-attempt timeout: a healthy delivery finishes in milliseconds, while a worst case where every attempt hangs to the timeout can run up to ~2 minutes before the worker gives up. It stops as soon as it gets a 2xx or determines further retries are pointless.
 
 ## Retry policy
 
-| Attempt | Delay before this attempt |
-| --- | --- |
-| 1 | none — fires immediately |
-| 2 | 200ms after attempt 1 ends |
-| 3 | 1 second after attempt 2 ends |
-| 4 | 5 seconds after attempt 3 ends |
+Retries follow an exponential-backoff schedule with ±50% jitter applied to every delay. The formula is the canonical [full-jitter pattern](https://aws.amazon.com/blogs/architecture/exponential-backoff-and-jitter/) — the *expected* delay is the value in the table below, while the *actual* delay is drawn uniformly from the jitter window so coordinated retries don't pile onto your endpoint at the same instant after a recovery.
+
+| Attempt | Expected delay before this attempt | Actual jittered range |
+| --- | --- | --- |
+| 1 | none — fires immediately | — |
+| 2 | 200ms after attempt 1 ends | `[100ms, 300ms)` |
+| 3 | 1 second after attempt 2 ends | `[500ms, 1500ms)` |
+| 4 | 5 seconds after attempt 3 ends | `[2.5s, 7.5s)` |
 
-Per-attempt timeout: **30 seconds** (configurable via `DELIVERY_TIMEOUT_MS` on our side). Treat it as a hard ceiling, not a target — acknowledge in well under a second and push slow work off the response path (see [Acknowledge fast](#acknowledge-fast-process-asynchronously) below).
+Per-attempt timeout: **30 seconds**. Treat it as a hard ceiling, not a target — acknowledge in well under a second and push slow work off the response path (see [Acknowledge fast](#acknowledge-fast-process-asynchronously) below).
 
 After attempt 4 fails, the event is logged and dropped. There is no persistent queue and no dead-letter destination — both are out of scope for v1.
 
 <Note>
 Multiple registered URLs receive the same event in parallel via `Promise.allSettled`. One slow or failing URL never delays delivery to the others.
 </Note>
 
+### Why jitter matters
+
+A naive deterministic schedule (`200ms, 1s, 5s` to the millisecond) means that when *every* project's deliveries flap at once — a rolling deploy on your side, a regional DB failover, a noisy upstream — every retry across every project queues at exactly the same offsets and lands on your first healthy moment as a coordinated herd. Jitter spreads each scheduled delay across a window twice as wide as the expected value, so the retry volume smears out and your connection pool / WAF / autoscaler get room to absorb the load gracefully.
+
+### Tunable on our side
+
+The retry schedule is operator-configurable. The Photon team can adjust these knobs per environment to trade latency for durability — useful, for example, if a regulated workload needs to tolerate a longer outage than the default ~6.5s budget covers. The full set:
+
+| Knob | Default | Effect |
+| --- | --- | --- |
+| Initial delay | 200ms | The `i = 0` term — delay before the first retry. |
+| Growth factor | 5× | Multiplier applied per retry index (`200ms → 1s → 5s → ...`). |
+| Per-attempt cap | 10 seconds | Ceiling applied to every computed delay before jitter, so the curve can't run away. |
+| Total attempts | 4 (initial + 3 retries) | Higher values trade wall-clock latency for more retries against a flaky endpoint. |
+
+These are *internal* env vars on the spectrum-webhook worker — customers can't set them per-webhook today. If you have a use case that needs different retry behaviour (more retries, longer ceiling), reach out and we'll discuss tuning the deployment-wide defaults or adding a per-project override. Open an issue on the [docs repo](https://github.com/photon-hq/docs) or message us in the [Discord](https://discord.gg/4c3VJzDfNA).
+
+<Tip>
+If you're seeing duplicates after long handler waits — say, attempt 1 takes 28 seconds and succeeds on your side, but our retry layer doesn't see the response in time — that's the per-attempt timeout, not the retry schedule. Tighten your handler (acknowledge first, process later) before asking us to widen our budget.
+</Tip>
+
 ## What your status codes mean to us
 
 | Status code(s) | Worker treats as | Result |
@@ -118,7 +141,7 @@ If your handler takes >30 seconds, the worker will time out the connection, mark
 
 ### Be idempotent
 
-At-least-once delivery means the same event can arrive more than once if your server hung after processing but before responding. Dedupe in your handler:
+At-least-once delivery means the same event can arrive more than once if your server hung after processing but before responding. Dedupe in your handler using a composite of the `X-Spectrum-Webhook-Id` header (the webhook config ID) and an event-scoped identifier from the payload — e.g. `payload.message.id` for the `messages` event:
 
 ```ts
 const dedupeKey = `${webhookId}:${payload.message.id}`;
@@ -131,7 +154,7 @@ await processOnce(payload);
 await markProcessed(dedupeKey);
 ```
 
-A short TTL (24-48 hours) on the dedupe table is enough — by then the worker has long since moved on.
+A short TTL (24-48 hours) on the dedupe table is enough — the retry budget is bounded to under a minute even with jitter, so anything we'd re-deliver lands well inside that window.
 
 ### Handle bursts
 
@@ -153,7 +176,7 @@ Returning `503` on overload is fine — we'll back off and retry. But it eats in
 | Webhook URL is `http://` (not HTTPS) | Dropped immediately by the URL guard, no retry. Every event lost until you re-register an `https://` URL. |
 | Webhook URL resolves to a private/internal IP | Dropped immediately, no retry (SSRF guard). Logged. |
 | Endpoint responds with a `3xx` redirect | Dropped immediately, no retry. Register the final URL instead. |
-| Endpoint down for >6 seconds | Dropped after 4 attempts. Event lost. |
+| Endpoint down for the full retry window (~6.5s default, more if you've requested tuning) | Dropped after the final attempt. Event lost — no DLQ today. |
 | Spectrum worker crashes mid-delivery | Event lost — no durable queue. Subsequent events resume after restart. |
 
 The "event lost" rows are why this is **at-least-once, with bounded retries**, not "guaranteed delivery." If your use case requires zero loss (financial transactions, audit logging), pair webhooks with periodic reconciliation against the [Spectrum API](/api-reference/introduction) — list messages on the space and backfill anything you missed.