|
| 1 | +--- |
| 2 | +title: Delivery and retries |
| 3 | +description: How Spectrum decides when to retry, when to give up, and what your endpoint should return |
| 4 | +--- |
| 5 | + |
| 6 | +This page describes what happens *after* the worker computes a signature and starts the `POST` to your URL. The contract is simple but worth knowing exactly, because it determines how fault-tolerant you need to be on your end. |
| 7 | + |
| 8 | +## The contract in one paragraph |
| 9 | + |
| 10 | +Spectrum tries to deliver each event for up to four attempts within a ~6.2 second window. Each attempt has a 10-second per-request timeout. Retries happen on `5xx`, `408`, `429`, network errors, and our own timeouts; other `4xx` codes mean "give up." Successful deliveries acknowledge with any `2xx`. After the budget is exhausted the event is dropped from the worker's memory — there is no durable retry queue. Customers who care about every event must dedupe and tolerate occasional misses. |
| 11 | + |
| 12 | +## What the worker does on each attempt |
| 13 | + |
| 14 | +```mermaid |
| 15 | +sequenceDiagram |
| 16 | + participant W as Spectrum worker |
| 17 | + participant Y as Your endpoint |
| 18 | +
|
| 19 | + W->>Y: POST (signed) — attempt 1 |
| 20 | + Y-->>W: 200 OK |
| 21 | + Note over W: ✓ delivered, stop |
| 22 | +``` |
| 23 | + |
| 24 | +If the first attempt fails, the worker waits and tries again: |
| 25 | + |
| 26 | +```mermaid |
| 27 | +sequenceDiagram |
| 28 | + participant W as Spectrum worker |
| 29 | + participant Y as Your endpoint |
| 30 | +
|
| 31 | + W->>Y: POST — attempt 1 |
| 32 | + Y-->>W: 503 |
| 33 | + W->>W: wait 200ms |
| 34 | + W->>Y: POST — attempt 2 |
| 35 | + Y-->>W: 503 |
| 36 | + W->>W: wait 1s |
| 37 | + W->>Y: POST — attempt 3 |
| 38 | + Y-->>W: 200 OK |
| 39 | + Note over W: ✓ delivered after retry |
| 40 | +``` |
| 41 | + |
| 42 | +Total wall-clock budget across all attempts: roughly 6.2 seconds (200ms + 1s + 5s of sleeps, plus per-attempt time on the network). The worker stops as soon as it gets a 2xx or determines further retries are pointless. |
| 43 | + |
| 44 | +## Retry policy |
| 45 | + |
| 46 | +| Attempt | Delay before this attempt | |
| 47 | +| --- | --- | |
| 48 | +| 1 | none — fires immediately | |
| 49 | +| 2 | 200ms after attempt 1 ends | |
| 50 | +| 3 | 1 second after attempt 2 ends | |
| 51 | +| 4 | 5 seconds after attempt 3 ends | |
| 52 | + |
| 53 | +Per-attempt timeout: **10 seconds** (configurable via `DELIVERY_TIMEOUT_MS` on our side, but you should treat 10s as the practical ceiling). |
| 54 | + |
| 55 | +After attempt 4 fails, the event is logged and dropped. There is no persistent queue and no dead-letter destination — both are out of scope for v1. |
| 56 | + |
| 57 | +<Note> |
| 58 | +Multiple registered URLs receive the same event in parallel via `Promise.allSettled`. One slow or failing URL never delays delivery to the others. |
| 59 | +</Note> |
| 60 | + |
| 61 | +## What your status codes mean to us |
| 62 | + |
| 63 | +| Status code(s) | Worker treats as | Result | |
| 64 | +| --- | --- | --- | |
| 65 | +| `2xx` | Success | Delivery complete. Stop. | |
| 66 | +| `5xx` | Retriable | Wait, retry up to 3 more times. | |
| 67 | +| `408 Request Timeout` | Retriable | Wait, retry. | |
| 68 | +| `429 Too Many Requests` | Retriable | Wait, retry. We don't honor `Retry-After` yet — use any 5xx/429 to backpressure. | |
| 69 | +| Any other `4xx` (e.g. `400`, `401`, `403`, `404`, `422`) | Fatal | Don't retry. The assumption is that the request will never succeed (auth bug, schema mismatch, missing route). | |
| 70 | +| Connection refused / DNS error / TCP reset | Retriable | Wait, retry. | |
| 71 | +| Per-attempt timeout (>10s) | Retriable | Wait, retry. | |
| 72 | + |
| 73 | +<Tip> |
| 74 | +**Return `4xx` deliberately.** Returning `400` or `401` from a real bug (e.g. signature verification failure) is correct — it tells us "stop retrying, this request will never work." Returning `500` for the same bug wastes our retry budget and your CPU cycles. |
| 75 | +</Tip> |
| 76 | + |
| 77 | +## What you should do on your end |
| 78 | + |
| 79 | +### Acknowledge fast, process asynchronously |
| 80 | + |
| 81 | +Return `2xx` as soon as you've **verified the signature and queued the work**. Do not block the response on slow downstream operations (LLM calls, third-party APIs, large database writes). |
| 82 | + |
| 83 | +```ts |
| 84 | +app.post('/spectrum-webhook', async (c) => { |
| 85 | + if (!verify(c)) return c.text('bad signature', 401); |
| 86 | + |
| 87 | + const payload = JSON.parse(await c.req.text()); |
| 88 | + void enqueueForProcessing(payload); |
| 89 | + |
| 90 | + return c.text('ok', 200); |
| 91 | +}); |
| 92 | +``` |
| 93 | + |
| 94 | +If your handler takes >10 seconds, the worker will time out the connection, mark it retriable, and `POST` again. Now you'll process the same event twice. |
| 95 | + |
| 96 | +### Be idempotent |
| 97 | + |
| 98 | +At-least-once delivery means the same event can arrive more than once if your server hung after processing but before responding. Dedupe in your handler: |
| 99 | + |
| 100 | +```ts |
| 101 | +const dedupeKey = `${webhookId}:${payload.message.id}`; |
| 102 | + |
| 103 | +if (await alreadyProcessed(dedupeKey)) { |
| 104 | + return c.text('ok', 200); |
| 105 | +} |
| 106 | + |
| 107 | +await processOnce(payload); |
| 108 | +await markProcessed(dedupeKey); |
| 109 | +``` |
| 110 | + |
| 111 | +A short TTL (24-48 hours) on the dedupe table is enough — by then the worker has long since moved on. |
| 112 | + |
| 113 | +### Handle bursts |
| 114 | + |
| 115 | +A noisy chat (group thread, mass DM) can produce many events per second. Make sure your handler can either: |
| 116 | + |
| 117 | +- Process events at the rate they arrive, or |
| 118 | +- Queue them durably (BullMQ, SQS, Postgres-backed queue, anything) and return `2xx` immediately. |
| 119 | + |
| 120 | +Returning `503` on overload is fine — we'll back off and retry. But it eats into your retry budget; queueing is preferable. |
| 121 | + |
| 122 | +## Failure modes and what they cost you |
| 123 | + |
| 124 | +| Scenario | Outcome | |
| 125 | +| --- | --- | |
| 126 | +| Endpoint returns `2xx` on first try | Best case. One delivery, one process. | |
| 127 | +| Endpoint returns `503`, recovers within 6s | Retried, eventually delivered. One process (assuming no `2xx` on the failed attempt). | |
| 128 | +| Endpoint times out after 10s, then succeeds | Retried, eventually delivered. **Possibly processed twice** — your handler ran during the timeout and again on retry. Dedupe required. | |
| 129 | +| Endpoint returns `400` (signature bug, etc.) | Dropped immediately, no retry. Event lost. Logged on our side. | |
| 130 | +| Endpoint down for >6 seconds | Dropped after 4 attempts. Event lost. | |
| 131 | +| Spectrum worker crashes mid-delivery | Event lost — no durable queue. Subsequent events resume after restart. | |
| 132 | + |
| 133 | +The "event lost" rows are why this is **at-least-once, with bounded retries**, not "guaranteed delivery." If your use case requires zero loss (financial transactions, audit logging), pair webhooks with periodic reconciliation against the [Spectrum API](/api-reference/introduction) — list messages on the space and backfill anything you missed. |
| 134 | + |
| 135 | +## Order and parallelism |
| 136 | + |
| 137 | +- **No global ordering guarantee.** Events from different projects, different spaces, or different platforms can arrive in any order. |
| 138 | +- **No per-space ordering guarantee.** A late retry for an earlier message can land after a successfully-delivered later message. |
| 139 | +- **Parallel deliveries to multiple URLs.** If you have multiple webhooks registered, they receive each event in parallel and may finish in any order. |
| 140 | + |
| 141 | +If your handler depends on order, sort by `message.timestamp` (which is the platform's send time, not the delivery time) and rely on dedupe to handle late arrivals. |
| 142 | + |
| 143 | +## What we *don't* deliver |
| 144 | + |
| 145 | +- **Outbound messages.** A message you send via the API does not echo back as a webhook. |
| 146 | +- **Read receipts, typing indicators, reactions.** Coming as separate event types in future versions — they are not in `messages` payloads today. |
| 147 | +- **Acknowledgements that you processed correctly.** Returning `2xx` only tells us the delivery succeeded; we don't track downstream state. |
| 148 | + |
| 149 | +## When to use the SDK loop instead |
| 150 | + |
| 151 | +If you find yourself working hard to compensate for delivery loss, consider running [`spectrum-ts`](/spectrum-ts/getting-started) directly instead of (or in addition to) webhooks. The SDK's `instance.messages` async iterable is a long-lived stream — slower events can't be lost to a delivery timeout because there is no delivery, just a `for await` loop running in your process. |
| 152 | + |
| 153 | +A common pattern: webhooks for low-latency push, and a periodic reconciliation worker that uses the SDK or API to backfill anything the webhook layer missed. |
0 commit comments