feat: make catch-up a guarantee rather than a hope

Three separate things stood between "a replay was requested" and "every
missed request was recovered". Each was measured against a live project
rather than reasoned about, and each was wrong.

The filter matched only half the cases. The two disconnect regimes look
different, and only one was covered:

    tunnel never existed          events_count 0, ignored_count 1
    tunnel connected, then killed events_count 0, ignored_count 0

The second is the hard crash. It matches neither `ignored_count >= 1`
nor `cli_events_count: 0` — measured directly, that field is not
populated when no CLI event was ever created, so filtering on it excludes
exactly the case catch-up is for. The query now keys on `events_count: 0`
alone, which is the true signature of a request that produced no
delivery. Live, the replay went from matching ~0 requests to 3 of 3.

The window did not survive a crash, fixed in the previous commit.

The outcome was assumed. The replay call returns as soon as the batch is
accepted, so `estimated_count` is a plan and not a result. The plugin now
polls the batch until Hookdeck reports it finished, and logs `N of M
request(s) replayed`; a shortfall warns and names it, and a batch that
does not finish inside the wait is reported as unknown rather than as
success. The wait is bounded, because this runs when a tunnel reconnects
and must not hold recovery open indefinitely.

One approach was tried and abandoned, and the reason is worth keeping: a
post-replay pass that re-read the window for requests still showing no
event. It can never pass. A replay re-ingests each request as a NEW
request with new events, so the original stays at events_count 0 for
ever — the check reported recovered requests as stranded, and no amount
of waiting would have changed that. The batch's own counts are the only
evidence available.

The honest limit is retention: 3 days on free, 7 on Team, 30 on Growth.
Beyond it the request is gone at the source and nothing reaches it.

669 tests, and the live suite is now 14/14.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>

Author: garethx

docs/durability.md

@@ -32,6 +32,27 @@ So while the transport is running the plugin records a liveness marker every `ca
 
 Over-shooting that window by up to one heartbeat is harmless: the catch-up query matches only requests that produced no event at all, so anything that did deliver is excluded by construction.
 
+## What catch-up guarantees
+
+Within Hookdeck's retention, catch-up recovers every request that arrived during an outage and produced no delivery. Three things make that a guarantee rather than a hope, and each was wrong at some point:
+
+**The filter matches every way a request can be stranded.** Measured against a live project, the two disconnect regimes look different:
+
+| | `events_count` | `ignored_count` |
+|---|---|---|
+| The tunnel never existed | 0 | 1 |
+| The tunnel connected, then the process was killed | 0 | **0** |
+
+The second is the hard-crash case. It matches neither `ignored_count >= 1` nor `cli_events_count: 0` — that field is not populated when no CLI event was ever created — so a filter using either excludes precisely the case catch-up exists for. The query keys on `events_count: 0` alone.
+
+**The outage window survives a crash.** See [above](#a-crash-that-never-ran-its-shutdown).
+
+**The result is confirmed, not assumed.** The replay call returns as soon as the batch is accepted, so its `estimated_count` is a plan. The plugin polls the batch until Hookdeck reports it finished and then logs `N of M request(s) replayed`. A shortfall is a warning naming it; a batch that does not finish within the wait is reported as unknown rather than as success.
+
+What it cannot do is recover anything Hookdeck has already aged out — 3 days on free, 7 on Team, 30 on Growth. Beyond that the request is gone at the source, and no filter or replay reaches it.
+
+> Do not try to verify recovery by re-reading the original requests. A replay re-ingests each one as a **new** request with new events, so the original stays at `events_count: 0` for ever. Checking it will always report a stranded request that was in fact recovered.
+
 ## Malformed bodies never reach the plugin
 
 Verified end to end: Hookdeck rejects an unparseable JSON body **at the edge**, answering the sender `400` with `rejection_cause: UNPARSABLE_JSON` and creating no event. The plugin's own `malformed_json` handling is therefore defence in depth rather than a path real traffic takes — it covers a body that survives the edge and fails here, such as one that is valid JSON but not valid UTF-8.

scripts/e2e-live.mjs

@@ -209,13 +209,37 @@ record(
 );
 
 log = await startGateway("restart");
-await sleep(20000);
+await sleep(25000);
 const catchUpLog = log.join("");
 record(
   "catch-up replay is issued on reconnect",
   /catch-up replay queued/.test(catchUpLog),
   (catchUpLog.match(/catch-up replay queued[^\n]*/) ?? ["not in log"])[0].slice(0, 80),
 );
+record(
+  "the replay matched the stranded request rather than nothing",
+  /catch-up replay queued \(~[1-9]/.test(catchUpLog),
+  (catchUpLog.match(/catch-up replay queued[^\n]*/) ?? ["not in log"])[0].slice(0, 60),
+);
+record(
+  "Hookdeck confirms the batch finished, with counts",
+  /catch-up replay finished: \d+ of \d+/.test(catchUpLog),
+  (catchUpLog.match(/catch-up replay finished:[^\n]*/) ?? ["not reported"])[0].slice(0, 70),
+);
+record(
+  "every request it planned to replay was replayed",
+  !/was not recovered/.test(catchUpLog),
+  (catchUpLog.match(/catch-up replay finished:[^\n]*/) ?? [""])[0].slice(0, 50),
+);
+// The event should now exist for our connection.
+const recovered = (await eventsForConnection()).filter(
+  (e) => e.created_at > new Date(t4 - 3000).toISOString(),
+);
+record(
+  "the event missed during the outage now exists",
+  recovered.length > 0,
+  `${recovered.length} event(s) created after the outage began`,
+);
 
 // ======================================================== 5. crash recovery
 const t5 = Date.now();

src/catchup.ts

@@ -34,14 +34,94 @@ export interface CatchUpQueryParams {
   sourceId?: string;
 }
 
+/** What Hookdeck reported about a replay batch once it settled. */
+export interface BatchOutcome {
+  /** True once Hookdeck reported the batch finished. */
+  complete: boolean;
+  replayed?: number;
+  planned?: number;
+  /** Set when the outcome could not be established. */
+  unknown?: string;
+}
+
+/**
+ * Waits for a bulk replay to finish, within a bound.
+ *
+ * Bounded because this runs when a tunnel reconnects: a batch that never
+ * completes must not hold the transport's recovery path open indefinitely. A
+ * timeout is reported as unknown rather than as success.
+ */
+async function waitForBatch(
+  options: CatchUpOptions,
+  batchId: string,
+): Promise<BatchOutcome> {
+  const deadline =
+    (options.now ?? Date.now)() + (options.verifyTimeoutMs ?? 30_000);
+  const sleep =
+    options.sleep ?? ((ms: number) => new Promise((r) => setTimeout(r, ms)));
+
+  while ((options.now ?? Date.now)() < deadline) {
+    const batch = await options.client.getBulkReplay(batchId);
+    if (!batch.ok) {
+      options.logger.warn(
+        `catch-up replay was requested but its progress could not be read: ${batch.message}`,
+      );
+      return { complete: false, unknown: batch.message };
+    }
+
+    if (batch.data.completed_at != null || batch.data.in_progress === false) {
+      const replayed = batch.data.completed_count;
+      const planned = batch.data.estimated_count;
+      const short =
+        replayed !== undefined && planned !== undefined && replayed < planned;
+
+      const line = `catch-up replay finished: ${replayed ?? "?"} of ${planned ?? "?"} request(s) replayed`;
+      if (short) {
+        options.logger.warn(
+          `${line}. The shortfall arrived during the outage and was not recovered — ` +
+            `replay it explicitly, or check it is still within Hookdeck's retention.`,
+        );
+      } else {
+        options.logger.info(line);
+      }
+
+      return {
+        complete: true,
+        ...(replayed !== undefined ? { replayed } : {}),
+        ...(planned !== undefined ? { planned } : {}),
+      };
+    }
+    await sleep(1000);
+  }
+
+  // Not an error, but not a success either: saying which is the whole point.
+  options.logger.warn(
+    `catch-up replay ${batchId} had not finished within the wait, so whether every ` +
+      `missed request was recovered is unknown. Check the batch in the Hookdeck dashboard.`,
+  );
+  return { complete: false, unknown: "timed out waiting for the batch" };
+}
+
 export function buildCatchUpQuery(
   params: CatchUpQueryParams,
 ): Record<string, unknown> {
   return {
-    // Requests that produced no CLI event and at least one ignored event: the
-    // signature of "arrived while nothing was listening".
-    cli_events_count: 0,
-    ignored_count: { gte: 1 },
+    /**
+     * Requests that produced NO EVENT AT ALL — the only filter that catches
+     * every way a request can be stranded.
+     *
+     * Measured against a live project, because the two disconnect regimes look
+     * different and only one of them was covered before:
+     *
+     *   tunnel never existed        events_count 0, ignored_count 1
+     *   tunnel connected then died  events_count 0, ignored_count 0
+     *
+     * The second is the hard-crash case, and it matches neither
+     * `ignored_count >= 1` nor `cli_events_count: 0` — that field is not
+     * populated when no CLI event was ever created, so filtering on it
+     * excludes precisely the case catch-up exists for.
+     */
+    events_count: 0,
     ingested_at: {
       gte: new Date(params.sinceMs).toISOString(),
       ...(params.untilMs !== undefined
@@ -61,12 +141,24 @@ export interface CatchUpOptions {
   sourceId?: string;
   /** Below this, an outage is not worth a bulk operation. */
   minGapMs?: number;
+  /** Set false to skip waiting for the batch and the post-replay check. */
+  verify?: boolean;
+  /** How long to wait for the batch before reporting the outcome unknown. */
+  verifyTimeoutMs?: number;
+  /** Injectable for tests, so waiting costs no wall-clock. */
+  sleep?(ms: number): Promise<void>;
   now?(): number;
 }
 
 export type CatchUpResult =
   | { ran: false; reason: "gap_too_small" | "no_connection" }
-  | { ran: true; batchId?: string; estimated?: number }
+  | {
+      ran: true;
+      batchId?: string;
+      estimated?: number;
+      /** What Hookdeck reported once the batch finished. */
+      recovered?: BatchOutcome;
+    }
   | { ran: false; reason: "failed"; message: string };
 
 export const DEFAULT_MIN_GAP_MS = 30_000;
@@ -109,11 +201,25 @@ export async function runCatchUp(
     return { ran: false, reason: "failed", message: result.message };
   }
 
+  // Waited for, because the replay call returns as soon as the batch is
+  // accepted. The batch's own counts are the evidence of recovery: a replay
+  // re-ingests each request as a NEW request with new events, leaving the
+  // original at events_count 0 forever — so re-reading the window can never
+  // show recovery, however long you wait for it.
+  const batch =
+    result.data.id !== undefined && options.verify !== false
+      ? await waitForBatch(options, result.data.id)
+      : undefined;
+
+  // Checked, not assumed. A bulk replay reports what it queued rather than
+  // what it recovered, so without this the only honest claim would be "a
+  // replay was requested".
   return {
     ran: true,
     ...(result.data.id !== undefined ? { batchId: result.data.id } : {}),
     ...(result.data.estimated_count !== undefined
       ? { estimated: result.data.estimated_count }
       : {}),
+    ...(batch !== undefined ? { recovered: batch } : {}),
   };
 }

src/hookdeck/client.ts

@@ -230,10 +230,31 @@ export interface HookdeckClient {
         id: string;
         verified?: boolean | null;
         rejection_cause?: string | null;
+        /** Zero means the request produced no delivery to anything. */
+        events_count?: number | null;
+        ingested_at?: string;
       }[]
     >
   >;
 
+  /**
+   * Progress of a bulk replay.
+   *
+   * The replay call returns as soon as the batch is accepted, so its
+   * `estimated_count` is a plan rather than a result. Only `completed_at`
+   * says the work is done, which is the difference between "a replay was
+   * requested" and "the events exist".
+   */
+  getBulkReplay(id: string): Promise<
+    ApiResult<{
+      id: string;
+      completed_at?: string | null;
+      in_progress?: boolean;
+      estimated_count?: number;
+      completed_count?: number;
+    }>
+  >;
+
   bulkReplayRequests(params: {
     query: Record<string, unknown>;
     target: { webhook_ids?: string[]; source_id?: string };
@@ -492,6 +513,10 @@ export function createHookdeckClient(
       return result.ok ? { ok: true, data: result.data.models ?? [] } : result;
     },
 
+    async getBulkReplay(id) {
+      return request("GET", `/bulk/requests/replay/${encodeURIComponent(id)}`);
+    },
+
     async bulkReplayRequests(params) {
       // `target` goes INSIDE `query`, and is required there. Sending it at the
       // top level is answered `422 query.target is required` — the whole call