fix: a hard crash left no record of the outage, so catch-up skipped it

Found by running the plugin end to end against a real Hookdeck project —
real source, real tunnel supervised by the plugin, real events, real
retries. `npm run test:e2e` is that harness; ten scenarios, all passing,
with every resource named openclaw-e2e-* and deleted afterwards.

The defect it found: `lastDisconnectAt` is written when the tunnel child
exits, which needs this process alive to notice. A kill -9 takes the
handler with it, so the outage most in need of catch-up left no record
and catch-up skipped it. Confirmed directly: the cursor file is unchanged
across a SIGKILL.

The transport now records a liveness marker while running and clears it
on a clean shutdown, so finding one at startup means the previous process
died without stopping, and its timestamp becomes the start of the outage.
Over-shooting by a heartbeat is harmless: the catch-up query matches only
requests that produced no event, so anything delivered is excluded by
construction. With the fix, a killed gateway now issues its replay on
reconnect.

Two facts the run established that were previously assumed:

Hookdeck rejects an unparseable JSON body AT THE EDGE — 400 with
rejection_cause UNPARSABLE_JSON, no event created. The plugin's
malformed_json handling is therefore defence in depth rather than a path
real traffic takes, and the docs now say so instead of implying it is
load-bearing.

A CLI destination accepts CUSTOM_SIGNATURE, and Hookdeck signs it with
byte-identical HMAC-SHA256/base64 in the same header as its own. That is
what makes this harness possible at all: the project signing secret is
dashboard-only, so the destination is configured with a secret we choose
and the verification path runs unchanged.

Also proven end to end rather than against fakes: a signed delivery
verified and dispatched with a matching ledger row; a manual retry
admitted rather than suppressed; an event arriving with no tunnel
stranded rather than lost; doctor's project comparison against the real
project; and doctor reading `verified` off real inbound requests.

662 tests.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>

Author: garethx

README.md

@@ -148,6 +148,7 @@ The README covers the common path. Everything else lives in [`docs/`](docs/):
 openclaw plugins install --link ./hookdeck-openclaw
 npm test           # no Gateway or Hookdeck account required
 npm run test:package   # loads the packed tarball in a real Gateway
+npm run test:e2e       # real project, real tunnel, real events (needs HOOKDECK_TEST_API_KEY)
 ```
 
 ## Learn more

docs/durability.md

@@ -23,3 +23,15 @@ Without `apiKey`, orphans are still detected, settled and dead-lettered — they
 `safety.allowRetryCancel` lets the plugin answer `Retry-After: -1` on permanently-invalid input — malformed JSON, a body that will never fit — which tells Hookdeck to stop retrying instead of burning all 50 attempts on something that cannot succeed.
 
 **It is off by default, and that default is deliberate.** A mistake here discards real traffic, and the events are gone once retention lapses (3 days on the free plan). Cancellation is only ever emitted from a closed allowlist of reasons, always dead-letters first, and never fires for anything a config change could fix — a missing secret, an unresolvable secretRef or a storage failure all stay retryable. Turn it on once you have watched the logs and seen what it *would* have cancelled.
+
+## A crash that never ran its shutdown
+
+`lastDisconnectAt` is written when the tunnel's child process exits, which requires this process to be alive to notice. A `kill -9`, an OOM kill or a power cut takes that handler with it, so the outage most in need of catch-up would leave no record of itself at all — and catch-up, finding no disconnect, would skip it.
+
+So while the transport is running the plugin records a liveness marker every `catchUp.heartbeatSeconds` (30 by default), and clears it on a clean shutdown. Finding one at startup means the previous process died without stopping, and its timestamp becomes the start of the outage window.
+
+Over-shooting that window by up to one heartbeat is harmless: the catch-up query matches only requests that produced no event at all, so anything that did deliver is excluded by construction.
+
+## Malformed bodies never reach the plugin
+
+Verified end to end: Hookdeck rejects an unparseable JSON body **at the edge**, answering the sender `400` with `rejection_cause: UNPARSABLE_JSON` and creating no event. The plugin's own `malformed_json` handling is therefore defence in depth rather than a path real traffic takes — it covers a body that survives the edge and fails here, such as one that is valid JSON but not valid UTF-8.

openclaw.plugin.json

@@ -105,6 +105,11 @@
             "maximum": 86400,
             "default": 30,
             "description": "Below this, an outage is not worth a bulk replay."
+          },
+          "heartbeatSeconds": {
+            "type": "integer",
+            "default": 30,
+            "description": "How often to record that the gateway is alive, so a crash that never ran its shutdown is still detectable as an outage."
           }
         }
       },

package.json

@@ -48,7 +48,8 @@
     "typecheck": "tsc --noEmit",
     "test:live": "vitest run test/live",
     "test:agent": "bash scripts/agent-smoke.sh",
-    "test:package": "bash scripts/test-package.sh"
+    "test:package": "bash scripts/test-package.sh",
+    "test:e2e": "node scripts/e2e-live.mjs . $(date +%s | tail -c 6)"
   },
   "homepage": "https://github.com/hookdeck/hookdeck-openclaw#readme",
   "bugs": {

scripts/e2e-live.mjs

@@ -0,0 +1,273 @@
+/**
+ * End-to-end against a real Hookdeck project.
+ *
+ * Real source, real tunnel supervised by the plugin itself, real events, real
+ * retries. The one thing not real is the signing secret: the project's own is
+ * dashboard-only, so the destination is configured with CUSTOM_SIGNATURE and a
+ * secret we choose. Verified beforehand to be byte-identical HMAC-SHA256/base64
+ * in the same header, so the verification path is exercised unchanged.
+ *
+ * Everything is named `openclaw-e2e-<run>` and deleted at the end.
+ */
+import { spawn, execFile } from "node:child_process";
+import { readFileSync, writeFileSync, mkdirSync, rmSync, existsSync } from "node:fs";
+import { promisify } from "node:util";
+
+const execFileAsync = promisify(execFile);
+const REPO = process.argv[2];
+const RUN = process.argv[3];
+const NAME = `openclaw-e2e-${RUN}`;
+const SECRET = `whsec_e2e_${RUN}`;
+const PORT = 18871;
+const ROOT = `/tmp/openclaw-e2e-${RUN}`;
+const KEY = /^HOOKDECK_TEST_API_KEY=(.+)$/m.exec(
+  readFileSync(`${REPO}/.env.local`, "utf8"),
+)[1].trim();
+
+const results = [];
+const record = (name, pass, detail) => {
+  results.push({ name, pass, detail });
+  console.log(`${pass ? "PASS" : "FAIL"}  ${name}${detail ? ` — ${detail}` : ""}`);
+};
+const sleep = (ms) => new Promise((r) => setTimeout(r, ms));
+
+const api = async (method, path, body) => {
+  const r = await fetch(`https://api.hookdeck.com/2025-07-01${path}`, {
+    method,
+    headers: { authorization: `Bearer ${KEY}`, "content-type": "application/json" },
+    ...(body ? { body: JSON.stringify(body) } : {}),
+  });
+  const text = await r.text();
+  return { status: r.status, body: text ? JSON.parse(text) : {} };
+};
+
+// ---------------------------------------------------------------- provision
+const conn = await api("PUT", "/connections", {
+  name: NAME,
+  source: { name: NAME, type: "WEBHOOK" },
+  destination: {
+    name: NAME,
+    type: "CLI",
+    config: {
+      path: "/hookdeck/e2e",
+      path_forwarding_disabled: true,
+      auth_type: "CUSTOM_SIGNATURE",
+      auth: { key: "x-hookdeck-signature", signing_secret: SECRET },
+    },
+  },
+  rules: [{ type: "retry", strategy: "linear", count: 3, interval: 10000, response_status_codes: ["500-599", "429", "408"] }],
+});
+if (!conn.body.id) {
+  console.log("PROVISION FAILED", conn.status, JSON.stringify(conn.body).slice(0, 300));
+  process.exit(1);
+}
+const CONNECTION_ID = conn.body.id;
+const SOURCE_URL = (await api("GET", `/sources?name=${NAME}`)).body.models[0].url;
+console.log(`provisioned ${CONNECTION_ID}  ${SOURCE_URL}\n`);
+
+// ------------------------------------------------------------------ gateway
+rmSync(ROOT, { recursive: true, force: true });
+mkdirSync(`${ROOT}/state`, { recursive: true });
+const config = {
+  gateway: { mode: "local", bind: "loopback", port: PORT },
+  plugins: {
+    load: { paths: [REPO] },
+    entries: {
+      hookdeck: {
+        enabled: true,
+        config: {
+          signingSecret: SECRET,
+          apiKey: KEY,
+          ingress: { basePath: "/hookdeck" },
+          maxConcurrent: 4,
+          transport: { mode: "cli", port: PORT, binaryPath: "/usr/local/bin/hookdeck" },
+          provisioning: { enabled: false },
+          catchUp: { enabled: true, minGapSeconds: 1 },
+          routes: {
+            e2e: {
+              source: NAME,
+              path: "/e2e",
+              connectionId: CONNECTION_ID,
+              dispatch: { mode: "wake", sessionKey: "main", text: "E2E {eventId}" },
+            },
+          },
+        },
+      },
+    },
+  },
+};
+writeFileSync(`${ROOT}/openclaw.json`, JSON.stringify(config, null, 2));
+
+let gw;
+const startGateway = async (label) => {
+  gw = spawn(`${REPO}/node_modules/.bin/openclaw`, ["gateway", "--allow-unconfigured"], {
+    env: { ...process.env, OPENCLAW_CONFIG_PATH: `${ROOT}/openclaw.json`, OPENCLAW_STATE_DIR: `${ROOT}/state` },
+    stdio: ["ignore", "pipe", "pipe"],
+  });
+  const log = [];
+  gw.stdout.on("data", (d) => log.push(String(d)));
+  gw.stderr.on("data", (d) => log.push(String(d)));
+  gw.log = log;
+  for (let i = 0; i < 40; i += 1) {
+    await sleep(1000);
+    if (log.join("").includes("tunnel connected")) break;
+  }
+  console.log(`  [${label}] gateway up, tunnel ${log.join("").includes("tunnel connected") ? "connected" : "NOT connected"}`);
+  return log;
+};
+
+const send = async (payload) =>
+  (await fetch(SOURCE_URL, {
+    method: "POST",
+    headers: { "content-type": "application/json" },
+    body: typeof payload === "string" ? payload : JSON.stringify(payload),
+  })).status;
+
+const ledger = () => {
+  const f = `${ROOT}/state/hookdeck/ledger.jsonl`;
+  if (!existsSync(f)) return [];
+  // Last-write-wins, as the store does. The append-only log keeps superseded
+  // rows, so reading it raw counts a settled row as still running.
+  const byId = new Map();
+  for (const line of readFileSync(f, "utf8").trim().split("\n").filter(Boolean)) {
+    const row = JSON.parse(line).d;
+    byId.set(row.eventId, row);
+  }
+  return [...byId.values()];
+};
+const rowFor = (id) => ledger().filter((r) => r.eventId === id).at(-1);
+
+const eventsForConnection = async () =>
+  (await api("GET", `/events?webhook_id=${CONNECTION_ID}&limit=50`)).body.models ?? [];
+
+// =========================================================== 1. core delivery
+let log = await startGateway("boot");
+const t1 = Date.now();
+await send({ scenario: "core", run: RUN });
+await sleep(8000);
+
+let evs = await eventsForConnection();
+const core = evs.find((e) => e.created_at > new Date(t1 - 5000).toISOString());
+record(
+  "a real signed delivery is verified and dispatched",
+  core?.status === "SUCCESSFUL" && core?.response_status === 200,
+  `hookdeck says ${core?.status}/${core?.response_status}`,
+);
+record(
+  "the ledger records it as succeeded",
+  rowFor(core?.id)?.status === "succeeded",
+  `row=${JSON.stringify(rowFor(core?.id) ?? null)?.slice(0, 90)}`,
+);
+
+// ====================================================== 2. manual retry (MANUAL)
+await api("POST", `/events/${core.id}/retry`);
+await sleep(8000);
+const afterRetry = rowFor(core.id);
+record(
+  "a manual retry is admitted, not suppressed as a duplicate",
+  afterRetry?.attempt >= 2 && afterRetry?.runCount >= 2,
+  `attempt=${afterRetry?.attempt} runs=${afterRetry?.runCount}`,
+);
+
+// ============================================================ 3. malformed body
+// Hookdeck screens this at the edge — 400 UNPARSABLE_JSON, no event created —
+// so the plugin's own malformed_json handling is never reached by a JSON-typed
+// source. Asserting what actually happens rather than what we assumed.
+const t3 = Date.now();
+const edge = await send("{not json");
+await sleep(6000);
+const reqs = (await api("GET", `/requests?limit=10`)).body.models ?? [];
+const rejected = reqs.find(
+  (r) => r.ingested_at > new Date(t3 - 3000).toISOString() && r.rejection_cause,
+);
+record(
+  "Hookdeck rejects a malformed body at the edge, before any delivery",
+  edge === 400 && rejected?.rejection_cause === "UNPARSABLE_JSON",
+  `edge answered ${edge}, request rejection_cause=${rejected?.rejection_cause}`,
+);
+record(
+  "and creates no event, so nothing reaches the gateway",
+  (rejected?.events_count ?? 0) === 0,
+  `events_count=${rejected?.events_count}`,
+);
+
+// ================================================= 4. catch-up after an outage
+// Kill the Gateway so the tunnel drops, send while nothing is listening, restart.
+gw.kill("SIGKILL");
+await sleep(3000);
+const t4 = Date.now();
+await send({ scenario: "catchup", run: RUN });
+await sleep(6000);
+const duringOutage = (await api("GET", `/requests?limit=20`)).body.models ?? [];
+const stranded = duringOutage.find(
+  (r) => r.ingested_at > new Date(t4 - 3000).toISOString() && (r.events_count ?? 0) === 0,
+);
+record(
+  "an event arriving with no tunnel is stranded rather than delivered",
+  stranded !== undefined,
+  stranded ? `request ${stranded.id} produced ${stranded.events_count} events` : "no stranded request found",
+);
+
+log = await startGateway("restart");
+await sleep(20000);
+const catchUpLog = log.join("");
+record(
+  "catch-up replay is issued on reconnect",
+  /catch-up replay queued/.test(catchUpLog),
+  (catchUpLog.match(/catch-up replay queued[^\n]*/) ?? ["not in log"])[0].slice(0, 80),
+);
+
+// ======================================================== 5. crash recovery
+const t5 = Date.now();
+await send({ scenario: "crash", run: RUN });
+await sleep(2500);
+gw.kill("SIGKILL"); // mid-flight, before the row can settle
+await sleep(2000);
+const beforeRestart = ledger().filter((r) => r.status === "running");
+log = await startGateway("recovery");
+await sleep(8000);
+const recoveryLog = log.join("");
+record(
+  "a crash mid-dispatch leaves a running row that recovery settles",
+  beforeRestart.length === 0 || /reconciling \d+ interrupted/.test(recoveryLog),
+  beforeRestart.length === 0
+    ? "no row was left running (dispatch completed first)"
+    : (recoveryLog.match(/reconciling [^\n]*/) ?? ["not reconciled"])[0].slice(0, 70),
+);
+
+// ============================================================== 6. the doctor
+const doctor = await execFileAsync(
+  `${REPO}/node_modules/.bin/openclaw`,
+  ["agent", "--local", "--session-key", "e2e", "-m", "Call hookdeck_doctor and print its raw JSON.", "--model", "anthropic/claude-haiku-4-5"],
+  { env: { ...process.env, OPENCLAW_CONFIG_PATH: `${ROOT}/openclaw.json`, OPENCLAW_STATE_DIR: `${ROOT}/state`,
+           ANTHROPIC_API_KEY: (/^AGENT_TEST_ANTHROPIC_API_KEY=(.+)$/m.exec(readFileSync(`${REPO}/.env.local`, "utf8")) ?? [,""])[1]?.trim() },
+    maxBuffer: 10 * 1024 * 1024 },
+).catch((e) => ({ stdout: String(e.stdout ?? e.message) }));
+const doctorOut = doctor.stdout ?? "";
+record(
+  "doctor reports the project match against the real project",
+  /both tm_/.test(doctorOut) || /cli\/api-key project/.test(doctorOut),
+  (doctorOut.match(/both tm_[A-Za-z0-9]+/) ?? ["see output"])[0],
+);
+record(
+  "doctor reports provider verification from real inbound requests",
+  /provider verification/i.test(doctorOut),
+  (doctorOut.match(/UNVERIFIED[^"]{0,60}/) ?? doctorOut.match(/were verified[^"]{0,20}/) ?? ["not reported"])[0].slice(0, 70),
+);
+
+// ================================================================== teardown
+gw?.kill("SIGKILL");
+console.log("\n--- teardown ---");
+for (const kind of ["connections", "sources", "destinations"]) {
+  const list = (await api("GET", `/${kind}?limit=255`)).body.models ?? [];
+  for (const item of list.filter((m) => String(m.name).startsWith("openclaw-e2e-"))) {
+    const d = await api("DELETE", `/${kind}/${item.id}`);
+    console.log(`  deleted ${kind}/${item.id} (${item.name}) ${d.status}`);
+  }
+}
+rmSync(ROOT, { recursive: true, force: true });
+
+console.log("\n=============================================");
+console.log(`${results.filter((r) => r.pass).length}/${results.length} scenarios passed`);
+for (const r of results.filter((r) => !r.pass)) console.log(`  FAILED: ${r.name} — ${r.detail}`);
+process.exit(0);

src/hookdeck/client.ts

@@ -224,10 +224,7 @@ export interface HookdeckClient {
    * over the API — the secret is never returned — so `verified` on the requests
    * that actually arrived is the only signal there is.
    */
-  listRequests(params?: {
-    limit?: number;
-    sourceId?: string;
-  }): Promise<
+  listRequests(params?: { limit?: number; sourceId?: string }): Promise<
     ApiResult<
       {
         id: string;

src/plugin/config-parse.ts

@@ -134,8 +134,9 @@ const configSchema = z.object({
     .object({
       enabled: z.boolean().default(true),
       minGapSeconds: z.number().int().nonnegative().max(86_400).default(30),
+      heartbeatSeconds: z.number().int().positive().max(3600).default(30),
     })
-    .default({ enabled: true, minGapSeconds: 30 }),
+    .default({ enabled: true, minGapSeconds: 30, heartbeatSeconds: 30 }),
   ingress: z
     .object({ basePath: z.string().min(1).default("/hookdeck") })
     .default({ basePath: "/hookdeck" }),
@@ -413,6 +414,7 @@ export function parseHookdeckConfig(raw: unknown): ConfigParseResult {
     catchUp: {
       enabled: value.catchUp.enabled,
       minGapSeconds: value.catchUp.minGapSeconds,
+      heartbeatSeconds: value.catchUp.heartbeatSeconds,
     },
     safety: { allowRetryCancel: value.safety.allowRetryCancel },
     routes,

src/plugin/config-types.ts

@@ -189,6 +189,8 @@ export interface CatchUpConfig {
   enabled: boolean;
   /** Below this, an outage is not worth a bulk replay. */
   minGapSeconds: number;
+  /** How often to record that this process is alive, so a crash is detectable. */
+  heartbeatSeconds: number;
 }
 
 export interface HookdeckPluginConfig {

src/store/cursor-store.ts

@@ -35,6 +35,16 @@ export interface CursorRecord {
    * treated as `shutdown` to preserve the original behaviour.
    */
   pauseReason?: "shutdown" | "operator";
+  /**
+   * Last time this process was known to be running and forwarding.
+   *
+   * Written periodically and cleared on a clean shutdown, so finding one at
+   * startup means the previous process died without running its teardown —
+   * a `kill -9`, an OOM kill, a power cut. That is the outage most in need of
+   * catch-up, and the one where nothing else records a disconnect, because the
+   * handler that would have done so died with the process.
+   */
+  lastSeenAt?: number;
   provisioningFingerprint?: string;
   connectionId?: string;
   updatedAt: number;