feat: diagnose a CLI/API-key project mismatch, escalate a dead tunnel

Applying the findings from the Hermes plugin's reviews, which faces the
same Hookdeck-shaped problems. Each was checked against this codebase
first rather than ported on faith; three applied.

The big one: in `cli` transport, "which project" has two independent
answers. Provisioning acts on the API key's project while `hookdeck
listen` looks for that connection in the CLI session's. Nothing
reconciled them, and this plugin deliberately will not force them
together — that means `hookdeck ci`, which rewrites the CLI's global
config and switches the active project for everything else on the
machine. So the split stays and is now visible instead:

- `hookdeck_doctor` compares the two, reading the CLI's project from its
  config file and the key's from `team_id` on any connection it can
  reach. A CLI with no session, or a key reaching no connections, is
  reported as unverified rather than as a mismatch: a project can
  legitimately be empty, and a missing session is a different failure
  with a different fix.
- The supervisor escalates. A tunnel that cannot stay up restarted
  forever at warn level, which reads as routine churn while nothing
  reaches the Gateway — and `maxConsecutiveFailures` was never set, so it
  never even reached the giving-up log. Three consecutive failed runs now
  log once, with the CLI's own last lines and, where recognisable, the
  cause. `no connection found matching filter` names the project
  mismatch; an auth failure names `hookdeck login`. A healthy run re-arms
  it.

Also `projectId`, sent as `X-Team-Id`. A project-scoped key implies its
project so this is optional today, but an organisation-scoped one does
not, and without it a call can act on whichever project holds a
same-named resource.

Checked and NOT applicable: a malformed signature answering 500 (ours
refuses cleanly and returns 401 — seven malformed inputs verified,
including non-ASCII and 100KB; and an unexpected throw is a deliberate
503, never a 500); state cleared only on success (in-flight releases in a
`finally`, auto-resume timers are cancelled globally); ambient
`HOOKDECK_*` env reads (already config-driven by an earlier fix).

`markHealthy` now returns whether it reset, because the supervisor needs
"was that run a success?" and the failure count has not yet been
incremented at that point — reading it there treated the first failure as
a success.

Verified live: the CLI's project and the API key's project both read
correctly on this machine and agree.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>

Author: garethx

docs/configuration.md

@@ -6,6 +6,8 @@ Every setting, its default, and what it changes.
 |---|---|---|
 | `headerPrefix` | `x-hookdeck` | Hookdeck's header prefix is white-labelable per project. Set it if yours differs. |
 | `signingSecret` | — | Inline string or a secretRef `{source, provider, id}`. Routes may override. Re-resolved on every request, so rotation needs no restart. |
+| `projectId` | — | Pins API calls to one project via `X-Team-Id`. Optional for a project-scoped key, which implies its project; needed for an organisation-scoped one. |
+| `transport.cliConfigPath` | `~/.config/hookdeck/config.toml` | Where the Hookdeck CLI keeps its session. Read by `hookdeck_doctor` to check both point at the same project. |
 | `apiKey` | — | Optional. Needed for provisioning, pause/resume, replay, issue management and re-queuing interrupted work. Without it the plugin runs ingress-only. |
 | `storage.enabled` | `true` | Persist the ledger and dead-letter log. Off means memory-only — see [Durability](#durability-and-recovery). |
 | `storage.deadLetterMaxEntries` | `500` | Dead-letter entries kept before the oldest are dropped. |

docs/transport.md

@@ -32,3 +32,21 @@ On shutdown the connection is **paused before** the listener is stopped. That or
 The `pausedByUs` marker is written *before* the pause call, so a crash in between still leaves the breadcrumb that unpauses on the next start — a connection left paused forever is a silent outage.
 
 `lastDisconnectAt` is written on **every** listener exit, clean or otherwise. It is the only durable evidence of an outage window, and the catch-up replay needs it to bound its query: `bulk/requests/replay` is the only path that can be time-scoped, since `bulk/ignored-events/retry` takes no date filter and there is no project-wide `GET /ignored-events` to enumerate with.
+
+## The two projects problem
+
+In `cli` transport, "which Hookdeck project" has two independent answers:
+
+- `hookdeck_setup` and every API call act on the project the **API key** belongs to.
+- `hookdeck listen` looks for that connection in the project the **CLI session** is logged into.
+
+Nothing reconciles them, and the plugin deliberately does not try: forcing them together means `hookdeck ci`, which rewrites the CLI's global config and switches the active project for every other use on the machine.
+
+When they differ, the failure is quiet in the worst way. The Gateway starts, logs that the transport is up, and receives nothing — the tunnel restart-loops on `no connection found matching filter` while every event becomes an ignored `CLI_DISCONNECTED`.
+
+Two things make it visible rather than silent:
+
+- **`hookdeck_doctor` compares them.** It reads the CLI's project from its config file and the API key's from `team_id` on any connection the key can reach, and fails with both ids and the fix. A CLI with no session, or a key that reaches no connections, is reported as unverified rather than as a mismatch — a project can legitimately be empty, and a missing session is its own separate failure.
+- **The supervisor escalates a standing failure.** After three consecutive runs that fail to stay up, it logs once — not once per restart — with the CLI's own last lines and, where the output is recognisable, the likely cause. A healthy run re-arms it, so a second outage is not silent.
+
+If your API key is organisation-scoped rather than project-scoped, set `projectId` as well: without it a call can act on whichever project happens to hold a resource of the same name.

index.ts

@@ -315,7 +315,14 @@ export default definePluginEntry({
             logger: log,
             ...(keyConfiguredButUnresolvable ? { apiKeyUnresolved: true } : {}),
             ...(apiKey !== undefined
-              ? { client: createHookdeckClient({ apiKey }) }
+              ? {
+                  client: createHookdeckClient({
+                    apiKey,
+                    ...(config.projectId !== undefined
+                      ? { projectId: config.projectId }
+                      : {}),
+                  }),
+                }
               : {}),
             configWarnings: () => parsed.warnings,
           };
@@ -461,7 +468,14 @@ export default definePluginEntry({
           return undefined;
         });
         const client =
-          apiKey !== undefined ? createHookdeckClient({ apiKey }) : undefined;
+          apiKey !== undefined
+            ? createHookdeckClient({
+                apiKey,
+                ...(config.projectId !== undefined
+                  ? { projectId: config.projectId }
+                  : {}),
+              })
+            : undefined;
 
         // Before serving anything: hand interrupted work back to Hookdeck.
         const summary = await reconcileOrphans({

openclaw.plugin.json

@@ -170,6 +170,10 @@
           }
         }
       },
+      "projectId": {
+        "type": "string",
+        "description": "Pins API calls to one Hookdeck project via X-Team-Id. Optional for a project-scoped API key, which implies its project; needed for an organisation-scoped one."
+      },
       "provisioning": {
         "type": "object",
         "additionalProperties": false,
@@ -621,6 +625,10 @@
             "type": "string",
             "format": "uri",
             "description": "Public base URL of the Gateway, required for http transport provisioning."
+          },
+          "cliConfigPath": {
+            "type": "string",
+            "description": "Where the Hookdeck CLI keeps its session, used by hookdeck_doctor to check that the CLI and the API key point at the same project. Defaults to ~/.config/hookdeck/config.toml."
           }
         }
       }

src/hookdeck/client.ts

@@ -19,6 +19,15 @@ export type FetchLike = (
 
 export interface HookdeckClientOptions {
   apiKey: string;
+  /**
+   * Pins every call to one project, via the same `X-Team-Id` header the
+   * Hookdeck CLI sends.
+   *
+   * A project-scoped key implies its project, so this is optional today. An
+   * organisation-scoped key does not, and without pinning it would act on
+   * whichever project happens to hold a matching resource name.
+   */
+  projectId?: string | undefined;
   baseUrl?: string;
   fetch?: FetchLike;
   timeoutMs?: number;
@@ -37,6 +46,8 @@ export type ApiResult<T> =
 
 export interface HookdeckConnection {
   id: string;
+  /** The project this connection belongs to; how we learn the key's project. */
+  team_id?: string;
   /** How a person refers to it. Issues carry only the id. */
   name?: string;
   paused_at?: string | null;
@@ -125,6 +136,8 @@ export interface HookdeckClient {
 
   getConnection(id: string): Promise<ApiResult<HookdeckConnection>>;
 
+  listConnections(limit?: number): Promise<ApiResult<HookdeckConnection[]>>;
+
   /**
    * Holds inbound events at status `HOLD` until unpaused, delivered then with
    * attempt trigger `UNPAUSE`. Nothing is dropped.
@@ -229,6 +242,9 @@ export function createHookdeckClient(
         headers: {
           authorization: `Bearer ${options.apiKey}`,
           "content-type": "application/json",
+          ...(options.projectId !== undefined
+            ? { "x-team-id": options.projectId }
+            : {}),
         },
         ...(body !== undefined ? { body: JSON.stringify(body) } : {}),
         signal: controller.signal,
@@ -321,6 +337,14 @@ export function createHookdeckClient(
       );
     },
 
+    async listConnections(limit = 1) {
+      const result = await request<{ models?: HookdeckConnection[] }>(
+        "GET",
+        `/connections?limit=${limit}`,
+      );
+      return result.ok ? { ok: true, data: result.data.models ?? [] } : result;
+    },
+
     async pauseConnection(id) {
       return request<HookdeckConnection>(
         "PUT",

src/plugin/config-parse.ts

@@ -81,6 +81,7 @@ const configSchema = z.object({
   headerPrefix: z.string().min(1).default(DEFAULT_HEADER_PREFIX),
   signingSecret: secretInputSchema.optional(),
   apiKey: secretInputSchema.optional(),
+  projectId: z.string().min(1).optional(),
   tools: z
     .object({ allowMutations: z.boolean().default(true) })
     .default({ allowMutations: true }),
@@ -106,6 +107,7 @@ const configSchema = z.object({
       mode: z.enum(["cli", "http", "none"]).default("none"),
       port: z.number().int().positive().max(65535).default(18789),
       binaryPath: z.string().min(1).default("hookdeck"),
+      cliConfigPath: z.string().min(1).optional(),
       allowUnsupportedVersion: z.boolean().default(false),
       publicUrl: z.string().url().optional(),
     })
@@ -374,6 +376,7 @@ export function parseHookdeckConfig(raw: unknown): ConfigParseResult {
     maxConcurrent: value.maxConcurrent,
     busyRetryAfterSeconds: value.busyRetryAfterSeconds,
     deferAttemptLimit: value.deferAttemptLimit,
+    ...(value.projectId !== undefined ? { projectId: value.projectId } : {}),
     dedupe: { ttlHours: value.dedupe.ttlHours },
     tools: { allowMutations: value.tools.allowMutations },
     storage: {
@@ -388,6 +391,9 @@ export function parseHookdeckConfig(raw: unknown): ConfigParseResult {
       mode: value.transport.mode,
       port: value.transport.port,
       binaryPath: value.transport.binaryPath,
+      ...(value.transport.cliConfigPath !== undefined
+        ? { cliConfigPath: value.transport.cliConfigPath }
+        : {}),
       allowUnsupportedVersion: value.transport.allowUnsupportedVersion,
       ...(value.transport.publicUrl !== undefined
         ? { publicUrl: value.transport.publicUrl }

src/plugin/config-types.ts

@@ -154,6 +154,11 @@ export interface TransportConfig {
   port: number;
   /** Resolved explicitly, because a shadowed binary defeats the version gate. */
   binaryPath: string;
+  /**
+   * Where the Hookdeck CLI keeps its session, for the project-mismatch check.
+   * Defaults to the CLI's own `~/.config/hookdeck/config.toml`.
+   */
+  cliConfigPath?: string;
   /** Downgrade the >=2.4.0 gate to a warning. Sub-2.3.2 silently stops delivering. */
   allowUnsupportedVersion: boolean;
   /** Public base URL of the Gateway, for `http` mode provisioning. */
@@ -197,6 +202,14 @@ export interface HookdeckPluginConfig {
    * cannot be re-queued.
    */
   apiKey?: SecretInput;
+  /**
+   * Pins API calls to one project, via `X-Team-Id`.
+   *
+   * A project-scoped key implies its project, so this is optional. An
+   * organisation-scoped key does not: without it, a call could act on whichever
+   * project happens to hold a resource of the same name.
+   */
+  projectId?: string;
   ingress: {
     /** Route prefix registered on the Gateway. */
     basePath: string;

src/tools/deps.ts

@@ -58,6 +58,8 @@ export interface ToolDeps {
    * someone to fix a config that is already correct.
    */
   apiKeyUnresolved?: boolean;
+  /** Injectable so `doctor` can read the CLI's config in a test. */
+  readFile?: (path: string) => Promise<string>;
   now?(): number;
 }
 

src/tools/doctor.ts

@@ -1,5 +1,9 @@
 import { uncoveredStatuses } from "../hookdeck/provision.js";
 import { RETRYABLE_STATUS_CODES } from "../protocol/outcome.js";
+import {
+  defaultCliConfigPath,
+  readCliProject,
+} from "../transport/cli-project.js";
 import { type ToolDeps } from "./deps.js";
 import { reportedPersistence } from "./status.js";
 
@@ -61,6 +65,15 @@ export async function doctorHandler(deps: ToolDeps) {
     });
   }
 
+  // In `cli` transport, "which project" has two independent answers:
+  // provisioning acts on the API key's project, while `hookdeck listen` looks
+  // for that connection in whichever project the CLI's session points at. When
+  // they differ the Gateway reports healthy and receives nothing, so the only
+  // thing standing between an operator and a silent outage is this check.
+  if (deps.config.transport.mode === "cli") {
+    checks.push(await projectMatchCheck(deps));
+  }
+
   checks.push({
     name: "api key",
     ok: deps.client !== undefined,
@@ -105,3 +118,71 @@ export async function doctorHandler(deps: ToolDeps) {
 
   return { ok: checks.every((c) => c.ok), checks };
 }
+
+interface Check {
+  name: string;
+  ok: boolean;
+  detail: string;
+}
+
+async function projectMatchCheck(deps: ToolDeps): Promise<Check> {
+  const name = "cli/api-key project";
+
+  const cli = await readCliProject(
+    deps.config.transport.cliConfigPath ?? defaultCliConfigPath(),
+    deps.readFile ??
+      (async (p) => (await import("node:fs/promises")).readFile(p, "utf8")),
+  );
+
+  if (cli.projectId === undefined) {
+    // Not a mismatch. A CLI with no session is its own failure — `hookdeck
+    // listen` cannot start at all — and reporting it as a mismatch would point
+    // at the wrong fix.
+    return {
+      name,
+      ok: true,
+      detail:
+        cli.reason === "no_config"
+          ? "unverified — no Hookdeck CLI config found, so the CLI's project is unknown. Run `hookdeck login`."
+          : "unverified — the CLI config holds no project, so no session is logged in. Run `hookdeck login`.",
+    };
+  }
+
+  if (deps.client === undefined) {
+    return {
+      name,
+      ok: true,
+      detail: `unverified — the CLI forwards from ${cli.projectId}, but with no API key there is nothing to compare it against`,
+    };
+  }
+
+  const connections = await deps.client.listConnections(1);
+  if (!connections.ok) {
+    return { name, ok: true, detail: `unverified — ${connections.message}` };
+  }
+
+  const apiProject = connections.data[0]?.team_id;
+  if (apiProject === undefined) {
+    // A project can legitimately be empty, and sending someone to fix that
+    // would be worse than saying nothing.
+    return {
+      name,
+      ok: true,
+      detail: `unverified — the API key reaches no connections, so its project cannot be read. The CLI forwards from ${cli.projectId}.`,
+    };
+  }
+
+  if (apiProject === cli.projectId) {
+    return { name, ok: true, detail: `both ${apiProject}` };
+  }
+
+  return {
+    name,
+    ok: false,
+    detail:
+      `MISMATCH: the CLI forwards from ${cli.projectId} but the API key acts on ${apiProject}. ` +
+      `hookdeck_setup creates connections in the API key's project while \`hookdeck listen\` looks ` +
+      `for them in the CLI's, so the Gateway will report healthy and receive nothing. Point the CLI ` +
+      `at the same project with \`hookdeck login\`, or configure an API key belonging to ${cli.projectId}.`,
+  };
+}

src/transport/backoff.ts

@@ -22,8 +22,13 @@ export interface Backoff {
   readonly failures: number;
   /** Delay for the next restart, or `undefined` once we have given up. */
   next(): number | undefined;
-  /** Called when the child has been connected long enough to count as healthy. */
-  markHealthy(connectedForMs: number): void;
+  /**
+   * Called when the child has been connected long enough to count as healthy.
+   * Returns whether that reset the counter, which is also the answer to "was
+   * that run a success?" — the supervisor needs it to tell a standing failure
+   * from ordinary churn.
+   */
+  markHealthy(connectedForMs: number): boolean;
   reset(): void;
 }
 
@@ -60,7 +65,9 @@ export function createBackoff(options: BackoffOptions = {}): Backoff {
     },
 
     markHealthy(connectedForMs) {
-      if (connectedForMs >= healthyResetMs) failures = 0;
+      if (connectedForMs < healthyResetMs) return false;
+      failures = 0;
+      return true;
     },
 
     reset() {