fix(onboard): announce and recover declared agent forward_ports (#5389)

laitingsheng · web-flow · commit 720bee9fed6e · 2026-06-13T10:33:04.000-07:00
## Summary Hermes onboard declares `forward_ports: [18789, 8642]` but the dashboard summary only printed the primary port and process recovery only re-established the primary forward. After the OpenShell gateway restarted during policy-preset apply, the secondary OpenAI-compatible API forward on port 8642 was silently dropped and never restored. ## Related Issue Fixes #5206 ## Changes - `printDashboardUi` now walks `agent.forward_ports` and emits a labelled block per non-primary entry. - `checkAndRecoverSandboxProcesses` now invokes a new `ensureDeclaredAgentForwardPortsHealthy` helper in all three branches. - Regression tests cover the print output and the recovery loop. ## Type of Change - [x] Code change (feature, bug fix, or refactor) - [ ] Code change with doc updates - [ ] Doc only (prose changes, no code sample modifications) - [ ] Doc only (includes code sample changes) ## Verification - [x] `npx prek run --all-files` passes - [x] `npm test` passes - [x] Tests added or updated for new or changed behavior - [x] No secrets, API keys, or credentials committed - [ ] Docs updated for user-facing behavior changes - [ ] `npm run docs` builds without warnings (doc changes only) - [ ] Doc pages follow the [style guide](https://github.com/NVIDIA/NemoClaw/blob/main/docs/CONTRIBUTING.md) (doc changes only) - [ ] New doc pages include SPDX header and frontmatter (new pages only) --- Signed-off-by: Tinson Lai <tinsonl@nvidia.com>  ## Summary by CodeRabbit * **New Features** * Sandbox recovery now re-establishes secondary agent-declared port forwards in addition to the primary dashboard forward. * Dashboard/onboarding output now shows forwarded OpenAI-compatible API and other secondary endpoint URLs with forwarding notes. * **Tests** * Added tests verifying secondary forwards are detected and re-established when missing and that onboarding prints secondary forward URLs correctly. * **Documentation** * Quickstart and troubleshooting guides updated to show the forwarded OpenAI-compatible API and how to verify/recover missing forwards.  --------- Signed-off-by: Tinson Lai <tinsonl@nvidia.com>
diff --git a/docs/get-started/quickstart-hermes.mdx b/docs/get-started/quickstart-hermes.mdx
@@ -101,9 +101,9 @@ Use the provider variables from [Inference Options](../inference/inference-optio
 
 ## Connect to Hermes
 
-When onboarding completes, NemoClaw prints the sandbox name, model, lifecycle commands, and Hermes dashboard URL.
+When onboarding completes, NemoClaw prints the sandbox name, model, lifecycle commands, the Hermes dashboard URL, and the OpenAI-compatible API URL.
 Hermes exposes its built-in browser dashboard on port `18789`.
-NemoClaw also forwards the OpenAI-compatible API on port `8642` for local clients.
+NemoClaw also forwards the OpenAI-compatible API on port `8642` for local clients, and the summary now announces both URLs.
 NemoClaw builds the Hermes dashboard assets into the sandbox image, so the dashboard starts without running `npm` as the sandbox user under `/opt/hermes`.
 Dashboard chat uses the prebuilt `/opt/hermes/ui-tui` bundle.
 If you need to recover the Hermes dashboard manually, use `hermes dashboard --tui --skip-build` so recovery does not try to rebuild assets under root-owned install paths.
@@ -122,6 +122,10 @@ Access
   Port 18789 must be forwarded before opening this URL.
   http://127.0.0.1:18789/
 
+  Hermes Agent OpenAI-compatible API
+  Port 8642 must be forwarded before connecting.
+  http://127.0.0.1:8642/v1
+
 Terminal:
   nemohermes my-hermes connect
 
diff --git a/docs/reference/troubleshooting.mdx b/docs/reference/troubleshooting.mdx
@@ -1624,6 +1624,20 @@ Expected output:
 Point an OpenAI-compatible client at `http://127.0.0.1:8642/v1` for chat completions.
 For terminal use, run `nemohermes <name> connect` and then `hermes` inside the sandbox.
 
+### `docker port` shows no mapping for 8642 even though forwarding works
+
+OpenShell port forwards are host-side relays managed by the OpenShell gateway process, not Docker `-p` publish mappings on the sandbox container.
+`docker port openshell-hermes-<id>` reflects only Docker-published ports, so it returns nothing for OpenShell-managed forwards even when the host bind is live.
+
+Use OpenShell's own view as the supported acceptance signal:
+
+```bash
+openshell forward list                       # shows the host bind for each forwarded port
+curl -sf http://127.0.0.1:8642/health        # confirms the relayed endpoint answers
+```
+
+If `openshell forward list` does not show port `8642`, run `nemohermes <name> connect --probe-only` (or `nemohermes <name> recover`) to ask the recovery path to re-establish every manifest-declared agent forward port that has gone missing.
+
 ### `nemohermes` reports `Sandbox 'X' already exists as OpenClaw`
 
 Each sandbox name maps to exactly one agent type.
diff --git a/src/lib/actions/sandbox/process-recovery.ts b/src/lib/actions/sandbox/process-recovery.ts
@@ -431,6 +431,60 @@ function ensureHermesDashboardPortForwardIfEnabled(sandboxName: string): boolean
   });
 }
 
+/**
+ * Re-establish every declared `forward_ports` entry on the active agent
+ * manifest that is not already owned by another recovery helper. The
+ * primary dashboard port is owned by `ensureSandboxPortForward`; the
+ * optional Hermes web dashboard port (a registry-recorded per-sandbox
+ * override that the manifest cannot statically declare) is owned by
+ * `ensureHermesDashboardPortForwardIfEnabled`. Skipping both here keeps
+ * the helpers orthogonal and avoids issuing duplicate `forward start`
+ * calls when an operator pins the Hermes dashboard to one of the
+ * manifest-declared ports.
+ *
+ * Without this helper, any remaining manifest-declared port (e.g.
+ * Hermes' OpenAI-compatible API on 8642) would be silently dropped after
+ * a gateway restart and never re-established by the recovery flow.
+ *
+ * Returns true when every covered declared port is healthy (probed or
+ * re-established), false when at least one declared port could not be
+ * re-established, and `null` when there is no active agent or no
+ * declared port left to manage after the skip set is applied.
+ */
+function ensureDeclaredAgentForwardPortsHealthy(
+  sandboxName: string,
+  primaryPort: number,
+): boolean | null {
+  const agent = agentRuntime.getSessionAgent(sandboxName);
+  if (!agent) return null;
+  const declared = (agent as { forward_ports?: unknown }).forward_ports;
+  if (!Array.isArray(declared) || declared.length === 0) return null;
+  const hermesDashboard = getHermesDashboardRecoveryConfig(sandboxName);
+  const skipSet = new Set<number>([primaryPort]);
+  if (hermesDashboard && Number.isInteger(hermesDashboard.publicPort)) {
+    skipSet.add(hermesDashboard.publicPort);
+  }
+  let sawCovered = false;
+  let allHealthy = true;
+  for (const candidate of declared) {
+    if (typeof candidate !== "number") continue;
+    if (!Number.isInteger(candidate) || candidate < 1 || candidate > 65535) continue;
+    if (skipSet.has(candidate)) continue;
+    sawCovered = true;
+    const health = isSandboxPortForwardHealthy(sandboxName, candidate);
+    if (health === true) continue;
+    if (health === "occupied") {
+      allHealthy = false;
+      continue;
+    }
+    if (!ensureSandboxPortForwardForPort(sandboxName, candidate)) {
+      allHealthy = false;
+    }
+  }
+  if (!sawCovered) return null;
+  return allHealthy;
+}
+
 function recoverHermesDashboardProcessIfEnabled(sandboxName: string): boolean | null {
   return recoverHermesDashboardProcess(sandboxName, { executeCommand: executeSandboxCommand });
 }
@@ -466,6 +520,10 @@ export function checkAndRecoverSandboxProcesses(
       }
       const forwardRecovered = ensureSandboxPortForward(sandboxName);
       const dashboardForwardRecovered = ensureHermesDashboardPortForwardIfEnabled(sandboxName);
+      const declaredForwardsRecovered = ensureDeclaredAgentForwardPortsHealthy(
+        sandboxName,
+        recoveryPort,
+      );
       if (!quiet) {
         if (forwardRecovered) {
           console.log(`  ${G}✓${R} Dashboard port forward re-established.`);
@@ -475,6 +533,9 @@ export function checkAndRecoverSandboxProcesses(
             `  Run \`openshell forward start --background <port> ${sandboxName}\` manually.`,
           );
         }
+        if (declaredForwardsRecovered === false) {
+          console.error("  One or more agent-declared port forwards could not be re-established.");
+        }
       }
       return {
         checked: true,
@@ -483,7 +544,8 @@ export function checkAndRecoverSandboxProcesses(
         forwardRecovered:
           forwardRecovered ||
           dashboardForwardRecovered === true ||
-          dashboardProcessRecovered === true,
+          dashboardProcessRecovered === true ||
+          declaredForwardsRecovered === true,
       };
     }
     if (forwardHealthy === "occupied") {
@@ -495,11 +557,21 @@ export function checkAndRecoverSandboxProcesses(
       return { checked: true, wasRunning: true, recovered: false, forwardRecovered: false };
     }
     const dashboardForwardRecovered = ensureHermesDashboardPortForwardIfEnabled(sandboxName);
+    const declaredForwardsRecovered = ensureDeclaredAgentForwardPortsHealthy(
+      sandboxName,
+      recoveryPort,
+    );
+    if (!quiet && declaredForwardsRecovered === false) {
+      console.error("  One or more agent-declared port forwards could not be re-established.");
+    }
     return {
       checked: true,
       wasRunning: true,
       recovered: false,
-      forwardRecovered: dashboardForwardRecovered === true || dashboardProcessRecovered === true,
+      forwardRecovered:
+        dashboardForwardRecovered === true ||
+        dashboardProcessRecovered === true ||
+        declaredForwardsRecovered === true,
     };
   }
 
@@ -530,6 +602,10 @@ export function checkAndRecoverSandboxProcesses(
     }
     const forwardRecovered = ensureSandboxPortForward(sandboxName);
     const dashboardForwardRecovered = ensureHermesDashboardPortForwardIfEnabled(sandboxName);
+    const declaredForwardsRecovered = ensureDeclaredAgentForwardPortsHealthy(
+      sandboxName,
+      recoveryPort,
+    );
     if (!quiet) {
       console.log(
         `  ${G}✓${R} ${agentRuntime.getAgentDisplayName(recoveryAgent)} gateway restarted inside sandbox.`,
@@ -542,12 +618,18 @@ export function checkAndRecoverSandboxProcesses(
           `  Run \`openshell forward start --background <port> ${sandboxName}\` manually.`,
         );
       }
+      if (declaredForwardsRecovered === false) {
+        console.error("  One or more agent-declared port forwards could not be re-established.");
+      }
     }
     return {
       checked: true,
       wasRunning: false,
       recovered,
-      forwardRecovered: forwardRecovered || dashboardForwardRecovered === true,
+      forwardRecovered:
+        forwardRecovered ||
+        dashboardForwardRecovered === true ||
+        declaredForwardsRecovered === true,
     };
   }
   if (!quiet) {
diff --git a/src/lib/agent/onboard.test.ts b/src/lib/agent/onboard.test.ts
@@ -188,6 +188,100 @@ describe("printDashboardUi — regression for #2078 (port 8642 is not a chat UI)
     expect(noteSpy).not.toHaveBeenCalled();
   });
 
+  it("announces manifest-declared secondary forward_ports alongside the primary dashboard", () => {
+    const hermesShipped = makeAgent({
+      name: "hermes",
+      displayName: "Hermes Agent",
+      forwardPort: 18789,
+      forward_ports: [18789, 8642],
+      healthProbe: { url: "http://localhost:8642/health", port: 8642, timeout_seconds: 90 },
+      dashboard: {
+        kind: "ui",
+        label: "Dashboard",
+        path: "/",
+        healthPath: "/api/status",
+        auth: "session",
+      },
+    });
+
+    printDashboardUi("hermes-box", null, hermesShipped, {
+      note: noteSpy,
+      buildControlUiUrls: buildUrlsLoopback,
+    });
+
+    const output = logSpy.mock.calls.map((args) => String(args[0])).join("\n");
+    expect(output).toContain("Hermes Agent Dashboard");
+    expect(output).toContain("Port 18789 must be forwarded before opening this URL.");
+    expect(output).toContain("http://127.0.0.1:18789/");
+    expect(output).toContain("Hermes Agent OpenAI-compatible API");
+    expect(output).toContain("Port 8642 must be forwarded before connecting.");
+    expect(output).toContain("http://127.0.0.1:8642/v1");
+  });
+
+  it("labels a non-health-probe secondary forward port as 'additional port' rooted at /", () => {
+    const dualAgent = makeAgent({
+      name: "experimental",
+      displayName: "Experimental",
+      forwardPort: 18789,
+      forward_ports: [18789, 9100],
+      healthProbe: { url: "http://localhost:18789/health", port: 18789, timeout_seconds: 30 },
+      dashboard: {
+        kind: "ui",
+        label: "Dashboard",
+        path: "/",
+        healthPath: "/health",
+        auth: "session",
+      },
+    });
+
+    printDashboardUi("agent-box", null, dualAgent, {
+      note: noteSpy,
+      buildControlUiUrls: buildUrlsLoopback,
+    });
+
+    const output = logSpy.mock.calls.map((args) => String(args[0])).join("\n");
+    expect(output).toContain("Experimental additional port");
+    expect(output).toContain("Port 9100 must be forwarded before connecting.");
+    expect(output).toContain("http://127.0.0.1:9100/");
+    expect(output).not.toContain("OpenAI-compatible API");
+    expect(output).not.toContain("http://127.0.0.1:9100/v1");
+  });
+
+  it("emits a URL for a secondary forward port that resolves to the scheme default", () => {
+    // Regression: `new URL("http://h:80").port === ""`. A strict equality
+    // filter against String(port) silently drops the URL line. The helper
+    // must normalise scheme-default ports before filtering.
+    const buildUrlsWithDefaultPort = (_token: string | null, port: number): string[] => {
+      if (port === 80) return ["http://127.0.0.1:80/"];
+      return [`http://127.0.0.1:${port}/`];
+    };
+
+    const httpAgent = makeAgent({
+      name: "experimental",
+      displayName: "Experimental",
+      forwardPort: 18789,
+      forward_ports: [18789, 80],
+      healthProbe: { url: "http://localhost:18789/health", port: 18789, timeout_seconds: 30 },
+      dashboard: {
+        kind: "ui",
+        label: "Dashboard",
+        path: "/",
+        healthPath: "/health",
+        auth: "session",
+      },
+    });
+
+    printDashboardUi("agent-box", null, httpAgent, {
+      note: noteSpy,
+      buildControlUiUrls: buildUrlsWithDefaultPort,
+    });
+
+    const output = logSpy.mock.calls.map((args) => String(args[0])).join("\n");
+    expect(output).toContain("Experimental additional port");
+    expect(output).toContain("Port 80 must be forwarded before connecting.");
+    expect(output).toMatch(/http:\/\/127\.0\.0\.1(:80)?\//);
+  });
+
   it("redacts tokenized URLs for UI-kind agents and shows the token retrieval command", () => {
     const token = "a".repeat(64);
     printDashboardUi("sandbox-y", token, uiAgent, {
diff --git a/src/lib/agent/onboard.ts b/src/lib/agent/onboard.ts
@@ -569,6 +569,7 @@ export function printDashboardUi(
       console.log(`  ${dashboardUrlForDisplay(url)}`);
     }
     printOptionalDashboardUi(agent, { ...deps, redactUrl: dashboardUrlForDisplay });
+    printAdditionalForwardPorts(agent, info.port, deps.buildControlUiUrls);
     return;
   }
 
@@ -578,6 +579,8 @@ export function printDashboardUi(
     for (const url of deps.buildControlUiUrls(null, info.port)) {
       console.log(`  ${dashboardUrlForDisplay(url)}`);
     }
+    printOptionalDashboardUi(agent, { ...deps, redactUrl: dashboardUrlForDisplay });
+    printAdditionalForwardPorts(agent, info.port, deps.buildControlUiUrls);
     return;
   }
 
@@ -598,4 +601,90 @@ export function printDashboardUi(
     }
   }
   printOptionalDashboardUi(agent, { ...deps, redactUrl: dashboardUrlForDisplay });
+  printAdditionalForwardPorts(agent, info.port, deps.buildControlUiUrls);
+}
+
+/**
+ * Print one block per manifest-declared `forward_ports` entry that is not
+ * the primary dashboard port. Each block announces the port and renders a
+ * loopback URL using the same `buildControlUiUrls` chain as the primary
+ * dashboard so WSL host-address fallbacks remain consistent.
+ *
+ * The label is sourced from the agent's `health_probe.port` match — that
+ * is the only manifest signal today that a declared secondary port is the
+ * OpenAI-compatible API surface (Hermes manifest sets
+ * `health_probe.port: 8642` alongside `forward_ports: [18789, 8642]`).
+ * Any other declared port gets a neutral "additional port" label.
+ *
+ * The URL filter normalises empty `URL.port` results to the scheme
+ * default. `new URL("http://h:80").port` returns `""` because WHATWG
+ * URL elides the default scheme port; a strict `urlPort === String(port)`
+ * comparison would silently drop URLs for ports 80 and 443 even though
+ * the underlying `forward_ports` validation accepts them. The
+ * normalisation keeps the filter sound while still excluding any URL
+ * whose port truly does not match the declared entry.
+ */
+function printAdditionalForwardPorts(
+  agent: AgentDefinition,
+  primaryPort: number,
+  buildControlUiUrls: (token: string | null, port: number) => string[],
+): void {
+  const declared = Array.isArray(agent.forward_ports) ? agent.forward_ports : [];
+  if (declared.length === 0) return;
+  const apiPort = agent.healthProbe.port;
+  for (const port of declared) {
+    if (!Number.isInteger(port) || port < 1 || port > 65535) continue;
+    if (port === primaryPort) continue;
+    const isApi = port === apiPort;
+    const sectionLabel = isApi ? "OpenAI-compatible API" : "additional port";
+    console.log("");
+    console.log(`  ${agent.displayName} ${sectionLabel}`);
+    console.log(`  Port ${port} must be forwarded before connecting.`);
+    const seen = new Set<string>();
+    for (const baseUrl of buildControlUiUrls(null, port)) {
+      const withoutHash = baseUrl.split("#")[0].replace(/\/$/, "");
+      const resolvedUrlPort = resolveUrlPort(withoutHash);
+      if (resolvedUrlPort !== port) continue;
+      const url = isApi ? `${withoutHash}/v1` : `${withoutHash}/`;
+      if (seen.has(url)) continue;
+      seen.add(url);
+      console.log(`  ${dashboardUrlForDisplay(url)}`);
+    }
+  }
+}
+
+/**
+ * Resolve the effective port of `candidate`, normalising the WHATWG
+ * URL behaviour that returns an empty string for the scheme-default
+ * port (`http://h:80` → `""`, `https://h:443` → `""`). Returns the
+ * integer port, or `null` when the input is unparseable or carries no
+ * recoverable port. The mapping is intentionally limited to `http` /
+ * `https` / `ws` / `wss` — the four schemes the dashboard URL builder
+ * emits — so an unknown scheme falls through to `null` instead of
+ * silently mapping to 80 or 443.
+ */
+function resolveUrlPort(candidate: string): number | null {
+  let parsed: URL;
+  try {
+    parsed = new URL(candidate);
+  } catch {
+    return null;
+  }
+  if (parsed.port !== "") {
+    const numeric = Number(parsed.port);
+    return Number.isInteger(numeric) ? numeric : null;
+  }
+  const protocol = parsed.protocol.replace(/:$/, "").toLowerCase();
+  switch (protocol) {
+    case "http":
+      return 80;
+    case "https":
+      return 443;
+    case "ws":
+      return 80;
+    case "wss":
+      return 443;
+    default:
+      return null;
+  }
 }
diff --git a/test/process-recovery.test.ts b/test/process-recovery.test.ts
