[hermes] Impossible to start the gateway after a reboot

[oparoz]

Investigation Summary

No response

Description

I was on v.0.80 and needed to reboot the GB10 box. Since then it's been impossible to properly recover the Hermes sandbox.

First attempt to connect failed and after trying to resume the onboarding, I ended up with a sandbox in provisioning mode, which was impossible to get out of without completely losing all the data.

Any attempt to recover the sandbox brought an error message about a config hash mismatch, which prevents the gateway from being started. That makes no sense to me since the shield was down.
I even set the correct hashes in hope I could start the gateway via docker exec, but it didn't change anything.

At some point after trying every command I could think of, the permissions in the box were set wrong and Hermes lost the ability to read certain files and folders. They are now owned by root or strange users. I'll post the output of the list command in the logs.

Today, I did a rebuild on 0.83 and most of the files were not backed up because of the permission problems, but even the new image has the same problem with files having the wrong permissions.
I restarted the gateway from the dashboard to see if things had improved, but they did not.
The mounted folder is unstable and attempts to recover or connect to that brand new sandbox also fail with the same GATEWAY_CONFIG_HASH_MISMATCH error.

Reproduction Steps

1. Use a working Hermes sandbox for a few days
2. Reboot
3. Try to connect to the sandbox

When everything fails, try with onboard, rebuild, recover, shields down. Explore the filesystem, try to launch the gateway from the inside. Try to restart the gateway from the Web dashboard

Environment

• Hardware: GB10 platform
• OS: Ubuntu 24.04 LTS
• OpenShell 0.0.72
• NemoClaw 0.0.83
• Hermes Agent 2026.7.1

Debug Output

═══ Sandbox Internals ═══

UID PID PPID C STIME TTY TIME CMD
root 1 0 0 18:26 ? 00:00:36 /opt/openshell/bin/openshell-sandbox
sandbox 85 1 0 18:26 ? 00:00:00 sleep infinity
sandbox 86 1 3 18:26 ? 00:03:31 bash /usr/local/bin/nemoclaw-start
sandbox 106 86 0 18:26 ? 00:00:00 tee -a /tmp/nemoclaw-start.log
sandbox 107 86 0 18:26 ? 00:00:00 tee -a /tmp/nemoclaw-start.log
sandbox 330 86 0 18:26 ? 00:00:28 /opt/hermes/.venv/bin/python /usr/local/bin/hermes.real gateway run
sandbox 346 86 0 18:26 ? 00:00:00 tail -n +1 -F /tmp/gateway.log
sandbox 347 86 0 18:26 ? 00:00:00 sed -u s/^/[gateway-log:] /
sandbox 636 86 0 18:26 ? 00:00:01 socat TCP-LISTEN:8642,bind=0.0.0.0,fork,reuseaddr TCP:127.0.0.1:18642
sandbox 787 86 0 18:26 ? 00:00:39 /opt/hermes/.venv/bin/python /usr/local/bin/hermes.real dashboard --host 127.0.0.1 --port 19119 --skip-build --no-open --tui
sandbox 803 86 0 18:26 ? 00:00:00 tail -n +1 -F /tmp/dashboard.log
sandbox 804 86 0 18:26 ? 00:00:01 sed -u s/^/[dashboard-log:] /
sandbox 884 86 0 18:26 ? 00:00:00 socat TCP-LISTEN:9119,bind=0.0.0.0,fork,reuseaddr TCP:127.0.0.1:19119
sandbox 81362 1 0 18:31 ? 00:00:00 /usr/lib/openssh/sftp-server
sandbox 1466948 787 0 19:52 ? 00:00:05 /opt/hermes/.venv/bin/python -m hermes_cli.main gateway restart
sandbox 1485204 884 0 19:53 ? 00:00:00 socat TCP-LISTEN:9119,bind=0.0.0.0,fork,reuseaddr TCP:127.0.0.1:19119
sandbox 1485213 884 0 19:53 ? 00:00:00 socat TCP-LISTEN:9119,bind=0.0.0.0,fork,reuseaddr TCP:127.0.0.1:19119
sandbox 1485215 787 0 19:53 pts/0 00:00:00 /usr/local/bin/node --expose-gc /opt/hermes/ui-tui/dist/entry.js
sandbox 1486435 884 0 19:54 ? 00:00:00 socat TCP-LISTEN:9119,bind=0.0.0.0,fork,reuseaddr TCP:127.0.0.1:19119
sandbox 1486847 1485215 0 19:54 pts/0 00:00:01 /opt/hermes/.venv/bin/python -m tui_gateway.entry
sandbox 1914292 1 99 20:19 ? 00:00:00 ps -ef
sandbox 1914293 86 0 20:19 ? 00:00:00 bash /usr/local/bin/nemoclaw-start
total used free shared buff/cache available
Mem: 124610 64225 39284 3751 25973 60384
Swap: 16383 0 16383

Logs

$ nemohermes hermes recover
  GATEWAY_CONFIG_HASH_MISMATCH

  Secret-boundary check did not complete cleanly for Hermes gateway in 'hermes'.
  Refusing recovery; inspect the validator output above before re-running `nemoclaw <sandbox> recover`.

  Probe failed: secret-boundary check did not complete for Hermes Agent gateway in 'hermes'.
  Inspect the validator output above and re-run `nemoclaw <sandbox> recover`.


Some error messages from the logs

2026-07-15 20:34:57,499 WARNING gateway.platforms.api_server: API server rejected invalid API key: remote='127.0.0.1' peer_ip='127.0.0.1' method='GET' path='/health/detailed' user_agent='Python-urllib/3.13'
2026-07-15 20:34:58,644 ERROR gateway.run: kanban dispatcher: tick failed on board default
Traceback (most recent call last):
  File "/opt/hermes/gateway/kanban_watchers.py", line 1009, in _tick_once_for_board
    conn = _kb.connect(board=slug)
  File "/opt/hermes/hermes_cli/kanban_db.py", line 1727, in connect
    apply_wal_with_fallback(conn, db_label=f"kanban.db ({path.name})")
    ~~~~~~~~~~~~~~~~~~~~~~~^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/opt/hermes/hermes_state.py", line 377, in apply_wal_with_fallback
    conn.execute("PRAGMA journal_mode=WAL")
    ~~~~~~~~~~~~^^^^^^^^^^^^^^^^^^^^^^^^^^^
sqlite3.OperationalError: attempt to write a readonly database
2026-07-15 20:34:59,634 WARNING gateway.drain_control: drain-control: failed to read /sandbox/.hermes/.drain_request.json: [Errno 13] Permission denied: '/sandbox/.hermes/.drain_request.json'

Listing of the mounted share:
$ la ~/.nemoclaw/mounts/hermes/.hermes/
ls: reading directory '/home/gb10/.nemoclaw/mounts/hermes/.hermes/': Permission denied
total 0

Let's try again (it works this time!):
$ la ~/.nemoclaw/mounts/hermes/.hermes/
total 328
  4 drwxr-xr-x 1 root            systemd-journal   4096 Jul 11 14:48 workspace
  4 drwxr-xr-x 1 root            systemd-journal   4096 Jul 11 14:48 skins
  4 drwx------ 1 systemd-network systemd-journal   4096 Jul 11 14:48 sessions
  4 drwxr-xr-x 1 root            systemd-journal   4096 Jul 11 14:48 profiles
  4 drwxrwx--- 1 systemd-network systemd-journal   4096 Jul 11 14:48 plans
  4 drwx------ 1 root            root              4096 Jul 11 14:48 pairing
  4 drwx------ 1 systemd-network systemd-journal   4096 Jul 11 14:48 memories
  4 drwx------ 1 systemd-network systemd-journal   4096 Jul 11 14:48 image_cache
  4 drwxr-xr-x 1 root            systemd-journal   4096 Jul 11 14:48 hooks
  4 drwx------ 1 systemd-network systemd-journal   4096 Jul 11 14:48 audio_cache
  4 drwxr-xr-x 1 systemd-network systemd-journal   4096 Jul 15 20:25 scripts
  0 -rw-r--r-- 1 systemd-network systemd-journal      0 Jul 15 20:25 auth.lock
  4 drwxr-xr-x 1 root            systemd-journal   4096 Jul 15 20:25 plugins
  4 drwxr-xr-x 1 root            systemd-journal   4096 Jul 15 20:25 weixin
  4 lrwxrwxrwx 1 root            systemd-journal     20 Jul 15 20:26 state.db-wal -> runtime/state.db-wal
  4 lrwxrwxrwx 1 root            systemd-journal     20 Jul 15 20:26 state.db-shm -> runtime/state.db-shm
  4 lrwxrwxrwx 1 root            systemd-journal     16 Jul 15 20:26 state.db -> runtime/state.db
  4 lrwxrwxrwx 1 root            systemd-journal     26 Jul 15 20:26 gateway_state.json -> runtime/gateway_state.json
  4 lrwxrwxrwx 1 root            systemd-journal     30 Jul 15 20:26 channel_directory.json -> runtime/channel_directory.json
 28 -rw------- 1 systemd-network systemd-journal  24752 Jul 15 20:26 response_store.db-wal
 32 -rw------- 1 systemd-network systemd-journal  32768 Jul 15 20:26 response_store.db-shm
  4 -rw------- 1 systemd-network systemd-journal   4096 Jul 15 20:26 response_store.db
  4 drwxr-xr-x 1 root            systemd-journal   4096 Jul 15 20:26 platforms
  4 drwxrwx--- 1 systemd-network systemd-journal   4096 Jul 15 20:26 bin
  4 -rw-rw---- 1 systemd-network systemd-journal     15 Jul 15 20:26 .tirith-install-failed
  4 drwxrws--- 1 dnsmasq         systemd-journal   4096 Jul 15 20:26 runtime
  4 drwxrwx--- 1 systemd-network systemd-journal   4096 Jul 15 20:26 kanban
  0 -rw-rw---- 1 systemd-network systemd-journal      0 Jul 15 20:26 kanban.db.init.lock
  0 -rw-rw---- 1 systemd-network systemd-journal      0 Jul 15 20:26 kanban.db.dispatch.lock
116 -rw-r----- 1 systemd-network systemd-journal 118784 Jul 15 20:26 kanban.db
  4 -rw------- 1 systemd-network systemd-journal   1178 Jul 15 20:27 auth.json
  4 drwxrwxr-t 1 root            systemd-journal   4096 Jul 15 20:27 ..
  4 drwx------ 1 systemd-network systemd-journal   4096 Jul 15 20:27 logs
  4 -rw-rw---- 1 systemd-network systemd-journal     68 Jul 15 20:27 .update_check
  4 -rw-r----- 1 systemd-network systemd-journal    513 Jul 15 20:27 SOUL.md
  4 -rw-r----- 1 systemd-network systemd-journal   3179 Jul 15 20:27 .hermes_history
  4 drwx------ 1 systemd-network systemd-journal   4096 Jul 15 22:20 dashboard-home
  4 drwxrwx--- 1 systemd-network systemd-journal   4096 Jul 15 22:25 cache
  4 drwxr-xr-x 1 root            root              4096 Jul 15 22:26 .
  4 drwxr-xr-x 1 root            systemd-journal   4096 Jul 15 22:26 skills
  4 drwxr-xr-x 1 root            systemd-journal   4096 Jul 15 22:26 cron
  4 -r--r--r-- 1 root            root              2067 Jul 15 22:26 config.yaml
  4 -r--r--r-- 1 root            root               221 Jul 15 22:26 .env
  4 -r--r--r-- 1 root            root               361 Jul 15 22:26 .config-hash

Backup cannot go through properly because of wrong permissions
  [rebuild 2026-07-15T18:26:36.084Z] Backup result: success=false, backed=; files=SOUL.md,.hermes_history,runtime/state.db, failed=memories,sessions,skills,plugins,cron,logs,skins,plans,workspace,profiles,cache,pairing,dashboard-home,platforms; failedFiles=
  ⚠ Partial backup: 0 dirs and 3 files OK; 14 dirs and 0 files failed
    Failed dirs: memories, sessions, skills, plugins, cron, logs, skins, plans, workspace, profiles, cache, pairing, dashboard-home, platforms
    Rebuild will continue — failed state could not be preserved.

Checklist

• I confirmed this bug is reproducible
• I searched existing issues and this is not a duplicate

[TonyLuo-NV]

Triage / disposition

This report has two distinct root causes; they've been split so each lands in the right place.

1. Rebuild data-loss — fixed ✅
The Partial backup … Rebuild will continue — failed state could not be preserved path destroyed every state directory when they all failed to back up (only loose files were saved). Fixed in #7005 (merged): rebuild now aborts by default when no state directory can be preserved and requires --force to proceed, with a diagnostic pointing at the likely ownership/permissions cause.

2. Gateway won't start / GATEWAY_CONFIG_HASH_MISMATCH — root cause is upstream in OpenShell ⏳
The config-hash refusal is working as intended: the Hermes preflight infers "sealed" from config.yaml/.env being root-owned + unwritable, then verifies the sealed hash and correctly refuses to start a gateway whose config it can't trust. It fires here because the inputs were corrupted, not because the check is wrong.

The corruption is the reported wrong ownership on the mounted .hermes/ tree (root/systemd-network/dnsmasq, drwx------). Traced to OpenShell: the rootless Podman/Docker drivers pin no user-namespace mapping and bind-mount host state as plain rbinds, so when a host reboot shifts the rootless subuid base, previously-written inodes resolve to unrelated host users — and nothing re-owns them. That also explains the in-sandbox attempt to write a readonly database and EACCES on .drain_request.json.

Filed upstream:

• Root cause: bug: sandbox bind-mount ownership breaks after host reboot (unpinned rootless userns mapping, no launch-time reconcile) OpenShell#2336
• Proposed fix (draft): fix(supervisor): opt-in reconcile of drifted sandbox state ownership (#2336) OpenShell#2338 — opt-in launch-time ownership reconcile as a stop-gap; durable fix is idmapped mounts / a pinned uid/gid map.

Suggested disposition: keep this open until the OpenShell fix lands (the primary "can't start the gateway after a reboot" symptom is not resolvable inside NemoClaw). The data-loss regression itself is fixed as of #7005.