Operations API Unix domain socket sometimes never created after deploy / clone (CLI unusable)

[kriszyp]

After a Harper restart triggered by a deploy (or during initial node clone), the operations API Unix domain socket (<rootPath>/operations-server) is sometimes never created on the filesystem, even though the startup banner advertises it and Harper otherwise appears to be running. CLI commands then fail with No domain socket found, unable to perform this operation. Restarting the container/process recreates the file.

Pattern

• Initial cloning (Tyler / Devin reproduction in original Jira): instance reported "still cloning" but harper was actually up and serving traffic. Socket file absent. Harper had been running ~1 hour. CLI was unusable. Restarting the instance created the file.
• Post-deploy (Devin, 2026-03-20, [redacted-customer-cluster] cluster): logs show Restarting http_workers after a deploy, the standard startup banner (which advertises unix socket: /home/harperdb/hdb/operations-server), TLS connection errors against the leader, replication eventually reconnects — but the operations-server socket file is never created. CLI commands return No domain socket found.
• Post-deploy (Devin, 2026-04-06, [redacted-customer-cluster]): same — harperdb cluster_status from inside the container returns No domain socket found.

Devin notes a previous race-condition fix appeared to resolve an earlier instance of this, but it has been observed on initial cloning and on post-deploy restarts since.

To investigate

• Trace the operations API server's UDS creation path. The startup banner is printed before the socket is bound — banner is informational, not proof of bind. Find where listen({ path: ... }) happens and look for paths where it can be skipped silently (caught exception, early-return, conditional gated on a flag that becomes false during cloning or http_workers restart).
• Check the http_workers restart path triggered by a deploy: which thread owns the UDS server? Does the restart tear down and re-create the listener? Is there a window where the old socket file is deleted but the new bind hasn't happened?
• Initial-clone path: between cloneNode and the eventual started state, is there a missed call to bind the UDS?
• File permissions / EACCES on the socket directory should not be the cause (Devin's container is the same one that produced a working socket on restart), but worth a sanity check in the trace.

Acceptance criteria

• After any restart path that keeps Harper running (deploy-triggered http_workers restart, initial clone completion), the operations-server UDS is present and accepts connections.
• A regression test exercises the deploy-restart path and asserts the socket is reachable afterwards.
• If the socket creation legitimately can't complete (config disabled, etc.), the banner reflects that rather than advertising a path that doesn't exist.

Notes for reviewer

• The "No domain socket found" check in bin/cliOperations.js:98 is the CLI-side observation; the root cause is server-side (operations API plugin / fastify route / http_workers lifecycle).

---

Tracked in Jira: CORE-3038

🤖 Filed by Claude on behalf of Kris.

[kriszyp]

Most likely fixed by #299