Free port 18789 before gateway spawn, fail /healthz when dead#219
Open
kbhuw wants to merge 1 commit into
Open
Conversation
The wrapper-supervised openclaw gateway occasionally hits its embedded run timeout (10 min) and the subsequent restart fails because port 18789 is still held by a zombie openclaw process — the supervisor's handle is null (fresh container life) so childProcess.kill can't reach it, and lsof/procps were missing from the runtime image so openclaw's own "port in use" check couldn't recover either. Net effect: the container stays up, WebSocket accepts connections, but chat.send fires an instant empty chat:final and every message looks stuck. - Install lsof + procps + psmisc in the runtime stage so PID-by-port lookups work. - Add a tryFreePort helper that probes with lsof (then fuser) and sends SIGTERM / SIGKILL to any pid holding the target port, before the supervisor spawns a new gateway or inside restartGateway. - /healthz now returns 503 when the gateway is configured but the internal port is unreachable, and railway.toml points the healthcheckPath at it instead of the always-200 /setup/healthz — so restartPolicyType=on_failure can actually recycle the container when the agent dies.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
gatewayProchandle is lost but the OS-level listener on port 18789 stays held by a zombie. The nextgateway runspawn fails withEADDRINUSE, but the wrapper process keeps running — so the container stays "up", WebSocket accepts connections, andchat.sendfires an instant emptychat:final. Every message looks stuck from the user's side.lsof/procps/psmisc, so openclaw's own "port in use" recovery path (which shells out to find the owning PID) had nothing to call — logs readPort is in use but process details are unavailable (install lsof or run as an admin user)./healthzalways returned 200, so Railway'srestartPolicyType=on_failurenever recycled the container even when the gateway was dead.Changes
lsof,procps,psmiscin the runtime stage.src/server.js:tryFreePort(port, { signal })helper — probes withlsof -t -i:PORT -sTCP:LISTEN(falls back tofuser), kills any matching PIDs.startGateway()now calls it (SIGTERM, then escalates to SIGKILL) right beforechildProcess.spawn, so zombies from a previous container life can't EADDRINUSE the new spawn.restartGateway()calls it after the SIGTERM + 750ms grace so a stuck child that escaped SIGTERM doesn't block the nextensureGatewayRunning()./healthzreturns 503 whenisConfigured()butprobeGateway()fails, so Railway's healthcheck can trigger a restart.railway.toml: pointhealthcheckPathat/healthz(the gateway-aware one) instead of/setup/healthz(always 200).Test plan
/healthzreturns 200 withgateway.reachable: true.restartGatewayvia/setup/api/console/run— confirm the supervisor kills the dummy listener and the new gateway spawn succeeds./healthzflips to 503 and Railway restarts the container.🤖 Generated with Claude Code