Skip to content

Integrate gracehttp for zero-downtime HTTP restarts - #1302

Draft
snadrus wants to merge 7 commits into
mainfrom
cursor/gracehttp-integration-122c
Draft

Integrate gracehttp for zero-downtime HTTP restarts#1302
snadrus wants to merge 7 commits into
mainfrom
cursor/gracehttp-integration-122c

Conversation

@snadrus

@snadrus snadrus commented Jun 11, 2026

Copy link
Copy Markdown
Contributor

Summary

Integrates github.com/facebookgo/grace/gracehttp for zero-downtime HTTP restarts across all Curio HTTP listeners (RPC, Web GUI, and market/public HTTPS).

  • All HTTP servers are now served through a single gracehttpsvc.Serve() call in rpc.ListenAndServe, enabling graceful handoff of listener file descriptors on restart.
  • Restart triggers (SIGUSR2):
    • curio run when another instance is already running (pid file check) — triggers restart and exits 0
    • curio cli restart — remote restart via new RPC Restart API (or --local via pid file)
    • Cluster restart (harmony_machines.restart_request) — sends SIGUSR2 after tasks drain; falls back to exit 100 for systemd if signal fails
    • systemctl reload curio — new ExecReload=/bin/kill -USR2 $MAINPID in the systemd unit
  • Shutdown (SIGTERM): curio cli stop and RPC Shutdown now trigger graceful HTTP drain via SIGTERM.
  • Pre-restart hook drains the task engine before forking the successor process.
  • PID file written early to $CURIO_REPO_PATH/curio.pid for duplicate-start detection and local restarts.

Duplicate start behavior

Running curio run while Curio is already running for the same repo no longer fails with port conflicts. Instead:

  1. Read $CURIO_REPO_PATH/curio.pid
  2. If the process is alive, send SIGUSR2 and exit successfully
  3. Otherwise claim the pid file and start normally

Skipped when:

  • LISTEN_FDS is set (gracehttp handoff child)
  • --halt-after-init is passed
  • CURIO_SKIP_ALREADY_RUNNING=1 is set

Integration tests are unaffected because they use StartCurioHarness, which calls tasks.StartTasks + rpc.ListenAndServe directly and never goes through runCmd.

Usage

# Starting while already running triggers restart
curio run

# Remote restart (via RPC, same as stop)
curio cli restart

# Local restart via pid file
curio cli restart --local

# Systemd zero-downtime reload
systemctl reload curio

RestartForceExitStatus=100 is retained as a fallback when graceful restart cannot signal the process.

Open in Web Open in Cursor 

cursoragent and others added 7 commits June 11, 2026 17:16
Wire facebookgo/grace/gracehttp into curio run so RPC, GUI, and market
HTTP servers share a single graceful serve loop. Restarts are triggered
via SIGUSR2 from curio cli restart, cluster restart requests, and systemd
ExecReload. Shutdown uses SIGTERM through the existing RPC API.

Co-authored-by: Andrew Jackson (Ajax) <snadrus@users.noreply.github.com>
When curio run finds a live process via the repo pid file, send SIGUSR2
to the running instance and exit instead of starting a second server.
Skip the check for gracehttp handoffs, halt-after-init, and when
CURIO_SKIP_ALREADY_RUNNING is set so integration tests stay unaffected.

Co-authored-by: Andrew Jackson (Ajax) <snadrus@users.noreply.github.com>
Spawn a subprocess worker serving the real /pdp/ping handler via gracehttp.
The parent continuously pings through a SIGUSR2 restart and asserts zero
failures while verifying the pid file reflects a new process.

Co-authored-by: Andrew Jackson (Ajax) <snadrus@users.noreply.github.com>
Co-authored-by: Andrew Jackson (Ajax) <snadrus@users.noreply.github.com>
Co-authored-by: Andrew Jackson (Ajax) <snadrus@users.noreply.github.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants