Integrate gracehttp for zero-downtime HTTP restarts - #1302
Draft
snadrus wants to merge 7 commits into
Draft
Conversation
Wire facebookgo/grace/gracehttp into curio run so RPC, GUI, and market HTTP servers share a single graceful serve loop. Restarts are triggered via SIGUSR2 from curio cli restart, cluster restart requests, and systemd ExecReload. Shutdown uses SIGTERM through the existing RPC API. Co-authored-by: Andrew Jackson (Ajax) <snadrus@users.noreply.github.com>
When curio run finds a live process via the repo pid file, send SIGUSR2 to the running instance and exit instead of starting a second server. Skip the check for gracehttp handoffs, halt-after-init, and when CURIO_SKIP_ALREADY_RUNNING is set so integration tests stay unaffected. Co-authored-by: Andrew Jackson (Ajax) <snadrus@users.noreply.github.com>
Spawn a subprocess worker serving the real /pdp/ping handler via gracehttp. The parent continuously pings through a SIGUSR2 restart and asserts zero failures while verifying the pid file reflects a new process. Co-authored-by: Andrew Jackson (Ajax) <snadrus@users.noreply.github.com>
Co-authored-by: Andrew Jackson (Ajax) <snadrus@users.noreply.github.com>
Co-authored-by: Andrew Jackson (Ajax) <snadrus@users.noreply.github.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
Integrates
github.com/facebookgo/grace/gracehttpfor zero-downtime HTTP restarts across all Curio HTTP listeners (RPC, Web GUI, and market/public HTTPS).gracehttpsvc.Serve()call inrpc.ListenAndServe, enabling graceful handoff of listener file descriptors on restart.curio runwhen another instance is already running (pid file check) — triggers restart and exits 0curio cli restart— remote restart via new RPCRestartAPI (or--localvia pid file)harmony_machines.restart_request) — sends SIGUSR2 after tasks drain; falls back to exit 100 for systemd if signal failssystemctl reload curio— newExecReload=/bin/kill -USR2 $MAINPIDin the systemd unitcurio cli stopand RPCShutdownnow trigger graceful HTTP drain via SIGTERM.$CURIO_REPO_PATH/curio.pidfor duplicate-start detection and local restarts.Duplicate start behavior
Running
curio runwhile Curio is already running for the same repo no longer fails with port conflicts. Instead:$CURIO_REPO_PATH/curio.pidSkipped when:
LISTEN_FDSis set (gracehttp handoff child)--halt-after-initis passedCURIO_SKIP_ALREADY_RUNNING=1is setIntegration tests are unaffected because they use
StartCurioHarness, which callstasks.StartTasks+rpc.ListenAndServedirectly and never goes throughrunCmd.Usage
RestartForceExitStatus=100is retained as a fallback when graceful restart cannot signal the process.