Skip to content

Install stop-signal handlers in the K8s manager CLI - #27

Open
dotsdl wants to merge 1 commit into
mainfrom
service-termination-signals
Open

Install stop-signal handlers in the K8s manager CLI#27
dotsdl wants to merge 1 commit into
mainfrom
service-termination-signals

Conversation

@dotsdl

@dotsdl dotsdl commented May 28, 2026

Copy link
Copy Markdown
Member

Summary

The manager start command ran manager.start() with no signal handling. Kubernetes terminates pods with SIGTERM (then SIGKILL after the grace period), so the manager process was killed without running ComputeManager.start()'s finally block — orphaning its ComputeManagerRegistration in the state store on every rollout, scale-down, kubectl delete pod, node drain, or eviction. Only an interactive SIGINT cleaned up.

This wires the command to alchemiscale.compute.signals.install_stop_handlers, which registers SIGHUP/SIGINT/SIGTERM handlers that call manager.stop() (now interruptible, so it wakes the inter-cycle sleep promptly) and let the loop unwind through its deregistration finally.

install_stop_handlers(manager)
try:
    manager.start(steal=steal)
except KeyboardInterrupt:
    pass

Notes

  • For SIGTERM to actually reach the process, ensure the container runs Python as PID 1 (or via exec/an init like tini) so the signal is delivered/forwarded — a sh -c "..." entrypoint without exec would swallow it.
  • The existing --steal flag remains a useful safety net for reclaiming a registration left behind by a hard SIGKILL.

Dependency

Requires the install_stop_handlers helper (and interruptible-sleep changes) added in the core repo:
OpenFreeEnergy/alchemiscale#503. Merge that first.

🤖 Generated with Claude Code

The 'manager start' command ran manager.start() with no signal handling. Kubernetes stops pods with SIGTERM (then SIGKILL after the grace period), which bypassed ComputeManager.start()'s finally block and orphaned the manager's ComputeManagerRegistration on every rollout/scale-down/pod deletion. It now installs SIGHUP/SIGINT/SIGTERM handlers via alchemiscale.compute.signals.install_stop_handlers, so stop() runs and the registration deregisters cleanly.

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant