Add utility for graceful Launchpad termination with external schedulers #352
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Summary
Fixes #311
When using Acme distributed experiments with external schedulers like Ray Tune's ASHA scheduler, the scheduler may terminate trials early. However, the Launchpad processes spawned by the experiment are not automatically terminated, leaving orphan processes running.
Problem
As described in #311, when Ray Tune's ASHA scheduler terminates a trial, the
mp.Processrunning the Launchpad program is killed, but the child processes spawned by Launchpad continue running as orphans. This happens because the termination signal is not forwarded to the Launchpad processes.Solution
Added two new utilities to
acme/utils/lp_utils.py:1.
LaunchpadProgramStopper(Context Manager)A context manager that registers signal handlers for
SIGTERMandSIGINT. When these signals are received, it callslp.stop()to gracefully terminate all Launchpad processes.2.
launch_with_termination_handler()(Convenience Function)A wrapper around
lp.launch()that automatically uses theLaunchpadProgramStoppercontext manager.Usage
Or using the context manager directly:
Testing
python3 -m py_compileacme/utils/signals.py