fix(cron): guard _cron_pool.submit() against interpreter shutdown race - #1510
Merged
Conversation
#1504) The unguarded _cron_pool.submit() inside _run_job_impl could raise RuntimeError('cannot schedule new futures after interpreter shutdown') during Python interpreter teardown, producing spurious cron failure records. Wrap the submit in try/except: the interpreter-shutdown RuntimeError is logged at debug and returns a clean failure tuple; any other RuntimeError re-raises after shutting down the pool. Mirrors the existing _submit_with_guard pattern (line ~5115) already used for the parallel dispatch pool.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
Fixes #1504
The
_cron_pool.submit()call inside_run_job_impl(cron/scheduler.py) was unguarded — the surroundingtry:block started after the submit call, so aRuntimeError: cannot schedule new futures after interpreter shutdownduring Python interpreter teardown would propagate uncaught, producing spurious cron failure records.This race was observed in production: 9 cron job failures across 2 job IDs in
~/.hermes/cron/failures/, all matchingcannot schedule new futures.Fix
Wrap
_cron_pool.submit()in atry/except RuntimeErrorguard that mirrors the existing_submit_with_guardpattern (line ~5115) already used for the parallel dispatch pool:cannot schedule new futures→ log at debug, shut down the pool, and return a clean failure tuple(False, "", "", "cannot schedule new futures after interpreter shutdown").RuntimeError→ shut down the pool and re-raise.Exception→ shut down the pool and re-raise.Also fixed: the delivery pool
pool.submit()at line ~2329 was already guarded by a genericexcept Exceptionand needed no change.Test plan
New test file
tests/cron/test_cron_pool_shutdown_guard.pywith 2 tests:test_shutdown_runtimeerror_returns_clean_failure— verifies the shutdown RuntimeError produces a failure tuple with the shutdown-specific error message and the pool is shut down withcancel_futures=True.test_other_runtimeerror_does_not_match_shutdown— verifies a different RuntimeError is NOT swallowed as a shutdown race (the error message does not containcannot schedule new futures).All 224 existing tests in
tests/cron/test_scheduler.pystill pass.Diff size
20 insertions in
cron/scheduler.py+ 129-line test file = under the 200-line cap.