Skip to content

fix(cron): guard _cron_pool.submit() against interpreter shutdown race - #1510

Merged
Lexus2016 merged 1 commit into
mainfrom
evolution/issue-1504-cron-shutdown-guard
Jul 30, 2026
Merged

fix(cron): guard _cron_pool.submit() against interpreter shutdown race#1510
Lexus2016 merged 1 commit into
mainfrom
evolution/issue-1504-cron-shutdown-guard

Conversation

@Lexus2016

Copy link
Copy Markdown
Owner

Summary

Fixes #1504

The _cron_pool.submit() call inside _run_job_impl (cron/scheduler.py) was unguarded — the surrounding try: block started after the submit call, so a RuntimeError: cannot schedule new futures after interpreter shutdown during Python interpreter teardown would propagate uncaught, producing spurious cron failure records.

This race was observed in production: 9 cron job failures across 2 job IDs in ~/.hermes/cron/failures/, all matching cannot schedule new futures.

Fix

Wrap _cron_pool.submit() in a try/except RuntimeError guard that mirrors the existing _submit_with_guard pattern (line ~5115) already used for the parallel dispatch pool:

  • If the error message contains cannot schedule new futures → log at debug, shut down the pool, and return a clean failure tuple (False, "", "", "cannot schedule new futures after interpreter shutdown").
  • Any other RuntimeError → shut down the pool and re-raise.
  • Any other Exception → shut down the pool and re-raise.

Also fixed: the delivery pool pool.submit() at line ~2329 was already guarded by a generic except Exception and needed no change.

Test plan

New test file tests/cron/test_cron_pool_shutdown_guard.py with 2 tests:

  1. test_shutdown_runtimeerror_returns_clean_failure — verifies the shutdown RuntimeError produces a failure tuple with the shutdown-specific error message and the pool is shut down with cancel_futures=True.
  2. test_other_runtimeerror_does_not_match_shutdown — verifies a different RuntimeError is NOT swallowed as a shutdown race (the error message does not contain cannot schedule new futures).

All 224 existing tests in tests/cron/test_scheduler.py still pass.

Diff size

20 insertions in cron/scheduler.py + 129-line test file = under the 200-line cap.

#1504)

The unguarded _cron_pool.submit() inside _run_job_impl could raise
RuntimeError('cannot schedule new futures after interpreter shutdown')
during Python interpreter teardown, producing spurious cron failure
records. Wrap the submit in try/except: the interpreter-shutdown
RuntimeError is logged at debug and returns a clean failure tuple;
any other RuntimeError re-raises after shutting down the pool.

Mirrors the existing _submit_with_guard pattern (line ~5115) already
used for the parallel dispatch pool.
@github-actions github-actions Bot added the fix Bug or fix label Jul 30, 2026
@Lexus2016
Lexus2016 merged commit 5bf1561 into main Jul 30, 2026
37 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

fix Bug or fix

Projects

None yet

Development

Successfully merging this pull request may close these issues.

[FIX] Cron interpreter-shutdown RuntimeError — ThreadPoolExecutor lifecycle bug in 9 cron runs

1 participant