Commit 1b2814b
committed
ci(rust): serialize core lib tests to dodge cross-runtime semaphore hang
Root cause of the long-standing intermittent hang on the `Rust Core
Tests + Quality` and `Rust Core Coverage` jobs: `cargo test -p openhuman`
runs `#[tokio::test]` tests in parallel, each on its own current-thread
runtime, but several modules share process-wide singletons:
* `scheduler_gate::LLM_PERMITS` — `tokio::sync::Semaphore`
* `scheduler_gate::STATE` — `OnceLock<RwLock<State>>`
* `local_ai::LOCAL_AI_TEST_MUTEX` — `std::sync::Mutex`
* `event_bus::testing::BUS_HANDLER_LOCK` — `tokio::sync::Mutex`
The `tokio::sync::Semaphore` waiter in test A registers a Waker tied to
runtime A. When test B drops the held permit from runtime B and that
runtime is in the process of tearing down, the cross-runtime wake does
not fire reliably — A's spawned future never resumes. On the CI runner
this presents as a hung `cargo test` step; locally on faster hosts the
prior assertion (`try_acquire_llm_permit().expect("must start free")`)
panics first and only manifests as a `FAILED` test.
Running `cargo test -p openhuman -- --test-threads=1` is a one-line
mitigation that eliminates the parallelism and brings the step from
"hangs indefinitely" to a deterministic ~90s on the same suite. The
deeper test-isolation refactor (swap the OnceLocks for resettable
cells, shared serial mutex around every `wait_for_capacity` callsite)
is queued as a follow-up — this commit unblocks CI immediately.
Applied to both `.github/workflows/test.yml` (`Test core crate`) and
`.github/workflows/coverage.yml` (`Rust Core Coverage`).1 parent 4a16c79 commit 1b2814b
2 files changed
Lines changed: 12 additions & 2 deletions
| Original file line number | Diff line number | Diff line change | |
|---|---|---|---|
| |||
84 | 84 | | |
85 | 85 | | |
86 | 86 | | |
87 | | - | |
| 87 | + | |
| 88 | + | |
| 89 | + | |
| 90 | + | |
88 | 91 | | |
89 | 92 | | |
90 | 93 | | |
| |||
| Original file line number | Diff line number | Diff line change | |
|---|---|---|---|
| |||
83 | 83 | | |
84 | 84 | | |
85 | 85 | | |
86 | | - | |
| 86 | + | |
| 87 | + | |
| 88 | + | |
| 89 | + | |
| 90 | + | |
| 91 | + | |
| 92 | + | |
| 93 | + | |
87 | 94 | | |
88 | 95 | | |
89 | 96 | | |
| |||
0 commit comments