Skip to content

Commit f2793e7

Browse files
committed
Split CPU unit tests into halves with per-half timeouts
Both full-suite attempts wedge before printing a summary: once the 100th failure trips PYTEST_OPTS' --maxfail, pytest-xdist's interrupt path stalls forever in mp pool teardown (no timeout guards _close_pool), and the 6h job limit cancels the run. Run the suite as two fresh-worker halves, override maxfail so all failures are listed, and cap each half with timeout so the sequential tail always runs. Signed-off-by: Guokai Ma <guokai.ma@intel.com>
1 parent f2b7d0b commit f2793e7

1 file changed

Lines changed: 9 additions & 2 deletions

File tree

.github/workflows/cpu-torch-latest.yml

Lines changed: 9 additions & 2 deletions
Original file line numberDiff line numberDiff line change
@@ -284,5 +284,12 @@ jobs:
284284
run: |
285285
unset TORCH_CUDA_ARCH_LIST # only jit compile for current arch
286286
cd tests
287-
HF_HOME=/tmp/hf_home/ pytest $PYTEST_OPTS --forked -n 4 unit/ --torch_ver="$TORCH_TEST_VERSION"
288-
HF_HOME=/tmp/hf_home/ pytest $PYTEST_OPTS --forked -m 'sequential' unit/ --torch_ver="$TORCH_TEST_VERSION"
287+
# The suite is split so each half gets fresh xdist workers: multi-rank pool
288+
# teardown eventually wedges a worker, and one 3400-test process never
289+
# reaches its summary inside the 6h job limit. maxfail is raised so every
290+
# failure is listed, and timeout caps each half.
291+
overall=0
292+
timeout 150m pytest $PYTEST_OPTS --maxfail=100000 --forked -n 4 unit/ --ignore=unit/v1 --torch_ver="$TORCH_TEST_VERSION" || overall=$?
293+
timeout 150m pytest $PYTEST_OPTS --maxfail=100000 --forked -n 4 unit/v1 --torch_ver="$TORCH_TEST_VERSION" || overall=$?
294+
HF_HOME=/tmp/hf_home/ pytest $PYTEST_OPTS --forked -m 'sequential' unit/ --torch_ver="$TORCH_TEST_VERSION" || overall=$?
295+
exit $overall

0 commit comments

Comments
 (0)