CI rerun filter does not retry GPU allocation failures wrapped in another exception

[albertvillanova]

CI fails with dev dependencies: https://github.com/huggingface/trl/actions/runs/32855030464/job/97824800816

RuntimeError: CUDA error: CUBLAS_STATUS_ALLOC_FAILED when calling cublasCreate(handle)
RuntimeError: Comparing

FAILED tests/test_sft_trainer.py::TestPatchChunkedCELMHead::test_forward_matches_reference_with_aux_loss[trl-internal-testing/tiny-Qwen3MoeForCausalLM] - RuntimeError: CUDA error: CUBLAS_STATUS_ALLOC_FAILED when calling `cublasCreate(handle)`
FAILED tests/test_sft_trainer.py::TestPatchChunkedCELMHead::test_backward_matches_reference[trl-internal-testing/tiny-DeepseekV3ForCausalLM-0528] - RuntimeError: Comparing
= 2 failed, 2424 passed, 149 skipped, 14 xfailed, 16 rerun in 600.57s (0:10:00)

Both failures are GPU memory exhaustion on the shared runner, not code defects: the same two tests passed in the three sibling jobs of that run (latest dependencies, minimum versions, without optional dependencies) and again on a re-run of the same job.

The problem is that neither was retried, even though make test already passes --only-rerun '(OSError|Timeout|HTTPError.*502|HTTPError.*504|OutOfMemoryError)' for exactly this situation, and 16 reruns did fire elsewhere in the same job.

Traceback:

    @pytest.mark.parametrize("model_id", _CHUNKED_CE_MODEL_IDS)
    def test_backward_matches_reference(self, model_id):
        ref_model, chunked_model, input_ids, labels, num_items = self._setup(model_id)
        ref_out = ref_model(input_ids=input_ids, labels=labels, num_items_in_batch=num_items)
        ref_out.loss.backward()
        out = chunked_model(input_ids=input_ids, labels=labels, num_items_in_batch=num_items)
        out.loss.backward()
        # lm_head gradient
>       torch.testing.assert_close(
            chunked_model.lm_head.weight.grad, ref_model.lm_head.weight.grad, atol=1e-5, rtol=1e-5
        )
E       RuntimeError: Comparing
E
E       TensorLikePair(
E           id=(),
E           actual=tensor([[-8.1022e-07, -4.9951e-07,  7.0211e-06,  ..., -8.0111e-06,
E                 7.4571e-06,  8.8054e-07],
E               ...
E           expected=tensor([[-8.1022e-07, -4.9951e-07,  7.0211e-06,  ..., -8.0111e-06,
E                 7.4571e-06,  8.8054e-07],
E               ...

tests/test_sft_trainer.py:2942: RuntimeError

with, as the direct cause:

    def _compare_regular_values_close(
        self,
        actual: torch.Tensor,
        expected: torch.Tensor,
        *,
        rtol: float,
        atol: float,
        equal_nan: bool,
        identifier: str | Callable[[str], str] | None = None,
    ) -> None:
        """Checks if the values of two tensors are close up to a desired tolerance."""
>       matches = torch.isclose(
            actual, expected, rtol=rtol, atol=atol, equal_nan=equal_nan
        )
E       torch.OutOfMemoryError: CUDA out of memory. Tried to allocate 20.00 MiB. GPU 0 has a total capacity of 44.39 GiB of which 16.06 MiB is free. [...] Process 2449 has 10.78 GiB memory in use. [...]

.venv/lib/python3.12/site-packages/torch/testing/_comparison.py:1113: OutOfMemoryError

Why the rerun filter did not fire

pytest-rerunfailures matches --only-rerun against the outermost exception only:

def _try_match_error(rerun_errors, excinfo):
    if excinfo:
        err = f"{excinfo.type.__name__}: {excinfo.value}"
        for rerun_regex in rerun_errors:
            if re.search(rerun_regex, err):
                return True
    return False

The chained __cause__ is never consulted. So:

• CUBLAS_STATUS_ALLOC_FAILED arrives as a plain RuntimeError that no pattern matches.
• The OOM arrives wrapped by torch.testing, which raises RuntimeError("Comparing\n\n{pair}\n\nresulted in the unexpected exception above. ...") and drops the original error text from the message. OutOfMemoryError is only the chained cause, so neither OutOfMemoryError nor CUDA out of memory can match.

Reproduced with synthetic failures for each shape: against the current filter, only a directly raised OOM is retried, while a cuBLAS allocation error and an OOM wrapped by assert_close are failed immediately.

[albertvillanova]

Fix:

• Retry tests failing on cuBLAS allocation errors the rerun filter missed #6916

[albertvillanova]

Partial fix:

• Retry tests failing on cuBLAS allocation errors the rerun filter missed #6916

That PR covers the CUBLAS_STATUS_ALLOC_FAILED half, which is unambiguous.

The wrapped-OOM half is not fixable on our side. torch.testing raises RuntimeError("Comparing\n\n{pair}\n\nresulted in the unexpected exception above. ...") and drops the original error text, so the outer message contains only the tensor pair repr and boilerplate, with OutOfMemoryError demoted to a chained cause. Nothing in it indicates memory pressure, so no --only-rerun regex can tell it apart from a real defect surfacing inside a comparison. Matching the wrapper text would retry genuine bugs, including nondeterministic ones that a narrow filter exists precisely to avoid masking.

The proper fix is for pytest-rerunfailures to match --only-rerun against the exception chain rather than only the outermost exception, so that a wrapped OutOfMemoryError is still recognized. Keeping this issue open for that.

[albertvillanova]

Reported upstream in pytest-rerunfailures:

• only_rerun does not match exceptions wrapped by another exception (__cause__ / __context__ not inspected) pytest-dev/pytest-rerunfailures#353

If they take the chain-walking match, our existing OutOfMemoryError pattern will cover the wrapped case on its own, with no wrapper-text heuristic needed and no further change to the Makefile.