Flaky backend tests: a bare `TestClient` closes the event loop before the background task finishes

[hysts]

Describe the bug

Three backend tests fail intermittently, and nothing in the failure points at the real cause:

• test/test_history.py::TestServerSideRecording::test_a_normal_prediction_is_recorded
• test/test_history.py::TestWorkflowRecording::test_internal_dependencies_are_not_recorded
• test/test_queueing.py::test_heartbeat_task_cancelled_after_stream_completes

The first two wait 30 seconds for a record that never arrives and fail with "no record was written". The third asserts that a heartbeat task is cancelled() or done() and finds it still pending.

This is live on main, not just on branches. The python run for 19cb8f7bc (run 33894903556) has test-ubuntu-latest-not-flaky at failure with FAILED test/test_history.py::TestServerSideRecording::test_a_normal_prediction_is_recorded - AssertionError: no record was written, while the workflow run itself concluded success and the commit carries a green tick. More on why that happens at the end.

The cause is in the tests rather than in the library. starlette's TestClient keeps one event loop for its whole lifetime only when it is entered as a context manager (starlette 1.6.0, testclient.py):

@contextlib.contextmanager
def _portal_factory(self):
    if self.portal is not None:   # set by __enter__
        yield self.portal
    else:                         # otherwise, one portal per request
        with anyio.from_thread.start_blocking_portal(**self.async_backend) as portal:
            yield portal

All three tests build a bare TestClient(app), so every request gets its own portal and that loop is closed as soon as the response is complete.

gradio/history.py::schedule_record_run files the record with asyncio.get_running_loop().create_task(_run()), deliberately without making the caller wait, and record_run then awaits anyio.to_thread.run_sync(save_record, ...). That task lives on the request's portal loop. If the offload has not come back by the time the response is done, the loop closes with the task still pending and it is never resumed. The heartbeat task the queueing test tracks is left pending the same way, which is why it is neither cancelled() nor done().

Under uvicorn the loop outlives the request, so this does not affect a running server. It only bites where a loop is created and torn down per request, which is what the bare TestClient does.

Instrumenting schedule_record_run confirms the mechanism rather than inferring it. On main's own sources the probe prints task created and task started with no wrote record after it. The same probe on a CI runner gets one step further and stops at the offload:

CIPROBE task created
CIPROBE task started, 8 threads: [...]
CIPROBE record_run: at limiter, borrowed=0/8
CIPROBE record_run: got limiter, offloading
                          <- never returns

borrowed=0/8 also rules out the "write pool saturated, dropping record" branch.

These tests are not new. #13763 landed the bucket-backed run history on 2026-09-02 and test/test_history.py came with it, so they have been flaky since they were added.

Why it went unnoticed: .github/workflows/test-python.yml sets continue-on-error: true on the test matrix job (line 103). The job's own check run is still recorded as failure, so a PR really is gated on it, but the check suite and the workflow run both roll up to success. The roll-up is what a commit's tick and gh run list show, so a failure on a push to main, where there is no PR to gate, leaves no visible trace at all. 19cb8f7bc above is exactly that: red job, green commit. Four test_load_assets failures reached main the same way until a release PR made them look like release breakage (#13697).

Reproduction

This is not an app-level bug, so the reproduction is the test suite itself. In a source checkout with the backend test requirements installed, run test/test_history.py under the same command CI uses, several times:

for i in 1 2 3 4 5; do
  python -m pytest -n auto test/test_history.py -m "not flaky and not serial" -q | tail -1
done

On main at 19cb8f7bc, 4 of 5 runs failed for me. The run time is a good tell: a failing run takes 37 to 43 seconds because it sits out the full 30 second _wait_for, while a clean run finishes in 7 to 17 seconds.

Logs

main run 1: 21 passed, 18 warnings in 12.48s :: none
main run 2: 1 failed, 20 passed, 18 warnings in 38.33s :: TestWorkflowRecording::test_internal_dependencies_are_not_recorded
main run 3: 1 failed, 20 passed, 18 warnings in 37.70s :: TestServerSideRecording::test_a_normal_prediction_is_recorded
main run 4: 1 failed, 20 passed, 18 warnings in 37.61s :: TestWorkflowRecording::test_internal_dependencies_are_not_recorded
main run 5: 2 failed, 19 passed, 19 warnings in 42.43s :: TestWorkflowRecording::test_internal_dependencies_are_not_recorded,TestServerSideRecording::test_a_normal_prediction_is_recorded

The same pattern shows up in the whole backend suite under CI's command (pytest -n auto -m "not flaky and not serial"): 4 of 5 runs failed, each time on one of the two recording tests.

System Info

gradio: source checkout, main @ 19cb8f7bc
python: 3.10
os: Ubuntu 22.04 (x86_64)
starlette: 1.6.0
anyio: 4.14.2
pytest: 9.1.1, run with pytest-xdist (-n auto)

Severity

I can work around it