Stabilize test suite: fix flaky timing tests, add rerun support, and handle Windows resource leaks

[rodrigobnogueira]

Description

What do these changes do?

Timing-based performance tests:

• test_import_time (test_imports.py): Simplify threshold logic — replace per-version/xdist/CI lookup with a single sys.version_info >= (3, 12) check (200ms / 300ms) to reduce flakiness.
• test_regex_performance (test_client_middleware_digest_auth.py): Raise threshold from 10ms to 80ms, add descriptive assertion message with elapsed time.
• test_forwarded_re_performance (test_web_request.py): Raise threshold from 10ms to 80ms, add descriptive assertion message with elapsed time.
• test_cookie_pattern_performance (test_cookie_helpers.py): Raise threshold from 10ms to 80ms, add descriptive assertion message with elapsed time.

Flaky test fix:

• test_uvloop_secure_https_proxy (test_proxy_functional.py): Replace connection to real example.com with a local TestServer using trustme certificates. Fixes CERTIFICATE_VERIFY_FAILED on macOS CI where the system trust store may lack public CAs.

Test infrastructure:

• pytest_configure (tests/conftest.py): Broaden Windows socket warning filter in Python < 3.12.
• make_client_request fixture (tests/conftest.py): Fix session leak by tracking multiple requests/sessions in lists, use asyncio.gather for parallel cleanup.

Are there changes in behavior for the user?

No user-facing changes. This only affects CI test behavior.

Is it a substantial burden for the maintainers to support this?

No — this actually reduces maintenance burden by making the test future-proof.

Related issue number

Fixes flaky test failures in PR #11990.

Checklist

• I think the code is well written
• Unit tests for the changes exist
• Documentation reflects the changes
• If you provide code modification, please add yourself to CONTRIBUTORS.txt
• Add a new news fragment into the CHANGES/ folder

[codecov]

Codecov Report

✅ All modified and coverable lines are covered by tests.
✅ Project coverage is 98.76%. Comparing base (9756a1f) to head (cbc3021).
⚠️ Report is 2 commits behind head on master.
✅ All tests successful. No failed tests found.

Additional details and impacted files

@@            Coverage Diff             @@
##           master   #11992      +/-   ##
==========================================
+ Coverage   98.73%   98.76%   +0.02%     
==========================================
  Files         127      127              
  Lines       44667    44675       +8     
  Branches     2372     2371       -1     
==========================================
+ Hits        44102    44122      +20     
+ Misses        402      393       -9     
+ Partials      163      160       -3     

Flag	Coverage Δ
CI-GHA	98.61% <96.87%> (+0.02%)	⬆️
OS-Linux	98.35% <93.75%> (+0.01%)	⬆️
OS-Windows	96.69% <53.12%> (-0.02%)	⬇️
OS-macOS	97.59% <81.25%> (+0.12%)	⬆️
Py-3.10.11	97.14% <84.37%> (+<0.01%)	⬆️
Py-3.10.19	97.62% <93.75%> (-0.01%)	⬇️
Py-3.11.14	97.82% <93.75%> (-0.01%)	⬇️
Py-3.11.9	97.34% <84.37%> (+<0.01%)	⬆️
Py-3.12.10	97.44% <81.25%> (+<0.01%)	⬆️
Py-3.12.12	97.92% <93.75%> (?)
Py-3.13.12	98.16% <93.75%> (+0.01%)	⬆️
Py-3.14.3	98.13% <62.50%> (+<0.01%)	⬆️
Py-3.14.3t	97.21% <62.50%> (?)
Py-pypy3.11.13-7.3.20	97.36% <50.00%> (?)
VM-macos	97.59% <81.25%> (+0.12%)	⬆️
VM-ubuntu	98.35% <93.75%> (+0.01%)	⬆️
VM-windows	96.69% <53.12%> (-0.02%)	⬇️

Flags with carried forward coverage won't be shown. Click here to find out more.

☔ View full report in Codecov by Sentry.
📢 Have feedback on the report? Share it here.

[codspeed-hq]

Merging this PR will not alter performance

✅ 59 untouched benchmarks

---

_{Comparing rodrigobnogueira:fix/import-time-test-python-3.14 (cbc3021) with master (100e5ed)}

[Dreamsorcerer]

I think the expectation was that import time would improve again in later versions. I think flaky failures today are mostly caused by regressions (e.g. idna: kjd/idna#188).

[rodrigobnogueira]

The CI failure on test_regex_performance popped up again in the macOS 3.12 job (as seen before—it's another timing-based test with a tight <10ms assertion).

I've updated it. Let me know if this looks good

[webknjaz]

Can we try using https://pypi.org/p/pytest-rerunfailures? It should be more generic.

[Dreamsorcerer]

The CI failure on test_regex_performance popped up again in the macOS 3.12 job (as seen before—it's another timing-based test with a tight <10ms assertion).

I think we can just double that to 20ms. It's meant to highlight an issue as an order of magnitude, so I clearly misjudged the performance of Mac OS.

[rodrigobnogueira]

Thanks @webknjaz for the suggesting pytest-rerunfailures.
A helper function was introduced for these time-based tests.
There are other tests that might be modified if you want: test_forwarded_re_performance and test_cookie_pattern_performance, both have a hardcoded 10ms threshold

[webknjaz]

@bdraco any ideas on designing this better?

[rodrigobnogueira]

As I mentioned before, there are two more tests evaluating the 10ms threshold. One of them just failed in the last commit.

Job Test (3.14, windows, false)

At this point it is easy to mark them to rerun.

================================== FAILURES ===================================
_______________________ test_cookie_pattern_performance _______________________
[gw2] win32 -- Python 3.14.2 C:\hostedtoolcache\windows\Python\3.14.2\x64\python.exe

    def test_cookie_pattern_performance() -> None:
        value = "a" + "=" * 21651 + "\x00"
        start = time.perf_counter()
        match = helpers._COOKIE_PATTERN.match(value)
        end = time.perf_counter()
    
        # If this is taking more than 10ms, there's probably a performance/ReDoS issue.
>       assert (end - start) < 0.01
E       assert (804.512782 - 804.4838781) < 0.01

end        = 804.512782
match      = None
start      = 804.4838781
value      = 'a====================================================================================================================...==================================================================================================================\x00'

tests\test_cookie_helpers.py:647: AssertionError

[rodrigobnogueira]

While trying to address the unclosed socket ResourceWarnings that raised in Windows test jobs, I noticed the make_client_request fixture was using single nonlocal variables to store the session and request created by the factory function. This meant that if a test called the factory multiple times, the reference to the previous session was overwritten and lost. Consequently, the fixture's teardown code only closed the last session created.
The issue is observed in tests/test_connector.py::test_connector_multiple_event_loop.

[webknjaz]

I think it's good but let me know if you'd like to drop that helper function before merging.

Urgh.. looks like my browser cached the previous diff and I commented on the wrong thing.

[rodrigobnogueira]

The test_uvloop_secure_https_proxy was failing on macOS CI because it connects to real https://example.com through a local secure proxy. Error verified: CERTIFICATE_VERIFY_FAILED.

Fixed by replacing the external request with a local TestServer using trustme-issued certificates, matching the pattern of test_secure_https_proxy_absolute_path.

[rodrigobnogueira]

Analysis of 20 PRs shows that 16 of them had failures that are addressed here.

Failure counts addressed:

test_uvloop_secure_https_proxy: 5 failures
test_cookie_pattern_performance: 4 failures
test_import_time: 4 failures
test_regex_performance: 3 failures

#12069: test_uvloop_secure_https_proxy
#12067: test_uvloop_secure_https_proxy
#12066: test_regex_performance
#12064: test_regex_performance
12063: test_middleware_uses_session_avoids_recursion_with_disabled_middleware (not addressed by this PR)
#12061: test_import_time
#12056: test_cookie_pattern_performance
#12055: test_cookie_pattern_performance
#12054: test_regex_performance
12046: CI passed
#12045: test_cookie_pattern_performance
12044: other errors
#12042: test_cookie_pattern_performance
#12041: test_uvloop_secure_https_proxy
#12038: test_import_time
#12030: test_uvloop_secure_https_proxy
11989: CI passed
#11966: test_uvloop_secure_https_proxy
#11947: test_import_time
#11942: test_import_time

[Dreamsorcerer]

test_uvloop_secure_https_proxy: 5 failures

That's not flaky, either the certificate changed or the runners changed.

[Dreamsorcerer]

I'm happy with this, so I'm going to merge this to get the CI unblocked again.

[patchback]

Backport to 3.13: 💔 cherry-picking failed — conflicts found

❌ Failed to cleanly apply 32924e8 on top of patchback/backports/3.13/32924e8fa4c9085dcbaf41b3b3c27fc773bfa213/pr-11992

Backporting merged PR #11992 into master

1. Ensure you have a local repo clone of your fork. Unless you cloned it
   from the upstream, this would be your origin remote.
2. Make sure you have an upstream repo added as a remote too. In these
   instructions you'll refer to it by the name upstream. If you don't
   have it, here's how you can add it:

   $ git remote add upstream https://github.com/aio-libs/aiohttp.git

3. Ensure you have the latest copy of upstream and prepare a branch
   that will hold the backported code:

   $ git fetch upstream
   $ git checkout -b patchback/backports/3.13/32924e8fa4c9085dcbaf41b3b3c27fc773bfa213/pr-11992 upstream/3.13

4. Now, cherry-pick PR Stabilize test suite: fix flaky timing tests, add rerun support, and handle Windows resource leaks #11992 contents into that branch:

   $ git cherry-pick -x 32924e8fa4c9085dcbaf41b3b3c27fc773bfa213

   If it'll yell at you with something like fatal: Commit 32924e8fa4c9085dcbaf41b3b3c27fc773bfa213 is a merge but no -m option was given., add -m 1 as follows instead:

   $ git cherry-pick -m1 -x 32924e8fa4c9085dcbaf41b3b3c27fc773bfa213

5. At this point, you'll probably encounter some merge conflicts. You must
   resolve them in to preserve the patch from PR Stabilize test suite: fix flaky timing tests, add rerun support, and handle Windows resource leaks #11992 as close to the
   original as possible.
6. Push this branch to your fork on GitHub:

   $ git push origin patchback/backports/3.13/32924e8fa4c9085dcbaf41b3b3c27fc773bfa213/pr-11992

7. Create a PR, ensure that the CI is green. If it's not — update it so that
   the tests and any other checks pass. This is it!
   Now relax and wait for the maintainers to process your pull request
   when they have some cycles to do reviews. Don't worry — they'll tell you if
   any improvements are necessary when the time comes!

🤖 @patchback
I'm built with octomachinery and
my source is open — https://github.com/sanitizers/patchback-github-app.

[patchback]

Backport to 3.14: 💔 cherry-picking failed — conflicts found

❌ Failed to cleanly apply 32924e8 on top of patchback/backports/3.14/32924e8fa4c9085dcbaf41b3b3c27fc773bfa213/pr-11992

Backporting merged PR #11992 into master

1. Ensure you have a local repo clone of your fork. Unless you cloned it
   from the upstream, this would be your origin remote.
2. Make sure you have an upstream repo added as a remote too. In these
   instructions you'll refer to it by the name upstream. If you don't
   have it, here's how you can add it:

   $ git remote add upstream https://github.com/aio-libs/aiohttp.git

3. Ensure you have the latest copy of upstream and prepare a branch
   that will hold the backported code:

   $ git fetch upstream
   $ git checkout -b patchback/backports/3.14/32924e8fa4c9085dcbaf41b3b3c27fc773bfa213/pr-11992 upstream/3.14

4. Now, cherry-pick PR Stabilize test suite: fix flaky timing tests, add rerun support, and handle Windows resource leaks #11992 contents into that branch:

   $ git cherry-pick -x 32924e8fa4c9085dcbaf41b3b3c27fc773bfa213

   If it'll yell at you with something like fatal: Commit 32924e8fa4c9085dcbaf41b3b3c27fc773bfa213 is a merge but no -m option was given., add -m 1 as follows instead:

   $ git cherry-pick -m1 -x 32924e8fa4c9085dcbaf41b3b3c27fc773bfa213

5. At this point, you'll probably encounter some merge conflicts. You must
   resolve them in to preserve the patch from PR Stabilize test suite: fix flaky timing tests, add rerun support, and handle Windows resource leaks #11992 as close to the
   original as possible.
6. Push this branch to your fork on GitHub:

   $ git push origin patchback/backports/3.14/32924e8fa4c9085dcbaf41b3b3c27fc773bfa213/pr-11992

7. Create a PR, ensure that the CI is green. If it's not — update it so that
   the tests and any other checks pass. This is it!
   Now relax and wait for the maintainers to process your pull request
   when they have some cycles to do reviews. Don't worry — they'll tell you if
   any improvements are necessary when the time comes!

🤖 @patchback
I'm built with octomachinery and
my source is open — https://github.com/sanitizers/patchback-github-app.

[rodrigobnogueira]

Thank you 👏