BlockingIOError and File descriptor xx is used by transport

[xiaoxiper]

Describe the bug

I encountered a similar issue while using the latest version of aiohttp
aiohappyeyeballs: 2.6.1

#10506 (comment)

BlockingIOError: [Errno 115] Operation now in progress

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
 File "<inner>.py", line 135, in plan
 agent_result, _ = await asyncio.gather(
 File "<inner>.py", line 175, in a_invoke
 result = await func(parameter)
 File "<inner>.py", line 1524, in load
 result.append(await func(parameter))
 File "<inner>.py", line 129, in convert
 enum = await func(parameter)
 File "<inner>.py", line 54, in enum
 await self.enum_func(parameter)
 File "<inner>.py", line 55, in wrapper
 result = await func(*args, **kwargs)
 File "<inner>.py", line 38, in load
 async with session.post(
 File "/usr/local/lib/python3.10/site-packages/aiohttp/client.py", line 1425, in __aenter__
 self._resp: _RetType = await self._coro
 File "/usr/local/lib/python3.10/site-packages/aiohttp/client.py", line 703, in _request
 conn = await self._connector.connect(
 File "/usr/local/lib/python3.10/site-packages/aiohttp/connector.py", line 548, in connect
 proto = await self._create_connection(req, traces, timeout)
 File "/usr/local/lib/python3.10/site-packages/aiohttp/connector.py", line 1056, in _create_connection
 _, proto = await self._create_direct_connection(req, traces, timeout)
 File "/usr/local/lib/python3.10/site-packages/aiohttp/connector.py", line 1383, in _create_direct_connection
 transp, proto = await self._wrap_create_connection(
 File "/usr/local/lib/python3.10/site-packages/aiohttp/connector.py", line 1116, in _wrap_create_connection
 sock = await aiohappyeyeballs.start_connection(
 File "/usr/local/lib/python3.10/site-packages/aiohappyeyeballs/impl.py", line 127, in start_connection
 raise first_exception
 File "/usr/local/lib/python3.10/site-packages/aiohappyeyeballs/_staggered.py", line 132, in run_one_coro
 result = await coro_fn()
 File "/usr/local/lib/python3.10/site-packages/aiohappyeyeballs/impl.py", line 208, in _connect_sock
 await loop.sock_connect(sock, address)
 File "/usr/local/lib/python3.10/asyncio/selector_events.py", line 499, in sock_connect
 self._sock_connect(fut, sock, address)
 File "/usr/local/lib/python3.10/asyncio/selector_events.py", line 515, in _sock_connect
 self._ensure_fd_no_transport(fd)
 File "/usr/local/lib/python3.10/asyncio/selector_events.py", line 248, in _ensure_fd_no_transport
 raise RuntimeError(
 RuntimeError: File descriptor 68 is used by transport <_SelectorSocketTransport fd=68 read=polling write=<idle, bufsize=0>>
'}

To Reproduce

It's difficult to reproduce directly - currently unable to reproduce with script-based stress testing; it can only be reproduced in the production environment and occurs after a period of testing

Expected behavior

Logs/tracebacks

-

Python Version

$ python --version
3.10.14

aiohttp Version

$ python -m pip show aiohttp
Name: aiohttp                                                                                                                                                                                                            
Version: 3.11.14                                                                                                                                                                                                         
Summary: Async http client/server framework (asyncio)                                                                                                                                                                    
Home-page: https://github.com/aio-libs/aiohttp                                                                                                                                                                           
Author:                                                                                                                                                                                                                  
Author-email:                                                                                                                                                                                                            
License: Apache-2.0                                                                                                                                                                                                      
Location: /usr/local/lib/python3.10/site-packages                                                                                                                                                                        
Requires: aiohappyeyeballs, aiosignal, async-timeout, attrs, frozenlist, multidict, propcache, yarl                                                                                                                      
Required-by: livekit-api

multidict Version

$ python -m pip show multidict
Name: multidict                                                                                                                                                                                                          
Version: 6.2.0                                                                                                                                                                                                           
Summary: multidict implementation                                                                                                                                                                                        
Home-page: https://github.com/aio-libs/multidict                                                                                                                                                                         
Author: Andrew Svetlov                                                                                                                                                                                                   
Author-email: andrew.svetlov@gmail.com                                                                                                                                                                                   
License: Apache 2                                                                                                                                                                                                        
Location: /usr/local/lib/python3.10/site-packages                                                                                                                                                                        
Requires: typing-extensions                                                                                                                                                                                              
Required-by: aiohttp, yarl

propcache Version

$ python -m pip show propcache
Name: propcache                                                                                                                                                                                                          
Version: 0.3.0                                                                                                                                                                                                           
Summary: Accelerated property cache                                                                                                                                                                                      
Home-page: https://github.com/aio-libs/propcache                                                                                                                                                                         
Author: Andrew Svetlov                                                                                                                                                                                                   
Author-email: andrew.svetlov@gmail.com                                                                                                                                                                                   
License: Apache-2.0                                                                                                                                                                                                      
Location: /usr/local/lib/python3.10/site-packages                                                                                                                                                                        
Requires:                                                                                                                                                                                                                
Required-by: aiohttp, yarl

yarl Version

$ python -m pip show yarl
Name: yarl                                                                                                                                                                                                               
Version: 1.18.3                                                                                                                                                                                                          
Summary: Yet another URL library                                                                                                                                                                                         
Home-page: https://github.com/aio-libs/yarl                                                                                                                                                                              
Author: Andrew Svetlov                                                                                                                                                                                                   
Author-email: andrew.svetlov@gmail.com                                                                                                                                                                                   
License: Apache-2.0                                                                                                                                                                                                      
Location: /usr/local/lib/python3.10/site-packages                                                                                                                                                                        
Requires: idna, multidict, propcache                                                                                                                                                                                     
Required-by: aiohttp

OS

PRETTY_NAME="Debian GNU/Linux 12 (bookworm)"
NAME="Debian GNU/Linux"
VERSION_ID="12"
VERSION="12 (bookworm)"
VERSION_CODENAME=bookworm
ID=debian
HOME_URL="https://www.debian.org/"
SUPPORT_URL="https://www.debian.org/support"
BUG_REPORT_URL="https://bugs.debian.org/"

Related component

Client

Additional context

No response

Code of Conduct

• I agree to follow the aio-libs Code of Conduct

[bdraco]

I don't think we will be able to do anything without a reproducer here as I've already been over the aiohttp and aiohappyeyeballs code line by line for many days looking for potential leaks and have not been able to find any.

The only thing that hasn't been audited is CPython loop.sock_connect for cancellation safety issues.

[xiaoxiper]

What other issues might be causing this exception? When this error occurs, all aiohttp-related code requests fail despite using independent client sessions. I've reviewed the surrounding logs and found no unusual activity.

Do you have suggestions for reproducing this issue?
What indicators should I monitor?
How can I mitigate this issue?
Should I consider switching to different versions?

Any leads would be greatly appreciated.

[bdraco]

What other issues might be causing this exception

Anything that creates an asyncio.Transport and than potentially reuses the fd. aiohttp is going to create them frequently so it could be that aiohttp is getting unlucky and losing a race against something else creating these.

? When this error occurs, all aiohttp-related code requests fail despite using independent client sessions. I've reviewed the surrounding logs and found no unusual activity.

Do you have suggestions for reproducing this issue? What indicators should I monitor?

I don't know how you are using aiohttp or if there are other things running in the event loop that are allocating file handles so its hard to give you a suggestion.

How can I mitigate this issue?

If you don't care about IPv6/IPv4 failover, you could try turning off the happyeyeballs by setting happy_eyeballs_delay=None on the TCPConnector
https://docs.aiohttp.org/en/stable/client_reference.html#aiohttp.TCPConnector

Should I consider switching to different versions?

Can you try Python 3.12+ and see if you can still reproduce the issue? We haven't had any reports with newer Python versions.

Any leads would be greatly appreciated.

[olindho1]

I've been seeing this issue in our CI systems as well last 2-3 weeks. As described, hard to reproduce, happens intermittently. To mitigate this issue we locked our package versions as per #10506 (comment) and #10561 .

(Python 3.10.12)

aiohttp==3.11.14
aiohappyeyeballs==2.4.8

Unfortunately, we are still seeing the issue.

[bdraco]

Do you have a backtrace that includes the BlockingIOError. It would be helpful to figure out where its being raised

Its got to be at https://github.com/python/cpython/blob/b8b4b713c5f8ec0958c7ef8d29d6711889bc94ab/Lib/asyncio/selector_events.py#L510

[bdraco]

Also are you running a Python version with python/cpython#88863 fixed?

[bdraco]

I got around to checking the cpython code for races:

So there is a race in cpython if loop.sock_connect gets cancelled (timeout in our case):

https://github.com/python/cpython/blob/b8b4b713c5f8ec0958c7ef8d29d6711889bc94ab/Lib/asyncio/selector_events.py#L518

self._sock_connect(fut, sock, address) gets called, a future gets returned, the connect times out, the future gets cancelled, but the writer is already added, but the cleanup doesn't happen until one event loop iteration later because its delivered via add_done_callback which uses call_soon https://github.com/python/cpython/blob/b8b4b713c5f8ec0958c7ef8d29d6711889bc94ab/Lib/asyncio/selector_events.py#L518 https://github.com/python/cpython/blob/b8b4b713c5f8ec0958c7ef8d29d6711889bc94ab/Lib/asyncio/selector_events.py#L530

But sock_connect returns before its actually torn down because of the delay, than we close the socket, and it gets reallocated before the call_soon cleans it up for asyncio so it thinks its still in use

[bdraco]

python/cpython#131728

[bdraco]

@xiaoxiper Can you try #10624 ?

[olindho1]

Do you have a backtrace that includes the BlockingIOError. It would be helpful to figure out where its being raised

Its got to be at https://github.com/python/cpython/blob/b8b4b713c5f8ec0958c7ef8d29d6711889bc94ab/Lib/asyncio/selector_events.py#L510

That's correct. The traceback (with proprietary parts redacted) is:

Traceback (most recent call last):
  File "/usr/lib/python3.10/asyncio/selector_events.py", line 509, in _sock_connect
    sock.connect(address)
BlockingIOError: [Errno 115] Operation now in progress

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "lib/python3.10/site-packages/aiohttp/client.py", line 1425, in __aenter__
    self._resp: _RetType = await self._coro
  File "lib/python3.10/site-packages/aiohttp/client.py", line 703, in _request
    conn = await self._connector.connect(
  File "lib/python3.10/site-packages/aiohttp/connector.py", line 548, in connect
    proto = await self._create_connection(req, traces, timeout)
  File "lib/python3.10/site-packages/aiohttp/connector.py", line 1056, in _create_connection
    _, proto = await self._create_direct_connection(req, traces, timeout)
  File "lib/python3.10/site-packages/aiohttp/connector.py", line 1383, in _create_direct_connection
    transp, proto = await self._wrap_create_connection(
  File "lib/python3.10/site-packages/aiohttp/connector.py", line 1116, in _wrap_create_connection
    sock = await aiohappyeyeballs.start_connection(
  File "lib/python3.10/site-packages/aiohappyeyeballs/impl.py", line 115, in start_connection
    raise first_exception
  File "lib/python3.10/site-packages/aiohappyeyeballs/impl.py", line 72, in start_connection
    sock = await _connect_sock(
  File "lib/python3.10/site-packages/aiohappyeyeballs/impl.py", line 197, in _connect_sock
    await loop.sock_connect(sock, address)
  File "/usr/lib/python3.10/asyncio/selector_events.py", line 499, in sock_connect
    self._sock_connect(fut, sock, address)
  File "/usr/lib/python3.10/asyncio/selector_events.py", line 515, in _sock_connect
    self._ensure_fd_no_transport(fd)
  File "/usr/lib/python3.10/asyncio/selector_events.py", line 248, in _ensure_fd_no_transport
    raise RuntimeError(
RuntimeError: File descriptor 53 is used by transport <_SelectorSocketTransport fd=53 read=polling write=<idle, bufsize=0>>

[olindho1]

Also are you running a Python version with python/cpython#88863 fixed?

At least for me, as I am running Python 3.10.12 (more specifically 3.10.12 (main, Feb 4 2025, 14:57:36) [GCC 11.4.0] installed/setup using uv venv) I believe I am running with a python version with this fixed. At least looking at python/cpython#88863 it should have been fixed by python/cpython#99722 which was merged in python/cpython@b191bc3 and is in tag v3.10.12.