Calling del on a not fully closed transport can generate a warning

[ffourcot]

Hello,

We found one pyroute2 related warning in our test case. The simplest reproducer is this code:

from pyroute2 import IPRoute

async def test_main():
    ipr = IPRoute()
    print(ipr.get_addr())
    ipr.close()

When we call pytest with -X dev option, a warning is emitted:

$ python3 -X dev -m pytest reproducer.py 
========================================= test session starts ==========================================
platform linux -- Python 3.11.2, pytest-7.2.1, pluggy-1.0.0+repack
rootdir: /notrelevant, configfile: tox.ini
plugins: asyncio-0.20.3,anyio-3.6.2
asyncio: mode=Mode.AUTO
collected 1 item                                                                                       

[...]

==================================== 1 passed, 2 warnings in 0.73s =====================================
/usr/lib/python3.11/asyncio/selector_events.py:843: ResourceWarning: unclosed transport <_SelectorDatagramTransport closing fd=12>
  _warn(f"unclosed transport {self!r}", ResourceWarning, source=self)
ResourceWarning: Enable tracemalloc to get the object allocation traceback

The easiest fix (but maybe not cleanest...) is this patch:

--- a/pyroute2/netlink/core.py
+++ b/pyroute2/netlink/core.py
@@ -462,6 +462,7 @@ class AsyncCoreSocket:
             if hasattr(self.local, 'transport'):
                 self.local.transport.abort()
                 self.local.transport.close()
+                self.event_loop.run_until_complete(asyncio.sleep(0))
                 del self.local.transport
                 del self.local.protocol
             if self.status['event_loop'] == 'new':

From my understanding, you are calling a close() without giving a chance to the event loop to really close the socket. Giving a chance to the event loop to run is enough to really clean the status.

Since it's only when ResourceWarning is enabled, it's probably never seen in production. We managed it in our code base with our own await asyncio.sleep(0) in tests teardown, but it's perhaps something to have a look on.

[svinota]

Thanks, it needs to be fixed indeed.

[ttarabula]

We're seeing this as well, but manifesting slightly differently - in our case as ValueError being thrown during some sort of destructor operation:

Exception ignored in: <function BaseEventLoop.__del__ at ...>
Traceback (most recent call last):
  File ".../lib/python3.12/asyncio/base_events.py", line 732, in __del__
    self.close()
  File ".../lib/python3.12/asyncio/unix_events.py", line 68, in close
    super().close()
  File ".../lib/python3.12/asyncio/selector_events.py", line 104, in close
    self._close_self_pipe()
  File ".../lib/python3.12/asyncio/selector_events.py", line 111, in _close_self_pipe
    self._remove_reader(self._ssock.fileno())
  File ".../lib/python3.12/asyncio/selector_events.py", line 298, in _remove_reader
    key = self._selector.get_key(fd)
          ^^^^^^^^^^^^^^^^^^^^^^^^^^
  File ".../lib/python3.12/selectors.py", line 190, in get_key
    return mapping[fileobj]
           ~~~~~~~^^^^^^^^^
  File ".../lib/python3.12/selectors.py", line 71, in __getitem__
    fd = self._selector._fileobj_lookup(fileobj)
         ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File ".../lib/python3.12/selectors.py", line 225, in _fileobj_lookup
    return _fileobj_to_fd(fileobj)
           ^^^^^^^^^^^^^^^^^^^^^^^
  File ".../lib/python3.12/selectors.py", line 42, in _fileobj_to_fd
    raise ValueError("Invalid file descriptor: {}".format(fd))
ValueError: Invalid file descriptor: -1

The event_loop.run_until_complete() patch above seems to fix the problem for us as well, as does pinning pyroute2 v0.9.2. Just an fyi - we'll wait for the official fix, as it doesn't sound like this way is the "cleanest" way to handle it.

Thanks!

Linux, Python 3.12.10, pyroute2 v0.9.5