Skip to content

Pre ping failing with uvloop #966

Open
@chmeldax

Description

@chmeldax

Describe the bug

When using aiomysql with uvloop==0.17.0, whenever MySQL closes a connection (due to wait_timeout), the subsequent pre-ping command fails with:

RuntimeError: unable to perform operation on <TCPTransport closed=True reading=False 0x62afa5bf29a0>; the handler is closed

This bug does not occur when uvloop is not used. Not sure whether to raise the issue with uvloop instead.

Might be related to #195?

To Reproduce

  1. Use uvloop==0.17.0 and uvicorn==0.23.2.
  2. Create a connection pool with pre_ping=True and pool_recycle not set.
  3. Fire some queries.
  4. Let MySQL close the connection due to the wait_timeout.
  5. Send a HTTP request to the app and let the pre-ping request fail.

Expected behavior

The pre-ping request should succeed, and the HTTP request should succeed as well.

Logs/tracebacks

ERROR [root] Error: unable to perform operation on <TCPTransport closed=True reading=False 0x62afa5bd73a0>; the handler is closed
 ERROR [root] Stack: Traceback (most recent call last):
   File "/usr/local/lib/python3.11/site-packages/anyio/streams/memory.py", line 97, in receive
     return self.receive_nowait()
            ^^^^^^^^^^^^^^^^^^^^^
   File "/usr/local/lib/python3.11/site-packages/anyio/streams/memory.py", line 92, in receive_nowait
     raise WouldBlock
 anyio.WouldBlock

 During handling of the above exception, another exception occurred:

 Traceback (most recent call last):
   File "/usr/local/lib/python3.11/site-packages/starlette/middleware/base.py", line 78, in call_next
     message = await recv_stream.receive()
               ^^^^^^^^^^^^^^^^^^^^^^^^^^^
   File "/usr/local/lib/python3.11/site-packages/anyio/streams/memory.py", line 112, in receive
     raise EndOfStream
 anyio.EndOfStream

 During handling of the above exception, another exception occurred:

 Traceback (most recent call last):
   File "/usr/local/lib/python3.11/site-packages/hisd_be4fe/api/errors.py", line 33, in catch_middleware_exceptions
     return await call_next(request)
            ^^^^^^^^^^^^^^^^^^^^^^^^
   File "/usr/local/lib/python3.11/site-packages/starlette/middleware/base.py", line 84, in call_next
     raise app_exc
   File "/usr/local/lib/python3.11/site-packages/starlette/middleware/base.py", line 70, in coro
     await self.app(scope, receive_or_disconnect, send_no_error)
   File "/usr/local/lib/python3.11/site-packages/starlette/middleware/cors.py", line 91, in __call__
     await self.simple_response(scope, receive, send, request_headers=headers)
   File "/usr/local/lib/python3.11/site-packages/starlette/middleware/cors.py", line 146, in simple_response
     await self.app(scope, receive, send)
   File "/usr/local/lib/python3.11/site-packages/starlette/middleware/exceptions.py", line 79, in __call__
     raise exc
   File "/usr/local/lib/python3.11/site-packages/starlette/middleware/exceptions.py", line 68, in __call__
     await self.app(scope, receive, sender)
   File "/usr/local/lib/python3.11/site-packages/fastapi/middleware/asyncexitstack.py", line 20, in __call__
     raise e
   File "/usr/local/lib/python3.11/site-packages/fastapi/middleware/asyncexitstack.py", line 17, in __call__
     await self.app(scope, receive, send)
   File "/usr/local/lib/python3.11/site-packages/starlette/routing.py", line 718, in __call__
     await route.handle(scope, receive, send)
   File "/usr/local/lib/python3.11/site-packages/starlette/routing.py", line 276, in handle
     await self.app(scope, receive, send)
   File "/usr/local/lib/python3.11/site-packages/starlette/routing.py", line 66, in app
     response = await func(request)
                ^^^^^^^^^^^^^^^^^^^
   File "/usr/local/lib/python3.11/site-packages/fastapi/routing.py", line 231, in app
     solved_result = await solve_dependencies(
                     ^^^^^^^^^^^^^^^^^^^^^^^^^
   File "/usr/local/lib/python3.11/site-packages/fastapi/dependencies/utils.py", line 622, in solve_dependencies
     solved = await call(**sub_values)
              ^^^^^^^^^^^^^^^^^^^^^^^^
   File "<makefun-gen-2>", line 2, in current_user_dependency
   File "/usr/local/lib/python3.11/site-packages/[redacted]/authentication/authenticator.py", line 143, in current_user_dependency
     user, _ = await self._authenticate(
               ^^^^^^^^^^^^^^^^^^^^^^^^^
   File "/usr/local/lib/python3.11/site-packages/[redacted]/authentication/authenticator.py", line 184, in _authenticate
     user = await strategy.read_token(token, user_manager, authorized, ignore_expired)
            ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
   File "/usr/local/lib/python3.11/site-packages/[redacted]/authentication/strategy/db/strategy.py", line 30, in read_token
     access_token = await self.access_token_db.get_by_token(token, max_age, authorized, ignore_expired)
                    ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
   File "/usr/local/lib/python3.11/site-packages/[redacted]/[redacted]/access_token.py", line 76, in get_by_token
     results = await self.session.execute(statement)
               ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
   File "/usr/local/lib/python3.11/site-packages/sqlalchemy/ext/asyncio/session.py", line 439, in execute
     result = await greenlet_spawn(
              ^^^^^^^^^^^^^^^^^^^^^
   File "/usr/local/lib/python3.11/site-packages/sqlalchemy/util/_concurrency_py3k.py", line 190, in greenlet_spawn
     result = context.throw(*sys.exc_info())
              ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
   File "/usr/local/lib/python3.11/site-packages/sqlalchemy/orm/session.py", line 2246, in execute
     return self._execute_internal(
            ^^^^^^^^^^^^^^^^^^^^^^^
   File "/usr/local/lib/python3.11/site-packages/sqlalchemy/orm/session.py", line 2131, in _execute_internal
     conn = self._connection_for_bind(bind)
            ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
   File "/usr/local/lib/python3.11/site-packages/sqlalchemy/orm/session.py", line 1998, in _connection_for_bind
     return trans._connection_for_bind(engine, execution_options)
            ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
   File "<string>", line 2, in _connection_for_bind
   File "/usr/local/lib/python3.11/site-packages/sqlalchemy/orm/state_changes.py", line 139, in _go
     ret_value = fn(self, *arg, **kw)
                 ^^^^^^^^^^^^^^^^^^^^
   File "/usr/local/lib/python3.11/site-packages/sqlalchemy/orm/session.py", line 1123, in _connection_for_bind
     conn = bind.connect()
            ^^^^^^^^^^^^^^
   File "/usr/local/lib/python3.11/site-packages/sqlalchemy/engine/base.py", line 3264, in connect
     return self._connection_cls(self)
            ^^^^^^^^^^^^^^^^^^^^^^^^^^
   File "/usr/local/lib/python3.11/site-packages/sqlalchemy/engine/base.py", line 145, in __init__
     self._dbapi_connection = engine.raw_connection()
                              ^^^^^^^^^^^^^^^^^^^^^^^
   File "/usr/local/lib/python3.11/site-packages/sqlalchemy/engine/base.py", line 3288, in raw_connection
     return self.pool.connect()
            ^^^^^^^^^^^^^^^^^^^
   File "/usr/local/lib/python3.11/site-packages/sqlalchemy/pool/base.py", line 452, in connect
     return _ConnectionFairy._checkout(self)
            ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
   File "/usr/local/lib/python3.11/site-packages/sqlalchemy/pool/base.py", line 1366, in _checkout
     with util.safe_reraise():
   File "/usr/local/lib/python3.11/site-packages/sqlalchemy/util/langhelpers.py", line 147, in __exit__
     raise exc_value.with_traceback(exc_tb)
   File "/usr/local/lib/python3.11/site-packages/sqlalchemy/pool/base.py", line 1304, in _checkout
     result = pool._dialect._do_ping_w_event(
              ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
   File "/usr/local/lib/python3.11/site-packages/sqlalchemy/engine/default.py", line 708, in _do_ping_w_event
     return self.do_ping(dbapi_connection)
            ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
   File "/usr/local/lib/python3.11/site-packages/sqlalchemy/dialects/mysql/mysqldb.py", line 171, in do_ping
     dbapi_connection.ping(False)
   File "/usr/local/lib/python3.11/site-packages/sqlalchemy/dialects/mysql/aiomysql.py", line 191, in ping
     return self.await_(self._connection.ping(reconnect))
            ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
   File "/usr/local/lib/python3.11/site-packages/sqlalchemy/util/_concurrency_py3k.py", line 125, in await_only
     return current.driver.switch(awaitable)  # type: ignore[no-any-return]
            ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
   File "/usr/local/lib/python3.11/site-packages/sqlalchemy/util/_concurrency_py3k.py", line 185, in greenlet_spawn
     value = await result
             ^^^^^^^^^^^^
   File "/usr/local/lib/python3.11/site-packages/aiomysql/connection.py", line 493, in ping
     await self._execute_command(COMMAND.COM_PING, "")
   File "/usr/local/lib/python3.11/site-packages/aiomysql/connection.py", line 724, in _execute_command
     self._write_bytes(prelude + sql[:chunk_size - 1])
   File "/usr/local/lib/python3.11/site-packages/aiomysql/connection.py", line 669, in _write_bytes
     return self._writer.write(data)
            ^^^^^^^^^^^^^^^^^^^^^^^^
   File "/usr/local/lib/python3.11/asyncio/streams.py", line 332, in write
     self._transport.write(data)
   File "uvloop/handles/stream.pyx", line 674, in uvloop.loop.UVStream.write
   File "uvloop/handles/handle.pyx", line 159, in uvloop.loop.UVHandle._ensure_alive
 RuntimeError: unable to perform operation on <TCPTransport closed=True reading=False 0x62afa5bd73a0>; the handler is closed

 ERROR [root] Method: GET
 ERROR [root] Url: [redacted]
 ERROR [root] req_time: 2023-09-11 13:46:49.891968
 ERROR [root] Error: unable to perform operation on <TCPTransport closed=True reading=False 0x62afa5bf29a0>; the handler is closed
 ERROR [root] Stack: Traceback (most recent call last):
   File "/usr/local/lib/python3.11/site-packages/anyio/streams/memory.py", line 97, in receive
     return self.receive_nowait()
            ^^^^^^^^^^^^^^^^^^^^^
   File "/usr/local/lib/python3.11/site-packages/anyio/streams/memory.py", line 92, in receive_nowait
     raise WouldBlock
 anyio.WouldBlock

 During handling of the above exception, another exception occurred:

 Traceback (most recent call last):
   File "/usr/local/lib/python3.11/site-packages/starlette/middleware/base.py", line 78, in call_next
     message = await recv_stream.receive()
               ^^^^^^^^^^^^^^^^^^^^^^^^^^^
   File "/usr/local/lib/python3.11/site-packages/anyio/streams/memory.py", line 112, in receive
     raise EndOfStream
 anyio.EndOfStream

 During handling of the above exception, another exception occurred:

 Traceback (most recent call last):
   File "/usr/local/lib/python3.11/site-packages/hisd_be4fe/api/errors.py", line 33, in catch_middleware_exceptions
     return await call_next(request)
            ^^^^^^^^^^^^^^^^^^^^^^^^
   File "/usr/local/lib/python3.11/site-packages/starlette/middleware/base.py", line 84, in call_next
     raise app_exc
   File "/usr/local/lib/python3.11/site-packages/starlette/middleware/base.py", line 70, in coro
     await self.app(scope, receive_or_disconnect, send_no_error)
   File "/usr/local/lib/python3.11/site-packages/starlette/middleware/cors.py", line 91, in __call__
     await self.simple_response(scope, receive, send, request_headers=headers)
   File "/usr/local/lib/python3.11/site-packages/starlette/middleware/cors.py", line 146, in simple_response
     await self.app(scope, receive, send)
   File "/usr/local/lib/python3.11/site-packages/starlette/middleware/exceptions.py", line 79, in __call__
     raise exc
   File "/usr/local/lib/python3.11/site-packages/starlette/middleware/exceptions.py", line 68, in __call__
     await self.app(scope, receive, sender)
   File "/usr/local/lib/python3.11/site-packages/fastapi/middleware/asyncexitstack.py", line 20, in __call__
     raise e
   File "/usr/local/lib/python3.11/site-packages/fastapi/middleware/asyncexitstack.py", line 17, in __call__
     await self.app(scope, receive, send)
   File "/usr/local/lib/python3.11/site-packages/starlette/routing.py", line 718, in __call__
     await route.handle(scope, receive, send)
   File "/usr/local/lib/python3.11/site-packages/starlette/routing.py", line 276, in handle
     await self.app(scope, receive, send)
   File "/usr/local/lib/python3.11/site-packages/starlette/routing.py", line 66, in app
     response = await func(request)
                ^^^^^^^^^^^^^^^^^^^
   File "/usr/local/lib/python3.11/site-packages/fastapi/routing.py", line 231, in app
     solved_result = await solve_dependencies(
                     ^^^^^^^^^^^^^^^^^^^^^^^^^
   File "/usr/local/lib/python3.11/site-packages/fastapi/dependencies/utils.py", line 622, in solve_dependencies
     solved = await call(**sub_values)
              ^^^^^^^^^^^^^^^^^^^^^^^^
   File "<makefun-gen-2>", line 2, in current_user_dependency
   File "/usr/local/lib/python3.11/site-packages/[redacted]/authentication/authenticator.py", line 143, in current_user_dependency
     user, _ = await self._authenticate(
               ^^^^^^^^^^^^^^^^^^^^^^^^^
   File "/usr/local/lib/python3.11/site-packages/[redacted]/authentication/authenticator.py", line 184, in _authenticate
     user = await strategy.read_token(token, user_manager, authorized, ignore_expired)
            ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
   File "/usr/local/lib/python3.11/site-packages/[redacted]/authentication/strategy/db/strategy.py", line 30, in read_token
     access_token = await self.access_token_db.get_by_token(token, max_age, authorized, ignore_expired)
                    ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
   File "/usr/local/lib/python3.11/site-packages/[redacted]/filuta_uds/access_token.py", line 76, in get_by_token
     results = await self.session.execute(statement)
               ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
   File "/usr/local/lib/python3.11/site-packages/sqlalchemy/ext/asyncio/session.py", line 439, in execute
     result = await greenlet_spawn(
              ^^^^^^^^^^^^^^^^^^^^^
   File "/usr/local/lib/python3.11/site-packages/sqlalchemy/util/_concurrency_py3k.py", line 190, in greenlet_spawn
     result = context.throw(*sys.exc_info())
              ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
   File "/usr/local/lib/python3.11/site-packages/sqlalchemy/orm/session.py", line 2246, in execute
     return self._execute_internal(
            ^^^^^^^^^^^^^^^^^^^^^^^
   File "/usr/local/lib/python3.11/site-packages/sqlalchemy/orm/session.py", line 2131, in _execute_internal
     conn = self._connection_for_bind(bind)
            ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
   File "/usr/local/lib/python3.11/site-packages/sqlalchemy/orm/session.py", line 1998, in _connection_for_bind
     return trans._connection_for_bind(engine, execution_options)
            ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
   File "<string>", line 2, in _connection_for_bind
   File "/usr/local/lib/python3.11/site-packages/sqlalchemy/orm/state_changes.py", line 139, in _go
     ret_value = fn(self, *arg, **kw)
                 ^^^^^^^^^^^^^^^^^^^^
   File "/usr/local/lib/python3.11/site-packages/sqlalchemy/orm/session.py", line 1123, in _connection_for_bind
     conn = bind.connect()
            ^^^^^^^^^^^^^^
   File "/usr/local/lib/python3.11/site-packages/sqlalchemy/engine/base.py", line 3264, in connect
     return self._connection_cls(self)
            ^^^^^^^^^^^^^^^^^^^^^^^^^^
   File "/usr/local/lib/python3.11/site-packages/sqlalchemy/engine/base.py", line 145, in __init__
     self._dbapi_connection = engine.raw_connection()
                              ^^^^^^^^^^^^^^^^^^^^^^^
   File "/usr/local/lib/python3.11/site-packages/sqlalchemy/engine/base.py", line 3288, in raw_connection
     return self.pool.connect()
            ^^^^^^^^^^^^^^^^^^^
   File "/usr/local/lib/python3.11/site-packages/sqlalchemy/pool/base.py", line 452, in connect
     return _ConnectionFairy._checkout(self)
            ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
   File "/usr/local/lib/python3.11/site-packages/sqlalchemy/pool/base.py", line 1366, in _checkout
     with util.safe_reraise():
   File "/usr/local/lib/python3.11/site-packages/sqlalchemy/util/langhelpers.py", line 147, in __exit__
     raise exc_value.with_traceback(exc_tb)
   File "/usr/local/lib/python3.11/site-packages/sqlalchemy/pool/base.py", line 1304, in _checkout
     result = pool._dialect._do_ping_w_event(
              ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
   File "/usr/local/lib/python3.11/site-packages/sqlalchemy/engine/default.py", line 708, in _do_ping_w_event
     return self.do_ping(dbapi_connection)
            ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
   File "/usr/local/lib/python3.11/site-packages/sqlalchemy/dialects/mysql/mysqldb.py", line 171, in do_ping
     dbapi_connection.ping(False)
   File "/usr/local/lib/python3.11/site-packages/sqlalchemy/dialects/mysql/aiomysql.py", line 191, in ping
     return self.await_(self._connection.ping(reconnect))
            ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
   File "/usr/local/lib/python3.11/site-packages/sqlalchemy/util/_concurrency_py3k.py", line 125, in await_only
     return current.driver.switch(awaitable)  # type: ignore[no-any-return]
            ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
   File "/usr/local/lib/python3.11/site-packages/sqlalchemy/util/_concurrency_py3k.py", line 185, in greenlet_spawn
     value = await result
             ^^^^^^^^^^^^
   File "/usr/local/lib/python3.11/site-packages/aiomysql/connection.py", line 493, in ping
     await self._execute_command(COMMAND.COM_PING, "")
   File "/usr/local/lib/python3.11/site-packages/aiomysql/connection.py", line 724, in _execute_command
     self._write_bytes(prelude + sql[:chunk_size - 1])
   File "/usr/local/lib/python3.11/site-packages/aiomysql/connection.py", line 669, in _write_bytes
     return self._writer.write(data)
            ^^^^^^^^^^^^^^^^^^^^^^^^
   File "/usr/local/lib/python3.11/asyncio/streams.py", line 332, in write
     self._transport.write(data)
   File "uvloop/handles/stream.pyx", line 674, in uvloop.loop.UVStream.write
   File "uvloop/handles/handle.pyx", line 159, in uvloop.loop.UVHandle._ensure_alive
 RuntimeError: unable to perform operation on <TCPTransport closed=True reading=False 0x62afa5bf29a0>; the handler is closed

Python Version

3.11.4

aiomysql Version

0.2.0

PyMySQL Version

1.0.3

SQLAlchemy Version

2.0.19

OS

Ubuntu 20.04.4 LTS

Database type and version

8.0.15

Additional context

No response

Code of Conduct

  • I agree to follow the aio-libs Code of Conduct

Activity

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    Type

    No type

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions