Response object reference to session returns None

[velomeister]

Describe the bug

I have a script that requests thousands of pages from an API at a rate of 1 req/sec due to rate limits. This causes that the session token that is created after authenticating into the API expires multiple times during execution.

To solve that implemented a custom raise_for_status function that refreshes a session token after getting an HTTP 500 error response code due to faulty implementation of the API that I'm consuming. Here's a barebones (and redacted) copy of the relevant script functions:

import backoff
import asyncio
import aiohttp

async def custom_raise_for_status(response: aiohttp.ClientResponse):
    if response.status >= 400:
        if response.status in (403, 500):
            logger.error("Token has expired. Refreshing...")
            await refresh_token(session=response._session)
        else:
            logger.error(f"Unexpected HTTP error code {response.status}")
        raise aiohttp.ClientResponseError(response.request_info, response.history)

# Token header is already defined in the session.
async def refresh_token(session: aiohttp.ClientSession):
    url = f"{API_URL}/auth_token/extend"
    async with session.get(url) as response:
        assert response.status == 200

@backoff.on_exception(
    backoff.expo, aiohttp.ClientResponseError, max_tries=20, logger=logger
)
async def get_api_endpoint(session: aiohttp.ClientSession):
    url = f"{API_URL}/path/to/endpoint"
    async with session.get(url) as response:
        data = await response.json()
    # Do stuff with data

async def main():
    # Do stuff
    headers = {"Content-Type": "application/json", "charset": "UTF-8"}
    ssl_context = ssl.create_default_context(cafile=certifi.where())
    connector = aiohttp.TCPConnector(ssl=ssl_context)
    async with aiohttp.ClientSession(
        headers=headers, raise_for_status=custom_raise_for_status, connector=connector
    ) as session:
        logger.info("Logging in...")
        await login(session=session)
        await get_api_endpoint(session=session)
    # Do more stuff

However, the reference to response._session is returning a None value.

To Reproduce

1. Implement the code showed previously.
2. Call to API responds with HTTP 500 after token expires.
3. An error ocurs.

Expected behavior

1. The HTTP 500 triggers raising ClientResponseError exception.
2. The backoff annotation calls the custom_raise_for_status function.
3. The response variable uses the reference to _session as a parameter to refresh the session token.
4. No exception is raised and the program continues.

Logs/tracebacks

on 585: ERROR:module.container_findings:Token has expired. Refreshing...
|███████▌⚠︎                               | (!) 585/3139 [19%] in 9:50.4 (0.99/s)
  + Exception Group Traceback (most recent call last):
  |   File "C:\path\to\project\main.py", line 115, in <module>
  |     main()
  |   File "C:\path\to\project\main.py", line 101, in main
  |     count_container_findings(SCORE_CARD)
  |   File "C:\path\to\project\module\client.py", line 188, in count
  |     asyncio.run(main())
  |   File "C:\Users\user\AppData\Local\Programs\Python\Python312\Lib\asyncio\runners.py", line 194, in run
  |     return runner.run(main)
  |            ^^^^^^^^^^^^^^^^
  |   File "C:\Users\user\AppData\Local\Programs\Python\Python312\Lib\asyncio\runners.py", line 118, in run
  |     return self._loop.run_until_complete(task)
  |            ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  |   File "C:\Users\user\AppData\Local\Programs\Python\Python312\Lib\asyncio\base_events.py", line 687, in run_until_complete
  |     return future.result()
  |            ^^^^^^^^^^^^^^^
  |   File "C:\path\to\project\module\client.py", line 178, in main
  |     hosts = await get_all_pages(session=session, pages=pages)
  |             ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  |   File "C:\path\to\project\module\client.py", line 118, in get_all_pages
  |     async with GatheringTaskGroup() as tg:
  |   File "C:\Users\user\AppData\Local\Programs\Python\Python312\Lib\asyncio\taskgroups.py", line 145, in __aexit__
  |     raise me from None
  | ExceptionGroup: unhandled errors in a TaskGroup (1 sub-exception)
  +-+---------------- 1 ----------------
    | Traceback (most recent call last):
    |   File "c:\path\to\project\.venv\Lib\site-packages\backoff\_async.py", line 151, in retry
    |     ret = await target(*args, **kwargs)
    |           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
    |   File "C:\path\to\project\module\client.py", line 103, in get_api_endpoint
    |     async with session.get(url) as response:
    |   File "c:\path\to\project\.venv\Lib\site-packages\aiohttp\client.py", line 1353, in __aenter__
    |     self._resp = await self._coro
    |                  ^^^^^^^^^^^^^^^^
    |   File "c:\path\to\project\.venv\Lib\site-packages\aiohttp\client.py", line 785, in _request
    |     await raise_for_status(resp)
    |   File "C:\path\to\project\module\client.py", line 57, in custom_raise_for_status
    |     await refresh_token(response._session)
    |   File "C:\path\to\project\module\client.py", line 74, in refresh_token
    |     async with session.get(url) as response:
    |                ^^^^^^^^^^^
    | AttributeError: 'NoneType' object has no attribute 'get'
    +------------------------------------

Python Version

$ python --version
Python 3.12.4

aiohttp Version

$ python -m pip show aiohttp
Name: aiohttp
Version: 3.10.3
Summary: Async http client/server framework (asyncio)
Home-page: https://github.com/aio-libs/aiohttp
Author: 
Author-email: 
License: Apache 2
Location: c:\Users\devel\Documents\Palo Alto\dish-score-cards\.venv\Lib\site-packages
Requires: aiohappyeyeballs, aiosignal, attrs, frozenlist, multidict, yarl
Required-by:

multidict Version

$ python -m pip show multidict
Name: multidict
Version: 6.0.5
Summary: multidict implementation
Home-page: https://github.com/aio-libs/multidict
Author: Andrew Svetlov
Author-email: andrew.svetlov@gmail.com
License: Apache 2
Location: c:\Users\devel\Documents\Palo Alto\dish-score-cards\.venv\Lib\site-packages
Requires:
Required-by: aiohttp, yarl

yarl Version

$ python -m pip show yarl
Name: yarl
Version: 1.9.4
Summary: Yet another URL library
Home-page: https://github.com/aio-libs/yarl
Author: Andrew Svetlov
Author-email: andrew.svetlov@gmail.com
License: Apache-2.0
Location: c:\Users\devel\Documents\Palo Alto\dish-score-cards\.venv\Lib\site-packages
Requires: idna, multidict
Required-by: aiohttp

OS

Windows 11

Related component

Client

Additional context

No response

Code of Conduct

• I agree to follow the aio-libs Code of Conduct

[Dreamsorcerer]

Hmm, that looks pretty awkward. You could use a closure as an alternative implementation:

async def main():
    async def custom_raise_for_status(...):
        ...
        await refresh_token(session)  # From the enclosing scope
        ...

    async with ClientSession(...) as session:
        ...

[Dreamsorcerer]

For the actual job you're trying to achieve, I think this is probably a duplicate of #6915 and the idea of adding middlewares to the client.