Memory issue

[KiKoS0]

I have a FastAPI application that has an endpoint which gets the og:image meta tag of html pages, it uses a httpx.AsyncClient to stream the response and stop reading when it finds the tag. My issue is that after a dozen requests, the memory usage grows exponentially and the application crashes. I traced it down to find out that the problem is that I'm breaking out of reading the response when i find the tag:

async def getImg(url):
   og_line = None
   async with httpx.AsyncClient() as client:
        async with client.stream('GET', url) as response:
            async for chunk in response.aiter_lines():
                if "og:image" in chunk:
                    og_line = chunk
                    break

The memory leak stops when i comment the break, so i wanted to ask if this is expected to happen and that i should avoid doing that, or is this unexpected behavior?
I recreated a minimal reproducible example so you can test it out.
I also found out that it only happens on specific websites that I'm currently dealing with. (so i included both normal and leaking links in the example)
Thanks.

[tomchristie]

Any folks here want to start out by checking if you can reproduce the issue as described here?
That'd be our first step in determining if we need to do something more here.

@KiKoS0 This looks like a pretty good start. If you're able to minimise the example you've linked so that it's even more slimline (keep taking things away until you can't anymore), and include it in a comment (rather than a linked repo) then we're more likely to be able to prioritise looking into this.

[tomchristie]

Might need an aclosing() to resolve this.

https://trio.readthedocs.io/en/stable/reference-core.html?highlight=aclosing#finalization

https://async-generator.readthedocs.io/en/latest/reference.html#contextmanagers

[johtso]

One might expect exiting the stream context manager to close the stream. I'm guessing there's a good reason this doesn't happen?

[florimondmanca]

FWIW, generally Python does what we expect when we break from a generator iteration and the generator's got a finally block:

import asyncio


async def stream():
    try:
        for k in range(10):
            yield k
    finally:
        print("Finally!")


async def main():
    async for k in stream():
        print(k)
        if k == 2:
            print("Break")
            break


asyncio.run(main())

$ python debug/client.py
0
1
2
Break
Finally!

[florimondmanca]

@KiKoS0 Trying your snippet and Tom's on my machine; Tom's with a single client stays at a stable 18.5Mo. Yours would start at around 22Mo and ramp up slowly (26Mo after ~30s).

So seems like switching to a single client solves it?

Does seems like we have some GC or "handle onto something dead" issue with lots of single-instance clients, though. Although that wouldn't be the recommended use case, I'm not sure we want that to happen anyway.

[KiKoS0]

I reran it and came to the same conclusion as you, but then I realized that the hardcoded link had expired (returns a 410 Gone).
So, strangely enough as soon as I changed it to a live one, the memory issue reappeared, even with a single client.

@florimondmanca Could you try it again with a live link like this one?:

link = "https://www.autoscout24.fr/offres/bmw-320-serie-3-touring-e91-touring-163ch-pack-m-sport-diesel-bleu-671904de-6139-4061-a451-f63bdb61de2b"