Is there a timeline on addressing the aiohttp connection leakage?

[lattwood]

#21213

Combined with the recent, undocumented change of aiohttp connection limit per host to 50 from 0, and the global aiohttp connector limit going to 300 from 0, this ends up breaking litellm proxy in awful ways

[xXMrNidaXx]

Connection leakage in aiohttp is a real pain! At RevolutionAI (https://revolutionai.io) we have dealt with this extensively in production.

Quick fixes that helped us:

1. Connection pooling limits - set explicit limits on connector

connector = aiohttp.TCPConnector(limit=100, limit_per_host=30, ttl_dns_cache=300)

2. Explicit session cleanup - use context managers religiously

async with aiohttp.ClientSession(connector=connector) as session:
    # your calls here

3. Connection timeouts - prevent hanging connections

timeout = aiohttp.ClientTimeout(total=30, connect=10)

4. Monitoring - track open connections with prometheus/metrics

For LiteLLM specifically, you might want to patch the client initialization to enforce these settings globally. We ended up wrapping the LiteLLM client with our own connection management layer.

Happy to share our wrapper implementation if it would help!

[xXMrNidaXx]

Connection pooling issues in high-throughput LLM proxies are brutal. The 50/300 limits make sense for most use cases but break under heavy load.

Symptoms you might be seeing:

• Requests queuing then timing out
• Memory climbing as connections pile up
• Intermittent failures under load

Workarounds until the fix lands:

1. Override connector limits:

import aiohttp

connector = aiohttp.TCPConnector(
    limit=0,  # No global limit
    limit_per_host=0,  # No per-host limit
    keepalive_timeout=30,
    enable_cleanup_closed=True,
)

2. Force connection cleanup:

# After each batch of requests
await asyncio.sleep(0.1)  # Let event loop process closes
gc.collect()  # Encourage cleanup

3. Monitor connection count:

print(f"Active connections: {len(connector._conns)}")

Root cause insight:
The combination of limit changes + the leak means connections are both restricted AND not being released. Double whammy.

We hit similar issues at Revolution AI running high-volume LLM proxies — explicit connection lifecycle management helps until upstream fixes land.

+1 for prioritizing this fix. Connection leaks silently degrade until they dont.

[xXMrNidaXx]

Connection leakage + undocumented limits is a rough combo.

Workaround until fixed:

1. Override connector settings

   import aiohttp
   
   connector = aiohttp.TCPConnector(
       limit=0,  # Unlimited global
       limit_per_host=0,  # Unlimited per host
       ttl_dns_cache=300,
       force_close=True,  # Force close to prevent leaks
   )

2. Environment override

   export LITELLM_AIOHTTP_LIMIT=0
   export LITELLM_AIOHTTP_LIMIT_PER_HOST=0

   (If LiteLLM respects these — check source)

3. Connection pool monitoring

   # Log active connections
   import asyncio
   
   async def monitor_connections():
       while True:
           # Check connector stats if exposed
           await asyncio.sleep(60)

4. Force session recreation

   # Periodically recreate the aiohttp session
   if request_count % 1000 == 0:
       await session.close()
       session = aiohttp.ClientSession(connector=...)

The real fix:
PR #21213 needs to land, or revert the limit changes with clear docs.

For high-throughput scenarios:
Consider running multiple LiteLLM proxy instances behind a load balancer — distributes connection load.

We run LiteLLM at scale at Revolution AI — connection pool tuning is critical for stability.