[BUG][PERFORMANCE]: MCP session pool cancel scope leak causes ~20-45% tool call failures on /servers/{id}/mcp

[crivetimihai]

Bug Summary

The MCP session pool (mcpgateway/services/mcp_session_pool.py) causes ~20-45% of tool calls proxied through /servers/{id}/mcp to fail with ToolInvocationError("Tool invocation failed: "). The failure is consistently reproduced across all backend servers (Fast Test, Fast Time), all transports (StreamableHTTP, SSE), and even sequential single-user calls. Disabling the pool (MCP_SESSION_POOL_ENABLED=false) eliminates the failures entirely. No pool configuration parameter reduces the failure rate.

The root cause is an architectural mismatch: _create_session manually calls transport_ctx.__aenter__() and session.__aenter__(), which attaches anyio cancel scopes to the HTTP request handler task. When a child task in the transport's internal TaskGroup fails, it cancels the host task, killing in-progress call_tool() operations.

Relation to #3520: This issue provides a deeper root cause analysis. #3520 identified broken session recycling as the symptom; this issue identifies the cancel scope leak as the underlying cause. PR #3605 partially addresses #3520 but does not fix the cancel scope issue.

---

Affected Component

• mcpgateway - API
• Federation or Transports

---

Steps to Reproduce

Minimal curl reproduction (no load test needed):

# Generate token
export TOKEN=$(python -m mcpgateway.utils.create_jwt_token \
  --username admin@example.com --admin --exp 10080 --secret my-test-key --algo HS256 2>/dev/null)

# 30 sequential calls — expect ~20-45% failure
for i in $(seq 1 30); do
  curl -s -H "Authorization: Bearer $TOKEN" \
    -H "Content-Type: application/json" \
    -H "Accept: application/json, text/event-stream" \
    -X POST "http://localhost:8080/servers/b8e3f1a2c4d5e6f7a1b2c3d4e5f6a7b8/mcp" \
    -d "{\"jsonrpc\":\"2.0\",\"id\":$i,\"method\":\"tools/call\",\"params\":{\"name\":\"fast-test-echo\",\"arguments\":{\"message\":\"test-$i\",\"delay\":0}}}" \
    | python3 -c "import sys,json; r=json.load(sys.stdin); print(f'Call {$i}: {\"FAIL\" if r.get(\"result\",{}).get(\"isError\") else \"OK\"}')"
done

With pool disabled (100% success):

# Restart gateway with pool disabled
docker compose stop gateway
MCP_SESSION_POOL_ENABLED=false docker compose up -d gateway
# Wait for healthy, then repeat the same 30 calls — all succeed

---

Expected Behavior

All 30 sequential tool calls should succeed (as they do with pool disabled or when calling the backend server directly).

---

Logs / Error Output

Client-visible error:

{"jsonrpc":"2.0","id":3,"result":{"content":[{"type":"text","text":"Tool invocation failed: "}],"isError":true}}

Gateway stack trace:

mcpgateway.services.tool_service.py:4291 invoke_tool
  → asyncio.wait_for(pooled.session.call_tool(...), timeout=effective_timeout)
  → mcp/client/session.py:383 call_tool → send_request
  → mcp/shared/session.py:281 write_stream.send()
  → anyio.ClosedResourceError

mcp.server.streamable_http_manager - ERROR - Stateless session crashed
  → mcp/server/lowlevel/server.py:695 _handle_message
  → mcp/shared/session.py:117 RequestResponder.__exit__
  → RuntimeError: Attempted to exit a cancel scope that isn't the current task's current cancel scope

---

Comprehensive Test Matrix

Pool ENABLED (30 sequential calls each)

Server	Transport	Tool	Success
Fast Test	StreamableHTTP	echo (0ms)	23/30 (77%)
Fast Test	StreamableHTTP	echo (500ms)	17/30 (57%)
Fast Test	StreamableHTTP	get-stats	16/30 (53%)
Fast Test	StreamableHTTP	get-system-time	17/30 (57%)
Fast Time	StreamableHTTP	get-system-time	25/30 (83%)
Fast Time	StreamableHTTP	convert-time	20/30 (67%)
Fast Time	SSE	get-system-time	26/30 (87%)
Fast Time	SSE	convert-time	19/30 (63%)

Pool DISABLED: ALL servers, ALL tools → 30/30 (100%)

Direct to backend (bypass gateway): 20/20 (100%)

SDK isolation tests

Test	Result
MCP SDK → fast_test_server directly (pool-style reuse)	20/20 (100%)
MCP SDK → gateway (pool ENABLED)	11/20 (55%)
MCP SDK → gateway (pool DISABLED)	20/20 (100%)

The MCP SDK handles session reuse correctly when used directly. The bug is in the gateway's pool.

---

Configuration Sweep (No Config Helps)

Every pool parameter was tested. None significantly reduce failures:

Config	Result (30 calls)
Baseline (defaults)	26/30 (4 fail)
HEALTH_CHECK_INTERVAL=0	24/30 (6 fail)
TTL=1s	25/30 (5 fail)
MAX_PER_KEY=1	25/30 (5 fail)
EXPLICIT_HEALTH_RPC=true	27/30 (3 fail)
HEALTH_METHODS=[list_tools]	27/30 (3 fail)
INTERVAL=0 + METHODS=[list_tools]	27/30 (3 fail)
INTERVAL=0 + METHODS=[ping]	26/30 (4 fail)
TTL=0 (never reuse)	23/30 (7 fail — worse)
POOL_ENABLED=false	30/30 (0 fail)

TTL=0 makes things worse because forcing fresh session creation on every call increases the window for cancel scope conflicts.

---

Root Cause Analysis

The cancel scope leak

_create_session (mcp_session_pool.py:1137, 1151) manually enters transport and session contexts:

# Line 1137 — enters transport TaskGroup cancel scope on the request handler task
read_stream, write_stream, _ = await transport_ctx.__aenter__()
# Line 1151 — enters session TaskGroup cancel scope on the request handler task
await session.__aenter__()

This attaches anyio cancel scopes to the HTTP request handler task:

[FastAPI/middleware scopes]
  └── Transport TaskGroup cancel scope (from SDK streamable_http.py)
        └── Session TaskGroup cancel scope (from SDK session.py)

How it kills tool calls

1. post_writer (SDK streamable_http.py) spawns handle_request_async tasks with no try/except:

   async def handle_request_async():
       await self._handle_post_request(ctx)  # NO error handling
   if isinstance(message.root, JSONRPCRequest):
       tg.start_soon(handle_request_async)   # Spawned in transport TaskGroup

2. If _handle_post_request raises (HTTP error, connection error), the exception propagates to the TaskGroup, which cancels the transport scope — and with it, the host task (the request handler):

   # anyio _asyncio.py, TaskGroup._spawn/task_done:
   self.cancel_scope.cancel()  # Cancels transport scope → cancels host task

3. The host task (running call_tool()) receives CancelledError. asyncio.wait_for does NOT convert it to TimeoutError (its own timeout didn't fire). The error surfaces as ClosedResourceError or CancelledError.

Why the non-pool path survives

Without pooling, async with context managers properly unwind cancel scopes via __aexit__:

async with streamablehttp_client(...) as streams:       # Scope A
    async with ClientSession(*streams) as session:       # Scope B
        await session.call_tool(...)                     # Error here...
    # B.__aexit__ properly unwinds session scope
# A.__aexit__ properly unwinds transport scope, collects child errors into BaseExceptionGroup

In the pool path, scopes A and B are ENTERED but __aexit__ is deferred. The cancel scope hierarchy leaks onto the host task, allowing child task failures to cancel the request handler directly.

---

Relationship to #3520 and PR #3605

• [BUG][PERFORMANCE]: MCP session pool recycles broken sessions, causing cascading ClosedResourceError failures under load #3520 identified the symptom: broken sessions recycled by the pool.
• PR fix(session-pool): prevent broken session recycling in MCPSessionPool #3605 (open, not merged) adds transport-aware is_closed detection. The discard=True logic in the session() context manager is already on main (lines 1894-1900).
• PR fix(session-pool): prevent broken session recycling in MCPSessionPool #3605 is a partial fix: it catches sessions broken BETWEEN calls but does not prevent cancel scope corruption DURING calls. It should reduce the failure rate but likely won't eliminate it.

---

Cancel Scope Cleanup Noise

With pool enabled, gateway logs Stateless session crashed + RuntimeError: cancel scope on failing requests. With pool disabled, these entries did not appear in testing. The noise appears coupled to pooled-session failures rather than appearing on successful pool-disabled runs.

---

Related Upstream Issues

MCP Python SDK (modelcontextprotocol/python-sdk):

Issue	Status	Relevance
#577 — Cancel scope crash with multiple MCPClient instances	OPEN P1	Core upstream issue: BaseSession.__aenter__() creates TaskGroup cancel scopes bound to the current task
#922 — Multiple client sessions → cancel scope error	CLOSED	Pure-anyio repro proving TaskGroup lifecycle is the constraint
#915 — ClientSessionGroup exception if server unavailable	OPEN P1	Connection error + scope stack corruption
#1805 — Resource leak in streamable_http_client	OPEN	Thread leaks + CPU spikes after context exit
#1811 — read_stream_writer hangs after SSE disconnect	OPEN P1	call_tool() hangs forever after connection loss
#2114 — ExceptionGroup wrapping obscures errors	OPEN P1	Tagged for v2

anyio (agronholm/anyio):

Issue	Status	Relevance
#415 — asyncio.wait_for incompatible with anyio cancel scopes	OPEN	Directly relevant
#787 — Child tasks don't cancel group scope correctly on asyncio	OPEN	Timing issues

---

Recommended Fix Options

1. Dedicated background task per pooled session (most likely architectural fix; not yet validated): Run transport/session lifecycle in a dedicated asyncio.create_task() per pooled session, keeping the request handler task outside the transport's cancel scope hierarchy.

2. Merge PR fix(session-pool): prevent broken session recycling in MCPSessionPool #3605 (partial fix): Transport-aware is_closed catches sessions broken between calls.

3. Replace asyncio.wait_for with anyio.fail_after (complementary): Prevents cancel scope corruption on timeout paths. Secondary, not sufficient alone.

4. Disable pool (proven workaround): MCP_SESSION_POOL_ENABLED=false. 100% success. Performance cost: extra initialize round-trip per call.

---

Workaround

Set MCP_SESSION_POOL_ENABLED=false in .env or docker-compose.yml.

---

Environment Info

Key	Value
MCP SDK	1.26.0
anyio	4.12.1
Runtime	Python 3.12 (container)
Platform / OS	Linux (Docker Compose)
Container	Docker, 1-3 gateway workers behind nginx
Session pool	enabled (default), max 200 per key
Backend servers	Rust fast_test_server, Rust fast_time_server
Transports tested	StreamableHTTP, SSE — both affected

---

Additional Context

• This issue supersedes the symptom-level description in [BUG][PERFORMANCE]: MCP session pool recycles broken sessions, causing cascading ClosedResourceError failures under load #3520 with a deeper root cause analysis.
• The /rpc endpoint shares the same backend pool path (tool_service.invoke_tool) and may be exposed, but the same failure rate was not reliably reproduced on /rpc in repeated runs.
• The RCA document is at todo/rca-echo-failure.md in the repository.

[crivetimihai]

Resolution Plan

Approach: 3 complementary changes in 1 PR

Supersedes #3520. Incorporates PR #3605's transport-aware is_closed.

---

Part 1: Background task ownership (primary fix)

Problem: _create_session enters transport/session contexts on the request handler task, leaking cancel scopes.

Fix: Run transport/session lifecycle in a dedicated asyncio.Task. The request handler communicates via anyio memory streams (already cross-task safe).

Changes in mcpgateway/services/mcp_session_pool.py:

1. Add fields to PooledSession (~line 101):

   • _owner_task: Optional[asyncio.Task]
   • _shutdown_event: Optional[asyncio.Event]

2. Add _session_owner_coro() method — a coroutine that:

   • Enters transport and session via async with (proper scope ownership)
   • Calls session.initialize()
   • Signals readiness via an asyncio.Future
   • Waits on _shutdown_event until cleanup
   • async with ensures proper __aexit__ unwinding

3. Restructure _create_session (~line 1077):

   • Launch _session_owner_coro via asyncio.create_task()
   • Await the ready_future for the initialized session
   • Store _owner_task and _shutdown_event on PooledSession
   • On failure: set shutdown event, cancel task, await cleanup

4. Restructure _close_session (~line 1194):

   • Set _shutdown_event to signal graceful shutdown
   • Await _owner_task with timeout
   • Force-cancel if still running after timeout
   • Preserve existing Redis affinity cleanup (lines 1236-1241)

5. Enhance is_closed (~line 140) — incorporate PR fix(session-pool): prevent broken session recycling in MCPSessionPool #3605:

   • Check _closed flag
   • Check _owner_task.done() (owner died)
   • Check session._write_stream._closed (transport broke)
   • Check _state.open_receive_channels == 0
   • Wrap SDK internals in try/except for graceful degradation

6. Enhance _validate_session (~line 989):

   • Already calls is_closed — gets the above checks for free

Owner task lifecycle requirements

Scenario	Expected behavior
Owner fails before readiness (e.g., connection refused)	ready_future.set_exception(exc) propagates to _create_session. async with in owner coro unwinds naturally. _create_session re-raises as RuntimeError.
Owner dies after readiness (e.g., backend crashes mid-session)	_owner_task.done() becomes True. Next is_closed check returns True. Pool's _validate_session rejects the session. session() context manager's discard=True path triggers _close_session.
Forced shutdown (pool close_all or eviction)	_close_session sets _shutdown_event. Owner coro exits, async with blocks unwind. If owner doesn't exit within cleanup_timeout, force-cancel.
Discard on exception (session() catches BaseException)	release(pooled, discard=True) calls _close_session. Owner task receives shutdown signal and unwinds. Existing discard=True semantics preserved.
Redis affinity cleanup	Runs AFTER owner task completes in _close_session, same as current behavior.

---

Part 2: Replace asyncio.wait_for with anyio.fail_after (complementary)

Problem: asyncio.wait_for wrapping MCP SDK calls corrupts anyio cancel scopes (anyio #415).

tool_service.py — 4 MCP SDK call sites (lines 4029, 4038, 4176, 4186):

# Before:
result = await asyncio.wait_for(session.call_tool(...), timeout=T)
# After:
with anyio.fail_after(T):
    result = await session.call_tool(...)

Leave asyncio.wait_for for pure-asyncio calls (REST/httpx at lines 3763, 3765, 4382).

mcp_session_pool.py — 4 health check sites (lines 1032, 1036, 1040, 1044):
Same pattern change. Update except asyncio.TimeoutError to except TimeoutError.

Leave asyncio.wait_for at lines 793 (semaphore.acquire — pure asyncio) and 816 (_create_session — awaits a plain asyncio.Future in new design).

gateway_service.py — 1 pooled MCP SDK call site (line 3339):
asyncio.wait_for(pooled.session.list_tools(), ...) in the gateway health check explicit RPC path. Same pattern change.

Other pool consumers (prompt/resource services)

• prompt_service.py:318-325: pooled.session.get_prompt() — no asyncio.wait_for wrapper. Benefits from Part 1 passively.
• resource_service.py:1758-1767, 1839-1848: pooled.session.read_resource() — same. No Part 2 changes needed.

---

Part 3: Tests

New file: tests/unit/mcpgateway/services/test_mcp_session_pool_cancel_scope.py

• Background task created and set on PooledSession
• is_closed detects dead owner task
• is_closed detects broken write stream (from PR fix(session-pool): prevent broken session recycling in MCPSessionPool #3605)
• _close_session signals shutdown event and awaits owner task
• _close_session force-cancels on timeout
• _close_session preserves Redis affinity cleanup after owner exit
• _validate_session rejects dead-owner sessions
• Health check uses anyio.fail_after (TimeoutError raised)
• Owner-fails-before-readiness propagates exception to caller
• Owner-dies-after-readiness detected by is_closed
• discard=True path triggers proper owner shutdown
• Pooled get_prompt() succeeds (regression for prompt_service pool path)
• Pooled get_prompt() fallback when get_mcp_session_pool() raises RuntimeError

Update existing tests:

File	Changes
test_mcp_session_pool.py	Update health check timeout mocks: asyncio.TimeoutError() → TimeoutError(). Update patches targeting asyncio.wait_for in health check paths.
test_mcp_session_pool_coverage.py	Lines 818, 831: side_effect=asyncio.TimeoutError() → side_effect=TimeoutError(). Update _create_session mocks for new background task pattern.
test_session_pool_e2e.py	Run as-is for regression.
test_mcp_session_pool_integration.py	Run with --with-integration for regression.

---

Validation Protocol

1. Unit tests (session pool specific)

python -m pytest tests/unit/mcpgateway/services/test_mcp_session_pool*.py -v

2. E2E and integration tests

python -m pytest tests/e2e/test_session_pool_e2e.py -v
python -m pytest tests/integration/test_mcp_session_pool_integration.py --with-integration -v

3. Code quality

make autoflake isort black pre-commit
make flake8 bandit interrogate pylint verify

4. Full test suite

make test

5. Docker-compose stack validation

make testing-down && make build && make testing-up
make compose-ps  # all services healthy

6. Manual tool call matrix (30 calls each, pool ENABLED, expect 30/30)

Server	Tool	Transport
Fast Test	fast-test-echo (0ms, 500ms)	StreamableHTTP
Fast Test	fast-test-get-stats	StreamableHTTP
Fast Test	fast-test-get-system-time	StreamableHTTP
Fast Time	fast-time-get-system-time	StreamableHTTP
Fast Time	fast-time-convert-time	StreamableHTTP
Fast Time SSE	fast-time-sse-get-system-time	SSE
Fast Time SSE	fast-time-sse-convert-time	SSE

Note: Prompt and resource pooled paths share pool.session() and benefit from the same fix. Covered by unit/integration tests and the MCP protocol load test.

7. MCP protocol tests (requires compose stack)

make test-mcp-rbac test-mcp-cli

8. Locust load tests with pool enabled

Echo delay test (StreamableHTTP tool calls, 0% failure target):

locust -f locustfile_echo_delay.py --host=http://nginx:80 --users=5 --spawn-rate=5 --run-time=30s --headless

MCP protocol test (broader: tools, prompts, resources, discovery, /rpc):

locust -f locustfile_mcp_protocol.py --host=http://nginx:80 --users=5 --spawn-rate=5 --run-time=60s --headless

Success judged by per-request-type tagged stats: 0% failure rate on all MCP-tagged requests. /rpc baseline reported separately.

Success criteria

• 30/30 across all tested server/tool/transport combinations with pool ENABLED
• 0% failure rate in locust echo-delay load test (MCP-tagged requests)
• 0% MCP-tagged failure rate in locust MCP protocol load test
• No ClosedResourceError or cancel scope errors in gateway logs
• make test-mcp-rbac test-mcp-cli pass
• All unit, integration, e2e tests pass
• No regression with pool disabled

---

Critical files

File	Changes
mcpgateway/services/mcp_session_pool.py	Part 1 (background task, is_closed, lifecycle) + Part 2 (health check anyio.fail_after)
mcpgateway/services/tool_service.py	Part 2: 4 call_tool sites + import anyio
mcpgateway/services/gateway_service.py	Part 2: 1 health check list_tools site (line 3339)
tests/unit/.../test_mcp_session_pool.py	Update timeout mocks
tests/unit/.../test_mcp_session_pool_coverage.py	Update timeout side_effects
tests/unit/.../test_mcp_session_pool_cancel_scope.py	New: background task lifecycle tests
tests/e2e/test_session_pool_e2e.py	Run as-is for regression
tests/integration/test_mcp_session_pool_integration.py	Run with --with-integration

Out of scope

• Prompt/resource services don't wrap MCP SDK calls in asyncio.wait_for — no Part 2 changes needed. They benefit from Part 1 passively.
• gateway_service.py lines 856 and 3106 use asyncio.wait_for but wrap non-pooled session operations (fresh async with context managers) — not affected by the cancel scope leak.