Sessions silently evicted under memory pressure, causing 404 "session not found" errors

[DavidBellamy]

Summary

During sustained timeout storms, SGLang engines evict sessions from memory. Subsequent requests to those sessions return 404 "session not found" errors, killing the affected trials.

Error

litellm.NotFoundError: OpenAIException - Error code: 404
{'error': 'session not found: session_id=c3000d57c24f4f35862bca25c0f058eb'}

Reproduction

In a 1h25m test run with 33 nodes and verbose logging:

• 99 session-not-found errors observed
• Concentrated in the 19:32-19:41 window, during sustained contention
• Multiple session IDs experienced repeated 404s across retries: 9cdcaab1..., c1194948..., 24cea266..., 62689204...

Mechanism

1. Timeout storms cause many in-flight requests to fail (see TITO prefill storms cause GPU contention that starves concurrent decode requests #920)
2. Failed requests leave orphaned session state on the engine
3. Engine memory pressure increases as new sessions are created while old ones still hold KV cache
4. Engine evicts older sessions to reclaim memory (likely via radix cache eviction with radix_eviction_policy='lru')
5. Client retries or sends the next turn to a session that no longer exists
6. Engine returns 404

Impact

• Each 404 kills the trial with AgentError, reward=0.0
• The 99 occurrences in this run were concentrated in a ~9 minute window, indicating a snowball effect where evictions beget more evictions
• Combined with rollback failures (127x, see filed issue), these two secondary failure modes accounted for 226 wasted rollouts on top of the 625 direct timeouts

Suggested fix

1. Return a retryable status code (e.g., 410 Gone or custom) with metadata about why the session was evicted, so the client can distinguish "session expired due to eviction" from "session ID was never valid"
2. Implement session pinning during active rollouts so sessions with in-flight requests are not eligible for eviction
3. Add session eviction logging so evictions are visible in the log with the session ID, eviction reason, and memory stats

Related: #920 (root cause), #936 (timeout measurement)