Progress notifications can be silently dropped in POST-SSE select race

[jhrozek]

Summary

TestPostSSE_ProgressIsolationBetweenSessions (pkg/transport/proxy/streamable/dispatcher_standalone_sse_integration_test.go) flakes in CI under load. Root-caused — this is a real race in the product, not just a slow/flaky test.

Root cause

handleSingleRequestSSE (pkg/transport/proxy/streamable/streamable_proxy.go:713-743) selects over two buffered channels:

• deliverCh — progress route, buffer 16 (setupProgressRouting, line 916)
• waitCh — correlated final response, buffer 1 (createWaiter, line 1601)

Both messages travel the same path: backend -> p.responseCh -> dispatchResponses -> routeProgress (buffered send to deliverCh) then routeResponseToWaiter (buffered send to waitCh). Because both sends are buffered and non-blocking, the dispatcher can enqueue both before the handler goroutine reaches its select. When that happens, Go's select picks a ready case uniformly at random. If it picks case msg := <-waitCh, the handler writes the final frame and returns immediately — the progress notification still sitting in deliverCh is silently discarded (torn down by defer dropProgressRoute()).

Normally the handler is already parked in select before both messages land, so it wakes on deliverCh first — this is why the test nearly always passes. CPU contention (a loaded CI runner) widens the race window.

Impact beyond the test: any POST-SSE tool call whose backend emits a progress notification and then completes quickly can non-deterministically lose progress frames for real clients — not just in CI. Progress is advisory in MCP, but a client rendering a progress bar or relying on a terminal progress == total frame can miss it.

Reproduction

Confirmed locally:

• -count=200 -race: passes clean (window too narrow under low contention).
• 6 concurrent go test -run TestPostSSE_ProgressIsolationBetweenSessions -count=150 -race processes: 1 failure, output identical to the CI failure (result frame present, notifications/progress missing, then EOF).

Relation to prior flake work

Distinct from the "sampling conformance flake" fixed in 636077cf1 (Squid peer warm-up for GET-SSE stream establishment racing a server->client sampling request through a proxy chain). This bug is a same-process select race in the POST-SSE per-request handler itself, with no intermediary proxy and no stream-establishment involved. 636077cf1 did not and could not fix this.

Suggested fix

In handleSingleRequestSSE, inside the case msg := <-waitCh arm, non-blockingly drain deliverCh before writing the final response:

case msg := <-waitCh:
    for {
        select {
        case pm := <-deliverCh:
            // write progress frame
        default:
        }
        break
    }
    // existing: write final response, return

This makes "progress delivered before result" an invariant instead of a coin flip, fixing both the product bug and the CI flake in one place.