Gate inference dispatch during weight updates to prevent wasted timeouts

[DavidBellamy]

Context

Analysis of a Harbor RL training job shows that ~33% of httpx.ReadTimeout bursts correlate with weight update events. With --update-weights-interval 2, weights are pushed to engines every 2 training steps.

Problem

During update_weights, SGLang engines freeze inference for 20-30s (measured: 22.1s sync) to load the new weight checkpoint. Any in-flight inference requests continue waiting, and any new requests dispatched during the freeze queue up. If an in-flight request was already 580+ seconds into a long generation, the 20-30s freeze pushes it past the proxy timeout (600s).

Example from the log:

• Weight update completed at 10:37:04 after 22.1s sync
• 119 ReadTimeout errors immediately followed
• All errors from the RolloutManager session server proxy

The engines themselves are healthy before and after the freeze (130-170 tok/s, <7% token usage). The timeouts are purely from the freeze duration stacking on top of existing request latency.

Proposed Fix

Add a dispatch gate around weight updates:

1. Before weight update: stop dispatching new inference requests to engines about to receive weights. Let in-flight requests drain (or at least stop adding new ones).
2. During weight update: hold new requests in the router queue (not forwarded to engines).
3. After weight update: resume dispatch once engines confirm they are ready to serve again.

This prevents wasting GPU-hours on requests that will inevitably timeout during the freeze, and avoids the post-update error burst.

Evidence

• Weight update sync: ~22s measured
• 5 weight updates observed in 7h run
• 33% of timeout bursts within 10 minutes of a weight update
• Correlation is strong for early updates, weaker for later ones (suggesting other factors also contribute)
• All tracebacks are identical: httpx.ReadTimeout in session_server.py:65 (do_proxy)

/cc @mingshanhee

[DavidBellamy]

Update from verbose logging test run (job 1529054)

Ran a 1h25m test with NCCL_DEBUG=info to try to catch weight update freezes in action. Relevant findings:

Weight update data

Only 1 weight update completed during the entire run:

[18:32:05] Timer update_weights start
[18:32:21] Timer update_weights end (elapsed: 15.5s)

The first timeout storm started at 18:43, 11 minutes after the weight update completed. The training loop never progressed to step 2 (insufficient successful rollouts due to timeout cascade from prefill contention, see #920).

Implication

In this run, weight updates were not the trigger for the timeout storms. The prefill contention pattern (#920) dominated. However, this doesn't invalidate the weight update gating proposal -- it just means the two issues are independent failure modes that can each independently cause timeout storms:

1. Prefill contention (TITO prefill storms cause GPU contention that starves concurrent decode requests #920): multiple long-context TITO prefills saturate GPU compute
2. Weight update freezes (Gate inference dispatch during weight updates to prevent wasted timeouts #921): engines freeze inference for 15-20s during update_weights_from_distributed

The run never reached a second weight update to test the gating hypothesis. A longer run with the prefill issue mitigated would be needed to isolate weight update behavior.