[RFC] Training-Inference Consistency for RL

[aoshen02]

[RFC] Training-Inference Consistency for RL

Anchors: #42259 (Logprobs), #39568 (R3), #45221 (Data Plane)

Problem

RL training correctness requires that for the same sample, weights, and routing strategy, the trainer and rollout engine observe identical results. Six sources of divergence:

1. Logprobs & token-ID replay — logprob mismatch → importance sampling ratio corruption → noisy policy gradient. trace_decode_token_ids (#46701) enables replaying a fixed token sequence while computing real logprobs.
2. R3 routing replay — MoE expert routing must be captured during inference and replayed during training. Mismatch → gradients computed on wrong experts.
3. DSA index replay — DeepSeek V3.2+ dynamic sparse attention indices must match between inference and training.
4. Data plane — logprobs, routed_experts, and DSA indices must be transported from inference GPU workers to trainer efficiently, bypassing the EngineCore/API Server bottleneck.
5. Hidden states replay — OPD requires exporting intermediate-layer hidden states from teacher to student, avoiding full [tokens, vocab] logit transfer.
6. Dtype (logits precision) — Inference must match training's fp32 precision for lm_head to maintain train-inference consistency.

Subsections

3.1 Logprobs & Token-ID Replay

Full tracker: #42259. Exit: #47680 + #42245 merged; #46701 accepted.

3.2 R3 Routing Replay

• #44214: FlashInfer + R3
• #45419: P/D disagg + R3
• #45635: KV offload + R3
• #37094: Align routed_experts with streamed output spans

Exit: R3 works on FlashInfer + P/D + KV-offload; R3 correctness test in RL CI.

3.3 DSA Index Replay

• #47280: RFC — --enable-return-indexer-topk
• #47279: Implementation

No downstream demand yet. Forward-looking for DeepSeek V4+ RL adoption. Exit: RFC accepted.

3.4 Data Plane

• #47809: Artifact transfer connector (bypasses EngineCore/API Server bottleneck)
• vime#320: TransferQueue trajectory ingestion for closed-loop training

Exit: at least one data plane backend merged; R3 transport E2E verified.

3.5 Hidden States Replay

• #37002: Observation Plugin — register plugins to intercept layer outputs via PyTorch hooks (Phase 1: API; Phase 2: model runner integration)

Exit: Observation Plugin Phase 2 merged.

3.6 Dtype Replay

• [Core] Support fp32 lm_head for generation models via head_dtype (RFC #48305 §3.6) #48390

Feedback Period

Two weeks from posting.