[Bug]: FFN CUDA graph replay illegal memory access in 2A1F DBO decode (persists after #270)

[yujuancao07]

Current environment

Python:3.12
afd-plugin commit: e37d76 (post-#270)
vLLM version/checkout: 0.26.0
CUDA/GPU:NVIDIA H100 80GB
Model: Qwen/Qwen3-30B-A3B (also fails for Qwen3.6-35B-A3B)
Scenario: tests/e2e afd-graph-dbo-2a1f — attention DP2 (TP1) + FFN DP1

Reproduction

Run the weekly E2E gate E2E · Qwen3 MoE (or directly
tests/e2e/models/qwen3_moe/test_qwen3_moe.py::test_qwen3_moe[afd-graph-dbo-2a1f]).

Expected behavior

The FFN runner replays a cached CUDA graph only for a step whose token
layout matches the captured one; the DBO 2A1F decode step executes
correctly

Actual behavior and logs

With `CUDA_LAUNCH_BLOCKING=1` the fault is pinned to the FFN graph replay
itself (local repro):


[ffn] ffn_worker.py:204 _run_ffn_server_loop
[ffn] ffn_model_runner.py:157 in execute_model
[ffn] torch/cuda/graphs.py:139 in replay
[ffn] torch.AcceleratorError: CUDA error: an illegal memory access was encountered


The illegal access executes **inside the replayed FFN CUDA graph** — not in
eager compute, NCCL, or Python-side kernel launches.

Timeline (identical in all three runs):


attention graph capture done → API ready → first GSM8K request
→ prefill (~1760 tok): FFN runs eager (#270 gating works — no replay), survives
→ first decode step (1 scheduled token): FFN selects REPLAY → IMA inside the graph


Attention-side dump at death:


SchedulerOutput(... num_computed_tokens=[1760], num_scheduled_tokens={...: 1}, ...)
SchedulerStats(num_running_reqs=1, ...)


Note the topology at the fault: a single request runs on attention DP rank 0;
**DP rank 1 is idle (0 tokens)** for the decode step.

AFD configuration

Before submitting

• I searched existing issues for related reports.
• I confirmed whether this reproduces against vLLM v0.26.0 or explained why not.