[crispyging]

PR body draft — [ROCm][RDNA2] Explicit TP2/TP4 expert-offload topology

(AI assistance: developed with an autonomous Hermes agent under operator
supervision; all claims below map to executed CPU tests or cited source.
Maintainer review pending. Initial submission claimed NO four-card GPU
qualification; the dated Oct-30 updates below carry the on-device evidence,
including the negative gates and the fault chronology.)

Summary

Follow-up to #146/#147: makes the expert-cache offload launch and
qualification path express and validate topologies other than the maintainer
host's TP2 pair, in its initial topology changes. Follow-up commits also add default-off communication workarounds (described below). The expert-cache runtime already
supports TP4 expert sharding by construction; the blockers were the launcher
pin and three TP2-hardcoded tools (sweep, qualify, AR checker).

Why the topology extension needed no expert-kernel changes (source audit)

• PackedExpertCache.logical_num_experts sizes from the rank's local weight
  count; local_to_slot is validated as int32[128 or 256]
  (csrc/rocm/mono_k2_rdna2.cu:2529), expert_base + numel <= 512 (L2542),
  Stage0 kLocalE=128, Stage1 capacity 256 (L894/L2241).
• Layout gate accepts local_experts in (128, 256) and a contiguous EP block
  (layout.py:292-336); the dense-MTP layout already demands TP4 geometry
  (128 experts / shared width 160) explicitly.
• Shared-expert width is runtime Ise (no 320 constant in the kernel); TP4's
  160 satisfies every Python gate. The -offload workspace sized for 320 is
  an over-allocation at TP4, safe.
• rdna_ar is "W ranks" by design; TP4 one-shot all-reduce was benchmarked
  in docs/rdna2/tp4-allreduce-sizes-2026-10-10.md.
  With --enable-expert-parallel (already in the flashnext recipes), TP4 ranks
  own 128 contiguous experts, i.e. exactly the kernel's TP4 shape.

Changes

1. tools/rdna/mono/serve_offload.sh: TP (2/4) x GPUS validated jointly
   (device count == TP, unique canonical HIP IDs); allocation-safety kept as
   ALLOWED_GPU_SETS allowlist, default 6,7 — unknown device sets still
   exit 2 exactly as before; SLOTS capped per rank at 512/TP (TP2 default
   96 unchanged; TP4 defaults to 48 = same 1/3-of-local budget); PRINT=1
   resolves and prints the full plan (TP, devices, local experts, slots,
   EXTRA_ARGS) for either serve or validate path without launching.
2. vllm/.../rdna_expert_cache_policy.py: pure
   validate_offload_topology(tp, devices, global_experts, slots) shared
   contract (rejects TP1/512-local, TP8/64-local, mismatched device counts,
   duplicate devices, slot overrun).
3. tools/rdna/mono/sweep_offload_policy.py: rank map derived from
   local/global expert counts (--local-experts/--global-experts); TP2
   defaults byte-compatible.
4. tools/rdna/mono/qualify_expert_copy.py: check_control_result() accepts
   any rank count via --expected-caches (was: exactly 2 ranks).
5. benchmarks/kernels/check_rdna2_offload_tp2_ar.py: the split-K2 fused
   all-reduce checker takes --tp 2|4 (default 2, TP2 invocation unchanged).
   rdna_ar_init itself validates world 2..8 (RDNA_AR_MAX_WORLD 8,
   csrc/rocm/rdna_allreduce.cu:90); the checker now derives handles
   (uint8[tp,64]), probe expectation (tp(tp+1)/2), rank ownership
   (512/tp experts, 640/tp shared slice) and worker count from --tp. At
   TP4 the independent oracle is the resident 128-expert mono_k2_rdna2
   entry with AR detached — the exact parity check four-card qualification
   needs, on kernels already in the tree.
6. serve_offload.sh GPUS export restored: the topology rework in this
   branch had assigned GPUS without export; the validation driver
   hard-requires it (: "${GPUS:?}"), so a VALIDATE=1 run relying on the
   wrapper default died in the child shell. Fixed with a behavior test that
   runs the real delegation with GPUS absent from the caller environment
   (red before, green after).
7. serve_validate.sh device pin becomes REFUSED_GPUS (default 0,1,
   preserving the maintainer host policy byte-for-byte): the unconditional
   refusal of HIP 0/1 is that host's allocation fact (W7800s, other job), not
   a vLLM constraint, and it silently blocked the VALIDATE path on any host
   whose V620s enumerate as HIP 0-3.
8. Tests + README topology section.

Test evidence (executed)

• 16 subprocess behavior tests run the real wrapper + the real qualification
  driver in stub trees: TP2 default plan preserved (slots 96, KV 16 GiB),
  TP4 plan (128 local, slots 48), refusals (unauthorized set, TP/device
  mismatch, duplicates, malformed lists, SLOTS>128 at TP4, TP1/TP3/TP8),
  serve/validate stubs receive TP=4, the validate delegation exports the
  default GPUS, and the driver's HIP 0/1 pin keeps its par1-cs25 default
  while REFUSED_GPUS="" admits a four-card 0,1,2,3 allocation.
• 6 pure-policy topology tests (ownership partition, slot bounds, invalid
  topologies, per-rank cache independence).
• 4 sweep tests incl. TP4 rank map + OUTSIDE-rank refusal; 6 qualify rank
  tests. 17 new behavior tests for the AR checker run the real main() with
  spawn/device-probe replaced: TP2/TP4 resolved geometries, slot bounds tied
  to the local expert count (129 refused at TP4, 128 accepted), stale-pair /
  insufficient-device refusals, _topology rejects TP1/3/5/8. Combined
  re-run of the full rdna offload test set (9 files: fused_moe
  test_rdna_expert_cache_policy.py/test_rdna_expert_cache.py/
  test_rdna_expert_offload_topology.py, tools/rdna
  test_rdna2_offload_ar_cli.py/test_serve_offload_topology.py/
  test_serve_validate_devices.py,
  models/qwen4_exp test_offload_policy_sweep.py/
  test_offload_policy_sweep_tp4.py/test_qualify_expert_copy_ranks.py):
  98 passed / 6 device-skipped
  under a CPU torch venv (existing TP2 suites included:
  test_rdna_expert_cache_policy.py 21, test_offload_policy_sweep.py,
  wrapper 12 + driver 4, qualify 5). On a no-torch interpreter the suite's skip-guard
  works (1 skipped); two pre-existing base-branch files
  (test_rdna_expert_cache.py, test_offload_policy_sweep.py)
  top-level-import torch and fail collection there — unchanged upstream
  behavior, not introduced by this branch.
• NOT run: MTP2/graph-replay qualification, perf cells, RuneBench turns (no
  coherent baseline exists on this host — see the hardware campaign section
  below, which supersedes this line: GPU gates WERE run, with negative
  results; four-card serving qualification remains not achieved).

Hardware campaign on a four-V620 host (honest, negative result)

Executed 2026-10-29 on a second host class: 4x Radeon Pro V620 32 GiB, no GPU
P2P (each card behind its own Navi21XL switch; can_device_access_peer == 0
for every pair), image local/vllm-v620-extras:d8529f06
(wheel 0.31.0+v620.extras.d8529f06.rocm714, torch 2.13.0+rocm10.0.0,
HIP 7.15.26333), restored AWQ weights + PLE int4. Physical HIP 0-3 only;
production lanes untouched. The graph-mode runs below had clean fault checks. A later TP2 eager ablation produced a new GPU page fault at 19:05:22 UTC; GPU work stopped and the test container was removed. No post-patch GPU confirmation has run.

• CPU suite (the 9 files above) run inside the real candidate image on the
  four-device host: 105 passed / 0 failed (device-dependent tests execute
  there instead of skipping).
• Launch/plan gates (items 1-2): PRINT=1 TP=4 GPUS=0,1,2,3 resolves
  local_experts=128 slots=48; the four serve ranks logged
  "128 logical experts, 48 GPU slots" each — cross-rank ownership geometry
  is CORRECT on hardware.
• Custom all-reduce checker --tp 4: rdna_ar_init fails
  hipDeviceEnablePeerAccess ("invalid device ordinal") — expected on a
  P2P-less host. Commits 92a55f861/94273f6cf (pushed, head is now
  94273f6cf) make the checker refuse with that reason pre-spawn instead of
  an opaque HIP error, and fix torch.cuda.device_count() == 0 pre-init on
  OEM V620s (no serial/GUID) that falsely tripped its own device guard.
  Serving is unaffected: it uses the RCCL host-bounce path
  (VLLM_RDNA_AR=0, disable_custom_all_reduce, NCCL_P2P_DISABLE=1/PXB).
• Offloaded TP4 MTP0: full boot + graph capture OK, then NaNs detected in logits on all four ranks (~824-925 NaN per rank of the 248,320-col gathered row
  shard) at the FIRST decode step of the first greedy request; engine aborts
  via the NaN guard. Same magnitude family as the "NaN 810/1024" caveat in
  perf(rdna2): batch expert copies and reduce cache planning overhead #147's notes — still uncaused there.
• Resident TP4 MTP0 (no monokernel/expert-cache/KV-offload): engine
  stable, NO NaN, but greedy correctness 0/3 (finite but incoherent:
  "capital of France" -> " Dubai", repetition-garbage in the prefix probe).
• TP2 OFFLOADED control on HIP 0,1 (maintainer-qualified geometry: 256
  local experts/rank, slots 96, native KV tier 16 GiB, same image/weights):
  boots in 320 s (17.94 GiB/rank, KV 196,608 tokens), greedy 0/3
  ("Marseille"/"Zurich"/empty), then NaNs detected in logits (22,184 NaN)
  kills the engine.
• TP2 RESIDENT control is not feasible on 32 GiB cards: deterministic
  OOM at weight materialization (31.14 GiB already allocated per rank before
  the MoE expert tensors), independent of load — 256-expert/rank resident
  FP16 W4A16 does not fit; that is precisely why the offload tier exists.

Interpretation (what this does and does not prove). The NaN-at-first-
decode and the incoherent-greedy symptoms reproduce at the maintainer's OWN
qualified TP2/EP2 geometry on this host class. So neither symptom is
attributable to TP4 topology, to the cross-rank expert ownership (which is
provably correct here), or by itself proof of an expert-ownership bug. At hardware-tested revision d8529f06, the PR diff was wrapper/CLI/bench/tests only; later CPU-tested communication patches have not been GPU-tested. The discriminator is the
host/environment: candidate axes are the ROCm/HIP stack in this image vs the
#147-qualified environment ("older HIP fallback UNQUALIFIED" applies to a
different stack), and/or this host's P2P-less RCCL host-bounce collectives,
and/or the restored-weight provenance. Four-card serving qualification
therefore remains NOT claimed; items 3-5 of a full qualification
(perf cells, MTP2 graphs, RuneBench turns) were not attempted because there
is no coherent baseline to measure. No performance numbers are claimed.

Follow-up CPU-only diagnosis (commit 363931a4a, 2026-10-29)

Source archaeology identified a strong, still GPU-unconfirmed explanation for part of the failures on this host class: the maintainer's own PR5 -> extras
campaign carries live-mounted, env-gated collective replacements for
gfx1030 hosts without GPU P2P — /data/stacks/projects/vllm-flashnext-pr5/ logits_processor.passing-910acae8.py (FLASHNEXT_HEAD_ALLREDUCE, comment:
"reproduced ROCm full-logit gather corruption"), the george-tree
linear.py patch (FLASHNEXT_GATHER_ALLREDUCE, "corruption also hits the
gather_output projections ... starves the drafter"), and
port-patches-into-extras.py: "our gfx1030 all_gather workarounds are NOT
upstream. So the patches are load-bearing, not optional." The migrated
extras trees never took them upstream, so a clean-tree image re-triggers
the corruption here. Both gated envs are exported by the launch scripts of
every passing lane config (launch-rdna-extras-3b59ee16e.sh:73-74,
docker-compose.qsa-fp8.yml), and the NaN observer
(gpu_model_runner._get_nans_in_logits) runs after the head gather, so
the gather sites are squarely in the failure path.

Commit 363931a4a ports both workarounds into the tree, gated exactly as
the lane ran them: unset env => every path byte-identical to base
(maintainer TP2 defaults and P2P-capable hosts untouched); GPU-resident
zero-pad + all-reduce, no unconditional host fallback. CPU oracle tests
(spawned gloo, worlds 2 and 4) prove the workaround layout is bit-equal to
reference all_gather and pin the gate contract: 9 new tests pass; full
rdna CPU suite 108 passed / 6 device-skipped. No GPU claim: the
in-image GPU discriminator is (D1) same d8529f0 image + live-mounted
passing file + env (no rebuild), (D2) a build of this commit + env;
coherent greedy output under either confirms causality. Weight-integrity
hashing against the HF manifests (133 entries) is running in parallel and
its results take precedence over any provenance hypothesis if clean.

Not in scope

#148's tiered budgets / live KV paging; model-agnostic offload; any kernel
change; any change to TP2 default behavior or the maintainer host's device
restriction (default resolution is byte-identical to the previous wrapper).

Update (Oct-29 cycle 12, CPU-only): closure evidence completed, third gather site ported

• The fogbox lane artifacts were read in full across three generations
  (Sept-17 george 955322d68 broad-correctness verdict.passed=true,
  33/33, 0 nonfinite at TP4-EP on this same host class; the extras
  3b59ee16e port script — "the patches are load-bearing, not optional";
  the frozen Oct-2 ab5ccf3d9f34 campaign closure). The logits/gather
  all-reduce closures are the ONLY patch content present in EVERY proven
  config across all three generations.
• A THIRD corrupted-collective site was found in the lane closures and is
  now ported (commit 91ff81019): LogitsProcessor.get_top_tokens's
  (value, index) pair all_gather is replaced by a MAX-then-MIN
  all_reduce pair under the same FLASHNEXT_HEAD_ALLREDUCE=1 gate
  (GPU-only, CUDA-graph-legal; default path untouched; inert at MTP=0,
  needed for MTP=1 parity). CPU gloo oracle worlds 2+4 including an
  exact-tie lowest-index contract: RED -> GREEN 12/12; full rdna CPU
  suite 111 passed / 6 skipped (torch 2.15.0+cpu).
• Two legacy patch candidates were CLOSED with evidence: the PLE
  worker.py FP8-numpy-view patch is a no-op for int4 layouts AND is
  superseded by a newer native in-tree fix already present in this base
  (is_fp8 + uint8 view + LUT), and the kv-manager ring-alignment patch
  is absent from the extras and Oct-2 closures (not mandatory).
• Restored-weight integrity: 133/133 sha256 verified against the public
  HF manifests — weight provenance is falsified as a failure axis.
• Two gate-config mismatches against every proven lane were identified
  (our launcher additionally sets NCCL_P2P_DISABLE=1 and runs
  max-num-seqs 8 where the passing lanes run 4–6). The next GPU
  discriminator (needs an operator-authorized host reboot; the host is
  faulted since an unrelated eager-ablation) must separate variables: first keep the recorded baseline configuration fixed and change only the head-gather workaround; test GDN prefill, P2P settings, sequence limits, and MTP-only workarounds in independent subsequent arms. Do not label a combined configuration change causal.
  No GPU-correctness claim is made here: the CPU tests prove layout/
  contract equivalence and gate behavior only.

Update (Oct-29 cycle 14, CPU-only): exact failing-gate call chain; attribution tightened; receiver-side corruption proven from published counts

Full source trace of what the failing gates actually executed (no code
change this cycle):

• Only the head gather was live. The launcher runs the failing gates with
  MTP=0, and rdna_mtp_args 0 passes no --speculative-config
  (rdna_launcher_common.sh:129), so the MTP drafter is never built: the
  gather_output=True fc pair (amd/mtp.py:222/231, ROCm dispatch proven via
  the platform __getattr__) and get_top_tokens never ran in any failing
  gate. The single active workaround-closure site in ALL failing runs is
  LogitsProcessor._gather_logits; FLASHNEXT_GATHER_ALLREDUCE and the
  top-tokens all-reduce are prophylactic for the future MTP=2 gate and are
  explicitly NOT claimed as explanations of the MTP=0 failures.
• NaN-observer citation corrected. The recipe exports
  VLLM_USE_V2_MODEL_RUNNER=1 (flashnext-mtp0.env:32), so the operative
  guard is the v2 sampler (gpu/sample/sampler.py:150, count before
  sampling; raise in gpu/async_utils.py:222) — not
  gpu_model_runner._get_nans_in_logits. It sees the full 248,320-vocab
  gathered tensor on every rank, so the post-gather conclusion stands.
• Positive evidence the corruption is receiver-side, from data already
  published above: the NaN counts 925 / 824 / 853 / 897 differ per rank on
  the SAME gathered tensor at the SAME step (digits corrected against the
  raw serve.log lines 730-761 in the late addendum below). Upstream NaNs plus a correct
  all-gather deliver a byte-identical tensor to every rank and therefore
  EQUAL counts; unequal per-rank counts can only mean the gathered buffers
  themselves diverged per receiver — the documented
  all_gather_into_tensor corruption signature (the collectives upstream of
  it on that path are all-reduces, which the passing lane configs prove
  unaffected on this fabric). This falsifies the "gather innocent, upstream
  NaN source" variant FOR THIS TP4 RUN; the late addendum below shows the
  TP2-offload NaN run has EQUAL per-rank counts, so an additional upstream
  numeric axis is real.
• Sharper GPU discriminator for the pending on-target test: with the head
  closure enabled, NaN-free coherent greedy output confirms head-gather
  causality; surviving NaNs with EQUAL per-rank counts would indicate an
  upstream generator (next axis VLLM_GDN_HIP_PREFILL, default flipped ON at
  base commit 147329b2f with an explicit TP=4-qualification note still
  outstanding in its own comment); NaNs with UNEQUAL counts after the closure
  would falsify the closure's coverage.

Late addendum (2026-10-29, CPU-only) — raw gate-log verification; TP2-offload run reclassified

Follow-up work read the exported gate logs and the raw knob source directly.
Refinements to the section above (a mis-quoted digit triple was corrected
inline there):

• VLLM_GDN_HIP_PREFILL activation mechanism corrected. The knob is a
  raw os.environ.get("VLLM_GDN_HIP_PREFILL", "1") read
  (qwen_gdn_linear_attn.py:306), default ON; the recipe
  flashnext-mtp0.env exports =1. The chain was therefore ACTIVE in every
  failing gate. (The serve-log "Unknown vLLM environment variable" warning
  means non-registry, not inert — an earlier draft of the previous section
  leaned on that warning as activation proof; the recipe export + raw read
  are the correct proof.) The base commit's own comment still owes a TP=4
  fidelity run.

• NaN numbers verified on the raw log: 925 / 824 / 853 / 897 (same step,
  same request, four ranks). Inequality — hence receive-side
  all_gather_into_tensor divergence for the TP4-offload run — stands,
  and the counts are source-proven post-gather (_get_nans_in_logits counts
  NaN rows of the gathered ModelOutput logits).

• New: the TP2-offload first-NaN run has EQUAL per-rank counts (22184 ==
  22184, tp2off-serve.log:1691-1704). On a post-gather tensor that means
  the 2-way gather output was consistent across receivers at that step: the
  TP2-offload NaN origin is upstream-consistent, not gather proof. The
  head-gather closure remains the only patch content present in every
  known-passing lane config, but H1 is proven for TP4-offload only; the never
  TP=4-qualified GDN-prefill axis covers the TP2-offload symptom. The
  pending on-target test (hot-synced closures + VLLM_GDN_HIP_PREFILL=0 +
  fogbox profile) discriminates per run: NaN-free coherent greedy on both
  geometries confirms the pair; EQUAL-count NaNs surviving point next at the
  mixed native stack (extensions compiled against ROCm 7.14.1 / RCCL 2.30.4
  headers with torch 2.13.0+rocm10.0.0 wheels at runtime, vs the coherent
  7.14 proven-image stack).

• Weight integrity CLOSED: all 133 serving files (5 AWQ + 128 PLE int4
  shards) sha256-match the HF manifests (complete=true, 133/133, 0 failed).

• Observer scope: the guard fires only under --monokernel (forces
  VLLM_RAISE_ON_LOGIT_NANS=1); the plain resident-TP4 run had no finiteness
  gate, so its "finite garbage" label is inferred from emitted text only.
  GPU confirmation is still NOT run: the host carries the unrebooted Oct-29
  19:05:22 gfxhub fault (eager TP2 ablation, PID 822920); the operator has not
  authorized recovery. No speed claims; nothing here is validated on RDNA
  hardware beyond the pre-existing maintainer TP2/EP2 qualification.

• Correction (cycle 16): vocab denominator fixed to 248,320 — proven from the model's own safetensors header (lm_head.weight shape [248320, 2560]); the earlier 60,288 figure was unverified. NaN percentages restated: TP4 ~0.37% of row, TP2-offload ~8.9%. Count site additionally source-proven post-gather in the v2 runner (batch sharder inactive for this model: no compute_logits_local), so the unequal-per-rank-counts argument holds.

Parent verification / evidence boundary

The focused CPU distributed-oracle suite was independently rerun at 91ff81019: 12 passed. The 133 weight-payload hash results were independently counted and verified. These validate CPU contracts and byte integrity, not quantization quality, loader semantics, ROCm collectives, or full-model correctness. Unequal post-gather NaN counts establish cross-rank inconsistency; they strongly implicate communication/buffer handling but do not alone distinguish a collective implementation defect from a surrounding race or lifetime bug. A matched on-device A/B remains required. All GPU testing remains stopped after the new fault.

Update (Oct-30, ON-DEVICE): RCCL-level corruption reproduced in isolation; TP4-MTP0 gate RECOVERED on the overlay at 91ff81019

Post-reboot cycle on the same 4x-V620 host (boot id changed; fn-fault clean
before and after every gate below; production lanes untouched; container via a
fixed scoped launcher). Two new evidence pieces, both on hardware:

1. Isolated RCCL repro of the corrupted collective (tp4_rccl_repro.py,
tiny tensors + the real vocab-248,320 head shapes, finite deterministic
rank-labeled inputs, worlds 2 and 4, fp16 and fp32, eager AND inside captured
CUDA graphs):

• all_gather_into_tensor: DETERMINISTIC corruption on every rank in both
  worlds — wrong-rank shard lands at offset 0, 64 Ki-element zero holes,
  identical corruption pattern inside graphs. No NaN inputs required; no
  vllm, no expert cache, no numerics involved.
• all_reduce SUM / MAX / MIN — including this PR's zero-pad local-vocab
  assemble and the top-1 (value-MAX / index-MIN) pair scheme from
  get_top_tokens — PASS exactly on every rank.
  This reproduces the "reproduced ROCm full-logit gather corruption" / "the
  patches are load-bearing, not optional" maintainer-lane comments at the RCCL
  layer on this image+host, and validates the workaround replacement
  (all-reduce of a zero-padded shard) as the correct collective on this fabric.

2. Full-model recovery. Re-ran the EXACT previously-failing gate config
(TP4/EP4, MTP0, monokernel expert-cache offload slots=48, native KV-offload
16 GiB, GMEM 0.92, MAXLEN 32768, MAXBAT 4096, GDN prefill ON,
FULL_AND_PIECEWISE graphs, same image d8529f06, same weights,
NCCL_P2P_DISABLE=1 recipe env) with ONLY two deltas: the branch's Python
files overlaid at 91ff81019 (SHA256-verified in place; runtime import
confirmed from the overlay) and FLASHNEXT_HEAD_ALLREDUCE=1 FLASHNEXT_GATHER_ALLREDUCE=1:

• boot READY 440 s (baseline: 441 s);
• greedy probes 3/3 coherent (Paris/Berlin chains) — baseline was 0/3 with
  NaN engine abort at first decode;
• PPL 4.83 on normal-length greedy continuations; prefix cache 16.3x with
  cold == warm text; probe-after-load PASS;
• serve log: zero "NaNs detected", zero memory faults; fn-fault clean after
  the whole campaign; cards released to ~16 MiB;
• batched-greedy concurrency matrix (48 prompt pairs): 30 identical /
  10 near-tie / 5 soft / 3 divergent, all 3 divergences on one prompt with
  mean log-prob deltas ~0.02–0.12 nats — the trajectory-flip pattern of
  batching-boundary greedy, not the old corruption signature (old resident
  arm: 28/48 fails + garbage repetition).

Attribution honesty: the same-config known-bad baseline was NOT re-run (its
NaN abort is established above), so this is stated as RECOVERY, not proof of
sole cause; the mechanism is independently supported by repro (1). MTP=0
keeps only the head-gather closure live — gather_output/get_top_tokens
paths go active at MTP>=1 and are being exercised next on-device; a
real-agent tool-use trial against this recovered server is the current gate.
No performance numbers claimed.

3. Real-agent tool-use trial — PASS. RuneBench Navigator (harbor, v2),
task woodcutting-xp-15m, endpoint = this recovered server (same image +
overlay + MTP0 config + tool flags): 16.0 min of continuous real agent
progress, 0 API errors, Woodcutting xp 0→14,375 (level 1→38), score = peak
113 XP/min; verifier artifacts + screen recording preserved; fn-fault clean
before/mid/after. Two earlier trials were infra-invalid (server started
without the tool-call flags; OpenCode's openai-compatible default requesting
32000 output tokens against the 32768 served context) — fixed
non-invasively, task instructions untouched. Agent turn latency under this
4-card lane is high; no throughput claim (owner benchmarks).

Update (Oct-30, ON-DEVICE): MTP2 arm on-device quality gates PASS; concurrency matrix + provisional perf cells

4. MTP2 arm (speculative MTP=2, full+piecewise graphs) — on-device quality gates PASS + real-agent trial.
serve_mtp2.sh differs from the MTP0 tools script by ONLY the recipe delta
(diff-verified): --speculative-config {"method":"mtp","num_speculative_tokens":2,"use_local_argmax_reduction":true}
with cudagraph capture sizes [3,6,12,24]; env = 1-line RECIPE change. Same
image d8529f06, same overlay at 91ff81019, same weights, same port.

• READY ~7 min; speculator graphs captured on all 4 ranks; 0 "NaNs
  detected"/Memory-Fault/Traceback lines; fn-fault clean pre-boot, post-boot,
  mid-trial and post-trial (orphaned VRAM-idle EngineCore/Worker processes from
  an earlier boot found and reaped before launch; VRAM confirmed idle first).
• Greedy 4/4 coherent (1+1 / Paris / Berlin / Rayleigh, temp 0 seed 0); tool
  probe finish_reason=tool_calls (chop_tree) + reasoning OK.
• Speculative decoding LIVE and numerically sound: mean acceptance length 2.59
  (probe window), per-position acceptance 0.898/0.695 — i.e. the draft fc and
  get_top_tokens MAX/MIN pair-gather paths, which the MTP0 arm left dormant,
  are ACTIVE on GPU with no NaN/fault across the whole campaign.
• PPL 5.10 on a fresh 104-token corpus (MTP0's 4.83 used a different 261-token
  corpus — same magnitude, NOT head-to-head comparable).
• RuneBench Navigator trial 4 (woodcutting-xp-15m, byte-identical invocation
  to trial 3): reward 304, 0 exceptions, 16m56s continuous progress (trial 3 on
  the MTP0 arm: reward 113). Single trial per arm => NO variance claim; faster
  decode plausibly contributes.
• Concurrency determinism (48-prompt solo-vs-batched identical-text greedy
  matrix, temp 0/seed 0, logprob deltas): 32/48 byte-identical; the 16 divergent
  are all coherent, factually correct, format/verbosity-only flips (bold/list/
  sentence-completion; mean |dlogprob| 0.02-0.45 nats), 0 garbage/repetition/NaN
  — batching-plus-spec-decode near-tie flips, not the corruption signature. Flip
  count exceeds the MTP0 arm's 3/48 (spec-decode accept/reject adds
  batch-dependent tie-break noise); flagged as an open quality item, not a gate
  failure. Raw per-prompt record retained.
• Provisional perf (MTP2 arm only; generation ends at natural EOS ~215 tokens;
  no matched MTP0 re-boot this cycle — boot risk not justified, so no
  arm-vs-arm claim): 1K-class (1296 prompt tok) c1/c4/c8 = 19.8/44.5/65.8
  aggregate out tok/s (mean TTFT 7.1/12.6/18.7 s); 16K-class (19,804 tok)
  = 7.9/40.5/56.0 (23.4/20.6/21.2 s; within-cell identical prompts make c4/c8
  TTFT prefix-cache-aided after the cold c1). All numbers PROVISIONAL; owner
  benchmarks.

With (1)-(4): the TP4/EP4 offload + MTP2 speculative serving configuration is
QUALIFIED on the four V620s at the quality gates tested here (greedy, PPL,
prefix, concurrency, draft-path GPU numerics, two real-agent trials);
performance is provisional and merge remains held for human review.

Flip attribution (CPU-side analysis of the retained per-prompt record; no
further GPU boots). The 16 divergent prompts fall into four pattern classes:
4 diverge at style onset (divpos 2 — terse-answer vs full-sentence opening);
4 are format/verbosity branches visible in the retained previews (bold/list/
expansion); 6 are late-trace branches (divpos 69–269, first 60 bytes identical
— divergence occurs mid-reasoning); 2 are minor wording variants. Median onset
position 37.5, max 269. Median mean |Δlogprob| 0.27 nats, max 0.45 — every flip
sits inside the near-tie band, consistent with batch-dependent draft
accept/reject tie-breaks under MTP2 spec-decode rather than numerical
corruption; no class shows truncation, garbage, or factual divergence.
Analysis script + raw per-prompt JSON retained on the test rig.

5. Server-side corruption counter across the whole MTP2 session

Independent of the output-level checks above, the engine's own scheduler-level
Corrupted: N reqs counter was read out of the live serve log for the entire
continuous MTP2 session (one server process, 03:52Z–05:16Z, spanning the perf,
concurrency and expert-cache-churn runs):

• 214 counter samples, all zero (corrupted_max = 0).
• Same-log marker grep: NaN detected 0, Memory Fault 0, HIP error 0,
  Traceback 0, out of memory 0.
• The session was not idle: the engine's Running: N reqs histogram reached the
  configured max_num_seqs=8 (10 samples at 8, 2 at 7, 2 at 6), so the zero
  result was not obtained at low load only.
• Spec-decode stayed genuinely active rather than collapsing to reject-all:
  mean acceptance length 2.49–3.00 across windows, avg draft acceptance
  74.5–85.3%.

Caveat kept explicit: this is an internal engine counter, not an external
numeric oracle, and it attests to scheduler-level integrity only — output
quality rests on the greedy/PPL/canary/concurrency/agent gates in sections
1–4. Raw extraction: corrupt_check.sh over the in-container serve log
(reproduced verbatim in the retained evidence file).

[BlivionIaG]

Thanks @crispyging — this is adapted and landed on rdna_extras (787b4fa018, merge of feat/moe-offload-topology). The PR's own commits are authored by an agent identity that this repo does not accept, so I applied your work as a verbatim squash (34727e4e1e) plus an adaptation commit (fffd733287, carries a Co-authored-by trailer for you).

What changed relative to the PR:

• FLASHNEXT_HEAD_ALLREDUCE / FLASHNEXT_GATHER_ALLREDUCE are now VLLM_RDNA_HEAD_ALLREDUCE / VLLM_RDNA_GATHER_ALLREDUCE; FLASHNEXT_HEAD_GLOO is dropped (host bounce in the hot loop, not graph-safe).
• One shared tensor_model_parallel_all_gather_via_all_reduce, routed through tensor_model_parallel_all_reduce.
• serve_offload.sh validates the topology through rdna_expert_cache_policy (TP2: 96 slots, TP4: 48 slots).
• The source-text test is replaced by thread-fabric oracles.
• New on top: a HIP one-shot all-gather and a fused vocab argmax on rdna_ar (bit-exact, 3-14x per collective, kill switch VLLM_RDNA_AR_GATHER=0).

Verified on 8x V620 with FULL_AND_PIECEWISE and prefix caching: TP2 offload (96 slots) and TP4 offload (48 slots), probes, PPL, and prefix-cache hits all pass.

Caveats: the zero-pad all-reduce fallback for hosts without P2P could only be covered by the oracles here (our box has working P2P). The TP4-offload evidence in the PR came from a different stack, so we re-qualified it ourselves rather than relying on it.

Write-up: docs/rdna2/moe-offload-topology-2026-11-02.md.

[BlivionIaG]

Closing: the work is on rdna_extras via 787b4fa018; the verbatim squash is 34727e4e1e and the adaptation fffd733287 (Co-authored-by crispyging), as described in the earlier comment. The PR itself is not merged only because its commits carry an agent identity.
Env-name mapping and what was dropped (FLASHNEXT_HEAD_GLOO) are in docs/rdna2/moe-offload-topology-2026-11-02.md ("Credit and adaptation of PR #149"). Thanks again, @crispyging.