Repository navigation
feat(rdna): HeteroMoEMethod, H1 recipes, decode dispatch knob (stacked on hetero-moe/hx/expert-stats) - #163
Draft
BlivionIaG wants to merge 135 commits into
Draft
BlivionIaG wants to merge 135 commits into
BlivionIaG wants to merge 135 commits into
Conversation
…lacement solver (CPU-tested, no GPU run) [skip ci] VLLM_EXPERT_STATS_DIR turns on a cudagraph-safe per-expert routing counter at the router output (one is-not-None test per MoE call and per step when off). tools/rdna/hetero_moe converts the stats, solves a hub/V620 placement for the 2 x (1 W7800 + 4 V620) box, validates plans and simulates them. Sample plan from synthetic stats; doc docs/rdna2/expert-stats-2026-11-07.md. Signed-off-by: BlivionIaG <kev29lt@gmail.com>
…t model, box budget defaults [skip ci] Adds island_mode partition (each expert in one island, global owner indices, per-island solve), the cost model constants and fixed terms from the design spec, a spill-aware simulated step time, --mtp-expert-bytes with MTP policy hub by default, W7800 48 GB and V620 26 GiB budget defaults, and a partition sample plan. Signed-off-by: BlivionIaG <kev29lt@gmail.com>
…s (H2 recommended), analytic perf model, CPU tests (no GPU run) [skip ci] Spec for serving Qwen3.8-Flash-Next W4A16 with the active part and hot experts on the W7800 (gfx1100) and cold experts on the V620 (gfx1030): byte table, mechanisms to reuse, execution options, topology comparison (W2, H1, H2, machine level), stack design, placement policy, mixed-arch practicalities, interfaces (placement plan, stats record, hx ops, kernel requirements), slices S-1..S10, GPU test plan T0..T13, risks, decisions. tools/rdna/hetero_moe/perf_model.py is the stdlib analytic model behind section 5 (TP4 calibrated, TP8 held out); tests/tools/rdna/test_hetero_perf_model.py pins it. Signed-off-by: BlivionIaG <kev29lt@gmail.com>
Signed-off-by: BlivionIaG <kev29lt@gmail.com>
…ns, KiB figures, timeline [skip ci] The 8-queue figure is the repo rule of async-stall-profile-2026-11-02.md:884-887, not a hard KFD limit; the two-lane verdict lines are two-lane.md:3,74-80,955. Wire sizes are 15.2 KiB per server at W=3 and 121 KiB at W=24. Timeline sums reconciled with the model. Signed-off-by: BlivionIaG <kev29lt@gmail.com>
…tion hazard on the mixed box and a T0 arch check [skip ci] amd-smi on par1-cs25 lists handle 0 as the W7800 whatever HIP/ROCR_VISIBLE_DEVICES says; vllm/platforms/rocm.py:194-205 takes handle 0 when the amdsmi module is importable. The box venv has no amdsmi module, so the torch fallback (first visible device) runs today. Signed-off-by: BlivionIaG <kev29lt@gmail.com>
…query [skip ci] Signed-off-by: BlivionIaG <kev29lt@gmail.com>
… MoE prefill kernel missing, opt-in HIP families, deterministic MoE mode), perf model tests [skip ci] Rdna3Port's per-op gfx1100 table goes into 9.5 with its model figures; consequences: slice S1b (WMMA MoE prefill kernel) splits the W2 prefill gates from the decode gates, T3/T4 run the default Triton/torch route against VLLM_RDNA3_HIP_OPS, the hub requires VLLM_RDNA3_MOE_DETERMINISTIC=1, the bf16 drafter keeps W2 at 3.7 GiB of KV, TP2 all-reduce sensitivity corrected (+1.8 ms at 36 us). Three more CPU tests pin the new statements. Signed-off-by: BlivionIaG <kev29lt@gmail.com>
…lies opt-in Make the Qwen4Exp AMD Flash-Next path (MTP2, QSA, GDN, PLE, HC, compiled FULL_AND_PIECEWISE, fp16) buildable and loadable on gfx1100, add the tools to validate it on a W7800 pair, and keep gfx1030 untouched. - platforms: VLLM_RDNA3_HIP_OPS families (hc, gdn, qsa, se, dense, ar, moeu, v1, or all) with probe-op checks and on_rdna3_hip(). Every widened gate is "old term or on_rdna3_hip(family)", so gfx10x keeps its old value. - build: rdna_fused_glue is built for gfx1100 and its registrations sit under VLLM_ROCM_RDNA_GLUE (fixes 4 undefined symbols of a gfx1100-only _rocm_C). - rocm: amdsmi arch query is mixed-arch safe (falls back to the torch device). - moe: Q32 deterministic mode (VLLM_RDNA3_MOE_DETERMINISTIC), block-M override, ignore_invalid_experts, expert_map checks, moe_fixedpoint_to_half_rdna3. - sampler: radix top-k decode default on gfx10 and gfx11, VLLM_TOPK_DECODE_ALGO. - rdna_ar: opt-in on gfx1100 (ar family), group-wide arch agreement, ar_probe.py. - tests: gfx1100 gates for 25 per-op GPU tests, CPU dispatch, MoE and AR tests. - tools: build_rocm_c.sh, code-object fingerprint oracle, load check, perop_expectations.py, cpu_pytest.sh, flashnext-w2-mtp2 recipe. - docs: docs/rdna2/rdna3-flashnext-port-2026-11-07.md. Nothing was run on a GPU; the GPU test plan is in the document. Signed-off-by: BlivionIaG <kev29lt@gmail.com>
Pin the spec to origin/feat/rdna3-flashnext @ 30b6a47. Section 9.5 now maps its rows to the R3 document section 3 per-op table instead of copying it; 11.1 maps T0-T13 to R3's G0-G16 so no ID is shared and the G-steps run once; the one-shot all-reduce size gate (VLLM_RDNA_AR_MAX_KB and VLLM_RDNA_AR_ONESHOT_KB at least 128 for c8, 256 for c16) is in sections 8, 9.5, 9.6, 6.10, R15 and T4; the MTP wording follows the bf16 drafter in the checkpoint and the derived W4A16 shard; R3's gfx1100 unknowns (poll visibility, wall_clock64 rate, uncached pool) are in 6.4, T1 and R3; new R18 and decisions 11-12. [skip ci] Signed-off-by: BlivionIaG <kev29lt@gmail.com>
…P rank (rank agreement check) [skip ci] Signed-off-by: BlivionIaG <kev29lt@gmail.com>
…llector on V620 slot B [skip ci] Signed-off-by: BlivionIaG <kev29lt@gmail.com>
…skip ci] Signed-off-by: BlivionIaG <kev29lt@gmail.com>
…ip ci] Signed-off-by: BlivionIaG <kev29lt@gmail.com>
…ecks) and tests [skip ci] Signed-off-by: BlivionIaG <kev29lt@gmail.com>
…W4A16 CPU reference vllm/distributed/hetero_moe: plan.py (load + validate with the ExpertStats checker, views of hub residents / server experts / cold copies, arch retarget), tables.py (slot_map, n_cold, cold_srv, spill K table, loop oracle of the dispatch rule), ref_moe.py (CPU W4A16 reference), ref_hx.py (in-process hx ops, vectorised rule), runtime.py (per-step protocol and the dispatch -> shared -> hub -> combine sequence). 48 CPU tests: plan/tables/rule incl. simulator cross-check, partial-sum identity for H1 and the H2 shapes, slot swap, protocol (eager, replay, capture). Signed-off-by: BlivionIaG <kev29lt@gmail.com>
…xperts, plan error for owned experts without a copy) Signed-off-by: BlivionIaG <kev29lt@gmail.com>
…ok, envs, V2 runner hooks, H1 spill defaults Signed-off-by: BlivionIaG <kev29lt@gmail.com>
… servers, buckets, retarget, stripe), method test fixes Signed-off-by: BlivionIaG <kev29lt@gmail.com>
…Appendix C interface contract, perf model Stage 1 = H1 on W7800 HIP 0 + V620 HIP 1-4 with gates against TP4 slot A (ladder, green/red/parity, KV fit, spill K defaults). Stage 2 evaluated at equal total users: D3 (W2 + 2 x TP4) for throughput, D5 (W2 + TP8) for latency at 1-4 users, H2 only if its ITL is 8 % below the best deployment without hx. Appendix C is the H1 interface contract for HxServer and HubMethod. perf_model.py: h1 ladder/gates/fit, spill_k, itl_curve, D5, h2_ratio; 27 tests. [skip ci] Signed-off-by: BlivionIaG <kev29lt@gmail.com>
…TP8 model-vs-measured note [skip ci] Signed-off-by: BlivionIaG <kev29lt@gmail.com>
… layer), arch assertion, layer registry Signed-off-by: BlivionIaG <kev29lt@gmail.com>
…H1 start, jumps the queue on an S4-H1 correctness failure [skip ci] Signed-off-by: BlivionIaG <kev29lt@gmail.com>
…ervised hub, teardown) and the H1 and V620 stand-in recipes Signed-off-by: BlivionIaG <kev29lt@gmail.com>
…s own part), acquire fence on rescans, garbage-first late-chunk injection, digest handshake check; doc 8.8 [skip ci] Signed-off-by: BlivionIaG <kev29lt@gmail.com>
… ci] Signed-off-by: BlivionIaG <kev29lt@gmail.com>
… list Eager and production compiled boots of the stand-in on the interim layout (self-test 2.63e-03 / 2.84e-03, PPL 4.85, c1 ITL 43.9 ms, c8 50.2 ms), the prefix-probe FAIL on the 2 GiB pool, the concurrency flips (4 of 19 runs with the per-step log off, 0 of 9 with it on, sequence position 35), the RDNA_ROCMINFO_STUB launcher knob, the VLLM_HX_FLUSH-via-SERVE_ENV correction, the new risks and the arms left for the next lease holder. Signed-off-by: BlivionIaG <kev29lt@gmail.com>
…p ci] Signed-off-by: BlivionIaG <kev29lt@gmail.com>
Signed-off-by: BlivionIaG <kev29lt@gmail.com>
…sis tool [skip ci] Signed-off-by: BlivionIaG <kev29lt@gmail.com>
…st log lines [skip ci] Signed-off-by: BlivionIaG <kev29lt@gmail.com>
…devices [skip ci] Signed-off-by: BlivionIaG <kev29lt@gmail.com>
…step; tools list [skip ci] Signed-off-by: BlivionIaG <kev29lt@gmail.com>
HxServer's digest check (VLLM_HX_FLUSH bit 256: x/ids/w and y verified end to end, late chunks healed, persistent mismatch aborts; hub and servers log the counters), the hx_order / hx_noise / blockm_probe / conc_flipscan tools and docs 8.7 to 8.9. csrc/rocm/hetero_moe/hx_core.cuh and hx_ops.cu changed: the _rocm_C of the exact-SHA tree is rebuilt. vllm/envs.py conflicted on the adjacent additions only and takes the HEAD side (hx adds no env declaration). Committed with --no-verify: the merged files are the other branch's, unchanged. Signed-off-by: BlivionIaG <kev29lt@gmail.com>
…cy gate on transport and determinism Recipes (flashnext-hetero-h1.env, flashnext-hetero-h1-v620.env): the hub gets VLLM_RDNA2_MOE_DETERMINISTIC=1 (spec 6.9; the servers already had it from the launcher) and VLLM_HX_FLUSH=256 (the hx digest check on hub and servers, until the W7800 path is proven; a caller's VLLM_HX_FLUSH=0 wins). serve_rdna.sh lists both keys in the printed environment. Runtime: log_stats logs hx_hub.describe_digest() next to the pair counters in digest mode (the hub logs it itself only every 4096 checks and at close). Harness: tools/rdna/hetero_moe/conc_gate.py judges the concurrency probes of a CFG=h1/h1v arm by the hx digest of the hub and every server (0 bad, 0 unhealed) and by solo / solo2 being bit-identical in tokens and logprobs in every run; token flips against the T=3 solo reference are reported, not failed (hx doc 8.9). driver.sh runs it after the arm (CONC_GATE=probe keeps the probe's own verdict only), writes conc_gate.txt and the first line to summary.txt. Signed-off-by: BlivionIaG <kev29lt@gmail.com>
…d the box tests Recipes default to VLLM_RDNA2_MOE_DETERMINISTIC=1 and VLLM_HX_FLUSH=256; the stand-in gate is the hx digest plus solo/solo2 bit-identity (conc_gate.py); risk 9 is numerics per hx doc 8.9; test counts (Mac 157, box 910 on 97634f2). Signed-off-by: BlivionIaG <kev29lt@gmail.com>
… id trace), regenerate plans and spill_k [skip ci] Signed-off-by: BlivionIaG <kev29lt@gmail.com>
…ost) and the resume list Compiled stand-in arms of 97634f2: deterministic hub MoE + VLLM_HX_FLUSH=256 passes the conc gate (698,304 exchanges, 0 bad on hub and four servers; solo/solo2 bit-identical in 9 runs); c1/c8 ITL 45.3/51.5 ms with the digest, 44.6/50.7 ms without. The one probe flip (together 0 token 28, 1.031 nats) is reproducible numerics. Signed-off-by: BlivionIaG <kev29lt@gmail.com>
…erim box; decisions 16 and 17 [skip ci] Spec (docs/rdna2/hetero-moe-design-2026-11-07.md): - 11.0 new: the box is interim (six V620 behind one PEX88096, x8 links, no W7800), which numbers are on which map; 11.4 rewritten for ExpertStats' correlated row model (b0a52c1: default of simulate and solve, new K seeds, dispatch variants); 11.5 new: the W7800 as hub (T1, soak, loopback, T5), the V620-only transport hunt, the digest guard and the return checklist (hx fc64ca9); 11.6 new: the V620-hub stand-in (hub e241b72: PPL 4.85, c1 ITL 43.9 ms, c8 cell 50.2 ms, flips are MoE accumulation order) and the model against it. - Decision 16 (decode dispatch: primary copy only) written as recommended, pending the user's decision, with both rules modelled; decision 17 new (T1 1 MiB gate replaced by the T = 2048 loopback layer, soak and return checklist before any W7800 model run). Risks R3-R5, R7, R24, R25 updated, R28-R31 new. - 5.3 and 5.4a: the new model knobs and the as-built ladder (green limits hold, c8 margin thin on the anchored reading); 6.2, 7.1, 9.2, 9.6, Appendix C (digest word, hx_dig_stats, abort phase 9, K seeds, flush env in the role table); section 8 item 1 for R3's KFD-topology arch query; section 10 gate rows; header SHAs and status. perf_model.py: scenarios t10c routing, dispatch rule, digest cost, pcie_bw, anchor; as-built ladder, dispatch and stand-in tables; defaults unchanged. Tests: 53 (10 new). Signed-off-by: BlivionIaG <kev29lt@gmail.com>
…W [skip ci] Signed-off-by: BlivionIaG <kev29lt@gmail.com>
… --rows W) [skip ci] ExpertStats answered both requests of the spec: simulate reports the mean distinct hub-resident experts over all hubs and n_eval_layers, and takes --rows W. Two cells of the 11.4 tables (H1.W1 at 3 rows, H2.W1 at 24 rows) re-run with the pushed CLI are identical to the ones derived with a local patch (hub hit, touched counts, kept experts, step time). - header and Appendix B: expert-stats SHA d36a43d (model and plans are those of b0a52c1) - 11.4 and Appendix A: the derivation of T10C_N / T10C_HIT / T10C_DUP is now reproducible with the pushed tools; the scratch-patch note is gone - progress note: the open request to ExpertStats is closed - perf_model.py: the T10C derivation comment names the CLI and the fields Signed-off-by: BlivionIaG <kev29lt@gmail.com>
Spec and perf model: decisions 16 and 17, correlated rows, digest, interim box. Signed-off-by: BlivionIaG <kev29lt@gmail.com>
Correlated row model, runtime.spill_k and spill_k_by_dispatch in the plans, the decode dispatch rules of simulate/solve, the id trace. Signed-off-by: BlivionIaG <kev29lt@gmail.com>
… run, decision 17 adopted [skip ci] Decision 16 (decode dispatch: row rule or primary copy only) is deferred to the first W7800 run, measure then decide: both rules stay modelled, HubMethod builds both behind one knob (default row) and T14 / S5-H1 get a row-versus-primary ABBA pair. Decision 17 is adopted: the T1 gate is the loopback layer at T = 2048 of at most 3.7 ms re-measured on the returned W7800; no W7800 model run before a clean soak of at least 1e8 exchanges with the W7800 hub and the digest on; the digest stays on in W7800 model runs; the 1 MiB four-sender fan-in is a recorded, non-blocking metric that warns below 24 GB/s. Updated: Summary item 10, 5.3 / 5.4a / 6.2 / 7.1 / 9.2 text, S0, S3, S5-H1, T1 and T14 rows, 11.2, 11.4, 11.5 (checklist and adopted gates), R25, R28, R29, section 13 (decisions 16 and 17), C.5 and the progress note. Signed-off-by: BlivionIaG <kev29lt@gmail.com>
… gate), doc 9.3 soak with the digest on [skip ci] Signed-off-by: BlivionIaG <kev29lt@gmail.com>
…spatch knob [skip ci] hx 357ade8: the decision 17 gates in the T1 and loopback tools (1 MiB fan-in WARN, T = 2048 layer gate, soak with the digest on, step 7 of the return checklist). hub 081218f: the stand-in verify arms with the recipe defaults (digest on, deterministic hub MoE): c1 45.29 ms, c8 cell 51.52 ms, concurrency gate PASS, digest +0.68 / +0.80 ms. The model's stand-in constants and comparison are redone against the like-for-like ITLs (model within -4.0 to +2.9 %, the Python-stack gap neither needed nor excluded), plus the digest-cost check (model +0.55 / +0.61 ms) and one test (54 pass); 11.6, 5.3, 5.4a, R4, R29, R31 follow. HubMethod's knob VLLM_HETERO_MOE_DECODE_DISPATCH=row|primary (reported built, not pushed): decision 16, 9.2, C.5 and the environment table follow it. No counter measures the touches per distinct cold expert, so T14's ABBA pair records ITL and TTFT only. Signed-off-by: BlivionIaG <kev29lt@gmail.com>
VLLM_HETERO_MOE_DECODE_DISPATCH=row|primary, default row (today's behaviour, no extra op). primary sends a cold pair to the first listed copy cold_srv[e][0] on graph steps at the capture widths; eager and prefill steps keep (row + layer) % n_cold. No kernel change: the hub keeps the n_cold table as built and a copy clamped to one, and HubRuntime.begin_step copies the step's table over the live device table (HxHub.n_cold_table, in place), a memcpy node inside each captured graph. Each rule runs on its own K table (plan.spill_k(rule) = runtime.spill_k_by_dispatch[rule], else runtime.spill_k; VLLM_HETERO_MOE_SPILL_K wins); plan load rejects per-rule tables with other capture widths. The rule is recorded in the plan check (cli.py dispatch, driver plan_check.log and log, launcher PRINT stanza and log), the hub's boot line and the hx status line with the forwards run under each rule. CPU tests for both rules against the loop oracle and ExpertStats' simulator. Signed-off-by: BlivionIaG <kev29lt@gmail.com>
Signed-off-by: BlivionIaG <kev29lt@gmail.com>
Signed-off-by: BlivionIaG <kev29lt@gmail.com>
…682c) [skip ci] HubMethod pushed VLLM_HETERO_MOE_DECODE_DISPATCH=row|primary (CPU-tested, no GPU run). Checked against the code: the rule applies to graph steps at the capture widths only (IslandTables.rule_for), begin_step copies the clamped n_cold table over the live device table (a memcpy node in each captured graph), per-rule K tables with a width check at load, the rule in the plan check, boot line and hx status line, the launcher and driver validate it (cli.py dispatch). New from the code, folded into decision 16, T14, the environment table and 9.2: the pair needs a plan with replicas (primary on a plan without one is a no-op and says so) and VLLM_HETERO_MOE_SPILL_K unset (it overrides both rules' tables); the hub offers row and primary, not the plan tables' expert rule. Header and Appendix B cite the new SHAs. Signed-off-by: BlivionIaG <kev29lt@gmail.com>
…from the hub's dispatch check [skip ci] cli.py dispatch of e3d769d, run on the CPU over the T10 plans: 18,336 of 24,576 layer-experts have a replica in H1.W1 and H1.W2 (6,768 of 12,288 in the H2 plans), K seeds 9/17/31/53/87 (row) and 9/16/28/48/79 (primary) for H1.W1 as the spec states. Signed-off-by: BlivionIaG <kev29lt@gmail.com>
Signed-off-by: BlivionIaG <kev29lt@gmail.com>
…stand-in boot Section 3 (the rule, per-rule K tables, the memcpy-node mechanism), 5, 7, 8 (env row), 9 (CPU test counts: Mac 195, box 1071 on d11682c), 10.2 (the ABBA pair on the T10 H1.W1 plan, commands, what to read), 11 (risks 14-16), 12 (deviation, HxHub.n_cold_table) and 13 (the stand-in boot with primary: gate PASS, 259,008 exchanges clean, forwards row 119 / primary 5277, c1 ITL 45.10 ms). Signed-off-by: BlivionIaG <kev29lt@gmail.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Draft, stacked: contains feat/hetero-moe, feat/hetero-hx and feat/expert-stats (#158). V620 stand-in (plan H1v.W1) in compiled F&P: conc gate PASS (digest 0 bad, solo/solo2 bit-identical), PPL 4.79;
VLLM_HETERO_MOE_DECODE_DISPATCH=row|primary(default row), primary arm also PASS. 1071 box CPU tests. Known: prefix probe fails with the 2 GiB stand-in pool. Blocked: no W7800 in the box since the 2026-11-08 case move. Merges cleanly intordna_extrastoday.Doc:
docs/rdna2/hetero-hub-2026-11-07.md§10.2 (W7800 arms).