Skip to content

Commit 54f1191

Browse files
committed
docs(kimi-k2): merge NVSHMEM arm into the headline 32-node table
One canonical 5-column table (NCCL | UCCL | NVSHMEM | UCCL delta | NVSHMEM delta) matching the qwen3-235b format, with per-column campaign provenance: NCCL+UCCL are a same-campaign A/B (2026-06-04), NVSHMEM is cross-campaign (2026-07-14, different cluster/storage) and its deltas are marked approximate. Replaces the separate DeepEP+NVSHMEM section.
1 parent d2e3b84 commit 54f1191

1 file changed

Lines changed: 39 additions & 57 deletions

File tree

  • 3.test_cases/megatron/megatron-bridge/kimi-k2/benchmarks

3.test_cases/megatron/megatron-bridge/kimi-k2/benchmarks/RESULTS.md

Lines changed: 39 additions & 57 deletions
Original file line numberDiff line numberDiff line change
@@ -15,43 +15,64 @@ NVSHMEM — is fixed by the image; see the arm table in [`../README.md`](../READ
1515
everything else (model, mock data, seq 4096, GBS 256, bf16, seed, EFA env,
1616
`MOE_A2A_OVERLAP` flags) is held fixed within a cell. 24 iterations per run,
1717
`log_interval=1`; the first 4 iterations (incl. the ~300 s compile) are dropped from
18-
perf stats. The June campaign measured UCCL vs NCCL same-campaign; the
19-
[DeepEP+NVSHMEM section](#deepepnvshmem-arm--32-node-pp8-2026-07-14-20260714t0515z-k2-nvshmem-pp8-32n-ue1)
20-
adds the third arm from a 2026-07-14 campaign (cross-campaign caveat noted there).
18+
perf stats. The headline table merges two campaigns — NCCL+UCCL (same-campaign A/B,
19+
2026-06-04) and DeepEP+NVSHMEM (2026-07-14) — with per-column provenance marked there.
2120

22-
- Image: `megatron-bridge-uccl:nemo-26.04.01-uccl-0dc87eb` (Bridge 0.4.2, Core 0.17.1)
21+
- UCCL/NCCL image: `megatron-bridge-uccl:nemo-26.04.01-uccl-0dc87eb` (Bridge 0.4.2, Core 0.17.1)
2322
- Raw logs (no overwrite, all ranks): `/fsx/megatron-bridge-bench/<campaign>/kimi-k2/<arm>-mb<m>-ovl<on|off>/`
2423
- Per-run gate: `NET/OFI Selected provider is efa, fabric is efa-direct` on **all** nodes;
2524
deepep arms additionally show the UCCL-EP proxy registration (high-throughput mode).
2625

2726
---
2827

29-
## Headline — 32-node PP8 (256× B300), campaign `20260604T083049Z-uccl-ab-pp8-32n-v2`
28+
## Headline — 32-node PP8 (256× B300), all three arms
3029

3130
Parallelism **TP8 / PP8 / EP32 / DP4** (the canonical layout, same as the published
3231
DSV3 numbers). Steady-state mean over iters 5–24, zero stalls, EFA active 32/32 in
33-
every run.
34-
35-
| cell | UCCL deepep | NCCL alltoall | UCCL delta (iter time) |
36-
|---|---|---|---|
37-
| **mb=4, overlap=on** | **3.834 s/iter · 219.1 TF/GPU · 273.5k tok/s** | 5.793 s/iter · 144.8 TF/GPU · 181.0k tok/s | **−33.8%** |
38-
| **mb=4, overlap=off** | **6.041 s/iter · 138.8 TF/GPU** | 9.303 s/iter · 90.1 TF/GPU | **−35.1%** |
39-
| **mb=1, overlap=off** | 14.98 s/iter · 56.0 TF/GPU | **12.88 s/iter · 65.1 TF/GPU** | +16.3% (NCCL faster) |
40-
| mb=1, overlap=on | *not measured* (see coverage note) | *not measured* ||
32+
every run. Two measurement campaigns feed this table:
33+
34+
- **¹ NCCL + UCCL**: same-campaign A/B, `20260604T083049Z-uccl-ab-pp8-32n-v2`
35+
(2026-06-04, EKS us-west-2, FSx). The UCCL Δ is a rigorous same-campaign delta.
36+
- **² DeepEP+NVSHMEM**: `20260714T0515Z-k2-nvshmem-pp8-32n-ue1` (2026-07-14, EKS
37+
`ml-clusters-shared-us-east-1`, 32× p6-b300 **us-east-1-atl-2a local-zone Capacity
38+
Block**, node-local NVMe via `STORAGE=hostpath`, `GDRCOPY_DEV=on` — see
39+
[`../README.md`](../README.md)). Image `megatron-bridge-uccl:nemo-26.04.01-deepep-nvshmem-567632d-cu13`
40+
(NVSHMEM v3.7.0-0, DeepEP `567632d`, the
41+
[`deepep-benchmark`](../../../../../micro-benchmarks/expert-parallelism/deepep-benchmark)
42+
NVSHMEM-over-libfabric/EFA build). Same bench entrypoint, layout, GBS, seq, mock
43+
data, iters, warmup. **The NVSHMEM Δ is cross-campaign** (different cluster/date/
44+
storage) and carries noise the UCCL Δ does not — read it as indicative, not exact.
45+
46+
| cell | NCCL alltoall¹ | DeepEP+UCCL¹ | DeepEP+NVSHMEM² | UCCL Δ¹ | NVSHMEM Δ² |
47+
|---|---|---|---|---|---|
48+
| **mb=4, overlap=on** | 5.793 s/iter · 144.8 TF/GPU · 181.0k tok/s | 3.834 s/iter · 219.1 TF/GPU · 273.5k tok/s | **3.742 s/iter · 224.6 TF/GPU · 280.2k tok/s** | **−33.8%** | ~−35.4% |
49+
| **mb=4, overlap=off** | 9.303 s/iter · 90.1 TF/GPU | **6.041 s/iter · 138.8 TF/GPU** | 6.632 s/iter · 126.4 TF/GPU · 158.1k tok/s | **−35.1%** | ~−28.7% |
50+
| **mb=1, overlap=off** | **12.88 s/iter · 65.1 TF/GPU** | 14.98 s/iter · 56.0 TF/GPU | 13.538 s/iter · 61.9 TF/GPU · 77.5k tok/s | +16.3% (NCCL faster) | ~+5.1% (NCCL faster) |
51+
| mb=1, overlap=on | *not measured* (see coverage note) | *not measured* | *not measured* |||
4152

4253
**Takeaways**
4354

44-
- **At mb≥4 — the throughput-efficient operating point for both dispatchers — UCCL-EP
45-
wins decisively on literal K2: −34/−35% iteration time in both overlap regimes.**
46-
Overlap does not close the gap (it helps both arms; the relative win is unchanged).
47-
- At **mb=1 no-overlap**, NCCL all-to-all is ~16% faster — the same unamortized
55+
- **At mb≥4 — the throughput-efficient operating point — both DeepEP transports win
56+
decisively on literal K2**: UCCL −34/−35% iteration time in both overlap regimes
57+
(same-campaign), NVSHMEM ~−35/−29% (cross-campaign). At the tuned point
58+
(mb=4+overlap) NVSHMEM edges the June UCCL number by ~2.4% — within cross-campaign
59+
noise; read the two DeepEP transports as **competitive**, not ranked.
60+
- At **mb=1 no-overlap**, NCCL all-to-all stays fastest — the same unamortized
4861
per-dispatch-overhead regime seen on DSV3 (~12.6%) and on the K2 16-node run (~12%).
49-
A tuned run operates at mb≥4, where UCCL wins.
62+
A tuned run operates at mb≥4, where the DeepEP arms win.
5063
- The mb4+overlap UCCL advantage is **larger at 32-node/PP8 (−34%) than at
5164
16-node/PP4 (−21%)** (table below). EP width is **identical in both (EP32)**
5265
the difference tracks the doubled node count (wider inter-node all-to-all span)
5366
and deeper pipeline, not expert-parallel width.
5467

68+
**NVSHMEM-arm validity** — every cell: `efa_ok` on all 32 nodes, 0 stalls, `n_steady`
69+
20/20, `parse-runs.py` transport = **nvshmem** (NVSHMEM-over-libfabric init banner on
70+
the 8 EP-group leader nodes; no UCCL proxy lines). Iteration-1 `lm loss` 13.43885 /
71+
13.43207 / 13.43359 (mb4-ovlon / mb4-ovloff / mb1-ovloff); mb4-ovlon matches the June
72+
arms' 13.438850/13.438960 to 5 significant figures — same numerical work, bf16
73+
round-off apart. As designed, every NVSHMEM run wrote `STATUS` then exited 1 at
74+
NVSHMEM finalize (benign; see `../run-ab-rawpods.sh`).
75+
5576
### Numerical (work) equivalence — confirmed
5677

5778
Per-iteration `lm loss` curves (24 iters, `log_interval=1`, last-PP-stage rank) for
@@ -81,45 +102,6 @@ The 32-node Capacity Block ended at 2026-06-04 ~10:51Z (EC2 reclaims instances
81102
remain the separate 2026-06-01 campaign in [`../../dsv3/benchmarks/RESULTS.md`](../../dsv3/benchmarks/RESULTS.md)
82103
(same image/layout, but not same-campaign apples-to-apples).
83104

84-
## DeepEP+NVSHMEM arm — 32-node PP8 (2026-07-14, `20260714T0515Z-k2-nvshmem-pp8-32n-ue1`)
85-
86-
The third dispatcher arm (see the arm table in [`../README.md`](../README.md)): **NVIDIA
87-
DeepEP v1 over NVSHMEM-libfabric/EFA** — the
88-
[`deepep-benchmark`](../../../../../micro-benchmarks/expert-parallelism/deepep-benchmark) build,
89-
image `megatron-bridge-uccl:nemo-26.04.01-deepep-nvshmem-567632d-cu13` (NVSHMEM v3.7.0-0,
90-
DeepEP `567632d`). Same bench entrypoint, layout (TP8/PP8/EP32/DP4, 256× B300), GBS 256,
91-
seq 4096, mock data, 24 iters, warmup 4 as the June campaign.
92-
93-
> **Cross-campaign comparison, not a same-campaign A/B.** The NVSHMEM cells ran on a
94-
> *different cluster and date* than the June UCCL/NCCL numbers: EKS
95-
> `ml-clusters-shared-us-east-1`, 32× p6-b300 **us-east-1-atl-2a local-zone Capacity
96-
> Block**, node-local NVMe (`STORAGE=hostpath`) instead of FSx, `GDRCOPY_DEV=on`
97-
> (see [`../README.md`](../README.md)). Same image lineage, config, and EFA env, but
98-
> deltas below carry cross-cluster noise the June UCCL-vs-NCCL deltas do not.
99-
100-
| cell | DeepEP+NVSHMEM (2026-07-14) | UCCL deepep (2026-06-04) | NCCL alltoall (2026-06-04) |
101-
|---|---|---|---|
102-
| **mb=4, overlap=on** | **3.742 s/iter · 224.6 TF/GPU · 280.2k tok/s** | 3.834 s/iter · 219.1 TF/GPU · 273.5k tok/s | 5.793 s/iter · 144.8 TF/GPU · 181.0k tok/s |
103-
| **mb=4, overlap=off** | 6.632 s/iter · 126.4 TF/GPU · 158.1k tok/s | **6.041 s/iter · 138.8 TF/GPU** | 9.303 s/iter · 90.1 TF/GPU |
104-
| **mb=1, overlap=off** | 13.538 s/iter · 61.9 TF/GPU · 77.5k tok/s | 14.98 s/iter · 56.0 TF/GPU | **12.88 s/iter · 65.1 TF/GPU** |
105-
106-
**What the numbers say** (with the cross-campaign caveat above):
107-
108-
- **NVSHMEM DeepEP is competitive with UCCL on literal K2 at the tuned operating point**:
109-
at mb=4+overlap it edges the June UCCL number (−2.4% iter time) and lands −35.4% vs the
110-
June NCCL baseline — the same "DeepEP-class dispatchers win decisively at mb≥4" picture.
111-
- At mb=4 no-overlap NVSHMEM trails UCCL (+9.8%) while still beating NCCL by −28.7%.
112-
- At mb=1 no-overlap the June ranking holds: NCCL fastest, but NVSHMEM (13.54 s) beats
113-
UCCL's June 14.98 s in the unamortized regime.
114-
115-
**Validity** — every cell: `efa_ok` on all 32 nodes (`Selected provider is efa`), 0 stalls,
116-
`n_steady` 20/20, `parse-runs.py` transport = **nvshmem** (NVSHMEM-over-libfabric init
117-
banner present on the 8 EP-group leader nodes; no UCCL proxy lines). Iteration-1 `lm loss`
118-
13.43885 / 13.43207 / 13.43359 (mb4-ovlon / mb4-ovloff / mb1-ovloff); the mb4-ovlon value
119-
matches the June arms' quoted 13.438850/13.438960 to 5 significant figures — same numerical
120-
work, bf16 round-off apart. As designed, every NVSHMEM run wrote `STATUS` then exited 1 at
121-
NVSHMEM finalize (benign; see `../run-ab-rawpods.sh`).
122-
123105
## Appendix — 16-node PP4 (128× B300), campaign `20260604T051726Z-uccl-ab-pp4-16n-v2`
124106

125107
Parallelism **TP8 / PP4 / EP32 / DP2***not directly comparable to the PP8 table*

0 commit comments

Comments
 (0)