@@ -15,43 +15,64 @@ NVSHMEM — is fixed by the image; see the arm table in [`../README.md`](../READ
1515everything else (model, mock data, seq 4096, GBS 256, bf16, seed, EFA env,
1616` MOE_A2A_OVERLAP ` flags) is held fixed within a cell. 24 iterations per run,
1717` log_interval=1 ` ; the first 4 iterations (incl. the ~ 300 s compile) are dropped from
18- perf stats. The June campaign measured UCCL vs NCCL same-campaign; the
19- [ DeepEP+NVSHMEM section] ( #deepepnvshmem-arm--32-node-pp8-2026-07-14-20260714t0515z-k2-nvshmem-pp8-32n-ue1 )
20- adds the third arm from a 2026-07-14 campaign (cross-campaign caveat noted there).
18+ perf stats. The headline table merges two campaigns — NCCL+UCCL (same-campaign A/B,
19+ 2026-06-04) and DeepEP+NVSHMEM (2026-07-14) — with per-column provenance marked there.
2120
22- - Image : ` megatron-bridge-uccl:nemo-26.04.01-uccl-0dc87eb ` (Bridge 0.4.2, Core 0.17.1)
21+ - UCCL/NCCL image : ` megatron-bridge-uccl:nemo-26.04.01-uccl-0dc87eb ` (Bridge 0.4.2, Core 0.17.1)
2322- Raw logs (no overwrite, all ranks): ` /fsx/megatron-bridge-bench/<campaign>/kimi-k2/<arm>-mb<m>-ovl<on|off>/ `
2423- Per-run gate: ` NET/OFI Selected provider is efa, fabric is efa-direct ` on ** all** nodes;
2524 deepep arms additionally show the UCCL-EP proxy registration (high-throughput mode).
2625
2726---
2827
29- ## Headline — 32-node PP8 (256× B300), campaign ` 20260604T083049Z-uccl-ab-pp8-32n-v2 `
28+ ## Headline — 32-node PP8 (256× B300), all three arms
3029
3130Parallelism ** TP8 / PP8 / EP32 / DP4** (the canonical layout, same as the published
3231DSV3 numbers). Steady-state mean over iters 5–24, zero stalls, EFA active 32/32 in
33- every run.
34-
35- | cell | UCCL deepep | NCCL alltoall | UCCL delta (iter time) |
36- | ---| ---| ---| ---|
37- | ** mb=4, overlap=on** | ** 3.834 s/iter · 219.1 TF/GPU · 273.5k tok/s** | 5.793 s/iter · 144.8 TF/GPU · 181.0k tok/s | ** −33.8%** |
38- | ** mb=4, overlap=off** | ** 6.041 s/iter · 138.8 TF/GPU** | 9.303 s/iter · 90.1 TF/GPU | ** −35.1%** |
39- | ** mb=1, overlap=off** | 14.98 s/iter · 56.0 TF/GPU | ** 12.88 s/iter · 65.1 TF/GPU** | +16.3% (NCCL faster) |
40- | mb=1, overlap=on | * not measured* (see coverage note) | * not measured* | — |
32+ every run. Two measurement campaigns feed this table:
33+
34+ - ** ¹ NCCL + UCCL** : same-campaign A/B, ` 20260604T083049Z-uccl-ab-pp8-32n-v2 `
35+ (2026-06-04, EKS us-west-2, FSx). The UCCL Δ is a rigorous same-campaign delta.
36+ - ** ² DeepEP+NVSHMEM** : ` 20260714T0515Z-k2-nvshmem-pp8-32n-ue1 ` (2026-07-14, EKS
37+ ` ml-clusters-shared-us-east-1 ` , 32× p6-b300 ** us-east-1-atl-2a local-zone Capacity
38+ Block** , node-local NVMe via ` STORAGE=hostpath ` , ` GDRCOPY_DEV=on ` — see
39+ [ ` ../README.md ` ] ( ../README.md ) ). Image ` megatron-bridge-uccl:nemo-26.04.01-deepep-nvshmem-567632d-cu13 `
40+ (NVSHMEM v3.7.0-0, DeepEP ` 567632d ` , the
41+ [ ` deepep-benchmark ` ] ( ../../../../../micro-benchmarks/expert-parallelism/deepep-benchmark )
42+ NVSHMEM-over-libfabric/EFA build). Same bench entrypoint, layout, GBS, seq, mock
43+ data, iters, warmup. ** The NVSHMEM Δ is cross-campaign** (different cluster/date/
44+ storage) and carries noise the UCCL Δ does not — read it as indicative, not exact.
45+
46+ | cell | NCCL alltoall¹ | DeepEP+UCCL¹ | DeepEP+NVSHMEM² | UCCL Δ¹ | NVSHMEM Δ² |
47+ | ---| ---| ---| ---| ---| ---|
48+ | ** mb=4, overlap=on** | 5.793 s/iter · 144.8 TF/GPU · 181.0k tok/s | 3.834 s/iter · 219.1 TF/GPU · 273.5k tok/s | ** 3.742 s/iter · 224.6 TF/GPU · 280.2k tok/s** | ** −33.8%** | ~ −35.4% |
49+ | ** mb=4, overlap=off** | 9.303 s/iter · 90.1 TF/GPU | ** 6.041 s/iter · 138.8 TF/GPU** | 6.632 s/iter · 126.4 TF/GPU · 158.1k tok/s | ** −35.1%** | ~ −28.7% |
50+ | ** mb=1, overlap=off** | ** 12.88 s/iter · 65.1 TF/GPU** | 14.98 s/iter · 56.0 TF/GPU | 13.538 s/iter · 61.9 TF/GPU · 77.5k tok/s | +16.3% (NCCL faster) | ~ +5.1% (NCCL faster) |
51+ | mb=1, overlap=on | * not measured* (see coverage note) | * not measured* | * not measured* | — | — |
4152
4253** Takeaways**
4354
44- - ** At mb≥4 — the throughput-efficient operating point for both dispatchers — UCCL-EP
45- wins decisively on literal K2: −34/−35% iteration time in both overlap regimes.**
46- Overlap does not close the gap (it helps both arms; the relative win is unchanged).
47- - At ** mb=1 no-overlap** , NCCL all-to-all is ~ 16% faster — the same unamortized
55+ - ** At mb≥4 — the throughput-efficient operating point — both DeepEP transports win
56+ decisively on literal K2** : UCCL −34/−35% iteration time in both overlap regimes
57+ (same-campaign), NVSHMEM ~ −35/−29% (cross-campaign). At the tuned point
58+ (mb=4+overlap) NVSHMEM edges the June UCCL number by ~ 2.4% — within cross-campaign
59+ noise; read the two DeepEP transports as ** competitive** , not ranked.
60+ - At ** mb=1 no-overlap** , NCCL all-to-all stays fastest — the same unamortized
4861 per-dispatch-overhead regime seen on DSV3 (~ 12.6%) and on the K2 16-node run (~ 12%).
49- A tuned run operates at mb≥4, where UCCL wins .
62+ A tuned run operates at mb≥4, where the DeepEP arms win .
5063- The mb4+overlap UCCL advantage is ** larger at 32-node/PP8 (−34%) than at
5164 16-node/PP4 (−21%)** (table below). EP width is ** identical in both (EP32)** —
5265 the difference tracks the doubled node count (wider inter-node all-to-all span)
5366 and deeper pipeline, not expert-parallel width.
5467
68+ ** NVSHMEM-arm validity** — every cell: ` efa_ok ` on all 32 nodes, 0 stalls, ` n_steady `
69+ 20/20, ` parse-runs.py ` transport = ** nvshmem** (NVSHMEM-over-libfabric init banner on
70+ the 8 EP-group leader nodes; no UCCL proxy lines). Iteration-1 ` lm loss ` 13.43885 /
71+ 13.43207 / 13.43359 (mb4-ovlon / mb4-ovloff / mb1-ovloff); mb4-ovlon matches the June
72+ arms' 13.438850/13.438960 to 5 significant figures — same numerical work, bf16
73+ round-off apart. As designed, every NVSHMEM run wrote ` STATUS ` then exited 1 at
74+ NVSHMEM finalize (benign; see ` ../run-ab-rawpods.sh ` ).
75+
5576### Numerical (work) equivalence — confirmed
5677
5778Per-iteration ` lm loss ` curves (24 iters, ` log_interval=1 ` , last-PP-stage rank) for
@@ -81,45 +102,6 @@ The 32-node Capacity Block ended at 2026-06-04 ~10:51Z (EC2 reclaims instances
81102 remain the separate 2026-06-01 campaign in [ ` ../../dsv3/benchmarks/RESULTS.md ` ] ( ../../dsv3/benchmarks/RESULTS.md )
82103 (same image/layout, but not same-campaign apples-to-apples).
83104
84- ## DeepEP+NVSHMEM arm — 32-node PP8 (2026-07-14, ` 20260714T0515Z-k2-nvshmem-pp8-32n-ue1 ` )
85-
86- The third dispatcher arm (see the arm table in [ ` ../README.md ` ] ( ../README.md ) ): ** NVIDIA
87- DeepEP v1 over NVSHMEM-libfabric/EFA** — the
88- [ ` deepep-benchmark ` ] ( ../../../../../micro-benchmarks/expert-parallelism/deepep-benchmark ) build,
89- image ` megatron-bridge-uccl:nemo-26.04.01-deepep-nvshmem-567632d-cu13 ` (NVSHMEM v3.7.0-0,
90- DeepEP ` 567632d ` ). Same bench entrypoint, layout (TP8/PP8/EP32/DP4, 256× B300), GBS 256,
91- seq 4096, mock data, 24 iters, warmup 4 as the June campaign.
92-
93- > ** Cross-campaign comparison, not a same-campaign A/B.** The NVSHMEM cells ran on a
94- > * different cluster and date* than the June UCCL/NCCL numbers: EKS
95- > ` ml-clusters-shared-us-east-1 ` , 32× p6-b300 ** us-east-1-atl-2a local-zone Capacity
96- > Block** , node-local NVMe (` STORAGE=hostpath ` ) instead of FSx, ` GDRCOPY_DEV=on `
97- > (see [ ` ../README.md ` ] ( ../README.md ) ). Same image lineage, config, and EFA env, but
98- > deltas below carry cross-cluster noise the June UCCL-vs-NCCL deltas do not.
99-
100- | cell | DeepEP+NVSHMEM (2026-07-14) | UCCL deepep (2026-06-04) | NCCL alltoall (2026-06-04) |
101- | ---| ---| ---| ---|
102- | ** mb=4, overlap=on** | ** 3.742 s/iter · 224.6 TF/GPU · 280.2k tok/s** | 3.834 s/iter · 219.1 TF/GPU · 273.5k tok/s | 5.793 s/iter · 144.8 TF/GPU · 181.0k tok/s |
103- | ** mb=4, overlap=off** | 6.632 s/iter · 126.4 TF/GPU · 158.1k tok/s | ** 6.041 s/iter · 138.8 TF/GPU** | 9.303 s/iter · 90.1 TF/GPU |
104- | ** mb=1, overlap=off** | 13.538 s/iter · 61.9 TF/GPU · 77.5k tok/s | 14.98 s/iter · 56.0 TF/GPU | ** 12.88 s/iter · 65.1 TF/GPU** |
105-
106- ** What the numbers say** (with the cross-campaign caveat above):
107-
108- - ** NVSHMEM DeepEP is competitive with UCCL on literal K2 at the tuned operating point** :
109- at mb=4+overlap it edges the June UCCL number (−2.4% iter time) and lands −35.4% vs the
110- June NCCL baseline — the same "DeepEP-class dispatchers win decisively at mb≥4" picture.
111- - At mb=4 no-overlap NVSHMEM trails UCCL (+9.8%) while still beating NCCL by −28.7%.
112- - At mb=1 no-overlap the June ranking holds: NCCL fastest, but NVSHMEM (13.54 s) beats
113- UCCL's June 14.98 s in the unamortized regime.
114-
115- ** Validity** — every cell: ` efa_ok ` on all 32 nodes (` Selected provider is efa ` ), 0 stalls,
116- ` n_steady ` 20/20, ` parse-runs.py ` transport = ** nvshmem** (NVSHMEM-over-libfabric init
117- banner present on the 8 EP-group leader nodes; no UCCL proxy lines). Iteration-1 ` lm loss `
118- 13.43885 / 13.43207 / 13.43359 (mb4-ovlon / mb4-ovloff / mb1-ovloff); the mb4-ovlon value
119- matches the June arms' quoted 13.438850/13.438960 to 5 significant figures — same numerical
120- work, bf16 round-off apart. As designed, every NVSHMEM run wrote ` STATUS ` then exited 1 at
121- NVSHMEM finalize (benign; see ` ../run-ab-rawpods.sh ` ).
122-
123105## Appendix — 16-node PP4 (128× B300), campaign ` 20260604T051726Z-uccl-ab-pp4-16n-v2 `
124106
125107Parallelism ** TP8 / PP4 / EP32 / DP2** — * not directly comparable to the PP8 table*
0 commit comments