Conversation
AutoEP built a DeepEP ElasticBuffer per MoE layer and kept every one of them for the life of the job. Each buffer reserves fabric resources that are not reported as device memory and that are exhaustible: on 32 H100s across four nodes, an isolated probe builds 27 buffers per rank and then fails the twenty-eighth inside ncclDevCommCreate with DOCA failure 21, on ten of the thirty-two ranks. A 48-layer model therefore could not start at all there, while the same probe builds 48 on 16 H100s. They are also slow: about 15 seconds each on 16 H100s and 22 on 32, so a 48-layer job spent over ten minutes building buffers it did not need. Every layer of a model passes the same EP group, expert count, top-k, hidden size, capacity, SM budget and queue-pair margin, and the buffer carries no per-layer state: dispatch allocates its output rather than returning a view into the buffer, the handle each layer replays in backward is held by its autograd node, and layers run one at a time. So hand them one buffer, keyed by those arguments and reference counted so an engine's teardown releases only its own layers' claims. Signed-off-by: Han Yu <hanyu@linkedin.com> Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> Signed-off-by: yh0903 <helloyu0903@gmail.com>
Codex Review SummaryThis comment shows the latest Codex review activity on this pull request.
ℹ️ About Codex in GitHubYour team has set up Codex to review pull requests in this repo. Reviews are triggered when you
Codex reacts with 👀 while any review is running, comments if it has suggestions, and reacts with 👍 once all reviews finish with no findings. |
![chatgpt-codex-connector[bot]](https://avatars.githubusercontent.com/in/1144995?s=60&v=4){kind=small}
There was a problem hiding this comment.
💡 Codex Review
Here are some automated review suggestions for this pull request.
Reviewed commit: 958813523e
ℹ️ About Codex in GitHub
Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you
- Open a pull request for review
- Mark a draft as ready
- Comment "@codex review".
If Codex has suggestions, it will comment; otherwise it will react with 👍.
Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".
Review asked whether sharing a buffer between two engines on one expert-parallel group lets their dispatches cross-match if the ranks schedule the engines in different orders. It does, but that schedule could not work before this change either: buffer construction is lazy, inside forward, and collective on the group, so one rank's construction would already have been matched against another's. Say so where the sharing happens. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> Signed-off-by: yh0903 <helloyu0903@gmail.com>
Review asked whether two engines on one expert-parallel group should share a buffer when their geometry matches. They should not. Layers of one model reach their dispatches in an order every rank agrees on because they are one forward pass; two models are driven independently, and a reinforcement-learning loop with a trainable actor and a frozen reference model is exactly that. A shared DeepEP communication context between them would make an order nothing enforces start to matter. So key the registry on a sharing scope as well as the geometry. AutoEP mints one scope per model it converts, and a layer built outside that path gets a scope of its own, which is what it had before sharing existed. The cost of the boundary is one more buffer per model, against a ceiling of 27 measured on 32 H100s, so it does not touch the reason for sharing in the first place. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> Signed-off-by: yh0903 <helloyu0903@gmail.com>
TestingUnit tests are in The GPU parity suite (
The extra pass is The two failures are the same two on all three revisions, including the one EP32 itself cannot be regression-tested from CI: the failure this fixes |
1 participant
AutoEP's DeepEP path builds one
ElasticBufferper MoE layer and neverreleases it, so a model's buffer count grows with its depth. On 16 H100s
that is merely wasteful. On 32 it is fatal: the twenty-eighth buffer
cannot be created, and a 48-layer model can no longer start at all.
This makes the buffers shared. Within one model, layers whose geometry
matches -- same expert-parallel group, hidden size, dtype, token
capacity and expert count -- now get the same buffer, reference-counted
so teardown still destroys it exactly once.
Sharing stops at the model. Two models converted separately keep their
own buffers even on one group with identical geometry, because they are
driven independently: an actor and a frozen reference model in an RL
loop need not reach their MoE layers in any fixed order relative to each
other, and a shared DeepEP communication context would make that order
matter. That costs one extra buffer per model against a ceiling of 27,
so it does not touch the reason for sharing.
Why it is the buffer count and not the depth
An isolated probe that does nothing but construct buffers, with no model
and no MoE layer, reproduces it:
The wall is at 27 per rank regardless of how many are asked for, and it
is not reached at EP16. The set of ranks that report the failure differs
between runs, which is what a per-rank resource ceiling looks like rather
than a deterministic bug.
The real 48-layer EP32 run fails the same way. Its logs show the buffer
construction warning at least 28 times, one every 25-30s over ten
minutes, before dying in
ncclDevCommCreate. The first buffer is fine;the twenty-eighth is not.
Construction is also not free: roughly 15s per buffer at EP16 and 22s at
EP32, so a 48-layer model spends a quarter-hour building buffers that are
all configured identically.
Why sharing is safe
The layers do not hold state in the buffer across each other. AutoEP runs
them strictly in order and does not use
async_with_compute_stream. Thedispatch output is a fresh tensor from
torch::empty, not a view intothe symmetric buffer, so a later layer cannot overwrite an earlier one's
result. The handles that must survive into backward are held by the
autograd nodes in
_DeepEPDispatchand_DeepEPCombine;last_handleis only a convenience read immediately after dispatch, and is now
documented as such.
A model whose layers genuinely differ -- different dtype, hidden size or
capacity -- still gets a buffer per distinct geometry, because the
registry is keyed on all of them, and on the model.
What this does not claim
No throughput or memory number. The buffers are not allocated through
the torch caching allocator, and a memory probe measured 0.0 GiB of
torch-visible memory for them, so this should not be read as a fix for
any previously observed memory difference. The claim is only that EP32
could not run and now can, and that 47 redundant constructions are gone.