Commit ced0016
perf(dpa4): batch the SO3/grid contractions over (D,F); keep use_amp through pt_expt assembly (deepmodeling#5960)
Users reported that compiled DPA4 training runs ~2x slower on `pt_expt`
than on `pt`. This PR is the result of chasing that: a performance bug
in how the SO3 contractions were lowered, and a correctness bug where a
configured `use_amp: false` was lost while `pt_expt` assembled the
model.
## Changes
**1. Weight broadcast across the node axis (`so3.py`, `lora.py`,
`grid_net.py`)**
`matmul(x[..., None, :], weight[None, ...])` makes the node count `N`
the matmul BATCH, so matmul broadcasts the weight to `(N, D, F, Cin,
Cout)` and autograd then reduces that whole expanded gradient
(`ExpandBackward0`) back to the parameter shape. At the water example's
sizes a 165 K-element weight expanded to 191 M elements (~0.8 GB) per
call, and the reduce was the single costliest kernel of a training step
(45.6 ms, 3x per step).
The fix batches the contraction over the small `(D, F)` axes so `N`
stays the GEMM ROW dimension and the weight is used in place.
Micro-benchmark, fwd+bwd at the real shapes: **16.48 ms -> 1.09 ms
(15x)**.
Two lookalike sites in `projection.py` are deliberately NOT changed:
their operands are `requires_grad=False` buffers, so no backward reduce
exists. Verified rather than assumed.
**2. The same lowering for the frame mixers (`_degree_batched_matmul`)**
Review found the `FrameContract` / `FrameExpand` mixers still on the
broadcast spelling. They now share one helper, `_degree_batched_matmul`,
written identically on the dpmodel side
(`dpmodel/descriptor/dpa4_nn/grid_net.py`) and the pt side
(`pt/model/descriptor/sezm_nn/grid_net.py`). Because it does no reshape,
an empty node axis (`N == 0`) flows through unchanged instead of hitting
a reshape error — pinned by a test.
**3. `use_amp` was lost during `pt_expt` model assembly (correctness)**
`use_amp` is a training-runtime policy, not model state, so it stays OUT
of the portable serialization record (this is the deepmodeling#5963 position, and
the jax deserializer actively rejects records carrying `use_amp: true`).
The bug was elsewhere: `pt_expt` assembled its model by converting an
already-populated dpmodel instance, and that conversion round-trips the
component through `deserialize(serialize())`. Anything that is
deliberately not in the portable record — `use_amp` among it — was
therefore dropped, and training silently ran under bfloat16 autocast
even when the input configured `use_amp: false`.
The fix is at the assembly boundary, not in the record: `pt_expt` now
constructs the wrapped class directly (`auto_wrapped_class(...)` in
`make_model.py`, `get_model.py`, and the bridging composition path), so
a live constructor-supplied component keeps its runtime state. The rule
is stated once, in the `auto_wrapped_class` docstring; the call sites
reference it.
An earlier revision of this PR instead added `use_amp` to `serialize()`.
That was reverted in review — it put a runtime knob into the portable
record and would have broken the jax contract.
**Also removed in review: an `enable_tf32` / `DP_TF32_INFER`
implementation for `pt_expt`.** It contributes nothing to the speedup
measured below (the benchmark card has no TF32 silicon), and deepmodeling#5958 owns
the `pt_expt` training-runtime alignment — including the documented
position that `pt_expt` always runs at `"highest"` matmul precision.
`pt_expt` therefore keeps master's warn-and-ignore behavior for
`enable_tf32`.
## Benchmark
DPA4 water example (`examples/water/dpa4`), one Tesla T4, torch 2.11,
fp32 (`use_amp: false`), batch size 6. Steady-state seconds per training
step, obtained by differencing the wall time of a 33-step and a 3-step
run of the same config, which cancels every one-time cost (import, data
load, statistics, `torch.compile` / make_fx lowering). All five arms
were measured in one session on the same machine; run-to-run variation
is about 2-3%.
**Provenance: measured at `ae720432b`**, the head at which this PR was
opened — i.e. BEFORE the review changes (change 2, the frame-mixer
lowering, and change 3's move from `serialize()` to the assembly
boundary). Change 1, which is where the entire speedup comes from, is
unmodified since. The numbers have not been re-measured on the current
head; a re-run is pending and I will post it rather than silently reuse
these.
| training mode | `pt` (reference) | `pt_expt` at master | `pt_expt` at
`ae720432b` | speedup vs master |
|---|---|---|---|---|
| eager | 0.891 s/step | 1.555 s/step | **0.921 s/step** | **1.69x** |
| compiled | 0.545 s/step | 1.611 s/step | **0.535 s/step** | **3.01x**
|
This reproduces the reported issue at master — `pt_expt` compiled was
3.0x slower than `pt` compiled, and even slower than its own eager path,
because the broadcast-weight contraction lowers to worse code under
inductor than under eager cuBLAS. After the fix `pt_expt` is at parity
with `pt`: eager within 3.4%, compiled within measurement noise.
## Known limitations
- **Backward numerics are covered for the frame mixers, not for the SO3
/ LoRA contractions.** `test_dpa4_frame_mixers.py` compares
`_degree_batched_matmul`'s weight gradient against the pt module's at
rtol/atol 1e-12. The rewritten SO3 and LoRA contractions are pinned on
the forward against an explicit `einsum` reference (rtol/atol 1e-12,
numpy and torch namespaces); their backward is still exercised only by
tracing, not compared by value.
- **The `pt` / `pt_expt` TF32 policy gap remains open.** On Ampere+
cards `pt` runs training matmuls under TF32 (`enable_tf32`, default
`True`) while `pt_expt` ignores the key with a warning; the two backends
are not speed-comparable there. Deferred to the deepmodeling#5958 training-runtime
series.
- **The residual compiled gap vs `pt` is not stable across sessions.**
An earlier session measured `pt_expt` compiled 10.8% slower than `pt`
compiled; the benchmark above measured it 1.7% faster. Both are within a
couple of run-to-run standard deviations, so I treat compiled as at
parity and the earlier gap as unconfirmed.
- The history contains churn at the GridBranch router (`7518a417c` ->
`01c58e665` -> `75459610a` -> `504bb2430` -> `157444204`): a matmul
spelling introduced, reverted, reintroduced, and finally restored to
master's line. The site is byte-identical to master in the final diff.
The degenerate GEMM that profiling found there existed only on this
branch, so it is not a fix — I have left the commits rather than
rewriting pushed history, and would squash them on request.
- Unrelated but found while benchmarking: **torch >= 2.11 ships no Volta
(CC 7.0) kernels**, and compiled training requires >= 2.11 via
`check_compile_torch_version`. Compiled DPA4 training is therefore
impossible on V100 with official wheels; T4 (CC 7.5) is the oldest card
that works.
## Tests
- `source/tests/common/dpmodel/test_dpa4_frame_mixers.py` —
`_degree_batched_matmul` vs the pt module: forward parity, the `N == 0`
contract, and weight-gradient parity.
- `source/tests/common/dpmodel/test_dpa4_lora.py` — new
`test_lora_so3_call_matches_einsum_contract`: `LoRASO3.call` against the
explicit `einsum("ndfi,difo->ndfo")` reference with a nonzero adapter,
on both the numpy and torch namespaces, for `n_focus` 1 and 2.
- `source/tests/pt_expt/model/test_get_model_dpa4.py` — `use_amp`
survives model assembly (both branches), for the plain and the
bridged/composed construction paths.
- `source/tests/common/dpmodel/test_descrpt_dpa4.py` — `use_amp` is
absent from the portable serialization record and defaults on
deserialize.
- Existing `test_grid_branch[1]`/`[2]` cover the changed SO3 contraction
against the pt implementation at rtol 1e-12.
- Run locally: 434 passed / 10 skipped across the dpa4 dpmodel, pt_expt
and cross-backend parity suites, plus the pt_expt model suite.
CUDA-gated precision-context cases were run on a T4 (29/29).
## Test status caveat — resolved
An earlier revision of this description flagged two locally failing
`pt_expt` AOTI-freeze tests
(`test_zbl_bridging.py::test_native_spin_with_bridging_graph_freeze_and_deep_eval`,
`test_dpa4_zbl_parallel.py::TestBridgedSpinGraphSelfComm::test_freeze_embeds_with_comm_artifact`)
as unadjudicated. They are now adjudicated as **pre-existing and
environmental, not caused by this branch**: a clean `upstream/master`
worktree on the same machine fails both with the identical
`InductorError: assert isinstance(index, CppCSEVariable) and
index.is_vec` (torch 2.11 CPU-SIMD codegen bug on an `atomic_add`
scatter buffer), and both tests pass on this branch with the known
workaround `torch._inductor.config.cpp.simdlen = 1` (2 passed). The same
bug is already documented in `source/tests/infer/gen_dpa4.py` /
`gen_dpa2.py`.
---------
Co-authored-by: Han Wang <wang_han@iapcm.ac.cn>
Co-authored-by: pre-commit-ci[bot] <66853113+pre-commit-ci[bot]@users.noreply.github.com>1 parent ed691aa commit ced0016
12 files changed
Lines changed: 438 additions & 34 deletions
File tree
- deepmd
- dpmodel
- descriptor/dpa4_nn
- model
- pt_expt
- model
- pt/model/descriptor/sezm_nn
- source/tests
- common/dpmodel
- pt_expt/model
| Original file line number | Diff line number | Diff line change | |
|---|---|---|---|
| |||
99 | 99 | | |
100 | 100 | | |
101 | 101 | | |
| 102 | + | |
| 103 | + | |
| 104 | + | |
| 105 | + | |
| 106 | + | |
| 107 | + | |
| 108 | + | |
| 109 | + | |
| 110 | + | |
| 111 | + | |
| 112 | + | |
| 113 | + | |
| 114 | + | |
| 115 | + | |
| 116 | + | |
| 117 | + | |
| 118 | + | |
| 119 | + | |
| 120 | + | |
| 121 | + | |
| 122 | + | |
| 123 | + | |
| 124 | + | |
| 125 | + | |
| 126 | + | |
| 127 | + | |
| 128 | + | |
| 129 | + | |
| 130 | + | |
| 131 | + | |
| 132 | + | |
102 | 133 | | |
103 | 134 | | |
104 | 135 | | |
| |||
493 | 524 | | |
494 | 525 | | |
495 | 526 | | |
496 | | - | |
497 | | - | |
498 | | - | |
| 527 | + | |
| 528 | + | |
499 | 529 | | |
500 | 530 | | |
501 | 531 | | |
| |||
575 | 605 | | |
576 | 606 | | |
577 | 607 | | |
578 | | - | |
579 | | - | |
580 | | - | |
| 608 | + | |
| 609 | + | |
581 | 610 | | |
582 | 611 | | |
583 | 612 | | |
| |||
| Original file line number | Diff line number | Diff line change | |
|---|---|---|---|
| |||
189 | 189 | | |
190 | 190 | | |
191 | 191 | | |
192 | | - | |
193 | | - | |
| 192 | + | |
| 193 | + | |
| 194 | + | |
194 | 195 | | |
195 | | - | |
| 196 | + | |
| 197 | + | |
196 | 198 | | |
197 | 199 | | |
198 | 200 | | |
| |||
| Original file line number | Diff line number | Diff line change | |
|---|---|---|---|
| |||
131 | 131 | | |
132 | 132 | | |
133 | 133 | | |
134 | | - | |
135 | | - | |
| 134 | + | |
| 135 | + | |
| 136 | + | |
| 137 | + | |
136 | 138 | | |
137 | | - | |
| 139 | + | |
| 140 | + | |
138 | 141 | | |
139 | 142 | | |
140 | 143 | | |
| |||
439 | 442 | | |
440 | 443 | | |
441 | 444 | | |
442 | | - | |
443 | | - | |
| 445 | + | |
| 446 | + | |
| 447 | + | |
444 | 448 | | |
445 | 449 | | |
446 | 450 | | |
447 | | - | |
| 451 | + | |
| 452 | + | |
448 | 453 | | |
449 | 454 | | |
450 | 455 | | |
| |||
| Original file line number | Diff line number | Diff line change | |
|---|---|---|---|
| |||
135 | 135 | | |
136 | 136 | | |
137 | 137 | | |
| 138 | + | |
138 | 139 | | |
139 | 140 | | |
140 | 141 | | |
| |||
162 | 163 | | |
163 | 164 | | |
164 | 165 | | |
| 166 | + | |
| 167 | + | |
| 168 | + | |
| 169 | + | |
| 170 | + | |
| 171 | + | |
| 172 | + | |
165 | 173 | | |
166 | 174 | | |
167 | 175 | | |
| |||
183 | 191 | | |
184 | 192 | | |
185 | 193 | | |
186 | | - | |
| 194 | + | |
187 | 195 | | |
188 | 196 | | |
| 197 | + | |
| 198 | + | |
189 | 199 | | |
190 | 200 | | |
191 | 201 | | |
| |||
433 | 443 | | |
434 | 444 | | |
435 | 445 | | |
| 446 | + | |
436 | 447 | | |
437 | 448 | | |
438 | 449 | | |
| |||
442 | 453 | | |
443 | 454 | | |
444 | 455 | | |
| 456 | + | |
445 | 457 | | |
446 | 458 | | |
447 | 459 | | |
| |||
478 | 490 | | |
479 | 491 | | |
480 | 492 | | |
| 493 | + | |
481 | 494 | | |
482 | 495 | | |
483 | 496 | | |
| |||
| Original file line number | Diff line number | Diff line change | |
|---|---|---|---|
| |||
343 | 343 | | |
344 | 344 | | |
345 | 345 | | |
| 346 | + | |
| 347 | + | |
| 348 | + | |
| 349 | + | |
| 350 | + | |
| 351 | + | |
| 352 | + | |
| 353 | + | |
| 354 | + | |
| 355 | + | |
| 356 | + | |
| 357 | + | |
| 358 | + | |
| 359 | + | |
346 | 360 | | |
347 | 361 | | |
348 | 362 | | |
| |||
387 | 401 | | |
388 | 402 | | |
389 | 403 | | |
390 | | - | |
| 404 | + | |
391 | 405 | | |
392 | 406 | | |
393 | 407 | | |
| |||
434 | 448 | | |
435 | 449 | | |
436 | 450 | | |
437 | | - | |
| 451 | + | |
438 | 452 | | |
439 | 453 | | |
440 | 454 | | |
| |||
| Original file line number | Diff line number | Diff line change | |
|---|---|---|---|
| |||
136 | 136 | | |
137 | 137 | | |
138 | 138 | | |
139 | | - | |
140 | | - | |
| 139 | + | |
| 140 | + | |
141 | 141 | | |
142 | 142 | | |
143 | 143 | | |
144 | 144 | | |
145 | 145 | | |
| 146 | + | |
| 147 | + | |
| 148 | + | |
| 149 | + | |
| 150 | + | |
| 151 | + | |
146 | 152 | | |
147 | 153 | | |
148 | | - | |
149 | | - | |
| 154 | + | |
| 155 | + | |
150 | 156 | | |
151 | 157 | | |
152 | 158 | | |
153 | | - | |
154 | | - | |
| 159 | + | |
| 160 | + | |
155 | 161 | | |
156 | | - | |
157 | 162 | | |
158 | 163 | | |
159 | 164 | | |
160 | 165 | | |
161 | 166 | | |
162 | 167 | | |
163 | 168 | | |
164 | | - | |
| 169 | + | |
| 170 | + | |
| 171 | + | |
| 172 | + | |
| 173 | + | |
| 174 | + | |
| 175 | + | |
| 176 | + | |
| 177 | + | |
| 178 | + | |
| 179 | + | |
| 180 | + | |
| 181 | + | |
| 182 | + | |
| 183 | + | |
| 184 | + | |
| 185 | + | |
| 186 | + | |
165 | 187 | | |
166 | 188 | | |
167 | 189 | | |
| |||
| Original file line number | Diff line number | Diff line change | |
|---|---|---|---|
| |||
9 | 9 | | |
10 | 10 | | |
11 | 11 | | |
12 | | - | |
13 | | - | |
| 12 | + | |
| 13 | + | |
| 14 | + | |
14 | 15 | | |
15 | 16 | | |
16 | | - | |
| 17 | + | |
17 | 18 | | |
18 | 19 | | |
19 | 20 | | |
20 | 21 | | |
21 | 22 | | |
22 | 23 | | |
23 | 24 | | |
| 25 | + | |
| 26 | + | |
| 27 | + | |
24 | 28 | | |
25 | 29 | | |
26 | 30 | | |
| |||
56 | 60 | | |
57 | 61 | | |
58 | 62 | | |
| 63 | + | |
| 64 | + | |
| 65 | + | |
| 66 | + | |
| 67 | + | |
| 68 | + | |
59 | 69 | | |
60 | 70 | | |
61 | 71 | | |
| |||
64 | 74 | | |
65 | 75 | | |
66 | 76 | | |
| 77 | + | |
67 | 78 | | |
68 | 79 | | |
69 | 80 | | |
| |||
| Original file line number | Diff line number | Diff line change | |
|---|---|---|---|
| |||
28 | 28 | | |
29 | 29 | | |
30 | 30 | | |
| 31 | + | |
31 | 32 | | |
32 | 33 | | |
33 | 34 | | |
| |||
465 | 466 | | |
466 | 467 | | |
467 | 468 | | |
468 | | - | |
| 469 | + | |
| 470 | + | |
| 471 | + | |
469 | 472 | | |
470 | 473 | | |
471 | 474 | | |
| |||
| Original file line number | Diff line number | Diff line change | |
|---|---|---|---|
| |||
265 | 265 | | |
266 | 266 | | |
267 | 267 | | |
| 268 | + | |
| 269 | + | |
| 270 | + | |
| 271 | + | |
| 272 | + | |
| 273 | + | |
| 274 | + | |
| 275 | + | |
| 276 | + | |
| 277 | + | |
| 278 | + | |
| 279 | + | |
| 280 | + | |
268 | 281 | | |
269 | 282 | | |
270 | 283 | | |
| |||
0 commit comments