Skip to content

Commit e35fc38

Browse files
author
Han Wang
committed
fix(pt_expt): compiled training runs the graph lower (eager==compiled); drop force_legacy_descriptor
Retarget _CompiledModel to compile forward_common_lower_graph for graph-eligible descriptors (dpa1 attn_layer==0), gated by the same mixed_types()+uses_graph_lower() predicate the eager default-flip uses; se_e2_a/dpa2/dpa3 keep compiling the dense forward_lower. _trace_and_compile_graph builds a synthetic NeighborGraph with prime-distinct nf/N/E axes (no make_fx duck-shape merge) and edge_vec as the autograd leaf; _forward_graph builds the carry-all graph eagerly and unravels flat (N,*) node outputs to (nf,nloc,*). cpp.simdlen=0 for the graph compile avoids an inductor CPU scatter-vectorizer crash on the per-frame virial atomic_add. Also fixes an eager autograd bug in dpa1 call_graph: xp.asarray(type_embedding, device=dev) DETACHES under torch, so the type-embedding weights never trained in the graph path (grad None despite a real finite-diff dependency). make_fx traced through it, so compiled != eager and the optimizer diverged after step 0. Use type_embedding directly (mirrors the dense path); the tebd net now trains and eager==compiled to 1e-10 across the varying-natoms trajectory. Drops the force_legacy_descriptor workaround + uses_graph_lower monkeypatch.
1 parent ce2fd12 commit e35fc38

3 files changed

Lines changed: 554 additions & 40 deletions

File tree

deepmd/dpmodel/descriptor/dpa1.py

Lines changed: 10 additions & 3 deletions
Original file line numberDiff line numberDiff line change
@@ -757,9 +757,13 @@ def call_graph(
757757
)
758758
# FLAT node axis (N, ...): no (nf, nloc) reshape -- ragged-native, spec.
759759
if self.concat_output_tebd:
760-
tebd = xp.asarray(type_embedding, device=dev)
760+
# Use type_embedding directly (mirrors the dense path's
761+
# ``xp.take(type_embedding, ...)``): ``xp.asarray(..., device=dev)``
762+
# DETACHES under torch, silently severing the type-embedding weight
763+
# gradient so the tebd net never trains; type_embedding already lives
764+
# on the model device, so the device cast was redundant anyway.
761765
atype_local = xp.asarray(atype, device=dev)
762-
atype_embd = xp.take(tebd, atype_local, axis=0) # (N, tebd_dim)
766+
atype_embd = xp.take(type_embedding, atype_local, axis=0) # (N, tebd_dim)
763767
grrg = xp.concat([grrg, atype_embd], axis=-1)
764768
return grrg, rot_mat
765769

@@ -1523,7 +1527,10 @@ def call_graph(
15231527
ss = rr[:, 0:1] # (E, 1)
15241528
# neighbor / center type embeddings (concat mode); ghost type == owner type
15251529
# so gathering by the LOCAL owner (src) reproduces the dense neighbor tebd.
1526-
tebd = xp.asarray(type_embedding, device=dev)
1530+
# NB: do NOT wrap in ``xp.asarray(..., device=dev)`` -- that DETACHES under
1531+
# torch and severs the type-embedding weight gradient (the tebd net would
1532+
# never train); type_embedding already lives on the model device.
1533+
tebd = type_embedding
15271534
atype_embd_nlist = xp.take(tebd, nei_type, axis=0) # (E, tebd_dim)
15281535
if not self.type_one_side:
15291536
atype_embd_nnei = xp.take(tebd, center_type, axis=0) # (E, tebd_dim)

0 commit comments

Comments
 (0)