Skip PhysX replicator for fully-heterogeneous 1:1 layouts (fixes mGPU double-free)#6559
Skip PhysX replicator for fully-heterogeneous 1:1 layouts (fixes mGPU double-free)#6559ooctipus wants to merge 2 commits into
Conversation
Since the replication-session refactor, InteractiveSceneCfg.replicate_physics was ignored: scenes configured with replicate_physics=False invoked native physics replication anyway, silently discarding per-environment USD differences (identified in isaac-sim#6541). The flag now lives where the policy executes: - CloneCfg.replicate_physics is the policy's home; InteractiveScene pipes its cfg flag into it and forwards it through ReplicateSession to replicate(). - REPLICATION_QUEUE holds bare asset cfgs: construction only registers which cfgs participate (required by ClonePlan.from_env_0 in direct workflows and to distinguish clonable assets from lights/terrain). - replicate() resolves how each cfg clones at dispatch: the cfg's cloning_contexts field when set, otherwise the active backend's default stack exported as isaaclab_<backend>.cloner.REPLICATION. PhysX pairs physics replication with USD clones (collision groups are authored on the per-env prims; PhysX has no kitless mode), Newton includes USD only under Kit, and OvPhysX replicates alone since its clone replay authors USD. - With replicate_physics=False, cloning is USD-only: the physics engine parses the per-env USD prims directly; an asset whose contexts are all physics-based is simply not cloned. Asset implementations know nothing about cloning beyond registering their cfg; per-asset overrides are a cfg assignment. The per-backend queue_*_replication helpers are removed in favor of this single path.
For scenes where every source maps only to its own environment (one variant per env, fully-heterogeneous 1:1), calling rep.replicate() once per source with a single self-target intermittently triggers native heap corruption (double-free / SIGABRT) under mGPU. Detect this layout by checking that all queued rows are self-only and skip register_replicator entirely. PhysX then parses the source prims directly from the stage, which is both correct and safe. Reproducer: mgpu_repro.zip -- heterogeneous mGPU with 192 variants and 4 GPUs, replicate_physics=True.
github-actions
Bot
added
bug
|
Folding this into #6550 directly. |
Greptile SummaryThis PR fixes an intermittent SIGABRT (
Confidence Score: 4/5The mGPU crash fix is targeted and empirically verified; the refactoring is broad but mechanical. The two remaining observations are documentation gaps, not correctness defects. The self-only detection logic is sound: it correctly identifies all-self-only layouts, clears the queue before any early return, and relies on PhysX parsing the stage directly. Two documentation gaps exist: the
Important Files Changed
Flowchart%%{init: {'theme': 'neutral'}}%%
flowchart TD
A[Asset constructor called] --> B[AssetBase.__init__]
B --> C[queue_replication cfg]
C --> D[REPLICATION_QUEUE stores cfg]
E[ReplicateSession.__exit__] --> F[replicate plan stage replicate_physics]
F --> G{For each cfg in queue}
G --> H{cfg.cloning_contexts set?}
H -- Yes --> I[Use cfg.cloning_contexts]
H -- No --> J[Load backend REPLICATION tuple]
I --> K{replicate_physics=False?}
J --> K
K -- Yes --> L[Filter out non-USD contexts]
K -- No --> M[Keep all contexts]
L --> N[backend_rows union]
M --> N
N --> O[For each BackendCtxCls call queue_mapping]
O --> P[PhysxReplicateContext.replicate]
P --> Q{Queue empty?}
Q -- Yes --> R[Return]
Q -- No --> S[Copy queue then clear]
S --> T{ALL entries self-only?}
T -- Yes --> U[Return: PhysX parses stage directly - mGPU crash avoided]
T -- No --> V[register_replicator + rep.replicate per source]
%%{init: {'theme': 'base', 'themeVariables': {"darkMode": true, "background": "#0d1117", "primaryColor": "#21262d", "primaryTextColor": "#e6edf3", "primaryBorderColor": "#8b949e", "lineColor": "#8b949e", "textColor": "#e6edf3", "edgeLabelBackground": "#161b22", "actorBkg": "#21262d", "actorBorder": "#8b949e", "actorTextColor": "#e6edf3", "actorLineColor": "#8b949e", "signalColor": "#8b949e", "signalTextColor": "#e6edf3", "noteBkgColor": "#373320", "noteBorderColor": "#d4a72c", "noteTextColor": "#f0e6c0", "labelBoxBkgColor": "#21262d", "labelBoxBorderColor": "#8b949e", "labelTextColor": "#e6edf3", "loopTextColor": "#e6edf3", "activationBkgColor": "#30363d", "activationBorderColor": "#8b949e"}}}%%
flowchart TD
A[Asset constructor called] --> B[AssetBase.__init__]
B --> C[queue_replication cfg]
C --> D[REPLICATION_QUEUE stores cfg]
E[ReplicateSession.__exit__] --> F[replicate plan stage replicate_physics]
F --> G{For each cfg in queue}
G --> H{cfg.cloning_contexts set?}
H -- Yes --> I[Use cfg.cloning_contexts]
H -- No --> J[Load backend REPLICATION tuple]
I --> K{replicate_physics=False?}
J --> K
K -- Yes --> L[Filter out non-USD contexts]
K -- No --> M[Keep all contexts]
L --> N[backend_rows union]
M --> N
N --> O[For each BackendCtxCls call queue_mapping]
O --> P[PhysxReplicateContext.replicate]
P --> Q{Queue empty?}
Q -- Yes --> R[Return]
Q -- No --> S[Copy queue then clear]
S --> T{ALL entries self-only?}
T -- Yes --> U[Return: PhysX parses stage directly - mGPU crash avoided]
T -- No --> V[register_replicator + rep.replicate per source]
|
1 participant
Summary
Fixes intermittent
double free or corruption(SIGABRT) inPhysxReplicateContext.replicate()when running fully-heterogeneous scenes (one variant per env) across multiple GPUs. Stacks onto #6550.Root cause: For a fully-heterogeneous 1:1 layout (N variants → N envs, each source mapped only to its own env),
replicate()callsrep.replicate()N times — once per source with a single self-target each. This causes intermittent native heap corruption under mGPU. The source prims are already in their correct env positions; no cross-env replication is needed. Each call has PhysX-internal per-call allocations that apparently sum to a problematic total across concurrent processes.Fix: Check whether all queued rows are self-only (
target_envs == (own_env_id,)). When true, skipregister_replicatorentirely — PhysX parses the source prims directly from the stage, which is correct and avoids the crash.Controls (from
mgpu_repro.zipin #6550):Failure signature (before fix)
🤖 Generated with Claude Code