[codex] Add Empire AI NVL72 training launcher

[jder]

Add an example Empire AI Beta NVL72 Slurm launcher for half-degree OM4 training. Updates to latest physics nemo base image.

Drive by: default W&B runs to the ocean_emulators entity instead of the stale samudra entity.

[jder]

@fomo-bot can you rebase on the latest main and remove the files mentioned below

[jder]

Please remove this file.

[jder]

Please remove this file.

[jder]

Please remove this file.

[fomo-bot]

Rebased PR #777 onto latest origin/main and force-with-lease pushed the PR branch.

Removed the three requested files from the PR diff:

• docs/empireai_nvl72_nccl_validation_failures.md
• scripts/empireai_nccl_smoke.sbatch
• scripts/empireai_validation_collective_stress.sbatch

New branch head: 90d56b2c (codex/empireai-nvl72-launcher).

Validation run:

• git diff --check origin/main...HEAD
• bash -n scripts/empireai_nvl72_train.sbatch scripts/slurm_apptainer_eval.sbatch scripts/slurm_apptainer_train.sbatch scripts/container/build_physicsnemo_26_05.sh scripts/container/run_cuda_tests_in_image.sh scripts/container/scrub_arm64_gpu_container_tests.sh

All passed. The review threads are now outdated on GitHub; I did not manually resolve them.

Debug info

• Repository: m2lines/Samudra
• Trigger: [codex] Add Empire AI NVL72 training launcher #777 (review)
• Request ID: 869a4eef-14be-4294-8ff9-e8a975c9c22b
• Accepted: true

[chatgpt-codex-connector]

💡 Codex Review

Here are some automated review suggestions for this pull request.

Reviewed commit: 448c47c8f6

ℹ️ About Codex in GitHub

Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you

• Open a pull request for review
• Mark a draft as ready
• Comment "@codex review".

If Codex has suggestions, it will comment; otherwise it will react with 👍.

Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".

[chatgpt-codex-connector]

💡 Codex Review

Here are some automated review suggestions for this pull request.

Reviewed commit: 59c0159faf

ℹ️ About Codex in GitHub

Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you

• Open a pull request for review
• Mark a draft as ready
• Comment "@codex review".

If Codex has suggestions, it will comment; otherwise it will react with 👍.

Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".

[chatgpt-codex-connector]

Exclude the cu13 nvshmem wheel on ARM builds

For aarch64 26.05 builds, the lockfile dependency is nvidia-nvshmem-cu13 from torch's ARM CUDA stack, not nvidia-nvshmem; uv sync --no-install-package skips the given package name exactly (checked uv sync --help). As written, the ARM64 container build still installs the 60MB PyPI nvidia-nvshmem-cu13 wheel into the venv instead of reusing PhysicsNeMo's image-provided CUDA libraries, which can shadow the base stack and defeats this Dockerfile's CUDA-package exclusions.

Useful? React with 👍 / 👎.

[jder]

@fomo-bot can you do this and review the lockfile for other missing packages please

[fomo-bot]

Pushed commit 67e62088 to PR #777.

Changed containers/Dockerfile.physicsnemo-26.05 to exclude the exact ARM lockfile package name:

nvidia-nvshmem-cu13

I also reviewed the lockfile against the Dockerfile’s --no-install-package list. All torch CUDA nvidia-* dependencies are now covered. The only unexcluded lockfile nvidia-* package is nvidia-ml-py, which I left alone because it is not a torch CUDA runtime wheel.

Checks passed:

• structured lockfile vs Dockerfile exclusion comparison
• uv lock --check
• git diff --check

The GitHub review thread is now outdated; I did not manually resolve it.

Debug info

• Repository: m2lines/Samudra
• Trigger: [codex] Add Empire AI NVL72 training launcher #777 (comment)
• Request ID: 08d47a0c-5f4f-4413-82c1-e839e70608c8
• Accepted: true