Support for Singularity / Apptainer in addition to Docker / Podman

[kkaytekin]

Hi team,

First of all, thank you for your hard work in maintaining this repository!

While reviewing the run_local_pretrain.sh script, I noticed that container support is currently limited to Docker and Podman. I was wondering if there's any interest or plan to add support for Singularity / Apptainer, which is often preferred in HPC environments.

It would be great to hear your thoughts on this. I'd be happy to help test or contribute if there's interest in exploring this further.

Best regards,

[wenxie-amd]

Hi @kkaytekin , thank you for your suggestions! As we don't have the experience using Singularity, and have a few questions:

1. Do you use Singularity to manage images when training large models? What are its advantages?
2. Is Singularity largely consistent with Docker in terms of usage, especially commands?

If possible, it would be great that you add Singularity support in this script and submit the code as a PR? We can review it together.

[kkaytekin]

Yes, Singularity is often used in HPC settings for its security and non-root execution. It can pull images directly from Docker Hub, but its commands differ from Docker, especially for building and running containers.

I was able to get the rocm/megatron-lm:v25.7_py310 image running with both Singularity and Apptainer. However, I have not yet been able to fully verify its reliability with Primus on the HPC system I use.

Looking at the training script and the ROCm tutorial, I noticed several InfiniBand-related environment configurations. Our system uses MI300A with Slingshot, and I am still quite new to the AMD ecosystem. I’m not sure whether the issues I encounter are due to incorrect container usage or necessary adjustments within the Primus scripts themselves.

At this point, I’d appreciate any suggestions or guidance.

Once I can verify reliable operation on both single-node and multi-node setups on our HPC system, I’d be happy to contribute Singularity support via a PR.