Why does Tensor-parallel Communication Overlap require MPI?

[mjkpolo]

Hello,
I was trying to replicate the MLCommons llama 2 70b training with h200s, but the cluster I am using doesn't support MPI. I got it working by setting

export TP_COMM_OVERLAP=0

and using a smaller model, because I noticed in the NeMo code:

NeMo/nemo/lightning/_strategy_lib.py

Line 92 in dc08edd

init_mpi_proc_group=getattr(parallel_config, "tp_comm_overlap", False)

I am not very familiar with training or HPC applications, but why does this feature require MPI, and cannot use NCCL? I read this blog post to try and understand what tensor parallel communication overlap is, but I can't figure out why I need MPI for it.

Thanks!

[ashors1]

Hi, MPI is used by default to bootstrap the user buffers (see the TransformerEngine documentation here). However, NCCL bootstrap should also be supported now. You can try setting tp_comm_bootstrap_backend="nccl" here.