CSC across multiple nodes with Slurm

[EthanMakaresz]

Hi,

I'm trying to run a CSC calculation on a HPC system managed with Slurm, but I'm running into issues with resource allocation. My dmft_config.toml file is attached (as a .txt file because I can't upload a .toml). I'm using VASP for the DFT with PLO for the projection. I'm on a very recent version of the unstable branch, compiled from source. I can provide the git hashes if necessary.

There are a series of issues I'm running into. I've already performed a one-shot calculation across multiple nodes, so I know that the QMC side of things can handle allocation across multiple nodes with no issues.

1: The resource allocation in the slurm job submission script is given by
#SBATCH --ntasks=X
#SBATCH --ntasks-per-core=1

Here I use srun to launch solid_dmft and pass the path to the srun command in the dmft_config.toml file, so that solid_dmft launches VASP using srun also. In this case, the output stalls after reporting that VASP is being launched. The Slurm error file contains the final line, "srun: error: Invalid numeric value "p" for --ntasks." This looks like solid_dmft has done something like "/path/to/srun -np Y /path/to/vasp", but srun takes -n as input and has interpreted p as the number of cores, not Y. So I switched to

2: Using mpirun to launch VASP but srun to launch solid_dmft. In this case, the output still stalls after reporting that VASP is being launched, but the error file says, "There are not enough slots available in the system to satisfy the 4 slots that were requested by the application: /path/to/vasp". I'm definitely requesting fewer cores for VASP than I am using to launch solid_dmft (in this specific case I used 8 for solid_dmft and 4 for VASP).

3: Using "#SBATCH --ntasks=8" and running solid_dmft with "srun -n 4 solid_dmft" in hopes that the other 4 cores are reserved for VASP. This had the same result as the previous attempt.

4: Restricting to a single node by using
#SBATCH --nodes=1
#SBATCH --ntasks-per-node=8
#SBATCH --ntasks-per-core=1

In this case VASP launches with mpirun and everything works as intended.

The problem is that the calculations I want to do can't really be done in a reasonable time on one node. I have a set of 5 d-orbitals with all the off-diagonal terms included, and I want to go to fairly low temperatures with CTHYB. Even running VASP on one CPU, that leaves only 95 CPUs to do the QMC on my system, which would take far too long. Is there a way to get solid_dmft to do a CSC calculation across multiple nodes with VASP?

dmft_config.txt

[the-hampel]

Hi Ethan,

yes I understand this has to work of course. I also was running jobs like this frequently via slurm in the past with QMC across multiple nodes. However, I usually restricted VASP to run on a single node. This is for the following reason: slurm does not like to run two mpi executables at the same time inside of one slurm job. But of course only either VASP or triqs use CPUs at the same time so technically this is possible. But only the first launched MPI job will be able to get a fast comm layer between nodes. The second one will have none or TCP at best.

Let's do this point by point:

1. using slurm srun. I guess you confirm that both VASP and solid_dmft work with srun. If you can use the latest unstable builds can you try this setting:

mpi_env = "slurm"
mpi_exe = "srun" 

this should then pass correctly to srun the arguments, but I never could really try this. It looks like this in mpi_helpers.py:

    if mpi_profile == 'slurm':
        # Default to srun when using SLURM; can be overridden via dft.mpi_exe in the config
        if os.path.basename(mpi_exe) == 'mpirun':
            mpi_exe = 'srun'
        # srun inherits the remaining time of the allocation automatically when -t is omitted
        return [mpi_exe, '-n', str(number_cores), '--export=PATH',
                '-N', os.getenv("SLURM_JOB_NUM_NODES"),
                '-A', os.getenv("SLURM_JOB_ACCOUNT"),
                '-p', os.getenv("SLURM_JOB_PARTITION"),
                '-w', f"./{hostfile}"] + shlex.split(dft_exe)

so it uses -n and not -np . For that you probably have to add #SBATCH --oversubscribe as a slurm option and best also #SBATCH --exclusive . Then hopefully if your cluster config allows it the second srun does not complain about insufficient resources. So when assuming your cluster has 32 cores per node you could do something like this in the slurm script:

#SBATCH --nodes=8
#SBATCH --ntasks-per-node=32
#SBATCH --ntasks-per-core=1
#SBATCH --oversubscribe
#SBATCH --exclusive

srun solid_dmft

and then in the solid_dmft config toml:

[dft]
...
mpi_env = "slurm"
mpi_exe = "srun" 
n_cores = 32
...

then VASP sould run on a single node (make sure that it does otherwise performance will be poor), which should be fine for VASP. And TRIQS/QMC should use all 8 nodes 32 cores each.

2. Using mpi_env = "openmpi-intra" is probably your best bet if using mpi_exe="mpirun" . But you need the --oversubscribe . The error you are seeing fits perfectly to that. It might even be that you have to add --oversubscribe --bind-to none to mpirun, i.e. dft_exec="--oversubscribe --bind-to none vasp_std" . This should work. You can even tell VASP directly which cores to use: dft_exec = "taskset -c 0-32 vasp_std . Try to experiment with these until the job runs. It can be very effective to get an interactive job session to test this and monitor via htop if the allocation works correctly. You can't add these options to mpi_exe they would be stripped away but adding to dft_exec it should work.

I will not go into detail for 3) and 4). Let's first see if one of these works for you. I think we should be able to get this working! Let me know how this goes.

Best,
Alex

[EthanMakaresz]

Hi Alex,

Thanks for the quick response! I tried your suggestion for using srun to launch vasp, but it's not working. The --oversubscribe and --exclusive options are mutually exclusive, so I can't set them both at the same time. I tried just using --oversubscribe, and a vasp job was launched, but it was waiting in the Slurm queue and the solid_dmft job was cancelled with no reason given. If the vasp job has to get through the queue every time it is launched, that will be a big problem, since it can take days if many other resources are being used (like those for the DMFT part).

The second option (mpi_env = "openmpi-intra" and dft_exec="--oversubscribe --bind-to none vasp_std") seems to be running, but VASP is running very slowly; at least an order of magnitude slower than it should. It took ~10 minutes to read the WAVECAR and in the 30 minutes since has not completed one iteration, whereas running VASP by itself for this system takes a minute or two per iteration.

I tried running with just mpirun, with mpi_env = "openmpi-intra" and making sure that there was a node with enough CPUs to run VASP, and this worked as intended. So the problem might have been that just using #SBATCH --ntasks distributed resources in such a way that there was no single node with enough CPUs available to run VASP.

Regards,
Ethan

[the-hampel]

I suspect that with the second option VASP only runs on one care. You probably have to do something like dft_exec="taskset -c 0-32 vasp_std" and bind VASP to specific cores to make this work reasonably fast. So does the last option you wrote via mpi_env = "openmpi-intra"work as intended, i.e. can you let the whole job run with many nodes and VASP grabs only one of them? Then we have a winner. Is that with--oversubscribe`?

Best,
Alex

[EthanMakaresz]

It seemed like the second option was running on all 4 cores I requested. A hostfile was generated that listed 4 CPUs. The slowdown was also much more that I would expect from reducing the CPUs from 4 to 1.

The test that worked was:
#SBATCH --nodes=2
#SBATCH --cpus-per-node=4
#SBATCH --ntasks-per-cpu=1
Then I set mpi_env = openmpi-intra, n_cores=4 for DFT and dft_exec=mpirun with no additional tags. This grabbed CPUs all from one node automatically for VASP.

Thanks for your help!

Regards,
Ethan

[EthanMakaresz]

Hi @the-hampel,

There is an issue I have noticed with this setup for charge self-consistent calculations. It seems that VASP is storing something large in memory - maybe the projectors - at each iteration but not properly releasing the memory, causing an OOM error.

Currently I have 150 GB of memory allocated for a monolayer calculation with 17*17 kpts (33 inequivalent after symmetry). With this, I can get 6 or 7 VASP iterations until VASP gets cancelled, which causes the output to stop but solid_dmft to keep running. I'm not able to see the actual error message from VASP, but looking up the job with sacct revealed that the VASP process ran out of memory. If I double the memory allocation, then VASP completes roughly double the iterations before running out of memory again. So maybe the projectors are being stored in memory at each iteration and that memory is not released after they are written?

Regards,
Ethan

[the-hampel]

Hi @EthanMakaresz ,

can you quickly check which hdf5 version you are using. There is a memory leak in hdf5 ver 1.12.3 and 1.14.x and I am not sure what the status is. There is some description of it here:
TRIQS/triqs#952

There are some issues for memory in triqs, but those should be addressed in triqs 3.3.2 and for sure on the 3.3.x branch:
TRIQS/triqs#955
TRIQS/cpp2py#55

Let's first cross check the h5 library. If you have options to install an older version that would help. If not I can also try to reproduce this. Can you somehow track live if VASP or TRIQS memory increases?

I think with the python script in the first issue you should be able to track it if spawned on the same node and you enter the pids. Something like this in python:

def process_memory():
    process = psutil.Process(os.getpid())
    mem_info = process.memory_info()
    print(f'{mem_info.rss/1024**2:.2f}')
    return mem_info.rss

Or any other method. That would be helpful to know where the problem sits.

best,
Alex

[EthanMakaresz]

Hi @the-hampel,

I'm using hdf5 version 1.14.6, so this could be the issue. I'm working on building an older version of hdf5 and I will let you know if this fixes the issue.

As for the memory, I can track it via sacct. The VASP process on 20 CPUs did not use much, so the issue must be with TRIQS. The sacct output for a recent test is attached:

Regards,
Ethan

[the-hampel]

phew. finding such memory leak in VASP would have been much harder and we know the reason most likely already. But it also means that the bug is still present in many hdf5 version supplied on clusters :-( I think on the TRIQS side everything is fixed. @Wentzell do you remember if we filed in the end a bug report on the hdf5 library? And if so what is the status? I tried to locate it on the hdf5 github site but could not find anything...