Running Databank on LUMI

[comcon1]

Databank computations can be run on LUMI in the following way now.

Make Singularity container with cotainr:

For that, we need conda-YAML environment description, let's name it e1.yml:

name: base
channels:
 - conda-forge
 - defaults
dependencies:
 - python==3.11
 - git
 - pip
 - gromacs
 - pip:
     - git+https://github.com/NMRlipids/Databank.git@you-can-specify-branch

And then run

module use /appl/local/csc/modulefiles
module load CrayEnv
module load cotainr
cotainr build nmdb_gmx.sif --system=lumi-c --conda-env=e1.yml

It will create sif-file with a container for further running on the nodes.

Creating Database on /scratch

cd /scratch/project_123456789/
git clone https://github.com/NMRlipids/BilayerData

Populating Database with symlinks using a simple bash-script:

#!/bin/bash

src="/pfs/lustrep4/appl/local/csc/datasets/NMRLipids/BilayerData/Simulations"
dst="$PWD/BilayerData/Simulations"

find "$src" -type f | while read -r file; do
    rel="${file#$src/}"
    mkdir -pv "$dst/$(dirname "$rel")"
    ln -sfv "$(realpath "$file")" "$dst/$rel"
done
# avoid downloading (optional)
cd BilayerData/Simulations
find . -type f -name 'README.yaml' -exec \
  sed -i -E 's/^DOI:.*$/DOI: localhost/' {} +

In the src of this script, we put the path where trajectories are stored, so that we make symlinks from it. The last two commands informs databank that everything is stored locally. We will not allow him then to download anything. It will modify README.yaml files, so do not commit them. But it is optional. Databank should know by itself that file is downloaded and skip downloading.

Setting up runner configuration and run

You can make run.sh slurm script as following. I use 2 cores just in case. Probably 1 is also enough.

#!/bin/bash
#SBATCH --partition=small
#SBATCH --account=project_123456789
#SBATCH --time=1-0:00:00
#SBATCH --nodes=1
#SBATCH --ntasks-per-node=1
#SBATCH --cpus-per-task=2 

OPTS=$@

module use /appl/local/csc/modulefiles
module load CrayEnv
module load cotainr

export OMP_NUM_THREADS=$SLURM_CPUS_PER_TASK
TRJPATH=/pfs/lustrep4/appl/local/csc/datasets/NMRLipids/BilayerData/Simulations

srun singularity exec \
       -B /scratch/project_123456789/:/scratch/project_123456789/ \
       -B ${TRJPATH}:${TRJPATH} \
       --env NMLDB_DATA_PATH=/scratch/project_123456789/BilayerData \
       nmdb_gmx.sif ${OPTS}

Then we can run it as, for example, sbatch -J nmldb run.sh nml_compute_databank --maicos -r 0-10. All the databank CLI commands are available here. They will be called from inside singularity container.

It is meaningful to set parallel run within ranges as following:

#!/usr/bin/env bash
# Run nml_compute_databank in N parallel chunks (0–1000)

# Total range and number of chunks
START=0
END=1000
CHUNKS=50

# Compute the size of each chunk
STEP=$(( (END - START) / CHUNKS ))

# Run chunks in parallel
for ((i=0; i<CHUNKS; i++)); do
   FROM=$(( i * STEP ))
   TO=$(( (i + 1) * STEP - 1 ))
   # Make sure last chunk reaches END
   if [[ $i -eq $((CHUNKS - 1)) ]]; then
       TO=$END
   fi
   CMD="sbatch -J maico${FROM} run.sh nml_compute_databank --maicos --thickness -r ${FROM}-${TO}"
   echo "Setting up: ${CMD}"
   $CMD
done
echo "✅ All parallel jobs set up."

So, now, everyone who has access to LUMI can try my settings.

Wrap up

You can commit JSON files by

git add **/*.json
git commit -m 'Recomputed on LUMI'

You should, of course, checkout a branch from your own fork to be able to commit.

After you are done for now, it's recommended to clean up. This command is used to clean up everything in all the subfolders which is not tracked.

git clean -fx

All the intermediate files which have been created -- they will eat your disk quota if not removed. It will also remove all the symlinks so do not forget to populate if you want to start working again.

[ohsOllila]

I tried this, but when running

sbatch -J nmldb run.sh nml_compute_databank --maicos -r 0-10

I get error

FATAL: "nml_compute_databank": executable file not found in $PATH

Is there some path definer missing?

[comcon1]

We have changed all the names, so everything should be updated accordingly. Let me check.

[comcon1]

First, this can be simplified to use pypi package (but use github if you want the latest or branch).

name: base
channels:
 - conda-forge
 - defaults
dependencies:
 - python==3.11
 - git
 - pip
 - gromacs
 - pip:
     - fairmd-lipids

Second, the shell paths NMLDB_ prefix should become FMDL_ prefix

#!/bin/bash
#SBATCH --partition=small
#SBATCH --account=project_123456789
#SBATCH --time=1-0:00:00
#SBATCH --nodes=1
#SBATCH --ntasks-per-node=1
#SBATCH --cpus-per-task=2 

OPTS=$@

module use /appl/local/csc/modulefiles
module load CrayEnv
module load cotainr

export OMP_NUM_THREADS=$SLURM_CPUS_PER_TASK
TRJPATH=/pfs/lustrep4/appl/local/csc/datasets/NMRLipids/BilayerData/Simulations

srun singularity exec \
       -B /scratch/project_123456789/:/scratch/project_123456789/ \
       -B ${TRJPATH}:${TRJPATH} \
       --env FMDL_DATA_PATH=/scratch/project_123456789/BilayerData \
       nmdb_gmx.sif ${OPTS}

Finally, the prefix was changed for CLI also nml_compute_databank should become fmdl_compute_databank