Slurm issue with different taskset options

[ufukozkan]

Hi all,

I am trying to run AWICM3-v3.3 on NHR@ZIB(formerly HLRN).

My branch is similar to 'origin/feat/awicm3_v3.3_and_v3.4_and_geomar'. We added cdo and nco modules, changed partition. We also modified configs/components/xios/xios.yaml file, since the names of the HDF5 libraries are different on NHR@ZIB.

My problem is that when I use 'taskset : True' as a setting, I get an error below;

Contents of hostfile_srun:
0-959 ./prog_fesom.sh
960-1151 ./prog_oifs.sh
1152-1152 ./prog_rnfmap.sh
1153-1156 ./prog_xios.sh

Submitting jobscript to batch system...
Output written by slurm:

cd /scratch/usr/hbkufuko/runtime/awicm3-v3.3//taskset-true/run_20000102-20000102/scripts/; sbatch taskset-true_compute_20000102-20000102.run
sbatch: error: --nodes is incompatible with --distribution=arbitrary
Exiting entire Python process!

If I manually submit the job (below) model is restarting without a problem.

sbatch run../scripts/...run

With the recommendation of @JanStreffing I tried 'taskset : False'. (Also there is another discussion one year ago; #1148)

However, the job never starts with a hetjob. Start time always iterates. When I kill the job and change the setting to True again, the job starts.

I have attached runscript and blogin yaml files. Let me know if something else is needed, please.

Kind regards,
Ufuk

blogin.txt

awicm3-v3.3-blogin-TL255L91-Arc01.txt

[JanStreffing]

Any input from the HLRN support? I seem to recall you mentioned something in the meeting.

[JanStreffing]

Can you upload the .run file?

[ufukozkan]

Any input from the HLRN support? I seem to recall you mentioned something in the meeting.

NHRsupport.txt

Can you upload the .run file?

hetjob.txt

taskset.txt

Let me know if anything else is needed, please.

Cheers,
Ufuk

[mandresm]

Hi @ufukozkan,

Sorry for the delay on replying to this. As I told you I was focusing on a development/improvement that was blocking the release of AWICM-3.3 and 3.4. I will have time this week to get Berlin's NHR HPC supported again.

I'll keep you posted starting from tomorrow.

[mandresm]

Hi @ufukozkan,

sbatch: error: --nodes is incompatible with --distribution=arbitrary
sbatch: error: --nodes is incompatible with --distribution=arbitrary
Exiting entire Python process!

If I manually submit the job (below) model is restarting without a problem.

sbatch run../scripts/...run

I had that exact same problem last week in Juwels, not in blogin. The problem was solved by deactivating taskset and using hethobs. However, I am also aware that if you use taskset: false and the sum of the number of MPI (nprocs) * openmp_num_threads in all your executable s (FESOM, XIOS, OIFS and RNFMAP) is not a multiple of the number of cores in a node, ESM-Tools is calculating incorrectly the resources that needs to request in slurm (that's a bug, I am starting to work on it).

In the mean time, the branch awicm3-v3.3_and_v3.4 got merged to release. I am not working on getting AWICM3-V3.3.0 to install and run on blogin. I'll keep you posted with the progress.

I have attached runscript and blogin yaml files.

And a recommention :) , it is normally better to simply add the contents of the files directly on the issue, or at lease part of them, inside one code block in markdown (see https://docs.github.com/en/get-started/writing-on-github/working-with-advanced-formatting/creating-and-highlighting-code-blocks)

[ufukozkan]

Hi @mandresm ,

Thank you for the recommendation. I will try that.

Furthermore, I have received another two e-mails by support. I have attached the e-mails.

Last_mails.txt

[mandresm]

Hi @ufukozkan,

Sorry again for the delay. Too many tasks these days, all very urgent...

I have started from release a new branch feat/awicm3_in_blogin. That branch in its current state can compile awicm3-v3.3.0 out of the box. It contains the changes suggested by Lars Nerger, plus some additions from my side.

Feel free to test and see if it compiles for you out-of-the-box with that branch, and please let me know.

Next step: fix the bug with the calculation of the computing resources.

[mandresm]

@christgau, thanks so much for the emails exchanged with @ufukozkan, and for taking the time to look into ESM-Tools repo and even making a PR.

I will be fixing the issues you found both for taskset: false and taskset: true.