Evaluation of parallel_file_movements: 'threads' with AWI-ESM3 on levante@dkrz.de

[JanStreffing]

I ran two simulations AWI-ESM3 at the same time (same HPC and filesystem load) on levante. One with:

general:
    parallel_file_movements: 'threads'

one without this option.

My findings over the first 4 years / simulation legs are:

Sequential

Run	Compute Time	Total Time	Cleanup Time
1	19m40s	26m09s	6m29s
2	19m31s	27m41s	8m10s
3	21m02s	26m53s	5m51s
4	19m34s	25m00s.	5m26s

Threaded

Run	Compute Time	Total Time	Cleanup Time
1	20m28s	24m45s	4m17s
2	20m56s	27m38s	6m42s
3	19m37s	22m58s	3m21s
4	19m31.s	23m34s	4m03s

Three observations:

1. The spread in cleanup time is quite large, likely due to file system load issues.
2. At equal load I seem to see a speedup.
3. That speedup may be in the range 1.3x - 1.5x.

I'll add more years to see if the differences hold. @mandresm

Experiment folders:
/work/bb1469/a270092/runtime/awiesm3-develop/run_dynamic_allocation_01/
/work/bb1469/a270092/runtime/awiesm3-develop/test_par_file_threads_01/

[mandresm]

Thanks for reporting @JanStreffing! If you provide me with the path to work of one of this simulations I can test how long does it take with my test dask script to copy those files somewhere else (a good estimation of what it'll take for ESM-Tools in the future once dask does the job). Then we can prioratize it's implementation if the speed up is satisfactory.

[mandresm]

oh! I've just seen you share the path already. Nevermind, I'll check how it goes.

[JanStreffing]

I think a decent chunk of them time is also the fesom bin restart tar and untar.

[JanStreffing]

Here are some more numbers from a month of daily runs:

Serial: {'min': '03:34', 'max': '07:57', 'mean': '05:03'}
Threads: {'min': '01:51', 'max': '03:40', 'mean': '03:18'}
Speedup (mean): 1.53x

/work/bb1469/a270092/runtime/awicm3-develop/copy_serial/log
vs
/work/bb1469/a270092/runtime/awicm3-develop/copy_threads/log

helper script: /work/ab0246/a270092/postprocessing/postp_time_stats/eval.py

[mandresm]

Good to know. Some numbers of copying a work folder (24G) just with cp and with the parallel approach and 2 nodes:
cp: ~2.5 min
dask: ~10 seconds (the script I am using could be even further parallelize, meaning the time could be even further reduced).

Initializing the dask workers takes between 30s (2 nodes) and 2 min (44 nodes). Reducing the dask worker initialization to a few seconds is going to be key. I will try in the future using dask_mpi and way less workers with more threads (maybe 4 workers per node with 32 threads?).

[nwieters]

Thank you @JanStreffing for testing this.
I also experienced slow file copying and tried it with setting parallel_file_movements: 'threads'. I did not checked if it was faster then, since I found in the documentation, that this is turned on by default https://esm-tools.readthedocs.io/en/latest/esm_variables.html ? So I guess this is not the case?

[mandresm]

The documentation is wrong, by default it is set to False, because threads has not been thoroughly tested. Once we implement the dask approach I am planning to do the thorough testing and we could consider setting up to dask always.

esm_tools/configs/defaults/general.yaml

Line 8 in 7aedaf7

parallel_file_movements: False

When do you need this feature the latest?

[JanStreffing]

I think this would be good to use once we start doing data heavy CMIP7 simulations. The plan is to start 1st of November, the latest. Obviously, it would be good to test a little bit before that. Maybe mid-October?

[mandresm]

Mid-October is doable I think.

[mandresm]

With https://gitlab.awi.de/hpc/tutorials/sbatch_dask_cluster_script.git and 44 nodes I have achieved the copying of /work/bb1469/a270092/runtime/awiesm3-develop/run_dynamic_allocation_01/ works directory in 30 s (including initialization of the dask cluster) in comparison with 1m 30s of a regular cp -r.

If I run the same copy with the dask cluster initialized it copies in 7 s. Most of the time is spent in copying the dirs in the work directory, but ESM-Tools never copies dirs in the loop, but files, meaning that if dask cluster is used in ESM-Tools for copying we could reduce the 7 s to possibly 3 s.

If ESM-Tools initializes the dask cluster right at the beginning of the tidy step, by the time it needs to start copying, the initialization of the cluster might be ready, so we are going to be looking at a x3-x30 speed up.