Performance evaluation

[jbusecke]

JOSS askes for If there are any performance claims of the software, have they been confirmed? (If there are no claims, please check off this item.)

What should/can we do to document performance of the package?

[NoraLoose]

We started the discussion on performance evaluation here and here.

[jbusecke]

Apologies for missing those. Should I close this issue?

[NoraLoose]

Not at all! I just posted these, so we don't need to start the discussion from scratch. :)

[jbusecke]

Perfect!

[NoraLoose]

Sorry, I didn't mean to kill this discussion. I will try to summarize the aspects that we had discussed to include in a performance evaluation.

• Compare performance of gcm-filters on CPUs and GPUs against fortran implementations of spatial filters, e.g., the ones implemented by @iangrooms and @arthurBarthe. Here is a list compiled by @rabernat how such a comparison could be done.
• Look at strong and weak scaling, see @rabernat's comment here.
• Compare computational speed when using increasingly complex Laplacians in gcm-filters, see for example the table here.
• ...

I am sure this list is incomplete, so please feel free to add/delete items. Maybe @iangrooms also saved the computational section that was originally part of the JAMES paper.

Question: Do we want to include the performance evaluation

1. in the JOSS paper,
2. in the gcm-filters documentation,
3. both (in some kind of combination)?

I think 3. would maybe work best.

[rabernat]

Thanks for the writeup. I think your list is perfect, and I favor option 3.

For me, the big question is the following. Most HPC system nodes have either ~24 CPU cores or 1 GPU. By using parallelism appropriately (e.g. parallelize across non-spatial dimensions), can the 24-core CPU speed get close to the 1 GPU speed?

I find this problem fun and would like to work on it a bit.

[rabernat]

I know I volunteered to do this. I still hope to work on it. Will try to squeeze it in over the next few days.

[rabernat]

Ok, so I started the benchmarking exercise in a a standalone repo:

https://github.com/ocean-eddy-cpt/gcm-filters-benchmarks

The repo is organized as follows:

• Main benchmarking notebook: benchmark_gcm-filters.ipynb - does all of the work and writes data to disk
• Data directory: data/ - CSV files with results
• plot_results.ipynb

One central goal is to figure out how many CPUs you need to use to replicate the GPU performance.

What I did so far

What I did was run the basic tutorial example on Casper with various numbers of CPUs and look at strong and weak scaling. I bypass questions about i/o performance by using random data. When x and y are contiguous (not chunked) the GCM filters approach is embarrassingly parallel, so in theory should scale perfectly (runtime inversely proportional to the number of CPUs used).

What I found was a bit different. Here is the weak scaling (from the plot_results.ipynb notebook). I used a single node of Casper, varying the number of CPUS from 1 to 36, with various combinations of workers and threads.

What the graph shows is decent scaling for low core counts, then a saturation at high core counts. As a reminder, this is weak scaling, which means we actually we use more cores, we make the problem bigger. The fact that the throughput saturates at a relatively low rate (200 MB/s) indicates that scaling is breaking down for some reason. Instead, as we add more cores, we should keep processing data faster. It would be great to dig into this a bit deeper to understand why this is happening. Perhaps that is something that Dask experts @gjoseph92 and @jrbourbeau could help us debug?

A useful comparison point is simply taking the mean of the same array, which exposes Dask's own scaling properties:

Although it is somewhat sub-linear, it is better than gcm-filters.

Next steps

Compare to GPU performance

We should compare these single-node CPU results to the GPU results. Unfortunately I couldn't do this because I broke my cupy installation in my Casper python environment 😢. Ideally someone with a working cupy could rerun the basic benchmark with the two different types of Casper GPUs. Note that, with a single GPU, we can't really look at scaling proper, since we can't use multiple GPUs. But we can still locate the GPU performance on the same axis (MB/s) as the CPU performance.

If we wanted to use a multi-GPU cluster, we could get into dask-cuda, but IMO this is overkill for the JOSS paper.

Look at distributed scaling on multiple nodes

It's a mystery why gcm-filters is not scaling linearly up to 36 cores on a single node. We should debug this and try to understand why. Perhaps it's related to memory usage? (There is no disk i/o in the example, so that can't be an issue.)

We could also look at scaling across multiple different nodes. Maybe if we keep the core count low on each node (e.g. use only 4 cores per node), but spread over several nodes, we could get better scaling. We would need to figure out the best way to launch a multi-node Dask cluster on Casper or Cheyenne (probably Cheyenne). Pining @andersy005, software engineer at CISL, for help with this.

Vary some of the array parameters

I used an array with chunks of shape (10, 1024, 1024) and dtype float64. Some other things to try

• Use single precision
• Use different chunk shape in the time dimension (e.g. 5 or 20 instead of 10)
• Use different chunk shape in the x, y dimensions (e.g. 256 or 2048 instead of 1024)
• Try different kernels. I just used REGULAR_WITH_LAND. Obviously other kernels are slower in absolute terms, but I hope that they have broadly similar scaling behavior, which is the focus here. This should be tested.

My hope is that, by getting this started and giving a structure to the benchmark code, this is easy to hand off to other folks. Please feel free to make PRs to the gcm-filters-benchmarks repo, updating the notebook and / or adding more CSV results to the data/ directory.

We are flying to Italy tonight, so I will be offline for a day or two while in transit. Will check back in later in the week.

Thanks for your patience, as I have been very slow with my task.

[gjoseph92]

Thanks @rabernat! I think @jrbourbeau will look into this soon. Two things that would help us investigate this faster:

1. Could you collect performance reports for some of runs? (Shameless plug: you can now use coiled.performance_report to capture and host these for you, which saves the annoyance of uploading to a gist and getting the githack link.)
2. Could you try letting threads_per_worker go up to 36? I'm curious if this could be a case of Co-assign neighboring tasks to neighboring workers dask/distributed#4892.

[mrocklin]

Checking in here. It would be useful to get a performance report of one anecdotal run. That might help us to better understand the cause of the sub-linear scaling. I would strongly recommend doing this before investing time in running larger parameter sweeps.