PTDS Benchmarks

[quasiben]

With cupy/cupy#4322 merged in we can begin profiling/benchmarking performance of more workers per GPU.

Setup

• Build latest cupy python -m pip install .
• CUPY_CUDA_PER_THREAD_DEFAULT_STREAM=1 dask-cuda-worker --nthreads=2
• Use at least 4 GPUs
• test nthreads: 1, 2, 4, 8, 32`

Test Operations

• sum
• mean
• svd
• Matrix Multiply
• Array Slicing
• transpose + sum: (x + x.T).sum()

Note @pentschev previously tested these operations on a single GPU:

• https://github.com/pentschev/pybench/blob/89d65a6c418a1fee39d447bd11b8a999835b74a9/pybench/benchmarks/benchmark_array.py
• https://github.com/pentschev/pybench/blob/master/examples/plot_array_example.ipynb

Example of CuPy with Dask

import cupy
import dask.array as da

rs = da.random.RandomState(RandomState=cupy.random.RandomState) 
x = rs.normal(10, 1, size=(500000, 500000), chunks=(10000, 10000))
x.sum().compute()

cc @charlesbluca

[jakirkham]

We've also used adding a matrix with it's transpose as a communication benchmark. Could be reasonable here to show the communication overhead goes to 0 while still computing work in parallel

[quasiben]

That's a good idea @jakirkham . I updated the test operations

[charlesbluca]

Thanks for writing this up @quasiben!

Some questions:

• Would this benchmark be better suited as a reproducible script, a one-off notebook with some plots to reflect the results, or both? Seems like it might be leaning towards notebook since the number of GPUs is flexible.
• Could pybench be applicable in any way towards putting this together, or does it serve as more of an example in implementing the tests?

[quasiben]

Would this benchmark be better suited as a reproducible script, a one-off notebook with some plots to reflect the results, or both? Seems like it might be leaning towards notebook since the number of GPUs is flexible.

Generally, yes! Some of these operations are already built in here:
https://github.com/rapidsai/dask-cuda/blob/branch-0.18/dask_cuda/benchmarks/local_cupy.py

Could pybench be applicable in any way towards putting this together, or does it serve as more of an example in implementing the tests?

Probably. Extending pybench seems like a reasonable way to automate running many of these operations. Much of the tests in the dask-cuda benchmark suite have nice output for testing a set of parameters be we don't have additional tooling for varying those parameters automatically. This would be very beneficial especially if plotting was also automated!

I also think doing a bit of a dive on some of these operations with a manual setup would be useful. We could use tools like pynvtx and confirm kernels are launching on multiple threads.

[charlesbluca]

After touching base with @pentschev, I think it may be easier to extend local_cupy.py with the additional test operations we want since it already takes in GPU count and nthreads as input - although I'm not too sure how to go about parametrizing the tests with pytest-benchmark.

For plotting pybench already has some tools for that, although @pentschev mentioned the process of generating the plots in his example notebook took some toil, so not sure how easy it would be to automate for wider use cases. I think a good start for now would be outputting benchmarking results in JSON that could be parsed by pybench's plotting utils, which could either be done:

• Somewhere in the implementation of local_cupy.py
• By pytest-benchmark using the --benchmark-json=PATH option

[charlesbluca]

Made a fork with some additional operations for testing and an argument to output JSON of the results:

• https://github.com/charlesbluca/dask-cuda/blob/ptds-benchmark/dask_cuda/benchmarks/local_cupy.py

Some more thought may need to go into device sync and warm up as in pybench, but for now I'll try this out and see if there are noticeable results.

[pentschev]

Looks great, when you're ready, be sure to open a PR! 😄

[charlesbluca]

Done! #524

From benchmarks using 4 GPUs and 1, 2, 4, 8 threads, there wasn't been a remarkable difference in performance; a plot of these results:

• https://gist.github.com/charlesbluca/009248ef106e027cfd106610656aa1f9

I'm going to scale back the number of GPUs and check out what the Nsys profile and Dask performance reports look like.

[jakirkham]

Interesting. It might be worth comparing notes with Peter to make sure things are configured correctly.

Couple questions. Were you able to see multiple streams in use ( cupy/cupy#4322 (comment) )? Also are using the changes from PR ( cupy/cupy#4322 ) (note this is not in a CuPy release yet)?

[pentschev]

IIRC, I was only able to see significant difference on transpose sum, and besides the CuPy PTDS support, I also needed rapidsai/rmm#633 and UCX. Could you try those out as well @charlesbluca ?

[charlesbluca]

Sure! I'll run the benchmarks again and see how it looks.