Benchmarking distributed GPU computations with nsight systems indicates that the bottleneck to improving strong scaling for simulations on immersed grids is now load balancing.
This arises from the use of the active cells map launching, which only performs computations on cells in the grid that are not immersed. For most bathymetrys (e.g. earth bathymetry) the naive partitioning of the grid into equal sized cells results in sub-grids containing very different amounts of active cells. This results in poor load balancing, causing the GPUs to become desynchronised and reducing utilisation.
I am opening this issue to track the outcome of different partitioning algorithms and their performance characteristics. Eventually this will result in a PR to add automatically balanced grid partitioning.
Benchmarking distributed GPU computations with nsight systems indicates that the bottleneck to improving strong scaling for simulations on immersed grids is now load balancing.
This arises from the use of the active cells map launching, which only performs computations on cells in the grid that are not immersed. For most bathymetrys (e.g. earth bathymetry) the naive partitioning of the grid into equal sized cells results in sub-grids containing very different amounts of active cells. This results in poor load balancing, causing the GPUs to become desynchronised and reducing utilisation.
I am opening this issue to track the outcome of different partitioning algorithms and their performance characteristics. Eventually this will result in a PR to add automatically balanced grid partitioning.