New parameter: algo.load_balance_knapsack_keep_fraction

[WeiqunZhang]

This new parameter can be used to reduce the cost of data movement during load balancing even though the resulting DistributionMapping may be less optimal.

[WeiqunZhang]

@hklion Could you give this a try? This depends on AMReX-Codes/amrex#3079. You can start with algo.load_balance_knapsack_keep_fraction=0.5. If it's 0, it should be the same as the existing knapsack strategy.

[ax3l]

@hklion you can compile with a local AMReX that has this PR checked out (e.g., -DWarpX_amrex_src=$HOME/src/amrex) or via -DWarpX_amrex_repo=https://github.com/WeiqunZhang/amrex.git -DWarpX_amrex_branch=knapsack_keep)

https://warpx.readthedocs.io/en/latest/install/cmake.html#build-options
Example:

cmake --fresh -S . -B build -DWarpX_amrex_repo=https://github.com/WeiqunZhang/amrex.git -DWarpX_amrex_branch=knapsack_keep
cmake --build build -j 8

[hklion]

Thanks for implementing this, @WeiqunZhang! I'm finding in almost all cases that algo.load_balance_knapsack_keep_fraction=0 (i.e. the original knapsack) is faster than higher values. Load balancing does take less time with a higher keep fraction, but the slowdown from load imbalance drags down the overall simulation performance. And we don't have the advantage of boxes in one region being contiguous to one another, like we have with SFC. I've mostly been playing around with Vay deposition, but with Esirkepov, it is ~2% faster to use a keep fraction of 0.5. So no massive performance improvements.

One question: How does the code decide which boxes to keep and which to redistribute?

[hklion]

• @RevathiJambunathan

[WeiqunZhang]

First, an average load is computed using the cost estimation. Then the average multiplied by the keep fraction gives us the target load we want to keep. On each process, we sort the current boxes from the heavies to the lightest and try to keep the heavy boxes as long as the load is less than the keep target. For the rest of the boxes, we redistribute with the knapsack algorithm and the kept boxes also contribute to the load during the knapsack redistribution.