#558: Restructured PR to add node level memory constraint post recent changes

[ppebay]

Resolves #558

[ppebay]

A first review would be appreciated @lifflander to make sure we agree on the design.

I am confident that the computation of the node-level memory usage is correct:

with indeed initially 4 blocks in the bottom row (node 0) and 1 block in top row (node 1), at 9B per block:

and in contrast, in rebalanced stage there are 6 blocks in bottom row (node 0) and 3 in top row (node 1) due to across-rank block replications:

[ppebay]

@lifflander @nlslatt this is an interesting case (synthetic-blocks, load-only + memory constraint):

Configuration 1:

We begin by enforcing the memory constraint at the node-level:

• 2 ranks per node;
• memory constraint of 36.0 (i.e. maximum of 4 blocks per node);
  where we readily obtain an imbalance of 0.0, in the following steps:

One first noticeable finding is that the final state not only each node has a maximum of 4 blocks, but also each rank has at most 2 blocks (memory = 36.0/18.0):

Therefore, it is in theory possible to achieve an equivalent per-rank memory constraint; however, the second important point is that this is achieved in the iterations above by traversing 2 intermediate ones with 3 blocks (i.e. memory=27.0) on some nodes.

Configuration 2:

We now try to enforce the equivalent memory constraint at the rank-level:

• 1 rank per node (i.e. no ranks_per_node optional parameter in the configuration file);
• memory constraint of 18.0 (i.e. maximum of 2 blocks per rank);
  in which case we now indeed cannot obtain an imbalance of 0.0, for instance in the following sequence (consistent with multiple other runs):

[... 6 identical iterations omitted ... ]

In other words, the iterative approach cannot attain the optimal configuration, and remains locked with a non-zero load imbalance:

This is because, with this iterative algorithm, one has to traverse a weaker configuration, where the constraint is enforced at the node level but no longer at the equivalently-formulated rank level.

Conclusion:

The important overall finding is that even when it appears to be the case that equivalent node and rank memory constraints can be achieved with optimal load imbalance, allowing for the weaker (node-level) constraint to be achieved during the course of the iterations can be the only way to reach the final optimum. This could have consequences for actual, non-synthetic cases where the node-level constraint would provide more degrees of freedom for better performance.

[ppebay]

@cwschilly could you please add the first configuration I was describing above to the integration tests? I think it's important to make sure we capture this corner case where relaxing the memory constraint to the node level allows convergence to 0 imbalance.

[cwschilly]

LGTM

[lifflander]

Looks good to me!