There is a PM request that we show the scalability features built into Armory. We should
- test armory on Noether with proper CUDA_VISIBLE_DEVICES set
- hoist a slurm one-node-cluster on Noether to confirm that Lightning does in fact distribute evaluation as we expect
We don't necessarily expect that on a one-node cluster simulation that wall-clock time for an evaluation would be less, but its success would empirically show that our use of the mechanism is valid.
As a proof of concept, this issue is fulfilled when we demonstrate code and metrics from a multi-chain evaluation, distributed over N compute nodes.
Higher order orchestration is not a goal for this issue.