SST PHOLD Parameter Sweep Function Parity

[MattToast]

Creates a simple python package for running parameter sweeps over the phold_dist.py python script. This is composed of a parsing module that creates parameter sets to sweep over and launching module that instructs how to launch a leg of the parameter sweep.

There are currently two parameter sweep strategies: Product and Random.

• Product will take the cross product of all of the parameter values supplied
• Random will randomly select N samples from a set of range of parameters

The parsing interface slightly re-organizes the original in how parameters are supplied to the sweep, but otherwise offers 1-to-1 functionality with the original script.

Six launching strategies have been included: SmartSim, DragonPG, DragonBatch, FluxBulkSubmit, FluxPyAPI, and Dry Run:

• SmartSim offers the 1-to-1 launching strategy as the original submit.sh script by adding jobs into the underlying WLM work queue.
• DragonPG will launch the sweeps with naive scheduling using the Dragon Process Group API. Note that this requires a dragon allocation and runtime be given up front.
• DragonBatch will launch the sweeps using the Dragon Batch API. This reduces control over placement of the ranks in favor of increased parallelism. Again the dragon runtime must receive a full allocation up front.
• FluxBulkSubmit will launch each of the jobs via the flux scheduler in a subprocess. This requires that the sweep be launched within a flux session, which also requires an allocation be given upfront.
• FluxPyAPI will also launch each of the job via the flux scheduler but instead of using a subprocess uses the Python API. The same caveats apply as with FluxBulkSubmit.
• Dry Run will simply print out that command as an srun command roughly equivalent to what was requested to be run.

Currently the SmartSim and Dry Run launching strategy offer 1-to-1 functionality with the original parameter sweeping script. The DragonPG and DragonBatch strategies allow us to explore what launching with Dragon may look like in the future. Similarly, the FluxBulkSubmit and FluxPyAPI strategies allow us to explore what launching with Flux may feel like in the name of being more portable than SLURM or PBS specific implementations.