Draft project document for benchmark submission manager

[arezaii]

This adds a tools directory and a PROJECT.md document to that directory. The document will be used to guide the development of tooling that facilitates running SST benchmarks with different configurations and collecting the data when complete.

[mstrout]

This is a great start on the design of an SST experimental tool design. I look forward to seeing the next iteration!

[mstrout]

Instead of container images, I would prefer there be ContainerFile s with instructions on how to build and use those containers.

[arezaii]

This point was meant to convey a way of defining the available versions and locations of SST on the system being used. Containers appear here just because we need to be able to point the launcher to them as well as the regular sst binary.

I have not brought the generation of new containers/files into scope with this tool. It can be topic for exploration, but possibly a different tool.

[mstrout]

Which project? The tool itself or a set of experiments being done with the tool?

[mstrout]

I think the Key Usage Scenarios should come before the Design Philosophy. Having concrete usage scenarios like this is wonderful. Please provide

1. the command line(s) that you propose the user will need for each scenario
2. a priority level of

• P0 for use cases that need to be in the minimal viable prototype, something you could use to run some checkpointing experiments
• P1 for enough use cases for doing a demo for the O.O team
• P2 for talking others on the team into using it for their experiments
• P3, polish features before sharing this with customers

Feel free to assign different meanings to the priority levels.

If you specify example command lines for the key usage scenarios and provide the tool with existing scripts you all already use for running experiments, I bet co-pilot could write most of the implementation and the testing for you. :)

[mstrout]

I really like this idea as long as it doesn't force the sst benchmarks themselves to adapt too much. There should be a more general way to specify what command line options those benchmarks take? In other words, would other people's sst-benchmarks be able to use this tool? The answer can be "not easily", but I want to know why.

[arezaii]

I imagine a base class that defines a general benchmark and then sub classes can define their own arguments/parameters and how to parse those from yaml config files. New configurations of existing benchmarks could be added as yaml files specifying the values.

Adding a new benchmark should be possible, but there will be a modest amount of code to update/add.

The main CLI would support launching experiments of different benchmarks by name and configuration file (.yaml) together. e.g. python run_batch_experiments phold single_node_sweep.yaml where all of phold's specific parameters would be stored in a section of the yaml file, which also has sections for the tool, for sst, for slurm, and others.

[mstrout]

It might be important to mention containers here.