Analytics cli

[bkorycki]

First attempt at an analytics CLI tool: run-csv.
To summarize, each run results in a new directory containing the output results file and a metadata json file.

The output structure is largely based off of the sample dvc directory in the modelbench-private dvc-experiments branch. However, I was not able to follow it 100%. Here are the key differences:

• The metadata lists suts instead of one sut. This is because the pipeline runner is shared with the already-exisiting run-csv-ITEMS which allows for multi-sut runs.
• I changed the prompts metadata key to input in order to generalize to runs where the input is prompts + responses.
• It is not currently possible to include provider and provider_id in the metadata. The best I could do is include the SUT initialization record. Annotators do not have initialization records, however, which maybe we should fix.
• I excluded the unique_count for responses in the metadata because it would be kind of difficult to keep track of. If it's super important, we can include it in a later PR.
• There is no cache (see convo in discord)

Some questions

• Would it be better to include the full path to the input source?
• Any advice on how to make the unit tests less repetitive would be appreciated.
• Also open to better names for the cli tool.

[github-actions]

MLCommons CLA bot All contributors have signed the MLCommons CLA ✍️ ✅

[rogthefrog]

That seems inconsistent with the comment right above: isn't a null or blank value a bad UID?

[bkorycki]

Yeah I can see how it looks a little counter intuitive. But this callback is called even when no argument is supplied. So it shouldn't return False when we aren't actually validating a UID.

Before, every click option that used this validate_uid callback either required exactly one argument or optionally allows a list of args. So validate_uid would never receive None.

However, my new cli has an optional option --sut. Since it's not a list, it's default value is None if no sut is specified, instead of an empty list.
That's why I need to add this line of code.

Basically, this function only validates UID strings that are actually provided. Does that make sense?

[rogthefrog]

What is Text? Is it the prompt?

[bkorycki]

Yes. This is identical to the already existing run-csv-items command.

[wpietri]

As long as we're changing things, do we want to change this so that will read our prompt files? Because I think that's going to be the most common use case.

[bkorycki]

I can do that but those changes will also affect the run-csv-items command that the eval folks use.

[wpietri]

• Also open to better names for the cli tool.

I think the CSV-ness of this is incidental to the purpose, so I might be more inclined to go with run-prompts or run-job. Or you could look for a more poetic name. Maybe something with a name derived from kitchen tools or hand tools?

[wpietri]

Looks good! I think not having everything in the metadata for now is fine. Let's get it into people's hands and see what they want once they're using it.

[wpietri]

I see what you're saying about the tests. Have you tried creating an object that holds all of those variables? You could at least move the boilerplate inside it, and I suspect it would be a good place for some convenience methods that would clear up your code some.

[bkorycki]

Yeah I thought about that but I'm having trouble setting up boiler plate because each test differs slightly in input file, cli args, and expected output file.

[bkorycki]

@wpietri I'm not very poetic so I re-named to run-job. Thanks for the suggestion!