Proposal: optional run audit sidecar for automation outputs

[mindbomber]

Proposal

Would mlperf-automations be open to an optional run audit/provenance sidecar for automation outputs?

This would not replace official MLPerf result files or change benchmark scoring. The goal is to make automation runs easier to reproduce, review, and cite by recording safe metadata about the environment and artifacts that produced a result.

Suggested sidecar shape

{
  "schema_version": "mlperf_automation.run_audit.v1",
  "automation_command": "mlcr run-mlperf,inference,...",
  "benchmark": "mlperf_inference",
  "benchmark_version": "v5.1",
  "model": "resnet50",
  "scenario": "offline",
  "implementation": "reference",
  "system_ref": "local-or-redacted-system-id",
  "result_paths": ["..."],
  "config_hashes": {
    "mlc_script": "sha256:...",
    "user_config": "sha256:..."
  },
  "provenance": {
    "automation_repo": "mlcommons/mlperf-automations",
    "automation_commit": "...",
    "created_at": "..."
  },
  "claim_status": "diagnostic",
  "redaction_status": "safe_for_public_log"
}

Why this may help

• separates diagnostic/local runs from official/reportable submissions
• records enough provenance to debug or reproduce runs later
• gives dashboards and result-summary tools a standard place to find audit-safe metadata
• avoids putting raw secrets, tokens, private filesystem paths, or full sensitive arguments in public artifacts
• complements result summary files without altering benchmark result semantics

Possible first PR scope

If maintainers think this is useful, I can prepare a small docs/example PR that:

• adds an example run_audit.example.json
• documents that the sidecar is optional and non-normative
• keeps official result files and scoring unchanged
• does not add external dependencies

This is motivated by AANA work around audit-safe AI evaluation artifacts, but the proposal is generic to MLPerf automation/reproducibility and does not require AANA as a dependency.

[arjunsuresh]

@mindbomber This is very good good to have. Are you proposing an extension of the official inference results json file including the automation recipe? https://github.com/mlcommons/inference_results_v6.0/blob/main/summary_results.json

[mindbomber]

@arjunsuresh Yes, that is the direction I had in mind, with one important constraint: I would not want to change the official result semantics or scoring fields.

A practical shape could be either:

1. an optional sidecar generated by mlperf-automations, e.g. run_audit.json, containing the automation recipe/provenance, config hashes, automation commit, redaction status, and related metadata; and/or
2. a very small optional extension in the official inference summary JSON that points to that sidecar or records its hash/reference.

So I see this as complementary metadata for reproducibility and auditability, not a replacement for summary_results.json or the existing submission/result structure. The recipe itself probably belongs outside the core result fields at first, because it may include automation-specific or locally redacted details. Then summary tooling could consume it when present.

If that direction sounds reasonable, I can prepare a small docs/example PR first: an example sidecar schema, guidance on safe/redacted fields, and a note about how it could map to a future optional summary-results reference.

[arjunsuresh]

Thank you @mindbomber for the explanation. The suggestion is good and it's something we always wanted to do. But we see less interest in reproducibility these days. So the main question is who will do the automation runs? I'm able to run smaller models on Nvidia RTX 4090 and unfortunately that's the only GPU I have access to. From MLCommons side I don't see any activity towards reproducibility these days.

[mindbomber]

@arjunsuresh That makes sense. I would frame the first step so it does not depend on MLCommons owning a large reproducibility campaign.

I can take on the initial lightweight part: a docs/example PR that defines the optional run_audit.json sidecar and shows how an automation run would record recipe/provenance metadata without changing official results or scoring.

For actual runs, I think the practical model is:

1. start with small, reproducible examples that can run on accessible hardware, including a single RTX 4090 or CPU-only where possible;
2. treat those as diagnostic/community reproducibility artifacts, not official submissions;
3. let future submitters or community members attach the sidecar when they run mlperf-automations; and
4. optionally add a reference/hash from summary tooling later, once the sidecar shape is stable.

So the proposal does not require MLCommons to provide GPU capacity up front. It gives people who already run automation recipes a standard, audit-safe way to publish enough provenance for others to inspect or rerun them.

If that sounds aligned, I can start with the small docs/example PR and keep the scope explicitly non-normative and reproducibility-oriented.