Result publication permits unresolvable provenance for composed metrics

[XuanYang-cn]

Problem Statement

Published benchmark results can contain metrics composed from separate runs without a source artifact that repository users can resolve or verify.

TestResult currently represents both a single benchmark run and a consolidated public artifact containing measurements from multiple runs. When measurements are composed, their provenance is stored in untyped additional parameters or JSON-encoded note strings. The result publication flow therefore permits machine-local paths, unavailable filenames, stale single-run metadata, and provenance that the frontend cannot explain.

Evidence from PR #815:

• Twelve Turbopuffer records did not execute the load stage but publish positive load metrics copied from six other runs. Their source references are machine-local /home/ubuntu/... paths.
• Forty filtered records identify separately measured serial metrics using a source filename and SHA-256, but those source files are not tracked or otherwise resolvable from the published artifacts.
• Consolidated files retain June or July top-level timestamps while containing measurements sourced in August.
• The frontend presents composed metrics as ordinary measurements and does not expose their provenance.
• Existing validation checks metadata shape, such as filename prefix and hash length, but does not resolve the source artifact or verify its content.

PR #815 removes the twelve absolute-path fields as immediate containment. It does not prevent the same class of invalid public metadata from recurring.

Solution

Define a clear publication interface for consolidated benchmark artifacts. Raw local results may retain execution-local metadata, but the published form must contain typed, portable, and verifiable provenance.

Published artifacts should distinguish direct measurements from composed measurements, identify the exact fields sourced from another run, reference a durable artifact through a repository path or stable URI, and verify its digest before publication. Artifact-level identity and timestamps must describe the consolidated publication rather than an arbitrary source run.

User Stories

1. As a benchmark reader, I want every source reference in a published result to be resolvable, so that I can audit where a measurement came from.
2. As a benchmark reader, I want composed measurements to be clearly identified, so that I do not assume all metrics were produced by one execution.
3. As a benchmark reader, I want source artifact hashes to be verified during publication, so that I can trust the referenced evidence.
4. As a benchmark maintainer, I want publication to reject machine-local paths, so that runtime filesystem details are not committed accidentally.
5. As a benchmark maintainer, I want provenance represented by typed models, so that required fields and invariants are validated consistently.
6. As a result publisher, I want one publication seam to sanitize and validate artifacts, so that every publication path follows the same rules.
7. As a frontend user, I want composed metrics and their sources to be visible, so that result comparisons are interpreted correctly.
8. As a result consumer, I want artifact identity and timestamps to describe the consolidated artifact, so that metadata remains honest when multiple runs are merged.
9. As a developer, I want existing result files to remain readable through an explicit compatibility path, so that the schema can evolve safely.
10. As a test author, I want to verify publication behavior through its public interface, so that internal refactoring does not weaken the provenance contract.

Implementation Decisions

• Establish one result-publication module and interface as the owning seam for consolidation, sanitization, provenance validation, and serialization.
• Keep single-run results distinct from consolidated published artifacts, either through separate models or an explicit artifact-level schema.
• Introduce typed measurement provenance that records whether composition occurred, which metric fields were composed, and the source run identity.
• Introduce a typed artifact reference containing a durable repository path or URI plus a cryptographic digest.
• Reject absolute local filesystem paths in published artifacts.
• Resolve every published artifact reference and verify its digest before writing the consolidated artifact.
• Replace JSON-encoded structured provenance in note strings with structured data.
• Define artifact-level identity and timestamp semantics for consolidated publications.
• Expose composed-measurement status and provenance through the result-reading interface used by the frontend.
• Preserve existing result compatibility through explicit schema migration or legacy parsing rather than weakening the new publication contract.

Testing Decisions

• Test through the highest publication interface: input run results in, validated public artifact out.
• Verify that absolute Unix and Windows filesystem paths are rejected during publication.
• Verify that composed measurements must list their source metric fields and source run.
• Verify that every artifact reference resolves and that digest mismatches fail publication.
• Verify that direct measurements do not require composed-measurement provenance.
• Verify that consolidated identity and timestamps do not silently inherit arbitrary source-run metadata.
• Verify that the frontend can distinguish direct and composed measurements through the result-reading interface.
• Verify compatibility behavior for existing committed result files.
• Prefer local fixture artifacts for tests so provenance verification is deterministic and does not require external benchmark services.

Out of Scope

• Re-running the benchmark data currently published by PR feat: FTS: add OSS OpenSearch backend, semantic FTS recall, filtered FTS cases #815.
• The immediate deletion of legacy absolute paths from PR feat: FTS: add OSS OpenSearch backend, semantic FTS recall, filtered FTS cases #815.
• Selecting or operating a new long-term external artifact storage product.
• Changing benchmark measurement algorithms or backend behavior.

Further Notes

The absolute paths found during PR #815 review are one symptom of the wider model mismatch. Removing them is the correct narrow fix for that PR; this issue owns the durable architecture and publication guardrails.

See also: #815