Define benchmark artifact, history, and comparison models

[tkilias]

Goal

Define the stable public models and storage contracts used by benchmark packaging, history, and comparison.

Scope

• Add versioned typed models for artifact metadata, runner samples, normalized cases, test-set collections, and comparison reports.
• Define portable artifact bundles and a Git-trackable benchmark-history layout.

Acceptance Criteria

• Models serialize and deserialize without information loss.
• The on-disk schema is versioned and documented.
• Compatibility expectations are documented for public model changes.

[tkilias]

Proposed implementation plan

Issue #10 — Versioned benchmark artifact and current-history contract

Summary

Add Pydantic v2 public models for individual runner-execution artifacts, normalized cases, logical test-set collections, and comparison reports. Git-tracked history holds only the current commit’s results, with each runner execution stored as its own artifact directory.

Key changes

• Add public Pydantic models and lossless JSON round-trip helpers:
  • ArtifactManifest and RunnerExecution for one produced benchmark result.
  • Normalized cases keyed by pytest-benchmark fullname.
  • In-memory test-set collection and comparison-report/result models.
• Require schema version, test-set ID, stable comparison_target, runner-execution ID, source-revision provenance, and structured platform metadata.
• Replace fixed database fields with an extensible attributes mapping (string keys to JSON values). Consumer projects can record database versions, implementation identifiers, side-by-side deployment details, or other comparison context without changing this package’s schema.
• Validate supported schema versions, IDs, JSON-safe attribute values, and duplicate identities when runner executions are collected.
• Add Pydantic v2 as a runtime dependency and export the public model types.
• Use the same portable and Git-trackable per-execution artifact layout:

benchmark-history/
└── <comparison-target>/
    └── <test-set-id>/
        └── <runner-execution-id>/
            ├── manifest.json
            └── benchmark.json

Example manifest.json:

{
  "schema_version": 1,
  "test_set_id": "tpch-sf10",
  "comparison_target": "onprem-standard",
  "runner_execution_id": "linux-x86_64-01",
  "source_revision": "8f12ab4",
  "platform": {
    "os": "ubuntu-24.04",
    "architecture": "x86_64"
  },
  "attributes": {
    "database": {
      "product": "Exasol",
      "version": "8.31.0"
    },
    "implementation": {
      "id": "native",
      "build": "2026.07.28"
    },
    "warehouse": "ci-performance-large"
  },
  "benchmark_file": "benchmark.json"
}

benchmark.json is unchanged pytest-benchmark output. No schema-version or revision directories are used, and no aggregate run file is stored.

• Treat the checked-out benchmark-history/ tree as the complete baseline for the current commit. A loader constructs logical collections from its artifact directories; Git itself preserves prior revisions.
• Document the artifact layout, manifest schema, attributes extension point, baseline identity rules, and compatibility expectations. Schema versions remain fields within JSON documents; incompatible changes require a new major schema version and readers support migration during transitions.

Test plan

• Verify every public model serializes/deserializes without information loss.
• Cover invalid schema versions, missing required metadata, non-JSON attribute values, duplicate runner-execution identities, and duplicate normalized cases.
• Verify arbitrary nested attributes—including database version and implementation IDs—round-trip without changing execution associations.
• Verify multiple targets, test sets, runner executions, and platforms round-trip through separate artifact directories.
• Verify the history loader reads only the current benchmark-history/ tree and builds collections without relying on revision-path structure.

Assumptions

• Pydantic v2 is the public validation and serialization dependency.
• comparison_target is the stable baseline selector; all project-specific comparison context belongs in attributes.
• Raw pytest-benchmark JSON remains byte-for-byte unchanged. Packaging, persistence operations, normalization, and comparison execution are implemented by dependent issues.