Skip to content

Publish Experiment Provenance To Immutable Storage #128

Description

@PSchmiedmayer

Use Case

An authorized operator should be able to retain experiment-run evidence outside the researcher-writable project and later verify which inputs, code, parameters, and environment produced an output.

Problem

Project-local provenance is useful for development but can be changed or removed by the same identity that runs the analysis.
Aim 1.2 requires authoritative retention, queries, reproducibility checks, and publication-ready reports.

Solution

  • Implement the experiment-provenance storage-sink contract against deployment-owned immutable or retention-locked object storage.
  • Use workload identity or a separately authorized service so agent tools cannot rewrite retained records.
  • Bind accepted run records to Heartwood audit checkpoints without merging the two schemas.
  • Add queries for output lineage, environment and input changes, failed or superseded runs, and exact reproducibility checks.
  • Produce a compact export suitable for research review and later publication reporting.
  • Keep storage provider, retention duration, and access policy in deployment configuration rather than project state.

Acceptance Criteria

  • A project identity can submit through the approved path but cannot overwrite or delete accepted records.
  • Retried publication is idempotent and interruption cannot create conflicting run histories.
  • Independent verification detects modified project records, missing retained records, and incorrect signer or deployment identity.
  • Query and report results are deterministic for the same retained run set.
  • Tests use synthetic records; live retention and compliance claims require deployment-owner evidence.

Alternatives considered

Owner-only project files do not provide independent retention.
Embedding complete experiment details in the minimized session audit would weaken both contracts.

Additional context

This follows the experiment-run foundation in #125 and completes the durable-storage portion of Aim 1.2.

Code of Conduct

  • I agree to follow this project's Code of Conduct and Contributing Guidelines

Activity

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    enhancementNew feature or request

    Type

    No type

    Projects

    • Status
      Backlog

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions