Skip to content

OpenAI Conformance Harness: runnable client with mock and real backend modes #1986

@Zigfreidish

Description

@Zigfreidish

Child of #1384.

Promote the in-process Conformance Matrix into a runnable Conformance Harness: a client that posts the same conformance rows to a live local endpoint or a configured remote Server Profile, with a deterministic mock-backend CI mode and a real-backend smoke mode, emitting the matrix report as a machine-readable operator evidence artifact.

Reuses the OpenAIConformanceReport schema; does not replace the in-process matrix as the authoritative proof (see docs/adr/0002-openai-conformance-at-control-plane-boundary.md).

Done when

  • Harness runs against a live local endpoint and a configured remote Server Profile.
  • Mock-backend mode is deterministic and CI-runnable without model weights.
  • Real-backend smoke mode produces an evidence artifact.
  • Failure output names the incompatible field/phase rather than a generic error.

Metadata

Metadata

Assignees

No one assigned

    Labels

    enhancementNew feature or requestrun-evidenceRun evidence, manifests, and reports

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions