Skip to content

Commit d7383d3

Browse files
committed
feat(evidence): emit qualified lifecycle evidence
1 parent 6225d0a commit d7383d3

12 files changed

Lines changed: 3346 additions & 17 deletions

CHANGELOG.md

Lines changed: 8 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -3,10 +3,18 @@
33

44
## v0.94.1 (2026-08-26)
55

6+
### Features
7+
8+
- **evidence**: Build signed six-class campaigns and content-addressed public lifecycle summaries.
9+
610
### Bug Fixes
711

812
- **evidence**: Match the retention digests the Cloud writer emits (#311) ([`df12b3c`](https://github.com/OpenAdaptAI/openadapt-evals/commit/df12b3ce72ad02992a7c7e389d8ccf2cfd01774c))
913

14+
### Continuous Integration
15+
16+
- **release**: Keep a durable App-authored draft and verify its exact bytes before publication.
17+
1018

1119
## v0.94.0 (2026-08-26)
1220

README.md

Lines changed: 46 additions & 11 deletions
Original file line numberDiff line numberDiff line change
@@ -1,13 +1,13 @@
11
# OpenAdapt Evals
22

33
> [!IMPORTANT]
4-
> **Status: Research infrastructure, not a required part of the product.** This
5-
> package is evaluation and benchmarking tooling for GUI agents and for the
6-
> OpenAdapt demonstration compiler. It is evidence-generating research work. It
7-
> is not required to record, compile, or replay a workflow, and no end user
8-
> needs it installed.
4+
> **Lifecycle: Support.** This maintained public repository produces
5+
> qualification and release evidence for OpenAdapt. The seven product targets
6+
> use signed admissions to determine their Production state. Evals supplies
7+
> evidence for those decisions and remains a separate operational tool.
98
>
10-
> The OpenAdapt product is the governed demonstration compiler,
9+
> End users don't install Evals to run a workflow. The OpenAdapt product is the
10+
> governed demonstration compiler,
1111
> [`openadapt-flow`](https://github.com/OpenAdaptAI/openadapt-flow), installed
1212
> via the [`OpenAdapt`](https://github.com/OpenAdaptAI/openadapt) launcher
1313
> (`pip install openadapt`). It compiles a demonstrated GUI workflow into a
@@ -43,10 +43,45 @@ dimensions that matter to a governed compiler:
4343
- **Cost and model-call accounting**: model calls, latency, and dollar cost per
4444
successful task, including the ~0-model-call healthy replay path.
4545

46-
This is internal research tooling. It is not a packaged end-user product and it
47-
is not needed to use OpenAdapt. If you want to record, compile, and replay a
48-
workflow, use the [`OpenAdapt`](https://github.com/OpenAdaptAI/openadapt)
49-
launcher, not this repository.
46+
Operators use this package to run qualification and publication gates. If you
47+
want to record, compile, and replay a workflow, use the
48+
[`OpenAdapt`](https://github.com/OpenAdaptAI/openadapt) launcher.
49+
50+
### Qualification and release evidence
51+
52+
The v3 campaign producer records six required classes for each task: healthy,
53+
safe halt, idempotency replay, uncertain delivery, declared attended, and
54+
governed repair. Each task and condition needs at least three signed trials.
55+
Evals derives silent incorrect success and over-halt from the signed runner,
56+
observer, and delivery facts. It doesn't trust author-supplied failure counts.
57+
58+
Campaign payloads and trial receipts stay inside the approved private evidence
59+
boundary. A public lifecycle summary uses the remote-safe decision receipt,
60+
the public qualification admission, and the production acceptance manifest. It
61+
contains aggregate class counts and commitments, without task names,
62+
application or environment values, or live identities.
63+
64+
`scripts/build_production_evidence.py` builds that summary and the paired
65+
content-addressed v2 references. The pair command preserves the raw Sigstore
66+
bundle bytes. Run it only after the referenced objects exist in an exact merged
67+
registry commit. The summary uses a later registry append, which avoids a
68+
commit-hash cycle.
69+
70+
```bash
71+
python scripts/build_production_evidence.py summary \
72+
--input public-summary-input.json \
73+
--output production-acceptance-summary.json
74+
75+
python scripts/build_production_evidence.py pair \
76+
--kind production-acceptance-summary \
77+
--object production-acceptance-summary.json \
78+
--sigstore-bundle production-acceptance-summary.sigstore.json \
79+
--registry-source-commit "$MERGED_REGISTRY_COMMIT" \
80+
--registry-revision "$REGISTRY_REVISION" \
81+
--registry-head-sha256 "$REGISTRY_HEAD_SHA256" \
82+
--output-root public-registry-candidate \
83+
--references-output production-acceptance-summary.references.json
84+
```
5085

5186
### Relationship to the rest of OpenAdapt
5287

@@ -79,7 +114,7 @@ does not claim maturity it has not measured.
79114

80115
## What is inside
81116

82-
- **openadapt-flow evaluation** (`openadapt_evals/flow/`): the paradigm-correct
117+
- **openadapt-flow evaluation** (`openadapt_evals/flow/`): the release-path
83118
eval for a demonstration compiler. A `replay` runner compiles one demonstration
84119
into an openadapt-flow bundle and replays it against the WAA in-guest server
85120
with roughly zero model calls, and a `hybrid` agent runs compiled replay first

0 commit comments

Comments
 (0)