|
1 | 1 | # OpenAdapt Evals |
2 | 2 |
|
3 | 3 | > [!IMPORTANT] |
4 | | -> **Status: Research infrastructure, not a required part of the product.** This |
5 | | -> package is evaluation and benchmarking tooling for GUI agents and for the |
6 | | -> OpenAdapt demonstration compiler. It is evidence-generating research work. It |
7 | | -> is not required to record, compile, or replay a workflow, and no end user |
8 | | -> needs it installed. |
| 4 | +> **Lifecycle: Support.** This maintained public repository produces |
| 5 | +> qualification and release evidence for OpenAdapt. The seven product targets |
| 6 | +> use signed admissions to determine their Production state. Evals supplies |
| 7 | +> evidence for those decisions and remains a separate operational tool. |
9 | 8 | > |
10 | | -> The OpenAdapt product is the governed demonstration compiler, |
| 9 | +> End users don't install Evals to run a workflow. The OpenAdapt product is the |
| 10 | +> governed demonstration compiler, |
11 | 11 | > [`openadapt-flow`](https://github.com/OpenAdaptAI/openadapt-flow), installed |
12 | 12 | > via the [`OpenAdapt`](https://github.com/OpenAdaptAI/openadapt) launcher |
13 | 13 | > (`pip install openadapt`). It compiles a demonstrated GUI workflow into a |
@@ -43,10 +43,45 @@ dimensions that matter to a governed compiler: |
43 | 43 | - **Cost and model-call accounting**: model calls, latency, and dollar cost per |
44 | 44 | successful task, including the ~0-model-call healthy replay path. |
45 | 45 |
|
46 | | -This is internal research tooling. It is not a packaged end-user product and it |
47 | | -is not needed to use OpenAdapt. If you want to record, compile, and replay a |
48 | | -workflow, use the [`OpenAdapt`](https://github.com/OpenAdaptAI/openadapt) |
49 | | -launcher, not this repository. |
| 46 | +Operators use this package to run qualification and publication gates. If you |
| 47 | +want to record, compile, and replay a workflow, use the |
| 48 | +[`OpenAdapt`](https://github.com/OpenAdaptAI/openadapt) launcher. |
| 49 | + |
| 50 | +### Qualification and release evidence |
| 51 | + |
| 52 | +The v3 campaign producer records six required classes for each task: healthy, |
| 53 | +safe halt, idempotency replay, uncertain delivery, declared attended, and |
| 54 | +governed repair. Each task and condition needs at least three signed trials. |
| 55 | +Evals derives silent incorrect success and over-halt from the signed runner, |
| 56 | +observer, and delivery facts. It doesn't trust author-supplied failure counts. |
| 57 | + |
| 58 | +Campaign payloads and trial receipts stay inside the approved private evidence |
| 59 | +boundary. A public lifecycle summary uses the remote-safe decision receipt, |
| 60 | +the public qualification admission, and the production acceptance manifest. It |
| 61 | +contains aggregate class counts and commitments, without task names, |
| 62 | +application or environment values, or live identities. |
| 63 | + |
| 64 | +`scripts/build_production_evidence.py` builds that summary and the paired |
| 65 | +content-addressed v2 references. The pair command preserves the raw Sigstore |
| 66 | +bundle bytes. Run it only after the referenced objects exist in an exact merged |
| 67 | +registry commit. The summary uses a later registry append, which avoids a |
| 68 | +commit-hash cycle. |
| 69 | + |
| 70 | +```bash |
| 71 | +python scripts/build_production_evidence.py summary \ |
| 72 | + --input public-summary-input.json \ |
| 73 | + --output production-acceptance-summary.json |
| 74 | + |
| 75 | +python scripts/build_production_evidence.py pair \ |
| 76 | + --kind production-acceptance-summary \ |
| 77 | + --object production-acceptance-summary.json \ |
| 78 | + --sigstore-bundle production-acceptance-summary.sigstore.json \ |
| 79 | + --registry-source-commit "$MERGED_REGISTRY_COMMIT" \ |
| 80 | + --registry-revision "$REGISTRY_REVISION" \ |
| 81 | + --registry-head-sha256 "$REGISTRY_HEAD_SHA256" \ |
| 82 | + --output-root public-registry-candidate \ |
| 83 | + --references-output production-acceptance-summary.references.json |
| 84 | +``` |
50 | 85 |
|
51 | 86 | ### Relationship to the rest of OpenAdapt |
52 | 87 |
|
@@ -79,7 +114,7 @@ does not claim maturity it has not measured. |
79 | 114 |
|
80 | 115 | ## What is inside |
81 | 116 |
|
82 | | -- **openadapt-flow evaluation** (`openadapt_evals/flow/`): the paradigm-correct |
| 117 | +- **openadapt-flow evaluation** (`openadapt_evals/flow/`): the release-path |
83 | 118 | eval for a demonstration compiler. A `replay` runner compiles one demonstration |
84 | 119 | into an openadapt-flow bundle and replays it against the WAA in-guest server |
85 | 120 | with roughly zero model calls, and a `hybrid` agent runs compiled replay first |
|
0 commit comments