docs: rewrite README for a human reader - #318
Merged
Merged
Conversation
At 571 lines this was the longest README in the org, and it opened with a fifteen-line callout that spent most of its words describing openadapt-flow. Someone who came for the eval harness read two paragraphs about a demonstration compiler before reaching anything they could run. The honest "this is research, you do not need it" point stays, as one sentence. Corrections, checked against openadapt-evals 0.94.0 installed from PyPI into a clean venv: - The `oa` CLI does not take `setup`, `status`, `start` and friends directly. Those sit under `oa evals vm`, so single-VM setup is `oa evals vm setup`. - The current evidence set was wrong. The README pointed at `current_flow_v1_31_0_local_20260818`, which PUBLISHED_EVIDENCE.json marks superseded along with six others. The current set is `current_flow_v1_33_0_local_20260826`, measured against Flow 1.33.0. - `oa-vm` has 59 subcommands, not "50+". The page now leads with `openadapt-evals mock --tasks 5` and the output it actually prints, then the dry-run cost estimate, both captured from that venv. The evidence section carries the real 1.33.0 numbers, including the rename condition where compiled replay went 3/3 and both Playwright selector controls went 0/3, and the part that argues against us: compiled replay is about thirty times slower per step than a working selector. The limits section now states what the 1.33.0 report states and the old README did not, which is that `scripts/eval_flow_on_waa.py` leaves `WAALiveAdapter.evaluate` unwired on the replay path, so the live path cannot independently score success today. The CLI tables, configuration, and AWS SSO setup move to docs/CLI.md. The package tree, the WAA stack diagram, and the demo-conditioned, full-eval, UI-Venus, GRPO, and custom-agent runbooks move to docs/WORKFLOWS.md. Nothing was deleted. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Seven agents ship became a count where the old README had a selective list. agents/__init__.py exports thirteen implementations. Say thirteen. Only two adapters exist is wrong: six BenchmarkAdapter subclasses ship, and the quickstart above that bullet runs one of them. The true claim is about benchmark families, so say that instead. oa-vm pool-wait, pool-run, and pool-auto have required --qualification-dir since the fresh-isolation-proof change. The quickstart command as written exits with an argparse error. Show the flag and say what the directory holds. docs/CLI.md listed vm setup-waa, which oa-vm does not have; the only reference in this repo points at openadapt-ml. Drop the row. The 59 subcommands count is 66 on this branch and 63 at v0.94.0, and the figure it replaced, 50+, was true. Drop the number. Steady time wraps the whole replay action loop, and one run executes 11 actions, so 6.9s against 0.21s is per run, not per step. Also: nine console scripts, not eight; pool-logs, pool-vnc, and pool-exec take no --cloud; the CLI reference was verified against the released wheel rather than this branch, which is what hid the pool-wait break; and the quickstart transcript is abridged, not verbatim.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
At 571 lines this was the longest README in the org. It opened with a fifteen-line callout that spent most of its words describing openadapt-flow, so a reader who came for the eval harness got two paragraphs about a demonstration compiler before anything they could run. The honest "this is research, you do not need it" point stays. It is one sentence now.
Corrections
I installed
openadapt-evals==0.94.0from PyPI into a clean venv and checked every command against that.oaCLI does not takesetup,status,startand the rest directly. They sit underoa evals vm, so single-VM setup isoa evals vm setup. The old text told you to runoa --helpand find them, and you cannot.current_flow_v1_31_0_local_20260818.PUBLISHED_EVIDENCE.jsonmarks that superseded, along with six others. The current set iscurrent_flow_v1_33_0_local_20260826, measured against Flow 1.33.0 on 2026-08-26. The page also still discussed 1.28 as the stale one and 1.31 as current, which stopped being true two releases ago.oa-vmhas 59 subcommands, not "50+".What the page leads with now
openadapt-evals mock --tasks 5and the output it prints, thenopenadapt-eval-flow --mode replay --tasks 154 --dry-runand its cost estimate. Both captured from that venv.The evidence section carries the real 1.33.0 table, including the
renamecondition where compiled replay went 3/3 and both Playwright selector controls went 0/3. It also carries the number that argues against us: compiled replay is roughly thirty times slower per step than a working selector, 6.9s against 0.21s. The 1.33.0 report says that plainly and the README should too.The limit that was missing
The 1.33.0 report states that
scripts/eval_flow_on_waa.pyleavesWAALiveAdapter.evaluateunwired on the replay path, so it cannot independently score success, and that the hybrid live path returns before execution. I confirmed there is noevaluatecall inopenadapt_evals/flow/replay_runner.pyin the shipped 0.94.0 wheel. The old README described that same command as "the paradigm-correct eval for a compiler" and said nothing about it. It says something now.What moved
docs/CLI.md: the three CLI tables, theopenadapt-eval-flowflag reference, configuration, AWS SSO setup, and the optional dependency groups.docs/WORKFLOWS.md: the package tree, the WAA stack diagram, and the demo-conditioned eval, full evaluation runner, UI-Venus grounder, GRPO training, and custom-agent sections.Nothing was deleted.
README: 571 lines to 203.