Skip to content

Commit 684e727

Browse files
phodalcodex
phodal
and
codex
committed
feat(studio): simplify live ACP comparison
Implements the default project, prompt, Agent, and message-flow Compare surface using docs/specs/2026-08-27-studio-selectable-acp-compare.md. Resource rows retain tool-call identity and expand recorded results in the originating AI lane, while Advanced evidence remains available. Validated with npm run check and 10 responsive Playwright scenarios. Co-authored-by: Codex (GPT 5.6 Sol) <codex@openai.com>
1 parent 3d6f3d7 commit 684e727

14 files changed

Lines changed: 1779 additions & 51 deletions

‎docs/specs/2026-08-26-studio-acp-resource-compare.md‎

Lines changed: 80 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,80 @@
1+
# Resource-oriented ACP comparison
2+
3+
## Traceability
4+
5+
- Spec ID: studio-acp-resource-compare
6+
- Status: Implemented
7+
8+
## Intent
9+
10+
Make a completed or running ACP comparison explain which project resources each
11+
AI inspected, changed, or verified. Keep the simple project-and-Prompt entry
12+
surface, but replace manual cross-reading of two transcripts with one
13+
resource-oriented comparison that retains the originating tool calls and their
14+
recorded results.
15+
16+
## Acceptance Scenarios
17+
18+
- AC-1: A canonical ACP tool call with multiple `parsed_cmd` entries projects
19+
into multiple observable operations without pretending that each operation
20+
was a separate provider call.
21+
- AC-2: Reads, searches, edits, listings, verification commands, and generic
22+
commands are associated with normalized resource keys when the recorded input
23+
or result supplies one; trial-root absolute paths are not rendered in the UI.
24+
- AC-3: Compare can show one row per resource with Baseline and Candidate
25+
operations aligned around that shared resource, including run-only resources
26+
and multiple operations from one call.
27+
- AC-4: Selecting an operation exposes its recorded status and a bounded,
28+
browser-safe result summary while preserving access to the two original
29+
message streams.
30+
- AC-5: The resource comparison updates from canonical tool-start and
31+
tool-result events during a live run and does not infer unobserved reads,
32+
edits, tests, or intent.
33+
- AC-6: The resource view has visible keyboard focus, labelled controls, no
34+
document-level horizontal overflow, and a usable bounded horizontal layout at
35+
1440x900, 1024x768, and 390x844.
36+
37+
## Non-goals
38+
39+
- Build a free-form node graph, infer causal intent, or rank a model from a
40+
single matched trial.
41+
- Add a new ACP provider payload to the browser contract.
42+
- Replace patch receipts or grader results with tool-call claims.
43+
- Implement an arbitrary repository file browser or editor.
44+
45+
## Plan and Tasks
46+
47+
- Add an operation projection beside the existing single-call normalization.
48+
Extract nested `parsed_cmd` operations, recorded changes, and narrow
49+
result-derived diff paths while retaining call identity.
50+
- Reuse that projection in the existing resource ledger so Advanced evidence
51+
no longer reports zero resources for composite ACP calls.
52+
- Add a default Resource map and a Messages switch to the simple Compare result
53+
area. Use docked rows and a bounded operation-result inspector rather than
54+
cards or a free-form graph.
55+
- Add focused model and component/browser coverage, then verify the live page at
56+
the three Studio layout widths.
57+
58+
## Test and Review Evidence
59+
60+
- AC-1/AC-2: model tests cover a compound list/search/read call, an absolute
61+
edit path, a diff/status verification call, and result summarization.
62+
- AC-3/AC-4: component/browser tests assert shared resource rows, lane-only
63+
operations, the Messages fallback, and an accessible operation inspector.
64+
- AC-5: stream-fold tests prove a running call becomes a completed operation
65+
with the recorded result without changing call identity.
66+
- AC-6: Playwright screenshots and console/page-error inspection at 1440x900,
67+
1024x768, and 390x844.
68+
- Risk: treating command output as fact can create false reads. Result parsing
69+
is limited to changed-file/diff evidence and never promotes arbitrary stdout
70+
paths to reads.
71+
- Risk: compound calls make raw call counts misleading. The UI labels operation
72+
rows with their parent call sequence instead of comparing operation count as
73+
a quality score.
74+
75+
Observed locally on 2026-08-26 with Node 24.15.0: Harness tests passed 172/172,
76+
Studio tests passed 283/283, and the focused Compare browser checks passed 2/2.
77+
The real `codex-acp` comparison projected five resources, four shared resources,
78+
one edited resource, and a Candidate-only README verification without exposing
79+
the temporary worktree path. Wide, compact, and narrow screenshots had no
80+
document-level horizontal overflow or browser console/page errors.

‎docs/specs/2026-08-26-studio-simple-acp-compare.md‎

Lines changed: 77 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,77 @@
1+
# Simple live ACP comparison
2+
3+
## Traceability
4+
5+
- Spec ID: studio-simple-acp-compare
6+
- Status: Implemented
7+
8+
## Intent
9+
10+
Make Compare understandable as one task: choose the currently bound project,
11+
write one user prompt, run two configured AI lanes, and watch both message
12+
streams. Preserve the existing checkpoint, trace, and verdict evidence as an
13+
advanced view without letting it compete with the primary flow.
14+
15+
## Acceptance Scenarios
16+
17+
- AC-1: Opening Compare shows one project control, one editable user Prompt,
18+
the two configured AI lanes, and one primary Run action without requiring a
19+
Builder or workbench-opening step.
20+
- AC-2: The exact submitted Prompt is passed to every fresh lane, retained in
21+
the experiment output, and used for the compare-set Prompt hash.
22+
- AC-3: Each lane streams assistant text, tool activity, run state, errors, and
23+
permission requests as the run progresses; raw provider payloads do not cross
24+
the browser boundary.
25+
- AC-4: The project control names the checkpoint-bound project honestly. It
26+
does not imply arbitrary workspace rebinding when only one project is
27+
available.
28+
- AC-5: Checkpoints, protocol counts, traces, and verdict tables remain
29+
available through one secondary Advanced evidence action and retain existing
30+
evidence-sufficiency semantics.
31+
- AC-6: The default surface has visible keyboard focus, no document-level
32+
horizontal overflow, and keeps Prompt plus Run visible at wide, compact, and
33+
narrow layouts without browser console or page errors.
34+
35+
## Non-goals
36+
37+
- Add arbitrary filesystem or remote-project discovery.
38+
- Change the configured lane count, models, ACP Agent command, grader, or
39+
checkpoint materialization contract from the browser.
40+
- Replace evidence-sufficiency rules with a conversational judgement.
41+
- Remove the existing evidence workbench or checkpoint history adapter.
42+
43+
## Plan and Tasks
44+
45+
- Add a small default Compare surface owned by `ExperimentView` and retain the
46+
existing workbench behind progressive disclosure.
47+
- Add a bounded Prompt override to the Studio run endpoint and Harness
48+
experiment runner, then persist the submitted bytes beside run evidence.
49+
- Project host-neutral message framing into a browser-safe assistant-message
50+
vocabulary and fold it into lane activity in event order.
51+
- Render the configured lanes as two docked conversation panes with inline ACP
52+
permission decisions and responsive stacking.
53+
- Update behavior tests and browser checks around the new default path.
54+
55+
## Test and Review Evidence
56+
57+
- AC-2: Harness runner test proves the override reaches both lane executors,
58+
the retained Prompt bytes match, and the compare-set hash changes with it.
59+
- AC-2/AC-3: Studio server tests prove bounded Prompt validation, runner
60+
forwarding, assistant-event projection, and provider-payload omission.
61+
- AC-3: comparison-model tests prove assistant deltas and tool calls preserve
62+
lane activity order.
63+
- AC-1/AC-4/AC-5: component/browser checks prove the default labels and the
64+
Advanced evidence escape hatch.
65+
- AC-6: Playwright checks at 1440x900, 1024x768, and 390x844; inspect focus,
66+
overflow, console/page errors, and screenshots.
67+
- Risk: a cosmetic Prompt field would create false evidence. Review the actual
68+
runner input, persisted bytes, and compare-set hash together.
69+
- Risk: streaming raw ACP frames could expose provider payloads. Keep the
70+
browser contract allow-listed and test that arbitrary payload fields are
71+
absent.
72+
73+
Observed locally on 2026-08-26: both real `codex-acp` lanes reached `finished`;
74+
the simple surface retained 9 assistant messages and 9 tool activities with no
75+
browser console warnings or errors. Harness tests passed 172/172, Studio tests
76+
passed 277/277, doc-link tests passed 8/8, and the focused Compare browser
77+
regression passed 2/2 across its responsive layouts.

‎docs/specs/2026-08-27-studio-selectable-acp-compare.md‎

Lines changed: 105 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,105 @@
1+
# Selectable ACP comparison agents
2+
3+
## Traceability
4+
5+
- Spec ID: studio-selectable-acp-compare
6+
- Status: Implemented
7+
8+
## Intent
9+
10+
Make live Compare easier to inspect and run. A selected file operation should
11+
show its retained Tool Call result beside that operation, and each fresh lane
12+
should let the user choose from the ACP Agents that this Studio server can
13+
actually launch for the current project.
14+
15+
## Acceptance Scenarios
16+
17+
- AC-1: Selecting a Read, Edit, Search, List, Verify, or Run operation expands
18+
its bounded recorded result inside the same resource row and lane; the view
19+
does not require the user to correlate a row with a detached page footer.
20+
- AC-2: Compare exposes one Agent selector per fresh lane. Each option reports
21+
a stable server-owned id, label, availability, and unavailable reason without
22+
exposing an arbitrary browser-controlled executable or environment.
23+
- AC-3: Starting a comparison sends only the selected Agent ids. The server
24+
rejects unknown or unavailable ids before materializing trials and launches
25+
each lane with the command and argv registered for that exact id.
26+
- AC-4: Studio discovers installed, protocol-backed presets for `qodercli
27+
--acp`, `codex-acp`, `pi-acp`, and `claude-agent-acp`; an environment-specific
28+
Agent such as DSH can be registered explicitly with a command and argv.
29+
Missing ACP bridges remain visible as unavailable rather than being treated
30+
as working Agents.
31+
- AC-5: Existing single `--acp-agent <command>` startup remains compatible and
32+
becomes the default selection. Preset discovery is bounded, deterministic,
33+
and portable across Windows, macOS, and Linux.
34+
- AC-6: Run evidence and lane headers identify the selected Agent separately
35+
from the requested model. A mixed-Agent comparison is labelled as such and
36+
does not claim that Agent identity was held constant.
37+
- AC-7: Keyboard focus, live status, inline result overflow, and the Agent
38+
controls remain usable at 1440x900, 1024x768, and 390x844 with no
39+
document-level horizontal overflow or browser console/page errors.
40+
41+
## Non-goals
42+
43+
- Install ACP adapters or Coding Agent CLIs from a browser action.
44+
- Treat a plain CLI or SDK as ACP when it does not expose an ACP stdio server.
45+
- Accept a command, argv, environment, path, or package name from the run API.
46+
- Infer file access, edits, verification success, or model quality beyond the
47+
retained canonical Tool Call and runtime evidence.
48+
- Add a new Coding Agent host support claim to the repository host matrix.
49+
50+
## Plan and Tasks
51+
52+
1. Add a server-owned ACP Agent catalog with stable ids, preset discovery, an
53+
explicit named-registration CLI contract, and backward-compatible single
54+
Agent handling.
55+
2. Project the catalog into the experiment preview, accept one selected id per
56+
execute lane, validate the bounded mapping, and select the matching executor
57+
at lane creation time.
58+
3. Record and stream the selected Agent identity without replacing the lane's
59+
model identity or leaking executable details.
60+
4. Add lane Agent selectors to the simple composer and place the selected Tool
61+
Call result inline in the originating resource row.
62+
5. Add focused contract, server, model, component/browser, portability, and
63+
real-Agent tests; then run a read-only qodercli review before commits.
64+
65+
## Test and Review Evidence
66+
67+
- AC-2/AC-3/AC-5: server and CLI tests cover catalog projection, backward
68+
compatibility, unknown/unavailable ids, browser command omission, and
69+
lane-specific executor selection.
70+
- AC-4: discovery tests simulate POSIX and Windows PATH/PATHEXT behavior without
71+
shell commands; local smoke records which preset executables are available.
72+
- AC-1/AC-6: component/model tests assert that one selected operation owns its
73+
inline result and that Agent and model labels remain distinct.
74+
- AC-3/AC-6: a two-lane fixture run proves two different registered commands
75+
are selected by lane and retained in canonical run evidence.
76+
- AC-7: Playwright screenshots and overflow/focus/error checks at the three
77+
Studio layout widths.
78+
- Risk: a preset name can overstate support. Availability requires the actual
79+
ACP entrypoint or explicit server registration; underlying CLI presence alone
80+
is insufficient for Pi, Claude, or DSH.
81+
- Risk: dynamic Agent selection adds a treatment axis. The UI and compare-set
82+
metadata must keep that confounder visible rather than presenting a pure model
83+
comparison.
84+
- Risk: browser-controlled process launch could become command injection. The
85+
request contains allow-listed ids only and process launch continues to use an
86+
argv array owned by the server.
87+
88+
### Implemented evidence
89+
90+
- Harness, Harness UI, and Studio suites pass 172/172, 31/31, and 291/291;
91+
generated Langium sources are current and the documentation link graph passes
92+
8/8 checks.
93+
- The Studio Playwright surface passes 10/10 checks across wide, compact, and
94+
390px layouts, including one roving Tab stop for Resources/Messages, inline
95+
result ownership, bounded overflow, and browser console/page-error checks.
96+
- A live mixed-Agent run selected `qodercli --acp` for AI 1 and `codex-acp` for
97+
AI 2. Both isolated trials passed, both changed only `README.md`, and their
98+
evidence records `agent-default` separately from the lane-selected
99+
`gpt-5.5` model.
100+
- Local discovery reports Qoder CLI and Codex ACP available. Pi ACP, Claude ACP,
101+
and DSH remain visible but unavailable because their ACP entrypoints are not
102+
installed or explicitly configured on this host.
103+
- A read-only qodercli review found no P1 issues. Its two P2 findings were fixed:
104+
already-exited Agent cleanup no longer waits through two grace periods, and
105+
experiment-wide preflight SSE failures are now visible in the Simple UI.

0 commit comments

Comments
 (0)