Skip to content

feat(eval): add an --engine flag with capability aware selection - #47

Merged
theonly1me merged 1 commit into
mainfrom
feat/eval-engine-flag
Sep 6, 2026
Merged

feat(eval): add an --engine flag with capability aware selection#47
theonly1me merged 1 commit into
mainfrom
feat/eval-engine-flag

Conversation

@theonly1me

Copy link
Copy Markdown
Owner

What changed

  • Added shadowclone eval --engine <id>, validated against the shared engineIds list with a clear error naming the known engines.
  • Extracted engine selection into src/eval/engine.ts as selectEvalRunner, keeping src/eval/run.ts under the line limit.
  • A rejected engine now says why: not installed, not authenticated, or unable to enforce a budget and a granular tool policy.
  • Exported engineIds from src/config so the CLI validates against one list rather than a copy.

Why

runEval hardcoded detectEngine({ purpose: "dispatch" }) with no way to choose, so the engine was whichever qualified first. On a machine with several authenticated CLIs there was no way to say which one should run the eval, and a failure printed a single generic line.

--engine narrows selection, it does not override the gate. A named engine still has to be permitted by managed policy, be authenticated, and support the purpose. Asking for one that cannot enforce a budget and a granular tool policy fails with that reason rather than silently falling back.

How to verify

bun run check
bun run cli eval --engine antigravity --sessions 1

The second command exits with The antigravity engine cannot run eval because it is not installed, since no Antigravity runner exists. --engine bogus lists the known engines.

selectEvalRunner takes the existing CommandProbe, so src/eval/engine.test.ts runs hermetically in 9ms instead of shelling out to every CLI.

@theonly1me
theonly1me merged commit a606fbd into main Sep 6, 2026
7 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant