feat(eval): add an --engine flag with capability aware selection - #47
Merged
Conversation
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
What changed
shadowclone eval --engine <id>, validated against the sharedengineIdslist with a clear error naming the known engines.src/eval/engine.tsasselectEvalRunner, keepingsrc/eval/run.tsunder the line limit.engineIdsfromsrc/configso the CLI validates against one list rather than a copy.Why
runEvalhardcodeddetectEngine({ purpose: "dispatch" })with no way to choose, so the engine was whichever qualified first. On a machine with several authenticated CLIs there was no way to say which one should run the eval, and a failure printed a single generic line.--enginenarrows selection, it does not override the gate. A named engine still has to be permitted by managed policy, be authenticated, and support the purpose. Asking for one that cannot enforce a budget and a granular tool policy fails with that reason rather than silently falling back.How to verify
bun run check bun run cli eval --engine antigravity --sessions 1The second command exits with
The antigravity engine cannot run eval because it is not installed, since no Antigravity runner exists.--engine boguslists the known engines.selectEvalRunnertakes the existingCommandProbe, sosrc/eval/engine.test.tsruns hermetically in 9ms instead of shelling out to every CLI.