feat(eval): add replay eval command, action fingerprints, and delta scoring (closes #14) - #36
Merged
Merged
Conversation
theonly1me
commented
Sep 6, 2026
Comment on lines
+9
to
+11
| for (let i = 0; i < arguments_.length; i++) { | ||
| const arg = arguments_[i]; | ||
| if (arg === "--json") { |
Owner
Author
There was a problem hiding this comment.
Suggested change
| for (let i = 0; i < arguments_.length; i++) { | |
| const arg = arguments_[i]; | |
| if (arg === "--json") { | |
| for (let argumentIndex = 0; argumentIndex < arguments_.length; argumentIndex++) { | |
| const argument = arguments_[argumentIndex]; | |
| if (argument === "--json") { |
| } | ||
| } | ||
|
|
||
| await runEval({ sessions, since, json, maxBudgetUsd }); |
Owner
Author
There was a problem hiding this comment.
Spawning 40 unattended agent runs without a cost preview can burn user budget unexpectedly. We should print the upper bound cost and prompt for confirmation when running interactively.
|
|
||
| const baselineRun = await runner({ | ||
| prompt, | ||
| cwd: os.tmpdir(), |
Owner
Author
There was a problem hiding this comment.
Running both the baseline and clone directly in os.tmpdir() means they share the machine's top-level temp dir. We should run each in an isolated mkdtemp folder and clean it up after the session finishes.
theonly1me
force-pushed
the
feat/shadowclone-eval
branch
from
September 6, 2026 11:25
7f49e9e to
64c9bd6
Compare
theonly1me
force-pushed
the
feat/shadowclone-eval
branch
from
September 6, 2026 11:25
64c9bd6 to
a7ef966
Compare
theonly1me
force-pushed
the
feat/shadowclone-eval
branch
from
September 6, 2026 11:25
a7ef966 to
62b3937
Compare
theonly1me
force-pushed
the
feat/shadowclone-eval
branch
from
September 6, 2026 11:25
62b3937 to
66af631
Compare
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
What changed
shadowclone evalcommand with--sessions,--since,--json, and--max-budget-usdflags.EngineRunfromstream-jsonparsing without capturing prompt or tool payloads.~/.shadowclone/eval/<evalId>.jsonfor reproducible replication.docs/design/eval-representation.md.Why
Measuring whether cloned profiles reproduce developer behavior requires reproducible, falsifiable metrics. Comparing clone replays against an unprofiled baseline isolates the profile's true behavioral lift from shared agent base rates. Action fingerprints prevent evaluating full tool inputs that could re-expose sensitive code or secrets.
How to verify
bun run typecheck bun test bun run checkData handling
redactSecrets