Skip to content

feat(eval): add replay eval command, action fingerprints, and delta scoring (closes #14) - #36

Merged
theonly1me merged 2 commits into
mainfrom
feat/shadowclone-eval
Sep 6, 2026
Merged

feat(eval): add replay eval command, action fingerprints, and delta scoring (closes #14)#36
theonly1me merged 2 commits into
mainfrom
feat/shadowclone-eval

Conversation

@theonly1me

Copy link
Copy Markdown
Owner

What changed

  • Implement shadowclone eval command with --sessions, --since, --json, and --max-budget-usd flags.
  • Record action fingerprints in EngineRun from stream-json parsing without capturing prompt or tool payloads.
  • Normalize file paths to repository-relative posix paths in replay evaluation.
  • Exclude empty dimensions from the similarity mean and report per-dimension deltas between clone and baseline.
  • Persist evaluation receipts to ~/.shadowclone/eval/<evalId>.json for reproducible replication.
  • Document privacy-safe action representation principles in docs/design/eval-representation.md.

Why

Measuring whether cloned profiles reproduce developer behavior requires reproducible, falsifiable metrics. Comparing clone replays against an unprofiled baseline isolates the profile's true behavioral lift from shared agent base rates. Action fingerprints prevent evaluating full tool inputs that could re-expose sensitive code or secrets.

How to verify

bun run typecheck
bun test
bun run check

Data handling

  • Any new capture source has an opt-in flag and a README entry in this PR
  • Everything that reaches the network passes redactSecrets
  • No raw capture in a log line, an error message, or a test fixture
  • A test exercises the real entry point, not just the redaction function

Comment thread src/cli/eval.ts Outdated
Comment on lines +9 to +11
for (let i = 0; i < arguments_.length; i++) {
const arg = arguments_[i];
if (arg === "--json") {

Copy link
Copy Markdown
Owner Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Suggested change
for (let i = 0; i < arguments_.length; i++) {
const arg = arguments_[i];
if (arg === "--json") {
for (let argumentIndex = 0; argumentIndex < arguments_.length; argumentIndex++) {
const argument = arguments_[argumentIndex];
if (argument === "--json") {

Comment thread src/cli/eval.ts Outdated
}
}

await runEval({ sessions, since, json, maxBudgetUsd });

Copy link
Copy Markdown
Owner Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Spawning 40 unattended agent runs without a cost preview can burn user budget unexpectedly. We should print the upper bound cost and prompt for confirmation when running interactively.

Comment thread src/eval/run.ts Outdated

const baselineRun = await runner({
prompt,
cwd: os.tmpdir(),

Copy link
Copy Markdown
Owner Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Running both the baseline and clone directly in os.tmpdir() means they share the machine's top-level temp dir. We should run each in an isolated mkdtemp folder and clean it up after the session finishes.

@theonly1me
theonly1me force-pushed the feat/shadowclone-eval branch from 7f49e9e to 64c9bd6 Compare September 6, 2026 11:25
@theonly1me
theonly1me force-pushed the feat/shadowclone-eval branch from 64c9bd6 to a7ef966 Compare September 6, 2026 11:25
@theonly1me
theonly1me force-pushed the feat/shadowclone-eval branch from a7ef966 to 62b3937 Compare September 6, 2026 11:25
Base automatically changed from feat/staleness-guard-and-dry-run to main September 6, 2026 11:25
@theonly1me
theonly1me force-pushed the feat/shadowclone-eval branch from 62b3937 to 66af631 Compare September 6, 2026 11:25
@theonly1me
theonly1me merged commit 67705bb into main Sep 6, 2026
5 checks passed
@theonly1me
theonly1me deleted the feat/shadowclone-eval branch September 6, 2026 11:26
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant