Skip to content

feat(eval): add automatic transfer evaluations for codex and claude code - #48

Merged
theonly1me merged 3 commits into
mainfrom
feat/automatic-transfer-evals
Sep 6, 2026
Merged

feat(eval): add automatic transfer evaluations for codex and claude code#48
theonly1me merged 3 commits into
mainfrom
feat/automatic-transfer-evals

Conversation

@theonly1me

Copy link
Copy Markdown
Owner

What changed

  • Add the shadowclone eval transfer command to run automatic offline transfer evaluations for Codex (gpt-5.6-sol) and Claude Code (sonnet).
  • Implement isolated snapshot execution with lockfile dependency restoration, OS-level bubblewrap and sandbox-exec boundaries, and agent context capture.
  • Implement dual-pass blind LLM judging with position reversal and verification test execution.
  • Add structured Zod schemas for transfer receipts, task candidate preparation, and preference check validation.
  • Fix candidate selection to exclude training evidence without dropping eligible tasks, and resolve starting commits from session timestamps.
  • Add unit tests for candidate preparation, evaluation evidence collection, dual-pass judging, and Codex action streaming.

Why

Evaluating clone prompt transfer currently requires manual end-to-end task setup and subjective review across engines. Automatic transfer eval isolates historical tasks into reproducible git snapshots and measures behavioral adherence without leaking test session data into training profiles.

How to verify

bun run typecheck
bun test
bun run lint

Data handling

  • Any new capture source has an opt-in flag and a README entry in this PR
  • Everything that reaches the network passes redactSecrets
  • No raw capture in a log line, an error message, or a test fixture
  • A test exercises the real entry point, not just the redaction function

@theonly1me
theonly1me merged commit 4eb0e92 into main Sep 6, 2026
6 of 7 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant