Skip to content

fix(eval): strip leading slash commands and surface replay errors - #43

Merged
theonly1me merged 4 commits into
mainfrom
fix/eval-slash-commands-and-error-surfacing
Sep 6, 2026
Merged

fix(eval): strip leading slash commands and surface replay errors#43
theonly1me merged 4 commits into
mainfrom
fix/eval-slash-commands-and-error-surfacing

Conversation

@theonly1me

Copy link
Copy Markdown
Owner

What changed

  • Strip leading slash commands such as /plan, /goal, and /boost from extracted prompt text while preserving Unix file paths.
  • Detect synthetic local command exits in parseClaudeStream and mark isError: true so failed command invocations are not treated as valid zero-turn runs.
  • Parse Claude CLI error arrays and subtype flags into errorMessage on EngineRun rather than falling back to a generic message.
  • Capture process stderr in runClaudeCode when an engine subprocess exits with a non-zero exit code.
  • Extract single-session replay into src/eval/replaySession.ts to keep src/eval/run.ts within line count limits.
  • Log prompt snippets and failure reasons when a session is skipped, and record skippedSessions in EvalReceipt.
  • Report skipped session counts in the shadowclone eval terminal summary.

Why

Prompts recorded from Antigravity frequently begin with slash commands like /plan. When piped to claude -p, Claude Code CLI treats leading slash tokens as interactive commands, printing <local-command-stdout>/plan isn't available in this environment.</local-command-stdout> and exiting with code 0. Because is_error was false, parseClaudeStream treated this as a successful run with zero actions, yielding 0% delta with an artificial 0.5 planning score. Stripping UI slash commands ensures the prompt text executes as the intended task.

In addition, replay errors were swallowed across multiple layers: runClaudeCode discarded stderr, parseClaudeStream ignored the errors string array on the result payload, and runEval printed a single generic line without the prompt or failure reason. Capturing errors across the runner and recording skippedSessions in receipts makes replay failures actionable.

How to verify

bun test src/eval/prompt.test.ts
bun test src/engine/parseClaude.test.ts
bun test
bun run check
bun run ./src/cli/index.ts eval --sessions 1 --max-budget-usd 0.50 --yes

Live validation on session 001bb52a-41b1-4c32-94aa-dee90ad403ae confirmed that /plan was stripped, Claude executed real tool calls (Bash, ToolSearch, WebFetch), and the session was scored in receipt 0b4a786f-b641-4bf5-bf80-f88910dd67f1.json.

Data handling

  • Any new capture source has an opt-in flag and a README entry in this PR
  • Everything that reaches the network passes redactSecrets
  • No raw capture in a log line, an error message, or a test fixture
  • A test exercises the real entry point, not just the redaction function

@theonly1me
theonly1me merged commit 439c889 into main Sep 6, 2026
7 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant