Skip to content

fix(eval): use uuid session ids, extract prompt text, and drop structural telemetry on deep learn - #42

Merged
theonly1me merged 2 commits into
mainfrom
fix/telemetry-and-eval-runner
Sep 6, 2026
Merged

fix(eval): use uuid session ids, extract prompt text, and drop structural telemetry on deep learn#42
theonly1me merged 2 commits into
mainfrom
fix/telemetry-and-eval-runner

Conversation

@theonly1me

Copy link
Copy Markdown
Owner

What changed

  • Use valid UUIDs for Claude Code replay session IDs in runEval so sessions are not rejected by the CLI.
  • Extract user prompt text from raw JSON transcript envelopes in extractPromptText.
  • Capture stderr in runClaudeCode and surface engine errors if an eval run fails.
  • Drop raw structural telemetry templates from the compiled profile when shadowclone learn --deep runs.
  • Move score dimension averaging into src/eval/score.ts to keep src/eval/run.ts within the line limit.

Why

Claude Code exited immediately on non-UUID replay session IDs and transcript JSON strings were passed directly as prompts, causing eval to report zero delta without running. In addition, deep distillation was concatenating coarse structural templates with semantic rules instead of replacing them.

How to verify

bun run check
bun test src/eval/prompt.test.ts
bun test src/eval/run.test.ts

@theonly1me
theonly1me merged commit ce9fb74 into main Sep 6, 2026
7 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant