feat: PDF-only evolution infrastructure + evolver prompt overhaul#23
Open
dastin-sentient wants to merge 35 commits into
Open
feat: PDF-only evolution infrastructure + evolver prompt overhaul#23dastin-sentient wants to merge 35 commits into
dastin-sentient wants to merge 35 commits into
Conversation
…ules - src/loop/trace_db.py: SQLite-backed trace storage with progressive disclosure (writes individual .md files per trace, generates lightweight index) - src/loop/reviewer.py: Background reviewer using harness.Agent with Haiku, extracts insights from successful solver traces into runtime_proposals table - src/loop/constraints.py: Skill constraint gating (size, growth, YAML, binary, answer leakage) — runs before expensive evaluation to discard bad mutations - src/tracing.py: Phoenix OTel tracing init (re-added from prior branch)
The unified evolver combines proposer + generator into a single Opus pass (cheaper than the split, and implementation is informed by full trace context). - src/schemas/skill_evolver.py: SkillEvolverResponse (action, skill_name, description, justification) - src/agent_profiles/skill_evolver/: uses build_options for harness-agnostic SDK support - Schemas and agent_profiles __init__ export the new names
…fields - src/harness/agent.py: Agent class now takes name: str = "agent" for clearer OTel span labels and downstream logging - src/schemas/agent.py: AgentResponse.final_answer Field description enforces clean atomic values (number/name/date) — prevents "approximately 18" leaking into scoring - src/loop/config.py: adds skill_unified evolution mode, cost_metric type, accuracy_threshold, and reviewer_enabled
- src/loop/helpers.py: failures are now 5-tuples including the question text.
build_proposer_query accepts past_traces_index and runtime_proposals sections,
and tells the evolver to Read specific trace files on demand.
- src/loop/runner.py:
* init TraceDB and BackgroundReviewer (auto-disabled on error)
* detect validation-train overlap (tiny datasets) and defer base eval
* failures carry the question for DB queries (5-tuple)
* persist every solver trace to DB + individual .md file
* fire reviewer fire-and-forget only when no failures (evolver handles failures)
* inject lightweight index + unconsumed runtime proposals into evolver query
* add skill_unified branch using SkillEvolverResponse
* phase 2 efficiency: once accuracy >= threshold and all pass, treat successful
traces as "expensive successes" and run evolver on them
* run constraint gate on all non-meta skills before commit
New flags: - --fresh: wipe program branches, frontier tags, feedback, checkpoint, trace DB - --reviewer_enabled: toggle background reviewer (default True) - --evolver_model: separate model for evolver (default opus) - --accuracy_threshold: enable phase 2 efficiency optimization - --data_root: configurable agent cwd/add_dirs (replaces previous hardcoding) - --mode skill_unified: opt into the unified evolver Also: - Agent instances get explicit name= for better span labels - skill_evolver agent built conditionally when mode == skill_unified - Phoenix tracing initialized at entry (init_tracing) - Fixed imports: Agent and set_sdk from src.harness (canonical path)
- tests/test_loop.py: all build_proposer_query calls updated to 5-tuple format (trace, agent_answer, ground_truth, category, question) - tests/test_opencode_skill_context.py: same 5-tuple migration - tests/test_trace_db.py: covers insert, progressive disclosure file write, index generation with cross-question context - tests/test_constraints.py: covers size/growth/frontmatter/binary/leakage gates All 468 tests pass (excluding 2 pre-existing failures on opencode_fix_clean unrelated to this integration, and test_opencode_runtime which has a pre-existing import error).
Previously Phoenix auto-instrumentation failures (e.g., wrapt/openinference version mismatch) would raise at entry and block the entire run. Now we: 1. Try full auto_instrument mode (default) 2. Fall back to manual-spans-only mode on failure 3. Silently continue with no tracing if Phoenix itself is broken The loop must run even without observability.
Previously the executor silently collected messages — no Phoenix spans beyond auto-instrumentation (which is fragile across wrapt versions) and no terminal progress. Added manual instrumentation so observability works regardless: - agent.run:<name> parent span wrapping the full query - <name>/turn.<N> spans per AssistantMessage - <name>/tool.<toolname> spans per ToolUseBlock - Prints "turn.N [model]: ToolName" or text preview to stdout for live feedback Agent now threads its .name attribute into the Claude executor so spans and logs are labeled (e.g., base/turn.5, skill_evolver/tool.Edit). Other executors (OpenCode, OpenHands, Codex, Goose) unchanged — they don't receive the agent_name kwarg.
…atch/ Previously --data_root was passed as the agent's cwd, so any Write/Bash from the solver polluted the officeqa data folder (e.g., scratch/debug_*.png). Fix: - cwd stays at EvoSkill project root (owns skills, cache, scratch) - --data_root is threaded into add_dirs, giving READ access without making it the working directory - System prompt explicitly tells the solver: data dirs are read-only, scratch goes into .cache/scratch/ within cwd
Previously all spans (turns + tools) appeared as flat siblings under agent.run:<name> because start_span() without explicit context picks up the outermost active span. Now we explicitly thread context: turn spans are children of run_span, and tool spans are children of their turn_span. Phoenix tree view now shows the expected hierarchy.
Previously tool inputs and text were capped at 300 chars via _preview, so long Bash scripts and Read-tool outputs were truncated in the Phoenix UI. Now: - tool.input contains the complete JSON-serialized input - text contains the full assistant text block - Raised OTel attribute length limit to 1MB (default was ~12KB) - Terminal print still uses short preview for at-a-glance visibility
1. All improver agents (skill_proposer, skill_generator, prompt_proposer, prompt_generator, skill_evolver) now default to opus. The --evolver_model flag overrides this single knob for all of them. --model only affects the solver/base agent. Rationale: the improvers benefit from stronger reasoning; the solver's model choice (sonnet/haiku) shouldn't drag them down to the same tier. 2. agent.run span captures the full query (agent.query attribute) instead of a 200-char preview. Evolver queries include failure traces + past trace index + runtime proposals — truncating them in Phoenix hid the actual context the evolver sees.
Phoenix displays input/output columns based on OpenInference semantic conventions. Our custom agent.query attribute wasn't picked up: - agent.run:* spans: openinference.span.kind=AGENT, input.value=<full query>, output.value=<final result string>, agent.total_cost_usd - turn spans: openinference.span.kind=CHAIN - tool spans: openinference.span.kind=TOOL, input.value=<full tool input JSON> Phoenix table and detail views now show the complete evolver prompt. Also defaulted --mode to skill_unified so the combined evolver runs out of the box (previously defaulted to skill_only which fires proposer + generator).
Previously only ToolUseBlock and TextBlock were handled — turns containing only extended thinking appeared empty in Phoenix (just model + turn attrs). Now: - ThinkingBlock: logged to 'thinking' span attribute + printed to terminal with "(thinking)" marker - Unknown block types: stored as 'unknown_block.<TypeName>' attribute (capped at 1000 chars) and printed with type name ThinkingBlock import is guarded (ImportError -> None) for older SDK versions.
Previously, individual turn spans showed no input/output (only model/turn attrs). Tool spans closed immediately with no output. You had to click the parent agent.run span to see the query, and tool results were invisible. Now each span shows what it did: - agent.run:<name>: input.value = full query, output.value = final result - <name>/turn.1: input.value = query (so drilling into turn.1 shows the question) - <name>/turn.N: output.value = concatenated text + tool call summary - <name>/tool.X: output.value = the tool result string, tool.is_error flag Tool spans now stay OPEN until the matching ToolResultBlock arrives (tracked by tool_use_id -> span map), so latency reflects actual tool execution time. Defensive cleanup at end closes any spans that didn't receive a result.
Phoenix hides OpenInference semantic attributes (input.value, output.value)
from the "All Attributes" pane — they render in the dedicated Input/Output
section instead. When users look at "All Attributes" they don't see them.
Set the same values under plain names ('query', 'input', 'output') that
will always appear in "All Attributes" regardless of Phoenix's filtering.
The semantic names stay for proper Input/Output pane rendering.
The skill evolver mutates files in-place during its run (Edit/Write tools
with acceptEdits permission). Reading .claude/program.yaml afterward can
fail if the working tree shifted.
Fix: read get_current() BEFORE the evolver runs, stash it, then use it to
create the child branch after the evolver completes its edits.
Before this fix, skill_unified runs crashed with:
FileNotFoundError: .claude/program.yaml
at manager.get_current() after evolver_trace completed.
Tool spans for the Skill tool were appearing empty in Phoenix. Making the tool-span path robust: - Handle missing/None block.name and block.input gracefully - Serialize block.input with default=str so unusual types don't fail silently - Always emit block.type and block.repr for debugging when a tool's shape is unexpected - Cast tool_use_id to str on both emit and lookup sides (so dict keys match) - When no id is available, set tool.output to an explanatory marker instead of leaving it blank
Previously summarize() only rendered the final ResultMessage.result string
(the last assistant text). Thinking, tool calls, tool results, and
intermediate reasoning were lost when the trace was persisted to the DB or
the per-question trace .md files.
Now summarize() walks self.messages and renders every block type per turn:
--- Turn 1 ---
(thinking): let me check the ToC first
(tool.Read): {"file_path": "...", "pages": "6-9"}
(tool.result): PDF pages extracted...
--- Turn 2 ---
text: Report page 5 is at PDF page 19
(tool.Read): {...}
(tool.result): ...
Tool results correlate with tool uses by tool_use_id (first pass builds map,
second pass renders inline). Per-result cap (default 4KB) prevents context
blowup on large file reads.
Fallback to the old "Full Result" view if messages are empty or the SDK's
block types can't be imported (non-Claude harnesses).
This finally makes the trace DB useful to the evolver — it can see what
the solver actually did on past iterations, not just its final answer.
wrapt 2.x removed the `module=` keyword from wrap_function_wrapper() (breaking API change). openinference-instrumentation-anthropic 1.0.0 still uses the old call form, so auto-instrumentation crashed with: TypeError: wrap_function_wrapper() got an unexpected keyword argument 'module' Our tracing.py catches this and falls back to manual-spans-only mode, but that loses all the LLM-level spans from auto-instrumentation. Pinning wrapt<2.0 restores the old API and lets auto-instrumentation work again. Can be removed once openinference publishes a release compatible with wrapt 2.x.
to current failures (prevents context bloat on larger datasets)
Bug fix
-------
_evaluate() passed result.ground_truth directly to the scorer. When the
CSV's ground_truth column is numeric, pandas hands us int64/float64, not
str. The scorer's fuzzy_match_answer then crashes on
`text.replace(...)` with AttributeError. Wrap in str() like the training
path already does.
Context bloat fix
-----------------
The evolver query inlined the full turn-by-turn transcript for each
current failure (~60KB/failure, dominated by thinking blocks). At
failure_samples=3 that's ~180KB per call, and grows with dataset size.
Now the evolver query contains, per failure:
- Question, agent answer, ground truth (short)
- Tool-call skeleton: just tool names in turn order, no thinking/
results (few dozen bytes)
- Reference to .cache/current_failures/<file> for the full transcript
Evolver reads specific transcripts on demand via the Read tool (same
pattern as past traces). Task instructions updated to point at the new
files and to prefer Write over Bash mkdir.
_fresh_reset also wipes .cache/current_failures/ now.
Two related gaps this closes:
1. Temporal causality for skills
Skills evolve across iterations. When the evolver later reads a past
trace that says "(tool.Skill): chart-counting", it had no way to know
what that skill's content WAS at the moment the solver invoked it —
only what the skill looks like NOW, possibly 5 edits later.
Now every trace persisted to the DB also embeds the exact contents
of each active SKILL.md at that iteration. The trace .md file has a
new "Active Skill Snapshots" section with the full markdown each
skill had when the solver ran.
This lets the evolver answer "what guidance was the agent following
when it failed iter-3?" vs "what changed that made iter-5 succeed?"
from evidence, not guesswork.
2. Comprehensive past-traces index
generate_index() previously prioritized failed_questions only and
sampled 5 "other" traces. Now it shows ALL persisted traces (up to
limit=200 by default), grouped by question, sorted chronologically
within each question.
The index is still compact — just metadata and file paths. The full
transcripts and skill snapshots live in the per-trace .md files that
the evolver reads on demand via the Read tool.
New in runner:
- _snapshot_active_skills() reads current SKILL.md contents and hands
them to trace_db.insert()
- insert() now accepts active_skill_contents and embeds them
Index column format:
| Iteration | Score | Turns | Active Skills | File |
…edup
Three related improvements to keep the evolver's context from exploding
as training-set size and iteration count grow.
A. Priority selection (per_question_cap=5)
When a question has more than K iterations, show only representative
rows: first (baseline), latest, best score, worst score, biggest score
change. Gaps are then filled to reach the cap. All rows stay in
chronological order. This caps per-question table size regardless of
iteration count.
B. Score trajectory line
Before each table, a compact one-liner showing ALL iterations of that
question:
Trajectory (10 iters): iter-1:0.00 → iter-2:0.10 → ... → iter-9:0.85
— best: iter-9 @ 0.85
Gives the evolver the full arc at a glance so it can decide which
specific iterations warrant a deeper Read.
C. Content-addressed skill snapshot dedup
Previously every trace embedded the full content of every active
skill, duplicating bytes across iterations where the skill was
unchanged. Now snapshots are stored by content hash at
.cache/skill_snapshots/<skill>_<hash>.md — 10 iterations using 3
distinct skill versions produces 3 snapshot files, not 30.
Trace files reference snapshots by path; evolver Reads them on
demand.
Result for 13 traces / 2 questions / 3 skill versions: index is 2.2KB
total, with full trajectory visible and 5 representative rows per
question. Scales sub-linearly with iteration count.
Side effects:
- _fresh_reset wipes the new .cache/skill_snapshots/ directory
- generate_index() new params: per_question_cap=5, limit=500
- New module helpers: _iter_num(), _select_representative_rows()
When the solver reads the same file multiple times in one run (or any tool returns identical output across turns), the transcript previously stored each copy in full — 4KB × N duplication per trace. With large PDFs being re-read 5-10 times this easily hit 40KB of redundant bytes in a single trace file. Now tool results are hashed (sha256 of the rendered text) as they're rendered. On first sight, content goes in as before. On subsequent matches the entry becomes: (tool.result): [identical to turn K's <ToolName> result — N chars omitted] Pointing at the earliest occurrence preserves the evidence that "the solver re-read the same file" — a diagnostic signal the evolver can use — without paying for duplicate bytes. Threshold: only dedup results ≥ 120 chars. Tiny results (single-line "hi", "ok", short error strings) aren't worth the indirection noise. Tested: 3 identical 542-char PDF reads compressed from ~1.5KB to ~1KB, while a trivial "hi" result passed through untouched.
…ties Documents the production launch incantation (env wipe, .env sourcing, nohup detach, key flags) so external users can reproduce experiments without reverse-engineering scripts/run_loop.py. Adds the inference-cache key explanation, workspace-separation rationale, Phoenix setup, and the .dataset/ helper scripts (audit compilation, hard/baby pool builders). Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
The "base agent" terminology was overloaded: it referred to both the
agent that solves benchmark questions AND the unevolved starting program
("base program") in the evolution loop. Renames the agent-related
identifiers to "solver" so traces, logs, and future code unambiguously
distinguish the two concepts. Span names already say "solver" — this
brings the module/function/symbol layer into alignment.
* Moved src/agent_profiles/base_agent/ → src/agent_profiles/solver/
* Renamed base_agent.py → solver.py
* Renamed BASE_AGENT_TOOLS → SOLVER_TOOLS,
{get,make,_build}_base_agent_options{,_from_task} → *_solver_options*,
base_agent_options → solver_options
The "base program" terminology in log strings (e.g., "Base score:",
parent="base", frontier="base") is unchanged because it refers to the
unevolved starting program — a distinct concept that should stay.
Imports updated across src/, scripts/, tests/.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
The base_agent → solver rename in 929e3f5 lost prompt.txt because .gitignore line 27 (*.txt) hid the renamed file from `git status`. The original base_agent/prompt.txt was force-added long ago, so the rename split into a deletion + an ignored add — only the deletion got staged. Force-add the file so the solver agent can read its system prompt at runtime. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
… cleanup Three cache-correctness bugs that caused stale or wrong results: 1. **Tree-hash didn't include the LIVE skills directory.** When workspace ≠ project source root, the cache key hashed the workspace's .claude/skills/ (which has only stub .disabled files) instead of the project's. Result: same cache key across iterations even as evolved skills changed → cache returned responses generated under DIFFERENT skill conditions → mid-gate produced bogus "fixed=0, regressed=0, same=N" artifacts. Adds `live_skills_dir` and `project_source_root` to CacheConfig and threads them through. 2. **`program.yaml` embedded a wall-clock timestamp.** Per-run `created_at` field made every fresh run produce a different yaml blob, busting the run-cache key on every `--fresh`. Drop `created_at` from base_metadata in options_to_config + ProgramConfig.child. (Same fix as PR sentient-agi#32, intentionally duplicated here so this bundle stands alone if sentient-agi#32 is merged separately.) 3. **Workspace branches left over from GATE-rejected mutations.** When mid-gate or val GATE discarded an iter, the workspace's program/iter-skill-N branch remained, crashing the next iter's `git checkout -b program/iter-skill-N` with "branch already exists". Adds defensive cleanup in ProgramManager._git_checkout_new plus skill-tree snapshot/restore so historical iters reproduce. Files: src/cache/run_cache.py, src/registry/{models,sdk_utils,manager}.py Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Bundles cohesive runtime improvements to the self-improving loop. Hard
to split further because most changes touch runner.py + executor.py +
agent.py simultaneously to wire context across modules.
**Mid-gate (cheap pre-val verification)** — re-runs the iter's train
samples on the newly-evolved program before paying for full val. If the
mutation didn't fix at least min_fixed train samples or regresses more
than max_regressions, discard without running val. Saves ~$25 of val
cost per failed mutation. Configurable via --mid_gate / --mid_gate_min_fixed
/ --mid_gate_max_regressions. Files: src/loop/{runner,config}.py,
scripts/run_loop.py.
**Pause/resume + clock-freeze** — `touch /tmp/evoskill_pause` halts the
loop at the next turn boundary; removing the file resumes within ~1s.
On resume, the wall-clock budget consumed during the pause is REFUNDED
to the active asyncio.timeout via tm.reschedule(when + elapsed), so a
30-min coffee break doesn't make a healthy run trip the 12-min cap.
New file: src/harness/pause.py. Wired via active_timeout contextvar in
src/harness/agent.py.
**Adaptive timeout extension** — the bundled CLI's API client retries
hung HTTP requests internally (API_TIMEOUT_MS=120s ×
CLAUDE_CODE_MAX_RETRIES=5). When the executor's stderr observer sees
"(attempt N/K)" banners, it extends the per-agent asyncio.timeout by
+120s, refunding the time the retry consumed. Closure-captured tm +
max_seen via ContextVar means each concurrent agent's extensions only
apply to itself. Worst-case effective budget: 12 + 2×5 = 22 min.
Files: src/harness/agent.py, src/harness/claude/executor.py.
**Phoenix span improvements** — per-iter prefix on every span
(`iter 3 | solver [val 12/15] [FAIL 0.79]`), score stamped onto run
spans post-trace via `eval_score_callback` contextvar, dropped the
noisy `agent.run:` prefix and the auto-instrumented `messages.create`
spans (auto_instrument=False). Per-turn spans now include the iter
prefix too. New contextvars in src/harness/utils.py:
eval_iter_label, eval_score_callback. Files: src/tracing.py,
src/harness/claude/executor.py, src/loop/runner.py (sets iter label).
**Cost tracking fix** — when all 3 train samples passed, the loop's
"continue" before accumulating _iter_cost into _total_cost made the
final report show "Total cost: $0.0000" after a real $30 run. Cost
now accumulated in the early-return branch.
**Train-time cache wrapper** — _run_one_train called agent.run()
directly instead of through the cache wrapper. Same UID re-evaluated
against the same parent program in subsequent runs paid full price.
Wrapped with the same cache.get/set pattern as evaluate_agent_parallel.
**Auto-cache for one-shot eval scripts** — evaluate_agent_parallel's
`cache` parameter now defaults to a sentinel `_AUTO_CACHE` that
auto-constructs a default RunCache. Existing callers passing
`cache=None` explicitly preserve no-cache behavior. Makes ad-hoc eval
scripts cache-by-default without each one having to remember to
construct a RunCache.
**Live tqdm postfix with running mean** — replaces tqdm_asyncio.gather
with manual asyncio.as_completed + tqdm + set_postfix, so the eval bar
shows `avg=0.XXX pass=N/M` updating per sample. Reads scorer from the
eval_score_callback contextvar set by the runner.
Files: src/loop/{runner,config,helpers,reviewer,trace_db}.py,
src/harness/{agent,utils,pause,claude/executor,claude/options}.py,
src/tracing.py, src/evaluation/{evaluate,eval_full}.py,
scripts/run_loop.py, src/cli/{commands/run,commands/init,config}.py.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Two changes:
1. **Prompt moved from prompt.py to prompt.md.** Editing a 2KB skill
spec inside a Python triple-quoted string was awkward — no markdown
preview, no syntax highlighting on the spec itself, and every edit
produced a Python diff. The .md file is read at agent-construction
time (same pattern as src/agent_profiles/solver/prompt.txt).
2. **Required YAML frontmatter for emitted SKILL.md.** The evolver
sometimes emitted SKILL.md without the `--- name/description ---`
header, which the GATE rejected → loop retried 3x with no fix →
wasted iterations. Added an explicit "REQUIRED file format" block
to the prompt so the model produces frontmatter on the first
attempt.
Files: src/agent_profiles/skill_evolver/{prompt.md,prompt.py,skill_evolver.py}
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Adds the OfficeQA Pro benchmark integration on top of the
task-universal evolution loop:
- **src/agent_profiles/officeqa_agent/** — domain-specific solver
prompt (PDF parsing, Treasury Bulletin context). Used by
scripts/run_loop.py when running on OfficeQA. The generic
src/agent_profiles/solver/ remains task-universal.
- **src/evaluation/officeqa_judge.py** — LLM-as-judge scorer for
OfficeQA's open-ended numeric/text answers. Handles unit
conversion, rounding tolerance, and equivalent phrasings.
- **scripts/eval_oneshot_unseen.py** — one-shot solver evaluation
over unseen UIDs. Writes per-question scores to oneshot_scores.csv
and failures to oneshot_failures.csv. Populates the run cache so
a subsequent evolution launch hits cache for these UIDs.
- **scripts/analyze_failures.py** — post-hoc failure analysis on a
saved eval pkl: groups by error pattern, surfaces the most common
failure modes for the evolver to target.
- **scripts/build_pro20_split.py / build_pro20_split_manual.py** —
build train/val splits from the OfficeQA Pro 20-question subset
(the 85 image-verified-correct UIDs).
- **scripts/llm_judge.py** — standalone CLI for running the LLM
judge over a (predicted, ground_truth) CSV.
- **scripts/run_skill_check.py** — sanity-check harness that runs
the solver agent on a single UID with a specific skill loaded,
for debugging skill effectiveness in isolation.
These are deliberately OfficeQA-specific — kept out of the
task-universal evolver per project constraint that
agent_profiles/{solver,skill_evolver}/ remain benchmark-agnostic.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
OpenTelemetry instrumentation libraries (e.g., openinference-anthropic) don't yet support wrapt 2.x, which broke Phoenix tracing on fresh installs. Pin to wrapt < 2.0 in the lockfile until upstream catches up. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
…kills When `_restore_skills(name)` was called for a program whose snapshot didn't exist (because `_snapshot_skills` early-returned when the project skills dir was empty), it no-op'd and left the live skills tree alone. Concretely: after iter N's child mutation was discarded, the loop called `switch_to(parent)` to revert. If parent was `base` (which has no skills), no snapshot existed, so the discarded child's skill files persisted in `.claude/skills/` — and iter N+1's evolver would see them and "edit" a skill that was supposed to be gone. We observed this in a 4-iter run where iters 3 and 4 emitted `edit:numerical-precision` even though their parent was `base`, because `numerical-precision` from iter 2 (which had been discarded at mid-gate) was still on disk. Fix: `_snapshot_skills` always creates the snapshot directory, even when empty. An empty snapshot is the correct record for a program with no skills, and `_restore_skills` uses it to wipe the live tree back to empty. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
…e sets
The round-robin sampler indexed pools linearly: for a pool of size P
sampled k-at-a-time, iters at the same offset within consecutive
cycles drew identical samples. With P=10 and k=5, iters 1+3 had the
same train set, and iters 2+4 had the same train set — the evolver
was re-attacking the same failures every other iteration.
Fix: maintain a shuffled order of pool indices per category, and
reshuffle each time `_per_cat_offset` wraps modulo pool size. Seed
the per-cycle RNG with `f"{cat}:{cycle}"` so the order is
deterministic across replays but different across cycles. Sampling
goes through a new `_next_train_sample(cat)` helper that handles
both the index lookup and the on-wrap reshuffle, so both the
proportional and round-robin paths share the logic.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Validated via 4-iteration opus evo experiment on OfficeQA Pro strict-94:
baseline 37.5% → 68.75% on val-16 (+31pp), 92.6% on holdout-68 (5 regressions).
Cache correctness:
- Tree_hash uses skills-tree only (no root_tree drift from program.yaml mutations)
- Effort included in cache key (different effort levels don't collide)
- Null-output cache entries replay as-is (same Q-M-E-S-K = same outcome)
- cached_transcript rendered at cache.set time (replays preserve full tool-call history)
Agent trace improvements:
- Partial messages preserved on timeout via _partial_messages contextvar
- AgentTrace.cached_transcript field + summarize() fallback for cache replays
- Turn spans include question tag (e.g. [val 13/16]/turn.1) for Phoenix readability
- No Phoenix truncation on system prompt, user query, or any span attribute
- tqdm mininterval=10s for cleaner log files
Evolver prompt overhaul:
- Compact failure index table (replaces verbose inline multi-KB sections)
- Trace files: clean format (question → GT → answer → reasoning → transcript)
- Dynamic summary ("N of M failed: X timeouts, Y wrong") replaces static Phase label
- Suppress empty sections (skills, feedback hidden when iter-1)
- CREATE-only skill policy (new name per iteration for clear version tracking)
- Mid-gate explanation in system prompt
- Base prompt header cleaned (removed redundant instruction)
- Relative trace paths (not absolute)
Loop infrastructure:
- Mid-gate mean-score policy (tolerates noise-driven single regressions)
- append_feedback parent mkdir fix (survives git branch switches)
- .claude/.keep committed to workspace (persists across program branches)
- manager.py logger import fix (NameError on merge-conflict cleanup path)
- env parameter wired through build_options → ClaudeAgentOptions
PDF-only workspace:
- build_pdf_only_workspace() with read-only APFS clone of PDFs
- Per-run PDF clone refresh (agent always sees latest user edits)
- Dedicated eval + evo runner scripts for sonnet and opus experiments
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
dastin-sentient
changed the title
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
Experiment result: baseline 37.5% → 68.75% on val-16 (+31pp), 92.6% on holdout-68 (5 regressions from skill overhead on easy questions).
New mechanisms introduced
Mid-gate mean-score policy (
src/loop/config.py,runner.py)New
mid_gate_policyconfig field with two modes:"counts"(default, backward-compatible): requiresfixed ≥ NANDregressed ≤ Mper-sample binary pass/fail"mean": passes ifmean(post_scores) ≥ mean(pre_scores)across re-evaluated samplesThe mean policy tolerates a single noise-driven regression as long as the net average improves — better signal when LLM scoring is noisy on borderline answers (chart reads, multi-component answers). Set via
LoopConfig(mid_gate_policy="mean").Skills-only cache tree_hash (
src/cache/run_cache.py)_git_behavior_token()now hashes ONLY the.claude/skills/tree SHA — not the workspace root tree. Previously, non-behavior-affecting mutations (e.g., ProgramManager committing "Update score" to.claude/program.yaml) shifted the tree_hash and caused spurious cache misses between consecutive runs with identical agent config.Cached transcript at cache-write time (
src/cache/run_cache.py,src/harness/agent.py)SDK message objects don't roundtrip through Pydantic JSON serialization. New approach:
_render_turn_transcript()is called atcache.set()time (while SDK objects are still in memory) and the rendered string is stored ascached_transcript. On cache replay,AgentTrace.summarize()falls back to this field — so failure-trace files for the evolver contain actual tool-call history instead of empty transcripts.Partial messages on timeout (
src/harness/agent.py,src/harness/claude/executor.py)New
_partial_messagescontextvar accumulates every SDK message as it streams. OnTimeoutError, theAgentTraceis built with these partial messages instead ofmessages=[]. The evolver now sees what the solver was actually attempting before it ran out of time.Compact evolver prompt (
src/loop/helpers.py)Replaced verbose inline failure sections (~12KB for 6 failures) with a compact markdown index table (~800 bytes). The evolver sees answer/GT/trace-ref at a glance and chooses which trace files to Read for diagnosis. Trace files contain: question, ground truth, solver answer, full reasoning (no truncation), and complete turn-by-turn tool-call transcript.
CREATE-only skill policy (
src/agent_profiles/skill_evolver/prompt.md)The evolver always creates a new skill with a new name — no editing existing skills. Each name = one version = one iteration. The loop manages version history. This gives clean attribution in the iteration history (score changes are tied to named skills).
PDF-only workspace (
src/officeqa/workspace.py)build_pdf_only_workspace()creates a clean workspace with:treasury_bulletin_pdfs/symlink to a read-only APFS clone (chmod 555).claude/skills/for skill loading/writing.cache/scratch/for intermediate filesThe APFS clone is instant + zero extra disk (copy-on-write). The chmod prevents agent writes to the corpus.
envparameter for Claude Agent SDK (src/harness/claude/options.py,utils.py)New
env: dict[str, str]parameter onbuild_options()→ forwarded toClaudeAgentOptions(env=...). Injects environment variables into the agent's Bash runtime without exposing literal paths in the system prompt.Test plan
🤖 Generated with Claude Code