feat: add engine-pluggable optimize_anything#346
Conversation
When run_dir is provided, save() now writes `gepa_state.json` alongside the existing pickle and JSON files. This single file gives autonomous agents (e.g. Claude Code) the full optimization picture: - All candidates with their texts, parent lineage, and metric call cost - Per-candidate per-datapoint scores (prog_candidate_val_subscores) - Per-candidate multi-objective scores - Pareto frontier details (instance, objective, hybrid, cartesian) - Derived analytics: best candidate, hardest examples, frontier members - Full iteration log including evaluation side_info/trajectories Also capture proposal evaluation data (eval_before, eval_after with scores, outputs, and trajectories/side_info) into the iteration trace, so agents can see WHY proposals were accepted or rejected. Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
- Serialize evaluation cache to gepa_state.json: per-candidate per-datapoint entries with score, output, and objective_scores (keyed by candidate idx and data id) - Add rejected_proposals section: each rejected proposal includes candidate text, parent ids, subsample data ids, before/after scores, and full eval_before/eval_after with trajectories/side_info - Move try_json_serialize to gepa_utils for shared use between engine and state export - Bump schema_version to 2 Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
Replaces the monolithic gepa_state.json export with a directory layout (candidates/, pareto/, iterations/, rejected_proposals/, eval_cache/) so agents can navigate run state without loading multi-MB blobs. Gated behind EngineConfig.write_agent_state / gepa.optimize(write_agent_state=), default off — when off, run_log.json / full_program_trace also stay lean (no per-iteration eval payloads or proposed_candidate copies). gepa_state.bin remains the single source of truth for resume; the tree is output-only.
… state When write_agent_state is enabled, valset evaluation now bypasses the eval cache and calls the adapter with capture_traces=True so trajectories come through. Each accepted candidate's outputs and trajectories are written to candidates/<idx>/outputs/<val_id>.json and candidates/<idx>/trajectories/<val_id>.json, covering both the seed and every reflective/merge acceptance. Drops the eval_cache/ export from the agent directory — the in-memory cache was not round-tripping trajectories and adds little over the other per-candidate files. Adds ValsetEvaluation.trajectories_by_val_id as the optional handoff channel from eval path to writer.
Introduces a unified Python API that dispatches a single optimization
call to any compatible backend (GEPA, Claude Code, Meta-Harness, …) over
a shared eval-server contract:
from gepa.omni import optimize_anything, Task
result = optimize_anything(
task=Task(name="t", initial_candidate="...", objective="..."),
evaluate=my_eval_fn,
backend="gepa",
config={"reflection": {"reflection_lm": "openai/gpt-5"}},
max_evals=200,
max_token_cost=5.0,
)
Design choices that drove the layout:
- Task is pure data (name, initial_candidate, objective, background,
train/val/test). No bundled eval_fn — code stays in Python.
- evaluate is a required, separate parameter; the eval server wraps it
and is the single budget/concurrency choke point.
- One Backend protocol — no Adapter sublayer. Backends call
server.evaluate (in-process) or POST to server.url (subprocess).
- max_token_cost is a single top-level field; the api threads it into
whichever per-backend knob enforces it (gepa: max_reflection_cost;
claude_code/meta_harness: --max-budget-usd).
- No Hydra inside gepa.omni — users call the function directly.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Replaces the ad-hoc kwarg sprawl on optimize_anything (max_evals,
max_token_cost, max_concurrency, output_dir, tracker, run_dir,
stop_at_score, effort, max_thinking_tokens, sandbox) with a single
typed OmniConfig dataclass:
config = OmniConfig(
backend="gepa",
max_evals=200,
max_token_cost=5.0,
run_dir="outputs/run1",
stop_at_score=0.95,
config={"reflection": {"reflection_lm": "openai/gpt-5"}},
)
optimize_anything(task, evaluate, config=config)
Cross-cutting fields (run_dir, stop_at_score, effort, max_thinking_tokens,
sandbox) are typed on OmniConfig directly. Backend-specific options live
in the free-form ``config: dict[str, Any]`` escape hatch.
Each backend now has a single construction path: ``__init__(config)``
reads cross-cutting fields directly and pops backend-specific keys from
``config.config`` onto self. ``run`` reads from self. The api never
injects attributes after construction. Backends warn about unknown keys
in ``config.config`` so typos surface immediately.
Other changes:
- Drop the dual from_config/__init__ split; one path only.
- Drop post-hoc threading of stop_at_score / effort / etc. — backends
read them off OmniConfig in __init__.
- optimize_anything always builds a timestamped default output_dir
(``outputs/omni/<task>/<backend>/<timestamp>/``) when unset.
- Drop run_dir auto-derivation from output_dir; explicit only.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
- Construct GEPA's nested dataclasses (EngineConfig, ReflectionConfig, MergeConfig, RefinerConfig) explicitly instead of relying on GEPAConfig.__post_init__'s dict-to-dataclass conversion. The runtime conversion is a feature, but the field annotations are dataclass-typed, so pyright (correctly) flagged the dict args. - Coerce gepa_result.best_candidate (str | dict[str,str]) to str before passing to omni's Result.best_candidate. We always feed a str seed through, so the runtime value is str — pyright just can't narrow it. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Datasets are now `list[Any]` (matching core GEPA's stance). EvalServer is
the single id authority: resolves `item.id` / `item["id"]` if present, else
assigns `f"{split}_{i}"`. Public `iter_split` yields `(id, item)` pairs that
backends and api consume so HTTP `example_id` lookups stay consistent.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
…ote) Compose multiple backends over the same task. Each helper takes the same ``(task, evaluate, configs: list[OmniConfig])`` shape as optimize_anything and calls it under the hood. Budgets are pre-partitioned per OmniConfig — no shared budget pool. - optimize_sequential: pipeline. Each backend's best becomes the next backend's seed. Monotonic — a regressing stage doesn't poison the chain (running best is fed forward). - optimize_parallel: run N backends concurrently, return the list of results in input order. Most flexible primitive; the others wrap it. - optimize_best_of: parallel + max(results, key=best_score). Uses each backend's own scoring during its run; no re-score pass. - optimize_vote: parallel, then re-score every best_candidate via ``evaluate`` (outside any budget) and pick the highest. Useful when backends have disjoint scoring quirks. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Ports the terrarium gepa_cc_agent into gepa.omni so the GEPA backend can run reflection through a sandboxed Claude Code subprocess instead of an LM callable. Plugged in via OmniConfig.config["claude_code_agent"]; the backend auto-enables write_agent_state so the proposer can read the iterations/, pareto/, and history.md tree off run_dir. Also factors copy_session_transcript into backends/claude_utils.py so both backends/claude_code.py and the new proposer share it. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
- sandbox: add macOS Seatbelt fallback (claude_settings_args) - claude_code: rewrite Strategy as explicit experiment loop, force tempdir when sandboxed, refactor program.md into composable helpers - meta_harness: rich colorized logging, propose/bench timing, val-split preference, failure summary rows, tracker integration, Seatbelt - gepa: install _ReflectiveDatasetDumpCallback when agent proposer is selected so iterations/NNNNN/reflective_dataset.json is browsable - claude_code_agent: append claude_settings_args to argv when sandboxed Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Splits the omni entry point in two: - optimize_anything(task, evaluate, config) — convenience. Builds EvalServer + BudgetTracker from config and runs (unchanged surface for existing callers). - optimize_anything_with_server(server, config) — primitive. Caller owns the EvalServer (construct + start + stop). Lets framework integrators (e.g. terrarium) keep their own outer eval server as the single source of truth for budget / persistence / tracking instead of nesting two servers. Both share a private _execute(server, backend, *, owns_server) helper that holds the actual run-loop + metadata + held-out test eval logic. The convenience path passes owns_server=True; the primitive trusts the caller's lifecycle. Backend.process_result now takes Path | None — when an EvalServer is constructed without an output_dir (typical for the primitive path), file-touching backends short-circuit instead of crashing. Updated the claude_code and meta_harness backends; gepa backend was already a no-op. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Two changes from a terrarium ↔ omni adapter audit:
1. backends/gepa.py: the dataset path was assigning task.test_set to
GEPA's valset. test_set is a held-out split for post-run eval and must
not enter the optimization loop. Switch to val_set, keeping
metadata["val_set"] as the secondary source.
2. backends/meta_harness.py: port two terrarium-side improvements:
- SKILL.md: add Step 0 post-eval reports, expanded anti-tuning
guidance with concrete mechanism examples, explicit budget-ownership
note, richer eval-trace instructions, and candidate / summary format
sections so the proposer no longer re-derives them.
- _parse_proposer_cost → _parse_proposer_result returning
(cost, payload); iter*_meta.json now also records the cmd and a
result_summary block (duration_ms, num_turns, usage, is_error,
subtype) so reviewers don't have to crack open stdout.json.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Both the claude_code and meta_harness backends now treat train+val as one combined "training" set: train+val are materialized together into work_dir/train/, scoring runs over the full pool, and the agents never learn that val existed. Test stays sealed for held-out reporting via the in-process Python API. claude_code: - drop validate.sh and the val-signal branch in _strategy_section / _rules_section / build_program_md; eval.sh is now the only signal. - eval.sh drops the positional split arg; only `./eval.sh <cand>` (full visible pool) and `./eval.sh <cand> --ids id1,id2` are accepted, so the prior `./eval.sh <cand> test` back-door is closed at the script layer. - materialize_sandbox writes train + val into train/ together; train_size in program.md reflects the combined count. meta_harness: - _score_candidate now evaluates on combined train+val ids (no more val- preferred / train-fallback ambiguity); _materialize_sandbox writes both splits into train/; _build_task_md drops the "hidden val set" wording. eval_server (HTTP): - /evaluate_examples now restricts callers to the agent-visible pool: the `split` parameter is rejected, and example_ids are validated against train+val (any id from test or unknown is 400). The default (no body fields) evaluates on the full visible pool. - /evaluate rejects example_id values outside the visible pool. - /task reports train_size = train+val combined and zeroes val_size / test_size so an HTTP probe cannot infer a sealed split exists. - The in-process evaluate_examples() API is unchanged so the runner can still score on the held-out test set for final reporting. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
frontier.json now records, in addition to the overall best, a
``per_example`` dict mapping each example id to the candidate that scored
highest on it (``{best_name, score, file}``). When different candidates
win different examples, the proposer can read both winners and design a
new candidate that fuses their mechanisms — same shape as the reference
text_classification example's per-task frontier.
- _update_frontier accepts per_example_scores and merges them in with a
strict-greater-wins rule (first writer keeps its slot on ties).
- The benchmark loop pulls per-example scores from the eval info dict
(skipped for single-task tasks where there's only ``_single``).
- SKILL.md Step 1 directs the proposer at the per-example frontier and
flags it as the strongest signal for combination candidates.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Mirror optimize_sequential / parallel / best_of / vote with primitive forms that take caller-owned EvalServer(s). Lets outer frameworks (e.g. terrarium) compose multiple omni backends while routing every inner eval through their own budget-tracking server. Sequential takes a single shared server (one budget pool); the parallel-family helpers take one server per config (caller pre-partitions the budget across N inner BudgetTrackers). Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Bare ``"sonnet"`` / ``"opus"`` aliases resolve CLI-side to whatever Anthropic currently points at, breaking experiment reproducibility across releases. Pin all three omni backend defaults (``claude_code``, ``meta_harness``, ``gepa``'s claude_code_agent proposer) to the versioned ID; the meta_harness backend also switches from opus to sonnet for parity. Aliases still work for callers who want auto-tracking of the latest default — explicitly pass ``model="sonnet"`` or ``model="opus"``. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Gepa's ``EngineConfig.cache_evaluation`` defaults to ``False``. Without caching, the reflective proposer's ``eval_curr`` re-evaluates the same candidate on the same minibatch every iteration the candidate is re-selected from the pareto frontier (and the seed in iteration 1, since iteration 1's curr_prog == seed). On single-instance tasks (terrarium per-problem frontier_cs) this multiplies real eval-budget spend by ~2-8× depending on selector behavior. Default to ``cache_evaluation=True`` in the omni gepa backend's engine_kwargs. Caller can override via ``config.engine.cache_evaluation=False`` to opt back out. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
When ClaudeCodeBackend's ``ralph`` config key is true, the backend resumes
the same pinned session via ``claude --resume <session_id>`` whenever the
CLI exits before the LLM/eval budget is exhausted. Each iteration's
``--max-budget-usd`` is set to the *remaining* budget so the cap doesn't
stack across iterations and the final iteration self-caps in the CLI.
Off by default — single-shot semantics preserved.
Stops when (a) cumulative cost ≥ ``max_token_cost − $0.05`` headroom,
(b) eval-count budget is exhausted, (c) claude exits non-zero, or
(d) ``stop_at_score`` is reached. No fixed iteration cap.
Records per-invocation ``{cost, score, returncode}`` in
``result.metadata["invocations"]`` so callers can see the loop's progress.
Ported from terrarium feat/cc-ralph-loop (Benzhang2004) into gepa.omni's
claude_code backend per the policy that all new adapter behavior lives in
gepa.omni rather than terrarium-side adapters.
Co-Authored-By: Benzhang2004 <zhangjialin04@sina.com>
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
…rver ``optimize_sequential_with_server`` now takes ``list[EvalServer]`` (one server per stage) instead of a single shared server, mirroring the parallel-family helpers. Each stage gets its own ``BudgetTracker``, so the caller pre-partitions ``total / N`` and a productive early stage cannot monopolize the pool — the same fair-share semantics already applied to ``parallel_with_server`` / ``best_of_with_server`` / ``vote_with_server``. Stage-to-stage seeding still works the same way: between stages the function mutates ``servers[i].task.initial_candidate`` to the running-best candidate from earlier stages, then restores the original tasks on exit. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
The omni gepa backend's LM-based proposer path constructs ``gepa.lm.LM(model, **reflection_lm_kwargs)`` and was only checking ``OmniConfig.max_thinking_tokens`` to populate the LM's thinking config — never reading ``OmniConfig.effort``. So a user passing top-level ``effort=medium`` to the gepa backend got extended thinking in the claude_code_agent path (which threads effort to ``--effort`` on the proposer subprocess) but no thinking config at all in the LM-based path: ``lm_kwargs`` stayed empty and litellm fell back to the provider's default. Auto-thread ``self.effort`` into ``lm_kwargs.reasoning_effort`` when ``max_thinking_tokens`` isn't set — matching how the native terrarium GEPAAdapter's ``_apply_effort`` runner hook already worked. Explicit ``reflection_lm_kwargs.reasoning_effort`` from the caller still wins via ``setdefault``. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
The two knobs control extended thinking via different mechanisms but
they're not actually mutually exclusive at the runtime level — the
claude CLI accepts both ``--effort`` and ``MAX_THINKING_TOKENS`` env
var simultaneously, and litellm accepts both ``reasoning_effort`` and
``thinking={…}`` kwargs. Forcing a hard ValueError prevented users
from passing both even when they had a reason to (e.g., comparing
adaptive-vs-fixed configs in a single run, or relying on whichever
the runtime picks).
Changes:
- ``optimize_anything`` and ``optimize_anything_with_server``: drop
the ``ValueError("config.effort and config.max_thinking_tokens are
mutually exclusive")`` raise.
- ``claude_code`` backend: drop ``max_thinking_tokens is None and``
guard around ``--effort`` — pass it whenever ``effort`` is set.
- ``meta_harness`` backend: same.
- ``gepa`` backend's LM path: was ``if max_thinking_tokens / elif effort``;
now both apply together (``setdefault`` still respects user-supplied
``reflection_lm_kwargs.reasoning_effort``).
- ``OmniConfig`` docstring: note both fields are independent.
Behavior with both set is whatever the runtime does — we just stop
preventing it.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
…ran in Sandboxed runs invoke claude with cwd=jail_root (a TemporaryDirectory), so the CLI writes its session jsonl under ~/.claude/projects/<slug-of-jail-root>/. The post-call copy was always keyed on self.run_dir, so the slug lookup missed and silently no-op'd via the if-not-src.exists() guard in claude_utils. Result: every sandboxed cca run lost its transcripts. Inline the copy into both branches with the matching cwd. The transcript file lives on the host fs (not in the jail), so calling copy inside the with-block before TemporaryDirectory cleanup is safe. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Per-save cost was O(N): _save_seed_iteration_dir and _save_proposal_iteration_dirs rewrote every existing iterations/NNNNN/ dir from scratch on every save, making total run cost O(N^2). Add an existence guard so each dir is written exactly once. Also drop the per-iteration trace.json: it duplicated content already covered by meta.json (subsample scores) and components/ (proposed candidate text), and its eval_before/eval_after carried fat 3-tuple outputs that bloated each rejected iteration to ~10KB. Update test to match. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
| val_ids = list(self.val_evaluation_policy.get_eval_batch(valset, state)) | ||
|
|
||
| # When write_agent_state is on, bypass the cache so we can capture | ||
| # trajectories (which the cache does not store). |
There was a problem hiding this comment.
Isn't a more principled fix to store these things in the cache rather than bypass? Also I think there's a separate optimize_anything cache and GEPA cache. Is GEPA cache even useful? if not, we should disable and remove it entirely. If it is useful, let's use it here by adding relevant data to it.
| from gskill.swe_fitness_fn import create_swe_fitness_fn | ||
|
|
||
| from gepa.optimize_anything import EngineConfig, GEPAConfig, ReflectionConfig, TrackingConfig, optimize_anything | ||
| from gepa.legacy_optimize_anything import EngineConfig, GEPAConfig, ReflectionConfig, TrackingConfig, optimize_anything |
There was a problem hiding this comment.
If we are confident on the parity/perf of the new version, perhaps better to migrate to new one.
Addresses two review comments on the write_agent_state output tree: - Merge _save_seed_iteration_dir and _save_proposal_iteration_dirs into one _save_iteration_dirs that shares the meta/components/val_scores writer while keeping the skip-before-recompute guard (O(1) per save). - Anchor each iteration directory on a random short id (uuid4().hex[:8]) stamped on the trace entry at slot creation, instead of state.i + 1. The loop counter isn't unique under concurrent/agent-swarm proposal backends (no shared counter to serialize on); a UUID needs none. Seed keeps the reserved id "seed". state.i stays the logical clock (ordering, logging, metrics step). Schema bumped v6 -> v7 with a migration that stringifies legacy integer ids. Proposers now receive only on-disk anchors (iteration_id, parent_iteration_id), not the sequence number; the file-based Claude Code proposer derives first-iteration from history.md presence. Co-authored-by: Isaac
|
@LakshyAAAgrawal iteration idx fixed. now everything will be uuid'd. thx! |
|
I had pointed my claude-code to refer to this PR, and set up meta-harness on some task I was doing locally. However, that implementation didn't add support for side info. Worth checking if all of that wiring works for all engines or not. |
…route Remove the legacy seed_candidate/evaluator dispatch from the public optimize_anything entry point. The function now takes the task fields directly (name, initial_candidate, objective, background, train/val/test sets, metadata) and builds the Task internally, with a fully documented signature. Merge the old _run_engine_optimization and ensemble's thin optimize_anything wrapper into a single public Task-based runner, optimize_task. The ensemble helpers, which carry a Task across stages, call optimize_task directly via a lazy import (avoiding the circular import with gepa.optimize_anything). Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
…odule docstring Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
The public optimize_anything entry point no longer dispatches to the legacy seed_candidate/evaluator API (removed in 9bb69f2); it takes the task fields directly and builds the Task internally. Replace the stale legacy-fallback / _LEGACY_API_ARGS / "cannot mix" tests with coverage of the actual new behavior: legacy kwargs and missing required args both raise a natural TypeError from arg binding. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
|
@LakshyAAAgrawal also removing the |
Drop the free-form Task.metadata field entirely — it was a confusing "magic reserved key" channel. Its only real consumers were: - gepa engine's metadata["val_set"] valset fallback — dead (no producer); removed. - autoresearch's metadata["optimize_anything_handoffs"] — moved to the engine's own engine_config["handoffs"], threaded explicitly through the sandbox/program-md builders. Added "handoffs" to _AR_CONFIG_KEYS (renamed from _CC_) and documented it in the engine docstring. - eval server /task scalar-metadata exposure — unconsumed; removed. Also rename the Task-based runner optimize_task -> optimize_anything_from_task (the from-a-Task counterpart of optimize_anything), and document where each engine's engine_config keys are enumerated. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
…> seed_candidate - OptimizeAnythingConfig gains an optional `name`, auto-generated as `<engine>-<uuid>-<timestamp>` when unset; drop the mandatory `name` argument from optimize_anything(). - Rename the Task/public `initial_candidate` field to `seed_candidate` and make it optional (None = seedless mode). - Thread the rename through the engines, the eval server `/task` endpoint, and the ensemble helpers; update tests. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
… EvalFn alias - Mirror gepa.legacy_optimize_anything's shape: leading positional `seed_candidate`, keyword-only `evaluator` / `dataset` / `valset` / `objective` / `background`. The new API only swaps in OptimizeAnythingConfig and adds `test_set`. Restores compatibility with the existing examples (circle_packing, aime_math, ...). - Remove the EvalFn type alias; inline Callable[..., tuple[float, dict[str, Any]]] at the use sites. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
…se on no budget - optimize_anything now builds the Task + server and runs inline rather than delegating to optimize_anything_from_task (which is now the thin Task-unpacking wrapper). - A missing budget (neither max_evals nor max_token_cost set) now warns and runs unbounded instead of raising ValueError. - Move optimize_anything_from_task and optimize_anything_with_server into oa/ensemble.py (still re-exported from gepa.optimize_anything); they delegate to optimize_anything / the run primitives via lazy imports. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
|
Did a full second-pass review against the current head ( Status of your prior comments
Additional issues I found on this passThese are net-new vs my June 18 review; I've ordered them by criticality. C1 — Backwards-compat:
|
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Rename src/gepa/legacy_optimize_anything.py to src/gepa/gepa_launcher.py and update all references (gepa.legacy_optimize_anything -> gepa.gepa_launcher) across source, tests, and generated API docs. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Drop the awkward "Legacy ... previously backed gepa.gepa_launcher" phrasing now that the module is named gepa_launcher. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
|
|
The OA tracker is untyped and its contract is undocumented OptimizeAnythingConfig.tracker: Any with the docstring "wandb/mlflow wrapper", but the required engine_config typo handling. Unknown keys warn (good, and documented) but warnings are Two same-named entry points. gepa.optimize_anything.optimize_anything With the introduction of the new API, all the code examples in the optimize_anything blogpost will break, but it should be cleanly migrated to the new API. |
|
Yeah agreed. Logger/tracker seem to have issues. I’ll add tests and fix
bugs.
…On Sat, Jun 27, 2026 at 18:46 Lakshya A Agrawal ***@***.***> wrote:
*LakshyAAAgrawal* left a comment (gepa-ai/gepa#346)
<#346 (comment)>
The OA tracker is untyped and its contract is undocumented
OptimizeAnythingConfig.tracker: Any with the docstring "wandb/mlflow
wrapper", but the required
methods — log_eval(used, score, best_score, cost) and log_metrics(metrics,
step=None) — are
discoverable only by grepping oa/eval_server.py and the callbacks. Given
Engine got a proper
Protocol, the tracker deserves one too (e.g. an OATracker Protocol), and
guides/experiment-tracking.md should document the OA tracker explicitly
and contrast it with the
core ExperimentTracker (they are different objects with different method
sets — a real source of
confusion, see #2 <#2>).
engine_config typo handling. Unknown keys warn (good, and documented) but
warnings are
easy to miss in long runs. Consider an opt-in strict mode, and recording
the accepted-vs-ignored
keys in the run summary so a silent typo is auditable after the fact.
Two same-named entry points. gepa.optimize_anything.optimize_anything
(OptimizeAnythingConfig) vs gepa.gepa_launcher.optimize_anything
(GEPAConfig) share a name
and most of a signature but take different config classes.
With the introduction of the new API, all the code examples in the
optimize_anything blogpost will break, but it should be cleanly migrated to
the new API.
—
Reply to this email directly, view it on GitHub
<#346?email_source=notifications&email_token=AGUCUPMUOHGKIJSSKQY4JDT5CB2JHA5CNFSNUABFM5UWIORPF5TWS5BNNB2WEL2JONZXKZKDN5WW2ZLOOQXTIOBSGM4TMNJUGA2KM4TFMFZW63VGMF2XI2DPOKSWK5TFNZ2KYZTPN52GK4S7MNWGSY3L#issuecomment-4823965404>,
or unsubscribe
<https://github.com/notifications/unsubscribe-auth/AGUCUPLCZV3VKJQXNOTMNBD5CB2JHAVCNFSNUABGKJSXA33TNF2G64TZHMYTAMZSGQYTSMZWGY5US43TOVSTWNBTGQ3TCMJZGI3TRILWAI>
.
You are receiving this because you authored the thread.Message ID:
***@***.***>
|
![devin-ai-integration[bot]](https://avatars.githubusercontent.com/in/811515?s=60&v=4){kind=small}
| if self.custom_candidate_proposer is not None: | ||
| return self.custom_candidate_proposer(candidate, reflective_dataset, components_to_update), empty, {} | ||
| new_texts = self.custom_candidate_proposer( | ||
| candidate, | ||
| reflective_dataset, | ||
| components_to_update, | ||
| metadata=metadata, | ||
| ) | ||
| return new_texts, empty, {} |
There was a problem hiding this comment.
🔴 Existing custom candidate proposers crash when called with the new metadata keyword argument
The custom proposer is now called with an unconditional metadata=metadata keyword argument (self.custom_candidate_proposer(...) at src/gepa/proposer/reflective_mutation/reflective_mutation.py:152-157) but existing user-defined proposers following the documented 3-positional-arg signature will not accept it, so they raise TypeError at runtime.
Impact: Any existing user of custom_candidate_proposer whose function does not accept **kwargs or an explicit metadata parameter will crash on the first iteration.
Mechanism: the call site unconditionally passes a new keyword argument
The old call was:
return self.custom_candidate_proposer(candidate, reflective_dataset, components_to_update), empty, {}The new call is:
new_texts = self.custom_candidate_proposer(
candidate,
reflective_dataset,
components_to_update,
metadata=metadata,
)The public API documentation at src/gepa/api.py:150 still describes the old signature as (candidate, reflective_dataset, components_to_update) -> dict[str, str]. The gepa/gskill/gskill/train_optimize_anything.py:680 also passes a custom_candidate_proposer=proposer that may not accept metadata.
While the ProposalFn Protocol at src/gepa/core/adapter.py:38-46 now includes metadata as a keyword-only parameter with a default, this is a typing-level contract. At runtime, any Python function that doesn't list metadata or **kwargs in its signature will reject the unexpected keyword argument.
The fix should either: (a) inspect the custom proposer's signature before passing metadata, or (b) wrap the call in a try/except to fall back to the 3-arg form, or (c) use **kwargs on the call side only when the callee accepts it.
Prompt for agents
In src/gepa/proposer/reflective_mutation/reflective_mutation.py around lines 151-158, the custom_candidate_proposer is called with metadata=metadata unconditionally. This breaks any existing custom proposer that doesn't accept **kwargs or an explicit metadata parameter. The fix should inspect the proposer's signature (using inspect.signature) to determine whether it accepts a 'metadata' keyword argument or **kwargs, and only pass metadata when supported. Alternatively, wrap the call in a try/except TypeError and fall back to the 3-arg form. The same pattern used by EvaluatorWrapper._filter_kwargs in gepa_launcher.py (signature inspection once at construction) would be appropriate here.
Was this helpful? React with 👍 or 👎 to provide feedback.
| # When write_agent_state is on, bypass the cache so we can capture | ||
| # trajectories (which the cache does not store). | ||
| if self.write_agent_state: | ||
| batch = valset.fetch(val_ids) | ||
| eval_result = self.adapter.evaluate(batch, program, capture_traces=True) | ||
| outputs_by_val_idx = dict(zip(val_ids, eval_result.outputs, strict=False)) | ||
| scores_by_val_idx = dict(zip(val_ids, eval_result.scores, strict=False)) | ||
| objective_by_val_idx = ( | ||
| dict(zip(val_ids, eval_result.objective_scores, strict=False)) | ||
| if eval_result.objective_scores is not None | ||
| else None | ||
| ) | ||
| trajectories_by_val_idx = ( | ||
| dict(zip(val_ids, eval_result.trajectories, strict=False)) | ||
| if eval_result.trajectories is not None | ||
| else None | ||
| ) | ||
| state.increment_evals(len(val_ids)) | ||
| return ValsetEvaluation( | ||
| outputs_by_val_id=outputs_by_val_idx, | ||
| scores_by_val_id=scores_by_val_idx, | ||
| objective_scores_by_val_id=objective_by_val_idx, | ||
| trajectories_by_val_id=trajectories_by_val_idx, | ||
| ) |
There was a problem hiding this comment.
🚩 write_agent_state bypasses evaluation cache entirely on every full-eval pass
When write_agent_state=True, the _evaluate_on_valset method at src/gepa/core/engine.py:239-260 skips the evaluation cache (state.cached_evaluate_full) and always calls self.adapter.evaluate() directly to capture trajectories. This means every accepted candidate triggers a fresh full-valset evaluation even if the same (candidate, example) pair was previously scored. For users who rely on cache_evaluation=True alongside write_agent_state=True, this silently nullifies the caching benefit on the full-eval path — potentially doubling the number of metric calls compared to the non-agent-state path. The tradeoff is documented in the comment ('the cache does not store trajectories') but users enabling both flags may not expect this cost increase.
Was this helpful? React with 👍 or 👎 to provide feedback.
| def stop(self) -> None: | ||
| if self._server: | ||
| self._server.shutdown() | ||
| self._server = None | ||
| self._pool.shutdown(wait=False) |
There was a problem hiding this comment.
🚩 EvalServer.stop() shuts down ThreadPoolExecutor with wait=False — in-flight evals may be abandoned
At src/gepa/oa/eval_server.py:367, self._pool.shutdown(wait=False) is called during server stop. This means any in-flight parallel evaluations submitted via evaluate_examples will be abandoned without waiting for completion. If an engine calls evaluate_examples and then the budget is exhausted (triggering BudgetExhausted → server.stop() via the optimize_anything runner), partially-completed batch results may never be collected. The eval_log and budget counter may undercount actual eval_fn invocations that completed but whose futures were abandoned.
Was this helpful? React with 👍 or 👎 to provide feedback.
| for val_id, output in outputs.items(): | ||
| path = os.path.join(out_dir, f"{val_id}.json") | ||
| with open(path, "w") as f: | ||
| json.dump(try_json_serialize(output), f, indent=2, default=json_default) | ||
| trajectories = valset_evaluation.trajectories_by_val_id or {} | ||
| if trajectories: | ||
| traj_dir = os.path.join(base, "trajectories") | ||
| os.makedirs(traj_dir, exist_ok=True) | ||
| for val_id, traj in trajectories.items(): | ||
| path = os.path.join(traj_dir, f"{val_id}.json") | ||
| with open(path, "w") as f: |
There was a problem hiding this comment.
🟥 Path traversal via unsanitized val_id in filesystem paths
User-controlled dataset item IDs (resolved via _resolve_id) are interpolated directly into filesystem paths when writing per-evaluation output files (_write_agent_iteration_files at src/gepa/core/engine.py:214-224). A dataset item with an id like ../../etc/passwd would write outside the intended directory tree.
Was this helpful? React with 👍 or 👎 to provide feedback.
| cmd += [ | ||
| "claude", | ||
| "--print", | ||
| "--output-format", | ||
| "json", | ||
| "--model", | ||
| self.model, | ||
| ] | ||
| if resume: | ||
| cmd.extend(["--resume", session_id]) | ||
| else: | ||
| cmd.extend(["--session-id", session_id]) | ||
| cmd.append(DENY_WEB_TOOLS) | ||
| # Single source of the permission posture (see claude_permission_args): | ||
| # bypassPermissions in the bwrap jail / unsandboxed, or the macOS | ||
| # Seatbelt settings whose --permission-mode default enforces the whitelist. | ||
| cmd.extend(claude_permission_args(work_dir, sandboxed=self.sandbox)) | ||
| if self.max_thinking_tokens is None and self.effort is not None: | ||
| cmd.extend(["--effort", self.effort]) | ||
| if self.max_token_cost is not None: | ||
| remaining = max(0.0, self.max_token_cost - adapter_cost) | ||
| cmd.extend(["--max-budget-usd", f"{remaining:.6f}"]) | ||
| cmd.append(prompt) |
There was a problem hiding this comment.
🟥 Subprocess command injection via unsanitized model name in claude CLI arguments
Engine-specific config values like model from OptimizeAnythingConfig.engine_config are passed directly into subprocess.Popen/subprocess.run command lists (src/gepa/oa/engines/autoresearch.py:575). While list-based subprocess calls prevent shell injection, the model string is not validated against an allowlist, and other string config values (like effort) flow into CLI arguments without sanitization.
Was this helpful? React with 👍 or 👎 to provide feedback.
| def start(self, port: int = 0) -> int: | ||
| """Start the HTTP server. Returns the bound port.""" | ||
| server_ref = self | ||
|
|
||
| class Handler(BaseHTTPRequestHandler): | ||
| def do_POST(self): | ||
| if self.path == "/evaluate": | ||
| server_ref._handle_evaluate(self) | ||
| elif self.path == "/evaluate_examples": | ||
| server_ref._handle_evaluate_examples(self) | ||
| elif self.path == "/validate": | ||
| server_ref._handle_validate(self) | ||
| else: | ||
| self.send_error(404) | ||
|
|
||
| def do_GET(self): | ||
| if self.path == "/status": | ||
| server_ref._handle_status(self) | ||
| elif self.path == "/task": | ||
| server_ref._handle_task_info(self) | ||
| else: | ||
| self.send_error(404) | ||
|
|
||
| def log_message(self, format, *args): | ||
| pass | ||
|
|
||
| # threading.HTTPServer would be better for high concurrency, but the | ||
| # underlying eval is already serialized through ``_eval_semaphore``; | ||
| # we keep the simple single-threaded HTTPServer to avoid surprising | ||
| # ordering, and rely on the thread pool inside evaluate_examples to | ||
| # parallelize batches. | ||
| from http.server import ThreadingHTTPServer | ||
|
|
||
| self._server = ThreadingHTTPServer(("localhost", port), Handler) | ||
| self._thread = threading.Thread(target=self._server.serve_forever, daemon=True) | ||
| self._thread.start() | ||
| return self.port |
There was a problem hiding this comment.
🟨 HTTP eval server binds without authentication on localhost
The EvalServer HTTP endpoint (src/gepa/oa/eval_server.py:358) binds to localhost with no authentication mechanism. Any local process can POST to /evaluate or /evaluate_examples to consume the eval budget, inject arbitrary candidates, or read task metadata via GET /task.
Was this helpful? React with 👍 or 👎 to provide feedback.
| def claude_permission_args( | ||
| work_dir: Path | str, | ||
| *, | ||
| sandboxed: bool, | ||
| extra_writable: list[Path | str] | None = None, | ||
| ) -> list[str]: | ||
| """Resolve the *single* tool-permission posture for a ``claude --print`` call. | ||
|
|
||
| Callers must use this instead of hardcoding ``--permission-mode`` so the | ||
| argv never carries two conflicting modes. Exactly one mode is emitted: | ||
|
|
||
| - **macOS + sandboxed** → ``--settings <seatbelt json> --permission-mode | ||
| default`` (via :func:`claude_settings_args`). ``default`` is what makes | ||
| the settings' ``permissions.allow`` whitelist enforce: in ``--print`` | ||
| mode every unlisted tool auto-denies (no human to approve), so the | ||
| allowlist becomes a strict tool-layer whitelist layered on top of the | ||
| Seatbelt filesystem confinement. | ||
| - **Linux + sandboxed** → ``--permission-mode bypassPermissions``. The | ||
| bwrap jail (:func:`bwrap_prefix`) is the OS-level confinement and there | ||
| is no human to answer prompts in ``--print`` mode, so tool permissions | ||
| are bypassed inside the jail. | ||
| - **unsandboxed** (either platform) → ``--permission-mode bypassPermissions``. | ||
| """ | ||
| if sandboxed: | ||
| # On macOS this carries its own ``--permission-mode default``; on Linux | ||
| # it is empty (bwrap handles confinement), so we fall through to bypass. | ||
| settings = claude_settings_args(work_dir, extra_writable=extra_writable) | ||
| if settings: | ||
| return settings | ||
| return ["--permission-mode", "bypassPermissions"] |
There was a problem hiding this comment.
🟨 Sandbox bypass permission sets bypassPermissions for unsandboxed runs
When sandbox=False, claude_permission_args returns ['--permission-mode', 'bypassPermissions'] (src/gepa/oa/sandbox.py:251), granting the Claude Code subprocess unrestricted file and tool access. This is the default mode since sandbox defaults to False in OptimizeAnythingConfig.
Was this helpful? React with 👍 or 👎 to provide feedback.
| def record(self, score: float) -> None: | ||
| """Record one eval call. Raises BudgetExhausted if over eval limit.""" | ||
| with self._lock: | ||
| if self.max_evals is not None and self._used >= self.max_evals: | ||
| raise BudgetExhausted(f"Eval budget exhausted: {self._used}/{self.max_evals} used") | ||
| self._used += 1 | ||
| self._log.append({"eval": self._used, "score": score, "time": time.time()}) |
There was a problem hiding this comment.
🟨 Budget check has TOCTOU race allowing over-budget evaluations
The BudgetTracker.record() method (src/gepa/oa/budget.py:56-60) checks the budget and increments the counter under a lock, but EvalServer.evaluate() calls budget.check() and budget.record() as two separate operations (src/gepa/oa/eval_server.py:155-162). Concurrent callers can pass the check simultaneously before any records, exceeding the budget.
Was this helpful? React with 👍 or 👎 to provide feedback.
…OATracker Config forwarding for the optimize_anything engines is now uniform and typed: - gepa engine: engine_config maps 1-to-1 to a GEPAConfig (built + validated in __init__, TypeError on unknown keys). tracking/reflection/merge/refiner/ callbacks/stop_callbacks all flow through untouched — fixes wandb/mlflow tracking being silently dropped. The OA layer only sets the eval budget, run_dir, and the stop_at_score / max_reflection_cost limits as EngineConfig fields; GEPA core installs the matching stoppers. - agent engines (autoresearch / meta_harness / best_of_n): each gets a typed config dataclass (AutoResearchConfig / MetaHarnessConfig / BestOfNConfig) built via Cls(**engine_config); the hand-maintained _*_CONFIG_KEYS tuples are gone. - effort / max_thinking_tokens are no longer top-level OptimizeAnythingConfig fields — they live in each agent engine's engine_config (the gepa engine takes reasoning knobs via reflection.reflection_lm_kwargs). - OATracker removed wholesale (module, config field, eval-server hook, engine calls); experiment tracking for gepa is configured through GEPAConfig.tracking. Core changes: - persist iterations/<id>/reflective_dataset.json under write_agent_state via a ReflectiveDatasetDumpCallback auto-registered by the launcher (was a private OA callback patching a core gap). - EngineConfig.stop_at_score field + its ScoreThresholdStopper. - MaxReflectionCostStopper binds to the effective cost source (custom_candidate_proposer or reflection_lm) so the cap fires in agent mode. Rename oa/_helpers.py -> oa/utils.py.
|
@LakshyAAAgrawal latest commit refactored a bunch of things: now |
4 participants
Porting terrarium's multi-backend into gepa. WIP
Introduces a unified Python API that dispatches a single optimization call to any compatible backend (GEPA, Claude Code, Meta-Harness, …) over a shared eval-server contract:
Design choices that drove the layout:
Results: