feat: comprehensive agent-readable state export in run_dir

[LakshyAAAgrawal]

Summary

When run_dir is provided, GEPA now writes gepa_state.json — a comprehensive, agent-readable export of the full optimization state. This is designed for consumption by autonomous agents (e.g. Claude Code) that need the complete picture to propose better candidates.

What's in gepa_state.json

{
  "schema_version": 1,
  "iteration": ...,
  "total_metric_calls": ...,
  "frontier_type": ...,
  "component_names": [...],

  "candidates": [
    {
      "idx": 0,
      "texts": {"system_prompt": "..."},
      "parent_ids": [],
      "avg_val_score": 0.85,
      "num_val_scored": 100,
      "metric_calls_to_discover": 0,
      "val_scores": {"0": 0.9, "1": 0.8, ...},
      "objective_scores": {"accuracy": 0.85}
    }, ...
  ],

  "pareto_front": {
    "instance_front": { "<val_id>": {"best_score": ..., "best_candidates": [...]} },
    "objective_front": { "<metric>": {"best_score": ..., "best_candidates": [...]} }
  },

  "summary": {
    "best_candidate_idx": 7,
    "best_avg_score": 0.92,
    "pareto_front_candidate_idxs": [0, 3, 5, 7],
    "hardest_examples": [{"val_id": "3", "best_score": 0.2, ...}, ...]
  },

  "iteration_log": [...]
}

Side-info capture

Proposal evaluation data (eval_before, eval_after) is now recorded in the iteration trace with:

• Per-example scores
• Outputs (best-effort JSON serialization)
• Trajectories/side_info (best-effort JSON serialization)
• Whether the proposal was accepted/rejected
• The proposed candidate text

This gives agents the feedback needed to understand WHY proposals succeeded or failed.

What's still missing (future work)

• Full valset evaluation outputs/side_info — the full validation eval uses capture_traces=False for performance, so only scores are available per-candidate per-datapoint, not outputs or side_info
• Rejected candidate texts — only accepted candidates are in program_candidates; rejected proposals are now in the iteration log but weren't before
• Evaluation cache export — the EvaluationCache is in the pickle but not JSON-exported

Test plan

• 447 tests pass (1 new test for agent state export)
• pyright: 0 errors
• ruff: 0 errors
• New test verifies JSON structure, candidate data, pareto front, and hardest examples

🤖 Generated with Claude Code

[semanticdiff-com]

Review changes with  

Changed Files

File	Status
src/gepa/proposer/reflective_mutation/reflective_mutation.py	41% smaller
src/gepa/core/engine.py	9% smaller
src/gepa/core/state.py	4% smaller
tests/test_state.py	1% smaller
.gitignore	Unsupported file format
src/gepa/api.py	0% smaller
src/gepa/core/adapter.py	0% smaller
src/gepa/gepa_utils.py	0% smaller
src/gepa/optimize_anything.py	0% smaller
tests/test_candidate_selector.py	0% smaller

[Shangyint]

A note to ourselves: currently, the gepa_state JSON will be quite large, which is not agent-friendly. We can possibly build a directory that represents the same information but in different files.