Unify evaluation caching: remove redundant adapter-level cache by lukedhlee · Pull Request #301 · gepa-ai/gepa

lukedhlee · 2026-04-03T01:32:36Z

What changed

Before this PR, GEPA had two independent evaluation caches running simultaneously when cache_evaluation=True:

	Core `EvaluationCache` (in `GEPAState`)	Adapter `_eval_cache` (in `OptimizeAnythingAdapter`)
Candidate key	Full 64-char SHA256	Truncated to 16 chars
Example key	Stable `DataId` from DataLoader	Content hash (fragile for non-serializable objects)
Thread-safe	No	Yes
Persistence	Serialized with `gepa_state.bin`	Per-pair `.pkl` files in `fitness_cache/`

The adapter cache always intercepted first (inside _call_evaluator), so the core cache never saw a miss — it was pure overhead. The two caches used different hash functions, making debugging and cross-referencing impossible.

After this PR, there is one cache: the core EvaluationCache. It now handles everything the adapter cache used to do (thread safety, disk persistence) plus new capabilities (trajectory caching).

Before → After

Adapter (`optimize_anything_adapter.py`)

Before: 90+ lines of cache logic (_eval_cache, _candidate_hash, _cache_key, _load_cache, _save_cache_entry, disk .pkl logic, threading.Lock)
After: _call_evaluator() is a 3-line pass-through. All caching handled by core.

Core cache (`core/state.py` → `EvaluationCache`)

Before: No thread safety, no disk persistence, no trajectory storage
After:
- threading.Lock on all reads/writes (with pickle-safe __getstate__/__setstate__)
- enable_disk_cache(path) for write-through .pkl persistence to {run_dir}/eval_cache/
- trajectory field on CachedEvaluation (with backward-compatible deserialization of old entries)
- New evaluate_batch_with_cache() method with require_trajectories flag

Proposer (`reflective_mutation.py`)

Before: Always evaluated fresh with capture_traces=True, then wrote results to cache without trajectories. On next iteration, cache hit → but no trajectory → had to re-evaluate anyway.
After: Calls evaluate_batch_with_cache(require_trajectories=True). Cached entries with trajectories are reused directly. Entries without trajectories are treated as cache misses and re-evaluated. Results (including trajectories) are stored back so the next iteration benefits.

`optimize_anything.py`

Before: Passed cache_mode/cache_dir to adapter constructor
After: Calls evaluation_cache.enable_disk_cache() on the core cache when disk mode is active. Adapter no longer needs cache config.

Resume (`initialize_gepa_state`)

Before: Disk cache was in adapter's fitness_cache/ directory, loaded by adapter
After: Disk cache is in eval_cache/ directory, re-enabled on the deserialized EvaluationCache via enable_disk_cache()

Public API

No changes. cache_evaluation and cache_evaluation_storage work exactly as before. The eval_cache/ directory replaces fitness_cache/ for disk persistence.

Tests

Added @pytest.mark.llm_call mock fixtures (per-file _mock_litellm autouse fixtures) so refiner/cache/best-evals tests run without real API keys — fixes the 6-hour CI timeout
Added 12 new tests: disk write-through, trajectory caching, evaluate_batch_with_cache (all-miss/all-hit/partial/require_trajectories), thread safety (4 threads × 100 concurrent puts)

semanticdiff-com · 2026-04-03T01:32:40Z

Review changes with

Changed Files

File	Status
src/gepa/proposer/reflective_mutation/reflective_mutation.py	36% smaller
src/gepa/core/state.py	22% smaller
src/gepa/optimize_anything.py	15% smaller
tests/test_cache_evaluation_storage.py	10% smaller
tests/test_refiner.py	7% smaller
src/gepa/adapters/optimize_anything_adapter/optimize_anything_adapter.py	5% smaller
src/gepa/api.py	0% smaller
src/gepa/core/engine.py	0% smaller
src/gepa/proposer/merge.py	0% smaller
tests/proposer/test_merge.py	0% smaller
tests/test_best_example_evals.py	0% smaller
tests/test_evaluation_cache.py	0% smaller

…vel cache The optimize_anything_adapter maintained its own evaluation cache alongside the core EvaluationCache in GEPAState. Both were activated by the same cache_evaluation=True flag, creating redundant caching with inconsistent identity (truncated 16-char hashes vs full SHA256, content-hashed examples vs stable DataIds). The adapter cache always intercepted first, making the core cache pure overhead. This commit removes the adapter's cache entirely and makes the core EvaluationCache the single source of truth: - Add thread-safe locking to EvaluationCache (for parallel evaluation) - Make candidate_hash() a public function in core/state.py - Remove ~90 lines of adapter cache code (_eval_cache, _candidate_hash, _cache_key, _load_cache, _save_cache_entry, disk pkl files) - Simplify _call_evaluator() to a direct pass-through - Deprecate cache_evaluation_storage config field (no longer needed) - Update tests to reflect unified caching via gepa_state.bin

Tests were directly constructing OptimizeAnythingAdapter with cache_mode="off" which no longer exists. Also updated test_refiner_with_disk_cache to check for gepa_state.bin instead of the removed fitness_cache/ directory.

…ion_storage The previous commit removed the adapter's disk cache but left no disk persistence option. This restores the cache_evaluation_storage="disk" behavior by adding write-through disk persistence directly to the core EvaluationCache: - EvaluationCache.enable_disk_cache(path): enables per-entry .pkl write-through and loads existing entries from disk on startup - optimize_anything() resolves cache_evaluation_storage and calls enable_disk_cache() when disk mode is active - initialize_gepa_state() re-enables disk cache on resumed runs - Disk entries use core's full SHA256 candidate hash + DataId (not the old adapter's truncated content hashes) - Directory changed from fitness_cache/ to eval_cache/ to avoid confusion with the old incompatible format

Write to a .tmp file first, then rename. Prevents corrupt/partial .pkl files when pickle.dump fails (e.g. non-picklable evaluator outputs).

The gepa.optimize() entry point (api.py) was creating EvaluationCache without enabling disk write-through, even when run_dir was available. Now it calls enable_disk_cache() automatically, matching the behavior of optimize_anything().

No code in this PR uses the public name. Can be made public in a future PR when cross-module hash usage is cleaned up.

The capture_traces=True evaluation in the reflective mutation proposer now checks the cache before calling adapter.evaluate(). Entries with cached trajectories are served directly, and only uncached examples (or entries without trajectories) trigger real evaluator calls. This makes total_num_evals accurate: cache hits no longer inflate the metric count, progress bar, or stop condition budget. Changes: - Add trajectory field to CachedEvaluation (backward-compat via __setstate__) - Add optional trajectory/trajectories params to put/put_batch - Proposer Step A: check cache, partition by trajectory presence, evaluate only what's needed, merge results

…CI hangs Tests in test_refiner.py, test_cache_evaluation_storage.py, and test_best_example_evals.py called real OpenRouter APIs, causing 6-hour hangs in CI (no API keys). Add an autouse conftest fixture that intercepts litellm.completion for @pytest.mark.llm_call tests, returning randomized JSON candidates. All 404 tests now pass in ~20s.

Each test file that calls optimize_anything (test_refiner.py, test_cache_evaluation_storage.py, test_best_example_evals.py) now has its own autouse _mock_litellm fixture. No changes to shared conftest or pyproject.toml — keeps the mock scoped to only the files that need it.

Move the 50-line cache-merge block from reflective_mutation.py into EvaluationCache.evaluate_batch_with_cache() with a require_trajectories flag. The proposer now calls one method instead of reimplementing cache hit/miss/merge logic inline.

…ch_with_cache, and thread safety

Shangyint · 2026-04-08T19:07:23Z

-        eval_curr = self.adapter.evaluate(ctx.minibatch, ctx.curr_prog, capture_traces=True)
-        total_evals += eval_curr.num_metric_calls if eval_curr.num_metric_calls is not None else len(ctx.subsample_ids)
+        if state.evaluation_cache is not None:
+            eval_curr, num_evals = state.evaluation_cache.evaluate_batch_with_cache(


Is this thread safe? execute proposals currently may run in multi-thread

Shangyint

LGTM! Left two minor comments

Shangyint · 2026-04-08T19:11:03Z


    # Evaluation caching
    cache_evaluation: bool = False
    cache_evaluation_storage: CacheEvaluationStorage = "auto"


can we have some explanation here how is "auto" resolved?

Is there any situation where run_dir is provided but the user would prefer to only keep in-memory?

LakshyAAAgrawal · 2026-04-08T21:47:54Z

Why is _save_to_disk is called outside the lock?
Similarly, why is _load_from_disk is called from enable_disk_cache without the lock?
evaluate_batch_with_cache accesses private _cache_dir from initialize_gepa_state. Could we ensure encapsulation?

Replace direct _cache_dir access in initialize_gepa_state with the public disk_cache_dir property.

…hed_evaluate

lukedhlee · 2026-04-08T22:53:58Z

Why is _save_to_disk is called outside the lock? Similarly, why is _load_from_disk is called from enable_disk_cache without the lock?

Great questions! It's faster this way as I/O is relatively slower. And it's actually safe as everything we need is loaded/updated with lock, and we simply need to store/load them in a disk, and concurrent writes/reads for different entries go to / come from different files.

evaluate_batch_with_cache accesses private _cache_dir from initialize_gepa_state. Could we ensure encapsulation?

Good point! Updated.

- TestEvaluateBatchWithCacheE2E: gepa.optimize() with counting adapter verifies cached run has fewer evaluate() calls than uncached - test_disk_cache_survives_restart: second run benefits from disk cache - Strengthen optimize_anything assertions: cache-on has fewer calls, cache-off has exactly matching calls

LakshyAAAgrawal · 2026-04-10T02:33:00Z


+    def evaluate_batch_with_cache(
+        self,
+        candidate: dict[str, str],


We should support a list of candidate, as I was saying in Shangyin's PR earlier. So paired list of candidate and example ids.

…evaluate through evaluate_batch_with_cache EvaluationCache now has a single cache-aware evaluation method (evaluate_batch_with_cache). GEPAState.cached_evaluate wraps its tuple-returning evaluator into the EvaluationBatch signature and delegates to the unified method.

…tion-cache

hash() is non-deterministic across Python runs (PYTHONHASHSEED), so the CountingAdapter could produce perfect scores (1.0) causing the engine to skip evaluations entirely, breaking the cache-reduces-calls assertion.

lukedhlee self-assigned this Apr 3, 2026

lukedhlee mentioned this pull request Apr 7, 2026

Cached eval still counts toward metric budget #316

Open

lukedhlee force-pushed the refactor/unify-evaluation-cache branch 2 times, most recently from 9f9a325 to b276da6 Compare April 8, 2026 03:14

lukedhlee added 11 commits April 8, 2026 11:44

Use atomic write for disk cache entries

660340f

Write to a .tmp file first, then rename. Prevents corrupt/partial .pkl files when pickle.dump fails (e.g. non-picklable evaluator outputs).

Revert candidate_hash rename — keep _candidate_hash private

d9eff3c

No code in this PR uses the public name. Can be made public in a future PR when cross-module hash usage is cleaned up.

Add end-to-end tests for disk cache, trajectory caching, evaluate_bat…

abcaa82

…ch_with_cache, and thread safety

lukedhlee force-pushed the refactor/unify-evaluation-cache branch from ac1a856 to abcaa82 Compare April 8, 2026 18:47

lukedhlee requested review from LakshyAAAgrawal and Shangyint and removed request for LakshyAAAgrawal April 8, 2026 18:49

Shangyint reviewed Apr 8, 2026

View reviewed changes

lukedhlee added 3 commits April 8, 2026 15:31

Add inline docs for cache_evaluation_storage explaining auto/memory/disk

0a5cde0

Add disk_cache_dir property to EvaluationCache for encapsulation

f4efee9

Replace direct _cache_dir access in initialize_gepa_state with the public disk_cache_dir property.

Remove unused cached_evaluate stub, rename cached_evaluate_full → cac…

a7a2479

…hed_evaluate

LakshyAAAgrawal reviewed Apr 10, 2026

View reviewed changes

lukedhlee added 3 commits April 11, 2026 12:44

Merge remote-tracking branch 'origin/main' into refactor/unify-evalua…

e45de92

…tion-cache

Fix flaky CI: use deterministic scores in E2E cache test

542a22b

hash() is non-deterministic across Python runs (PYTHONHASHSEED), so the CountingAdapter could produce perfect scores (1.0) causing the engine to skip evaluations entirely, breaking the cache-reduces-calls assertion.

Benzhang2004 mentioned this pull request Jun 5, 2026

feat: ReflectionLM protocol + batch-based parallel proposals (#329 Phases 1 & 4) #369

Open

3 tasks

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Uh oh!

Unify evaluation caching: remove redundant adapter-level cache#301

Unify evaluation caching: remove redundant adapter-level cache#301
lukedhlee wants to merge 18 commits into
mainfrom
refactor/unify-evaluation-cache

lukedhlee commented Apr 3, 2026 •

edited

Loading

Uh oh!

semanticdiff-com Bot commented Apr 3, 2026 •

edited

Loading

Uh oh!

Shangyint Apr 8, 2026

Uh oh!

Shangyint left a comment

Uh oh!

Shangyint Apr 8, 2026

Uh oh!

LakshyAAAgrawal Apr 10, 2026

Uh oh!

LakshyAAAgrawal commented Apr 8, 2026

Uh oh!

lukedhlee commented Apr 8, 2026

Uh oh!

LakshyAAAgrawal Apr 10, 2026

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

3 participants

Uh oh!

Conversation

lukedhlee commented Apr 3, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

What changed

Before → After

Adapter (optimize_anything_adapter.py)

Core cache (core/state.py → EvaluationCache)

Proposer (reflective_mutation.py)

optimize_anything.py

Resume (initialize_gepa_state)

Public API

Tests

Uh oh!

semanticdiff-com Bot commented Apr 3, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

Shangyint Apr 8, 2026

Choose a reason for hiding this comment

Uh oh!

Shangyint left a comment

Choose a reason for hiding this comment

Uh oh!

Shangyint Apr 8, 2026

Choose a reason for hiding this comment

Uh oh!

LakshyAAAgrawal Apr 10, 2026

Choose a reason for hiding this comment

Uh oh!

LakshyAAAgrawal commented Apr 8, 2026

Uh oh!

lukedhlee commented Apr 8, 2026

Uh oh!

LakshyAAAgrawal Apr 10, 2026

Choose a reason for hiding this comment

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

3 participants

lukedhlee commented Apr 3, 2026 •

edited

Loading

Adapter (`optimize_anything_adapter.py`)

Core cache (`core/state.py` → `EvaluationCache`)

Proposer (`reflective_mutation.py`)

`optimize_anything.py`

Resume (`initialize_gepa_state`)

semanticdiff-com Bot commented Apr 3, 2026 •

edited

Loading