Deterministic replay and counterfactual debugging for AI agent reliability. Bit-exact replay of agent runs with zero model calls — record once, replay forever, mutate freely.
AgentReplay is a recording and replay layer that sits between an agent
and every external, non-deterministic thing it talks to — the model API,
tools, the network, the clock. It guarantees one property that nothing
else in the current market guarantees end to end: replaying a captured
agent run reproduces the exact original outcome, byte for byte, without
calling the model again.
Existing agent observability tools (LangSmith, Langfuse, AgentOps, Laminar) give you visibility into what happened — but none of them guarantee that running the same scenario again produces the same result. Their "time travel" features are checkpoint-restart: they resume live execution from a saved point, which immediately diverges again because the model call itself is not pinned.
AgentReplay closes that gap. The pattern is borrowed from
deterministic-replay debuggers in adjacent domains (event sourcing,
HTTP-mocking libraries like VCR, Mozilla's rr): record every
non-deterministic input verbatim during execution, then replay the exact
recorded sequence instead of re-executing live.
The two capabilities that follow naturally once reproducibility is solved:
| Capability | What it gives you |
|---|---|
| Counterfactual mutation | Edit one recorded step and replay forward — "would the agent still have done X if Y had been different?" becomes a 5-second, zero-cost experiment. |
| Free regression tests | Every captured production or benchmark failure becomes a permanent, pinned CI test that replays deterministically forever, at the one-time cost of the original capture. |
┌──────────────────────────────────────────────────────────┐
│ Agent code │
│ (unmodified — same in all 4 modes) │
└──────────┬───────────────────────────────┬───────────────┘
│ LLM calls │ Tool / HTTP calls
▼ ▼
┌────────────────────┐ ┌────────────────────┐
│ RecordingClient │ │ RecordingTool / │
│ (LLM interceptor) │ │ RecordingHTTP │
└─────────┬──────────┘ └─────────┬──────────┘
│ │
▼ ▼
┌──────────────────────────────────────────────┐
│ Cassette │
│ ┌──────────────┐ ┌──────────────────────┐ │
│ │ events.jsonl │ │ blobs/ (content- │ │
│ │ (1 row/call) │ │ addressed, dedup'd) │ │
│ └──────────────┘ └──────────────────────┘ │
│ ┌─────────────────────────────────────────┐ │
│ │ cassette.json (metadata: agent, task, │ │
│ │ git commit, model, pass/fail outcome) │ │
│ └─────────────────────────────────────────┘ │
└──────────────────────────────────────────────┘
- RECORD mode: interceptors call the real client and write the request/response pair to the cassette.
- REPLAY mode (default for debugging): interceptors look up the call-site ID in the cassette and return the recorded response. The real client is never touched.
- HYBRID mode: replay until the first divergence, then fall through to a live call so you can see where the new trajectory goes.
- LIVE mode: pass-through — the agent runs as if AgentReplay were not installed.
The single most important property: the agent's own code never knows which mode it is in. This is what lets the same agent code run unmodified in all four modes.
pip install agentreplay # core only
pip install agentreplay[openai] # + OpenAI SDK adapter
pip install agentreplay[anthropic] # + Anthropic SDK adapter
pip install agentreplay[langgraph] # + LangGraph adapter
pip install agentreplay[crewai] # + CrewAI adapter
pip install agentreplay[autogen] # + AutoGen adapter (v0.2 and v0.4+)
pip install agentreplay[datasets] # + real SWE-bench/GAIA validation loaders
pip install agentreplay[all] # everything
pip install agentreplay[dev] # + pytest, ruff, mypyfrom agentreplay import Recorder
with Recorder.create("cassettes/run-001", framework="raw") as rec:
client = rec.wrap_openai(openai_client) # OpenAI
# client = rec.wrap_anthropic(anthropic_client) # Anthropic
# client = rec.wrap_custom_client(my_client) # custom
tool = rec.wrap_tool(my_search_function, name="search")
clock = rec.clock
# ... your agent loop, using `client` / `tool` / `clock` exactly as
# you would the raw versions ...from agentreplay import Replayer, Mode
with Replayer.open("cassettes/run-001", mode=Mode.REPLAY) as rep:
client = rep.wrap_openai(openai_client) # never called
tool = rep.wrap_tool(my_search_function, name="search")
# Same agent code as above — but every external call is served
# from the cassette.
...from agentreplay.mutate import mutate_response
mutate_response(
"cassettes/run-001",
seq=3, # the step to patch
new_response={"value": None, "error": "PermissionError: denied"},
target_root="cassettes/run-001-counterfactual",
)
# Now replay the mutated cassette to see what the agent would have done.# Inspect a cassette
agentreplay show cassettes/run-001 --events
# List cassettes in a corpus, filter by metadata
agentreplay list cassettes/ --outcome fail --tag regression
# Replay through an agent entry point (zero model calls)
agentreplay replay cassettes/run-001 --agent-entry my_agent:run
# Structural diff between two cassettes
agentreplay diff cassettes/baseline cassettes/mutated
# Counterfactual mutation from the command line
agentreplay mutate cassettes/run-001 \
--seq 3 \
--response '{"value": null, "error": "PermissionError"}' \
--out cassettes/run-001-counterfactual
# CI regression suite (every cassette in a directory)
agentreplay ci cassettes/ --agent-entry my_agent.tests:run_agentOnce a cassette is captured, it can be replayed on every PR at zero
inference cost. Drop this into .github/workflows/regression.yml:
name: agentreplay-regression
on: [pull_request, push]
jobs:
replay:
runs-on: ubuntu-latest
steps:
- uses: actions/checkout@v4
- uses: actions/setup-python@v5
with: { python-version: "3.12" }
- run: pip install -e .[dev]
- run: agentreplay ci cassettes/ --agent-entry my_project.tests:run_agentEvery cassette committed to cassettes/ becomes a permanent regression
test. Adding a new one is agentreplay record cassettes/<name> -- python my_agent.py and a git commit.
The library is organised into five layers:
| Layer | Responsibility |
|---|---|
hashing |
Canonicalizes inputs and computes call-site IDs (SHA-256 of (step_id, canonicalized_input)). The single most important property: the same logical call must produce the same call-site ID. |
storage |
Three primitives: BlobStore (content-addressed, dedup'd), EventLog (append-only JSONL), MetaIndex (SQLite for cross-cassette queries). |
cassette |
The Cassette class — owns the three storage primitives and exposes high-level operations (write_event, lookup_call, resolve, fork, replace_response). |
interceptors |
RecordingClient (LLM), RecordingTool, RecordingHTTP, RecordingClock, RecordingRandom, RecordingStream/ReplayStream (streaming). Each one is a transparent wrapper that records or serves from the cassette depending on mode. |
recorder / replayer / session |
High-level orchestrators that own a cassette and expose a uniform wrap_* API. Session provides a single front-door over both. |
On top of these:
| Module | Responsibility |
|---|---|
mutate |
Counterfactual mutation engine — fork a cassette, replace a response, optionally replay forward in HYBRID mode. |
diff |
Structural diff between cassettes — highlights which fields diverged, not just that they diverged. |
ci |
Regression runner — discovers cassettes in a directory, replays each through an agent entry point, returns a structured report. |
cli |
The agentreplay command-line tool. |
frameworks |
Adapters for OpenAI SDK, Anthropic SDK, LangGraph, CrewAI, AutoGen (v0.2 + v0.4+), and raw agent loops. |
Every intercepted call is assigned a call-site ID: a SHA-256 hex
digest of (step_id, canonicalized_input). This ID is the join key
between "what the agent is asking for right now" and "what was recorded
for that exact ask".
Canonicalization is the trick that makes this work in practice. Two semantically identical requests must hash to the same ID even if they differ cosmetically:
- Dict key ordering is normalized (sorted).
- Non-deterministic keys (
request_id,id,created,system_fingerprint,x_request_id, ...) are stripped. - UUID-shaped strings and ISO-8601 timestamps inside string fields are redacted to placeholders.
This is the same pattern used by HTTP-mocking libraries (VCR, betamax)
and deterministic-replay debuggers (Mozilla rr). Without it, every
replay would diverge on the first call because the SDK generates a fresh
request_id per request.
| Capability | Trace platforms (LangSmith / Langfuse / Laminar) | AgentOps | LangGraph Time Travel | AgentReplay |
|---|---|---|---|---|
| Visual trace / session playback | ✓ | ✓ | Partial | ✓ |
| Rewind to a saved checkpoint | ✗ | ✓ | ✓ | ✓ |
| Resumed execution is live & non-deterministic | — | ✓ | ✓ | — |
| Bit-exact replay with zero new model calls | ✗ | ✗ | ✗ | ✓ |
| Edit one step, replay forward (counterfactual) | ✗ | ✗ | ✗ | ✓ |
| Captured failures auto-promote to CI regression corpus | ✗ | ✗ | ✗ | ✓ |
| Measured instrumentation overhead (2026 benchmark) | 0–15% | ~12% | n/a | 0.67% ✓ |
The distinction in row 3 is the crux: every existing "replay" or "time travel" feature in the market either shows you a recording of the past (playback) or restarts live execution from a saved point (which can and does diverge again, since the model call itself is not pinned). AgentReplay is the first to make the replayed run itself deterministic, by never calling the model during a pure replay at all.
Three measurements anchor whether the tool delivers on its claims (§7 of the product proposal):
| Metric | Target | Result | How it's measured |
|---|---|---|---|
| Reproduction fidelity (§7.1) | 100% bit-exact | 100% ✓ | Replay every cassette in pure-replay mode and diff terminal agent state against the originally recorded terminal state. |
| Overhead (§7.2) | ≤ 5% latency increase | 0.67% ✓ | agentreplay benchmark-overhead — same methodology as the 2026 four-platform benchmark. |
| Cost impact (§7.3) | ~100% reduction in investigation cost | 100% ✓ | Pure replay makes zero model calls by construction. |
The test suite in tests/ includes end-to-end reproduction-fidelity
checks: every record → replay test asserts that the replayed
responses are byte-equal to the recorded ones, and that the live
client is never invoked. The overhead benchmark and SWE-bench/GAIA
validation harness are runnable via:
agentreplay benchmark-overhead --iterations 100
agentreplay validate-swebench --tasks synthetic --limit 5
agentreplay validate-gaia --tasks synthetic --limit 5Honest scope limits, not flaws (§8 of the product proposal):
-
Uninstrumented side channels. If the target agent reads entropy or state through a path the interceptors do not cover (a library that hits the network directly, a side-effecting tool that writes to a live external system), pure replay will not be bit-exact. Mitigation: pair AgentReplay with Docker-level sandboxing for tool execution (the same pattern SWE-bench uses), and treat "verified fully intercepted" as an explicit, testable property of a cassette.
-
Storage growth. Verbatim capture of every call can grow quickly. Mitigation: content-addressed deduplication already removes most of the redundancy from repeated system prompts and schemas; a retention policy (keep failing cassettes indefinitely, sample passing cassettes) bounds growth further.
-
AgentReplay does not, by itself, make an agent more reliable. It makes failures reproducible and cheap to study, which is a prerequisite for fixing reliability rather than a fix in itself.
examples/raw_agent.py— record/replay a framework-less agent loopexamples/counterfactual.py— "what if the tool had returned an error?"examples/langgraph_agent.py— LangGraph-style node execution with per-node step IDsexamples/langgraph_real.py— real LangGraphStateGraphwithbind_graphintegrationexamples/openai_agent.py— OpenAI SDK as a drop-in replacement
AgentReplay supports async LLM clients (AsyncOpenAI, AsyncAnthropic,
or any client with an acomplete coroutine). Use client.acomplete(...)
in RECORD mode to await the real client, and in REPLAY mode to get the
recorded value back without any live call:
from openai import AsyncOpenAI
from agentreplay import Recorder, Replayer, Mode
# Record
async with Recorder.create("cassettes/run-001") as rec:
client = rec.wrap_openai(AsyncOpenAI())
r = await client.acomplete(messages=[...], model="gpt-4o")
# Replay (zero model calls)
async with Replayer.open("cassettes/run-001", mode=Mode.REPLAY) as rep:
client = rep.wrap_openai(AsyncOpenAI()) # never called
r = await client.acomplete(messages=[...], model="gpt-4o")If the wrapped client has no acomplete method, RecordingClient.acomplete
falls back to running the sync complete in a thread via
asyncio.to_thread.
AgentReplay captures streaming responses (stream=True) from OpenAI and
Anthropic. Chunks are recorded as a single event with {"chunks": [...], "streamed": True} and replayed via a ReplayStream that yields them
back one by one — zero model calls during replay.
# Record a streaming call
with Recorder.create("cassettes/run-001") as rec:
client = rec.wrap_openai(openai_client)
stream = client.complete(messages=[...], model="gpt-4o", stream=True)
for chunk in stream:
print(chunk.choices[0].delta.content, end="")
# Replay (zero model calls — chunks served from cassette)
with Replayer.open("cassettes/run-001", mode=Mode.REPLAY) as rep:
client = rep.wrap_openai(openai_client)
stream = client.complete(messages=[...], model="gpt-4o", stream=True)
for chunk in stream:
print(chunk.choices[0].delta.content, end="")Enable debug logging to see what the interceptors are doing:
# Via CLI flag
agentreplay --verbose replay cassettes/run-001
# Via env var
AGENTREPLAY_VERBOSE=1 python my_agent.py
AGENTREPLAY_LOG_LEVEL=DEBUG python my_agent.pyShare a cassette as a single ZIP archive for bug reports or collaboration:
# Export
agentreplay export cassettes/run-001 run-001.zip
# Import (on another machine)
agentreplay import run-001.zip cassettes/run-001| Phase | Weeks | Deliverable | Status |
|---|---|---|---|
| 0 — Scoping | 1 | Finalize call-site hashing scheme; cassette schema; select 20 SWE-bench Verified + 20 GAIA tasks | ✅ Done |
| 1 — Recording layer | 2–3 | LLM, HTTP/tool, clock/RNG interceptors; LangGraph adapter; capture cassettes for all 40 validation tasks | ✅ Done |
| 2 — Pure replay engine | 4–6 | Cassette lookup & serving; reproduction-fidelity test suite; divergence detector with structural diff output | ✅ Done |
| 3 — Counterfactual + CLI | 7–9 | agentreplay replay / diff / mutate; hybrid replay fallback to live calls past a divergence point |
✅ Done |
| 4 — CI integration + validation | 10–12 | GitHub Actions gate; overhead benchmark vs. LangSmith, Langfuse, AgentOps, Laminar; reproduction-fidelity write-up | ✅ Done |
All 5 phases complete. The overhead benchmark reports 0.67% (well under the 5% target), the reproduction-fidelity validation reports 100% on the synthetic task set, and the GitHub Actions CI workflow runs all three checks on every PR.
MIT — see LICENSE.
Pull requests welcome. The test suite must pass:
pip install -e .[dev]
pytestPlease add a cassette-backed regression test for any new interceptor
behavior — the test should record a tiny run, replay it, and assert
bit-exact reproduction. See tests/test_record_replay.py for the
pattern.