Commit 6df7e11
test: end-to-end agent tests with a scripted LLM
Previously every test either mocked at the `invoke()` level (the outer
entry point) or at `build_agent` (the factory). Nothing exercised the
actual deepagents graph — so wiring regressions could slip through CI
and only surface on a real LLM call (expensive to debug).
This adds 11 tests that run the real deepagents graph — filesystem
middleware, subagent middleware, sandbox, skill loader — with the chat
model swapped for a scripted `FakeMessagesListChatModel` subclass that
replays pre-canned `AIMessage`s (including tool_calls).
New helper: `ScriptedLLM` (subclass of `FakeMessagesListChatModel`).
- Adds the `bind_tools` / `with_structured_output` no-ops that
deepagents middleware requires; the base class raises
NotImplementedError on bind_tools.
- Tools still get routed by the graph when our AIMessage carries
`tool_calls` — they just aren't inspected by the fake model.
Test groups:
TestWritePathIsRealDisk write_file lands under root_dir
read_file → write_file sequence
TestCurateUpdatesManifest wiki-curator flow: read raw, write
wiki page, edit_file the manifest
to flip pending → curated. Locks in
that in-place manifest updates must
use edit_file (write_file refuses
to overwrite existing files).
TestStyleLearnerE2E ls → read sample → write STYLE.md
TestDrafterE2E read STYLE.md + wiki → write
chapter with [src:…] citation
TestAgentGracefulOnToolError ToolMessage error is observable;
agent recovers on next turn
TestDeterministicOutput scripted final message is what the
caller sees back
TestSandboxViaFilesystemBackend parent traversal (`..`) blocked;
Windows absolute paths blocked
TestRecursionLimitStopsRunaway an LLM that never emits a final
message is bounded by
recursion_limit; graph raises
GraphRecursionError
TestBuildAgentWiringUsesScriptedLLM
`build_agent` accepts scripted
model when `make_model` is patched;
1 main + 3 subagents = 4 model
calls; each subagent gets its
`role=` tag.
Production change alongside: pinned `FilesystemBackend(virtual_mode=True)`
in `agent.build_agent()` because deepagents will flip this default in
0.5.0 and the deprecation warning was noisy. Virtual mode maps
`/research/wiki/x.md` (as the LLM sees it) onto `<root_dir>/research/wiki/x.md`
on disk, and bans `..` + bare absolute paths — which our sandbox tests
now exercise.
Stats: 11 new tests, 301 total passing, 1 skipped, ruff clean.
Run time for the e2e module: ~2s (no network, no real LLM).
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>1 parent 0dd7c45 commit 6df7e11
2 files changed
Lines changed: 431 additions & 1 deletion
| Original file line number | Diff line number | Diff line change | |
|---|---|---|---|
| |||
123 | 123 | | |
124 | 124 | | |
125 | 125 | | |
126 | | - | |
| 126 | + | |
| 127 | + | |
| 128 | + | |
| 129 | + | |
| 130 | + | |
| 131 | + | |
| 132 | + | |
| 133 | + | |
127 | 134 | | |
128 | 135 | | |
129 | 136 | | |
| |||
0 commit comments