Two SEO agents do the same job — audit a web page for a target keyword — so we can prove that the one with memory beats the one without.
- Agent A — no memory. Same skills every run. The baseline / control.
- Agent B — Agent A plus two things: it remembers lessons from past audits and applies them, and it flags problems nobody asked about.
Everything else is identical — same tools, same steps, same prompt. So if B does better, it's the memory and proactivity, nothing else. That isolation is the whole point: it makes any difference attributable to learning.
Each audit covers four jobs: keyword research, on-page fixes, content-gap analysis, rank tracking.
- Quickstart
- Configuration
- The two screens
- How it works
- Proving B beats A: the eval
- API
- Project layout
- Tech stack
- Commands
- Known limits / planned
make up # starts everything: MongoDB, Redis, backend, worker, scheduler, frontend
# open http://localhost:5173
make test # run the backend tests
make down # stop everythingmake up brings the stack up healthy with no API keys. But a fresh clone has an empty database, so to actually run audits you need live keys in .env:
ANTHROPIC_API_KEY=...
SERP_API_KEY=...
REPLAY_MODE=false
- Live mode does real page fetches, Google (SERP) lookups, and LLM calls — and records every one into the database.
- Replay mode (default) serves those recordings back with no keys and no network — so it only has data after a live run. On a clean clone it starts empty.
Planned, not yet shipped: a committed seed (recorded data + a pre-trained playbook) so a fresh clone runs — and shows B beating A — keyless out of the box. Until then, a clean clone needs keys for the first runs.
There's no CORS setup — the frontend reaches the backend same-origin through Vite's /api proxy.
All via environment (.env):
| Variable | Default | Meaning |
|---|---|---|
REPLAY_MODE |
true |
Replay recordings (no keys/network) vs live. |
MONGO_URL |
mongodb://mongo:27017/learning_seo_agent |
MongoDB. |
REDIS_URL |
redis://redis:6379/0 |
Redis (job queue). |
ANTHROPIC_API_KEY |
— | Needed for live runs. |
SERP_API_KEY |
— | SerpAPI; needed for live SERP. |
AGENT_MODEL |
claude-sonnet-4-6 |
LLM both agents use. |
AGENT_MAX_TOKENS |
4096 |
Per-call output cap. |
SEED_IDLE_ENABLED |
true |
Background self-learning on/off. Runs only when keys are present. |
SEED_IDLE_INTERVAL_SECONDS |
1800 |
How often the background loop runs. |
- Audit — type a real URL + keyword, hit Run. Agent A and Agent B appear side by side. Runs go through a job queue (async) and are saved in the database, so the page shows the latest run plus a history list — click any past run to reopen its full A/B comparison. Reloading the page keeps everything.
- B's Memory — what B has learned. Two grids: the playbook (trusted, promoted rules) and the diary (raw observations). Each playbook rule shows its confidence, how many runs it recurred across, and citations back to the diary rows it came from.
Both agents run the same four-stage pipeline over the same tools:
- Keyword research — find related phrases worth targeting.
- On-page audit — fetch the page, run SEO checks (title/headings/meta/length/comparison table).
- Content gap — search the SERP, compare against the top competitors, find missing features.
- Rank tracking — find where the page sits for the keyword.
Each stage is a short ReAct loop (think → call a tool → observe → submit typed recommendations). The agents emit a fixed set of recommendation types (add keyword to title/heading, add/fix meta description, expand content, add comparison table, fill content gap, improve internal linking, target keyword, track rank).
Tools (identical for both): fetch_page (scrape title/headings/body; degrades gracefully to SERP-only if a site blocks bots with a 403), search_serp (SerpAPI), run_seo_checks (deterministic on-page checks).
B differs from A only by: the playbook injected into its prompt, a separate learning step, and a proactive check. Nothing else.
Two layers:
- Diary — every raw observation B writes, one per audit. Append-only, messy. Each entry is structured, not free text:
page_kind,rec_type,target, a one-linewhy, and a 0–100confidenceB assigns itself. - Playbook — the trusted, distilled subset that gets fed back into B's prompt.
The janitor turns one into the other:
- groups diary entries by the exact key
(page_kind, rec_type, target)— structured, so nothing drifts; - promotes a pattern only once it recurs across ≥3 distinct runs;
- scores it by a recency-weighted average of B's own confidences, bucketed low/medium/high;
- keeps the top ~10 rules, each stamped with provenance (the diary rows it came from);
- skips rec types whose target is just the page's own keyword (rank tracking, keyword suggestions) — those would memorize a keyword, not learn a pattern.
So the diary grows with every run; the playbook stays a small, trusted, page-kind-specific set.
- B audits a page (exactly like A).
- In a separate reflection step (its own LLM call, after the audit so it can't distort the recommendations), B writes one structured lesson to the diary.
- The janitor consolidates. When a pattern has recurred enough, it's promoted to the playbook.
- The next audit of that kind of page injects the top playbook rules into B's prompt.
B surfaces problems the audit didn't, via a blind-spot scan: a deterministic pass that runs every on-page check and reports real failures the recommendations missed (capped, severity-ranked). A never runs it, so it's purely B's edge — e.g. catching a missing meta description the LLM dropped.
This is currently a fixed checklist, not the agent deciding for itself. Making it memory-driven (check the page against B's learned playbook) and/or an LLM "what am I missing?" step is planned — see
DESIGN.md.
B doesn't only learn from your audits. A scheduler runs a learning batch on a timer:
- for each seed topic (a given list of keywords across page kinds),
- it searches the live SERP for real pages, skips bot-blocked ones (a cheap check, no LLM wasted),
- queues the rest as low-priority "audit + learn" jobs,
- then runs the janitor.
So B improves on its own between jobs. It never crawls the open web — discovery is anchored to the given seed topics. (Runs only when keys are present; off otherwise.)
User audits and background learning share one Redis queue with a single worker:
- User runs = high priority — they jump ahead of all queued background work.
- Background learning = low priority.
- The worker never preempts the job in flight, so a user waits at most for the one background page currently being processed — never the whole batch.
User runs are async: POST /run returns a job_id immediately; the worker fills in the result; the UI polls until done. Every run is stored, so it's reload-safe.
Every LLM call is cached in MongoDB, keyed by the exact request (model + prompt + tools). Live mode calls the API and records; replay mode serves the recording with no key and no network. This is what makes the eval deterministic and (once seeded) keyless.
The graded centerpiece is the eval harness (app/eval/), not the agents. It scores A and B against a typed answer key on four metrics:
- Coverage — did it catch the recommendations that matter (severity-weighted)?
- Correctness — were its recommendations right (precision against the key)?
- Novelty — did it surface less-obvious wins (rarer recommendations weigh more)?
- Efficiency — value per cost (fewer tool calls / tokens for the same result).
Key properties:
- Frozen matcher — a recommendation matches the key when
type == typeand normalizedtarget == target. - Deterministic — runs off recordings (replay), so the score is the same every time, in CI, with no key.
- Train/eval split + leak guard — B learns only on the train split; both agents are scored cold on a held-out eval split; a leak guard fails loudly if any eval page sneaks into memory. This is what makes "B > A" a real claim, not luck.
It compares A vs B with memory off vs B with memory on, so the lift is isolated to memory.
All under /api:
| Method | Path | What |
|---|---|---|
| GET | /health |
Status + mode (replay/live). |
| POST | /run |
Enqueue an A-vs-B audit {url, keyword} → {job_id}. |
| GET | /run |
Recent runs (newest first). |
| GET | /run/{job_id} |
One run's status + full result. |
| GET | /memory/playbook |
Promoted rules. |
| GET | /memory/diary |
Raw observations (provenance behind the rules). |
| GET | /eval/report |
The A-vs-B scored comparison (cached). |
backend/app/
core/ config, db (lazy Mongo), logging, exceptions
models/ Pydantic models (page, serp, recommendation, memory, run, job, ...)
repositories/ one per Mongo collection (pages, serps, completions, diary, playbook, runs, run_results)
agents/ agent_a, agent_b, pipeline (4 stages), loop (ReAct), llm (Anthropic + record/replay), prompts, blind_spot
tools/ fetch_page, search_serp, seo_checks, memory_tools, registry
memory/ janitor (diary → playbook consolidation)
jobs/ queue (Redis HIGH/LOW), worker, scheduler, handlers, seed_topics, discovery, seed_runner
eval/ testset, matcher, premise, scorers, runner, leak_guard, comparison, report
api/ health, run, memory, eval, monitor
frontend/src/ App, api, types, components/ (AgentColumn, MemoryPanel, RunModal)
FastAPI · MongoDB · Redis · React + Vite + TypeScript · uv (Python) · Anthropic (claude-sonnet-4-6). Everything runs via Docker Compose.
| Command | Does |
|---|---|
make up |
Build + start the whole stack. |
make down |
Stop everything. |
make logs |
Tail logs. |
make test |
Backend test suite. |
- No committed seed yet — a fresh clone needs keys for the first runs (see Quickstart). Shipping recorded fixtures + a pre-trained playbook so a clone shows B>A keyless is the top planned item.
- Proactivity is a fixed checklist, not the agent deciding for itself. Planned: memory-driven blind spots + an LLM proactive pass.
- The monitor (re-check already-seen pages over time, alert/re-audit unprompted) is designed but not wired live.
- Background learning uses a fixed seed-topic list. Planned: active learning — B picks its weakest page kind to study next.
- Minor: de-duplicate B's recommendations; turn the deterministic checks into recommendations for both agents so obvious defects are never missed; add a rate-limit backoff so the background loop can't exhaust the SERP quota.
See DESIGN.md for the reasoning behind every decision (and the open questions), and transcripts/ for real example runs.