You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
Two SEO agents do the same job — audit a web page for a keyword — so we can prove that the one with memory beats the one without.
3
+
Two SEO agents do the **same job** — audit a web page for a target keyword — so we can prove that the one **with memory** beats the one **without**.
4
4
5
-
-**Agent A** — no memory. Same skills every run. The baseline.
6
-
-**Agent B** — Agent A plus two things: it **remembers** lessons from past audits, and it **flags problems nobody asked about**.
5
+
-**Agent A** — no memory. Same skills every run. The baseline / control.
6
+
-**Agent B** — Agent A **plus** two things: it **remembers** lessons from past audits and applies them, and it **flags problems nobody asked about**.
7
7
8
-
Everything else (the tools, the steps, the prompt) is identical. So if B does better, it's the memory — nothing else.
8
+
Everything else is identical — same tools, same steps, same prompt. So if B does better, it's the memory and proactivity, nothing else. That isolation is the whole point: it makes any difference *attributable* to learning.
9
9
10
-
Each audit covers four jobs: keyword research, on-page fixes, content-gap analysis, and rank tracking.
10
+
Each audit covers four jobs: **keyword research, on-page fixes, content-gap analysis, rank tracking.**
-[Proving B beats A: the eval](#proving-b-beats-a-the-eval)
28
+
-[API](#api)
29
+
-[Project layout](#project-layout)
30
+
-[Tech stack](#tech-stack)
31
+
-[Commands](#commands)
32
+
-[Known limits / planned](#known-limits--planned)
33
+
34
+
---
35
+
36
+
## Quickstart
13
37
14
38
```bash
15
-
make up # starts everything (database, queue, backend, frontend)
39
+
make up # starts everything: MongoDB, Redis, backend, worker, scheduler, frontend
16
40
# open http://localhost:5173
17
-
make test# run the tests
18
-
make down # stop
41
+
make test# run the backend tests
42
+
make down # stop everything
19
43
```
20
44
21
45
`make up` brings the stack up healthy with **no API keys**. But a **fresh clone has an empty database**, so to actually run audits you need live keys in `.env`:
@@ -26,33 +50,176 @@ SERP_API_KEY=...
26
50
REPLAY_MODE=false
27
51
```
28
52
29
-
How the data layer works:
30
-
-**Live mode** (`REPLAY_MODE=false`, keys set) does real page fetches, Google (SERP) lookups, and LLM calls — and **records every one into the database**.
31
-
-**Replay mode** (`REPLAY_MODE=true`, the default) serves those recordings back with no keys and no network. So replay only has something to serve **after** you've run live at least once; on a clean clone it starts empty.
53
+
-**Live mode** does real page fetches, Google (SERP) lookups, and LLM calls — and **records every one into the database**.
54
+
-**Replay mode** (default) serves those recordings back with no keys and no network — so it only has data **after** a live run. On a clean clone it starts empty.
55
+
56
+
> **Planned, not yet shipped:** a committed **seed** (recorded data + a pre-trained playbook) so a fresh clone runs — and shows B beating A — keyless out of the box. Until then, a clean clone needs keys for the first runs.
57
+
58
+
There's no CORS setup — the frontend reaches the backend same-origin through Vite's `/api` proxy.
59
+
60
+
## Configuration
32
61
33
-
> Planned, not yet shipped: a committed **seed** (recorded data + a pre-trained playbook) so a fresh clone runs — and shows B beating A — keyless out of the box. Until then, a clean clone needs keys for the first runs.
62
+
All via environment (`.env`):
63
+
64
+
| Variable | Default | Meaning |
65
+
|---|---|---|
66
+
|`REPLAY_MODE`|`true`| Replay recordings (no keys/network) vs live. |
|`SEED_IDLE_ENABLED`|`true`| Background self-learning on/off. Runs only when keys are present. |
74
+
|`SEED_IDLE_INTERVAL_SECONDS`|`1800`| How often the background loop runs. |
34
75
35
76
## The two screens
36
77
37
-
-**Audit** — type a real URL and keyword, hit Run. You get Agent A and Agent B side by side. Every run is saved, so you can reload the page or click any past run to see it again.
38
-
-**B's Memory** — what B has learned. Two lists: the **playbook** (trusted rules) and the **diary** (raw notes). Each rule shows how confident B is and which notes it came from.
78
+
-**Audit** — type a real URL + keyword, hit Run. Agent A and Agent B appear side by side. Runs go through a job queue (async) and are saved in the database, so the page shows the **latest run** plus a **history list** — click any past run to reopen its full A/B comparison. Reloading the page keeps everything.
79
+
-**B's Memory** — what B has learned. Two grids: the **playbook** (trusted, promoted rules) and the **diary** (raw observations). Each playbook rule shows its confidence, how many runs it recurred across, and **citations** back to the diary rows it came from.
80
+
81
+
---
82
+
83
+
## How it works
84
+
85
+
### The agents
86
+
87
+
Both agents run the **same four-stage pipeline** over the **same tools**:
88
+
89
+
1.**Keyword research** — find related phrases worth targeting.
90
+
2.**On-page audit** — fetch the page, run SEO checks (title/headings/meta/length/comparison table).
91
+
3.**Content gap** — search the SERP, compare against the top competitors, find missing features.
92
+
4.**Rank tracking** — find where the page sits for the keyword.
93
+
94
+
Each stage is a short ReAct loop (think → call a tool → observe → submit typed recommendations). The agents emit a fixed set of recommendation types (add keyword to title/heading, add/fix meta description, expand content, add comparison table, fill content gap, improve internal linking, target keyword, track rank).
95
+
96
+
**Tools** (identical for both): `fetch_page` (scrape title/headings/body; degrades gracefully to SERP-only if a site blocks bots with a 403), `search_serp` (SerpAPI), `run_seo_checks` (deterministic on-page checks).
97
+
98
+
**B differs from A only by:** the playbook injected into its prompt, a separate learning step, and a proactive check. Nothing else.
99
+
100
+
### Memory: diary → janitor → playbook
101
+
102
+
Two layers:
103
+
104
+
-**Diary** — every raw observation B writes, one per audit. Append-only, messy. Each entry is **structured**, not free text: `page_kind`, `rec_type`, `target`, a one-line `why`, and a 0–100 `confidence` B assigns itself.
105
+
-**Playbook** — the trusted, distilled subset that gets fed back into B's prompt.
106
+
107
+
The **janitor** turns one into the other:
108
+
- groups diary entries by the exact key `(page_kind, rec_type, target)` — structured, so nothing drifts;
109
+
- promotes a pattern only once it **recurs across ≥3 distinct runs**;
110
+
- scores it by a **recency-weighted average** of B's own confidences, bucketed low/medium/high;
111
+
- keeps the top ~10 rules, each stamped with **provenance** (the diary rows it came from);
112
+
- skips rec types whose target is just the page's own keyword (rank tracking, keyword suggestions) — those would memorize a keyword, not learn a pattern.
113
+
114
+
So the diary grows with every run; the playbook stays a small, trusted, page-kind-specific set.
115
+
116
+
### How B learns
117
+
118
+
1. B audits a page (exactly like A).
119
+
2. In a **separate reflection step** (its own LLM call, *after* the audit so it can't distort the recommendations), B writes one structured lesson to the diary.
120
+
3. The janitor consolidates. When a pattern has recurred enough, it's promoted to the playbook.
121
+
4. The next audit of that kind of page injects the top playbook rules into B's prompt.
122
+
123
+
### Proactivity
124
+
125
+
B surfaces problems the audit didn't, via a **blind-spot scan**: a deterministic pass that runs every on-page check and reports real failures the recommendations missed (capped, severity-ranked). A never runs it, so it's purely B's edge — e.g. catching a missing meta description the LLM dropped.
39
126
40
-
## How B learns
127
+
> This is currently a fixed checklist, not the agent deciding for itself. Making it memory-driven (check the page against B's *learned* playbook) and/or an LLM "what am I missing?" step is planned — see `DESIGN.md`.
128
+
129
+
### Background self-learning
130
+
131
+
B doesn't only learn from your audits. A **scheduler** runs a learning batch on a timer:
132
+
133
+
- for each **seed topic** (a given list of keywords across page kinds),
134
+
- it searches the live SERP for real pages, skips bot-blocked ones (a cheap check, no LLM wasted),
135
+
- queues the rest as low-priority "audit + learn" jobs,
136
+
- then runs the janitor.
137
+
138
+
So B improves on its own between jobs. It never crawls the open web — discovery is anchored to the **given seed topics**. (Runs only when keys are present; off otherwise.)
139
+
140
+
### The job queue
141
+
142
+
User audits and background learning share **one Redis queue with a single worker**:
143
+
144
+
-**User runs = high priority** — they jump ahead of all queued background work.
145
+
-**Background learning = low priority.**
146
+
- The worker never preempts the job in flight, so a user waits at most for the **one** background page currently being processed — never the whole batch.
147
+
148
+
User runs are **async**: `POST /run` returns a `job_id` immediately; the worker fills in the result; the UI polls until done. Every run is stored, so it's reload-safe.
149
+
150
+
### Live vs replay
151
+
152
+
Every LLM call is cached in MongoDB, keyed by the exact request (model + prompt + tools). **Live mode** calls the API and records; **replay mode** serves the recording with no key and no network. This is what makes the eval deterministic and (once seeded) keyless.
153
+
154
+
---
155
+
156
+
## Proving B beats A: the eval
157
+
158
+
The graded centerpiece is the **eval harness** (`app/eval/`), not the agents. It scores A and B against a typed **answer key** on four metrics:
159
+
160
+
-**Coverage** — did it catch the recommendations that matter (severity-weighted)?
161
+
-**Correctness** — were its recommendations right (precision against the key)?
162
+
-**Novelty** — did it surface less-obvious wins (rarer recommendations weigh more)?
163
+
-**Efficiency** — value per cost (fewer tool calls / tokens for the same result).
164
+
165
+
Key properties:
166
+
-**Frozen matcher** — a recommendation matches the key when `type == type` and normalized `target == target`.
167
+
-**Deterministic** — runs off recordings (replay), so the score is the same every time, in CI, with no key.
168
+
-**Train/eval split + leak guard** — B learns only on the train split; both agents are scored cold on a **held-out** eval split; a leak guard fails loudly if any eval page sneaks into memory. This is what makes "B > A" a real claim, not luck.
169
+
170
+
It compares **A** vs **B with memory off** vs **B with memory on**, so the lift is isolated to memory.
171
+
172
+
---
173
+
174
+
## API
175
+
176
+
All under `/api`:
177
+
178
+
| Method | Path | What |
179
+
|---|---|---|
180
+
| GET |`/health`| Status + mode (replay/live). |
181
+
| POST |`/run`| Enqueue an A-vs-B audit `{url, keyword}` → `{job_id}`. |
182
+
| GET |`/run`| Recent runs (newest first). |
183
+
| GET |`/run/{job_id}`| One run's status + full result. |
184
+
| GET |`/memory/playbook`| Promoted rules. |
185
+
| GET |`/memory/diary`| Raw observations (provenance behind the rules). |
186
+
| GET |`/eval/report`| The A-vs-B scored comparison (cached). |
187
+
188
+
## Project layout
189
+
190
+
```
191
+
backend/app/
192
+
core/ config, db (lazy Mongo), logging, exceptions
1. After an audit, B writes **one lesson** to its diary — e.g. *"on recipe pages, suggest adding ingredient substitutions."*
43
-
2. A cleanup step (the **janitor**) reviews the diary. When the same lesson shows up across **3+ different pages**, it gets promoted to the **playbook**.
44
-
3. The next audit feeds the playbook back into B, so it applies what it learned.
204
+
## Tech stack
45
205
46
-
B also learns in the background: between your audits it picks real pages for a fixed list of **topics** and audits them on its own. It never wanders the open web — only those given topics.
A separate **eval** scores A and B against a fixed answer key, on pages B has never seen, across four measures: coverage, correctness, novelty, and efficiency. It's repeatable and needs no API key. Live, B's edge shows up as extra correct fixes A misses, plus the problems it proactively flags.
210
+
| Command | Does |
211
+
|---|---|
212
+
|`make up`| Build + start the whole stack. |
213
+
|`make down`| Stop everything. |
214
+
|`make logs`| Tail logs. |
215
+
|`make test`| Backend test suite. |
51
216
52
-
## What's inside
217
+
## Known limits / planned
53
218
54
-
-**Backend** (FastAPI + MongoDB + Redis): the agents, tools, memory, the job queue, and the eval.
55
-
-**Frontend** (React + Vite): the two screens.
56
-
-**Model**: `claude-sonnet-4-6` (change via `AGENT_MODEL`).
219
+
-**No committed seed yet** — a fresh clone needs keys for the first runs (see Quickstart). Shipping recorded fixtures + a pre-trained playbook so a clone shows B>A keyless is the top planned item.
220
+
-**Proactivity is a fixed checklist**, not the agent deciding for itself. Planned: memory-driven blind spots + an LLM proactive pass.
221
+
-**The monitor** (re-check already-seen pages over time, alert/re-audit unprompted) is designed but not wired live.
222
+
-**Background learning uses a fixed seed-topic list.** Planned: active learning — B picks its *weakest* page kind to study next.
223
+
- Minor: de-duplicate B's recommendations; turn the deterministic checks into recommendations for both agents so obvious defects are never missed; add a rate-limit backoff so the background loop can't exhaust the SERP quota.
57
224
58
-
See `DESIGN.md` for why it's built this way, and `transcripts/` for real example runs.
225
+
See **`DESIGN.md`** for the reasoning behind every decision (and the open questions), and **`transcripts/`** for real example runs.
0 commit comments