Skip to content
Merged
Show file tree
Hide file tree
Changes from 2 commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
274 changes: 31 additions & 243 deletions .agents/NEXT.md

Large diffs are not rendered by default.

7 changes: 2 additions & 5 deletions .agents/TODO.md
Original file line number Diff line number Diff line change
Expand Up @@ -7,11 +7,8 @@ gates, and documentation. Upstream service defects go to `improvements/` instead
Add an item when you find work you are not doing now. Delete it when it is done; git history is the
archive. Each item states what is wrong, how it was found, and what "done" means.

Updated 2026-09-09 during issue #141 and upstream handoff verification.
Rejected-candidate retirement and OAuth completion were recorded on 2026-09-10.
The OAuth record is [the consent completion ledger](rounds/2026-09-10-oauth-consent.md).
The current ledger is `.agents/rounds/2026-09-14-truth-maintenance.md`. The ranked handoff is
`NEXT.md`.
The latest maintenance review is [the September 14 ledger](rounds/2026-09-14-truth-maintenance.md).
[NEXT.md](NEXT.md) ranks the work and holds open owner decisions.

## Improvements follow-up

Expand Down
8 changes: 3 additions & 5 deletions ARCHITECTURE.md
Original file line number Diff line number Diff line change
@@ -1,10 +1,8 @@
# Architecture — how `search` and `execute` actually work

The end-to-end mechanics of the two tools, verified against the code as of 2026-07-03
(`codemode.skill.run` surface added and code-verified 2026-07-06). Read
[`PLAN.md`](./PLAN.md) first for *why* the design is shaped this way; this doc is the *how*,
with file paths for every claim. Nothing here is aspirational — if the code moves, this doc
is wrong until refreshed.
This document describes the implemented request paths and their source files.
Read [PLAN.md](PLAN.md) for product scope and [README.md](README.md) for connection and operation.
The source code owns current behavior; dated research records explain earlier decisions and experiments.

Two tools, one Worker: `search` is a host-side ranked query over a generated catalog;
`execute` runs LLM-authored JavaScript in a network-less Dynamic Worker isolate whose only
Expand Down
471 changes: 111 additions & 360 deletions PLAN.md

Large diffs are not rendered by default.

3 changes: 3 additions & 0 deletions eval/README.md
Original file line number Diff line number Diff line change
@@ -1,5 +1,8 @@
# Search-routing eval

Use [How to run](#how-to-run) for commands and `gates.json` for the accepted thresholds and source fingerprints.
The dated result sections preserve historical evidence; they do not describe the current deployment or authorize another experiment.

> **Start at [`eval/EVALS.md`](./EVALS.md)** — the one-page map of all eval instruments,
> which numbers are gates vs diagnostics, and the rules that keep them targeted.

Expand Down
2 changes: 1 addition & 1 deletion eval/discovery/README.md
Original file line number Diff line number Diff line change
Expand Up @@ -62,7 +62,7 @@ policy; its exact stamp and aggregate are retained here.

## Seed Pools

- `extended-strict-misses`: the 12 current extended-lane strict top-5 misses derived from `eval/routing-cases.json`. They are the cases where strict labels still miss while broader acceptable families can be defensible.
- `extended-strict-misses`: 12 cases selected from extended-lane strict top-5 misses when this pool was authored. Their source references remain in `cases.json`; membership does not track the current routing miss count.
- `issue-9-exemplars`: vague/status/current-recommendation questions from GitHub issue #9's exemplar class, authored against exact manifest ids.
- `lumenloop-agentic-misses`: the 8 LumenLoop-labelled cases (`expected_service: lumenloop`) from the real agentic run `eval/agentic/results/agentic-2026-07-04-drift.json` (local-only, gitignored). Ground truth is authored from each case's `expected_cards`, so `expectedFamilies` is `["lumenloop"]` for the seven LumenLoop-only-card cases and `["lumenloop","scout"]` only for `blend-tvl`, whose authoritative card list itself includes `scout_analyze`. This pool deliberately measures **LumenLoop-family** discovery: a miss means one-shot search did not surface the intended LumenLoop editorial/directory source, even in cases where Scout may still return a factually usable answer. Scout was dropped from `phoenix-scf` (SCF submission history is a LumenLoop-specific dataset Scout cannot serve).
- `round-844-real-user`: representative real-user alias/error/tooling questions from the round-844 lane context in `eval/README.md`, using committed routing-corpus questions as durable refs.
Expand Down
2 changes: 1 addition & 1 deletion eval/playground/README.md
Original file line number Diff line number Diff line change
Expand Up @@ -94,4 +94,4 @@ Run that regrade after playground prompt or tool-loop changes. Treat this as a s

For live loopback runs, the harness reads `MCP_SERVER_SECRET` from the environment or `.dev.vars` without logging it and mints an ephemeral run-scoped demo cookie. This exercises the normal cookie-authenticated per-subject throttle while avoiding collisions in the shared `dev-loopback` bucket; it never uses this cookie against a non-loopback URL.

One run-scoped subject is intentionally limited to the real `DEMO_CAPS.chatsPerHour` ceiling (currently 30). Selection over that cap, including `--full` against the 469-case main battery, is rejected before the harness mints a cookie or sends a request. Use `--sample 30`, a <=30-case named contract, or an explicit `--ids` shard; it never rotates subjects to evade the safeguard.
One run-scoped subject is intentionally limited to the real `DEMO_CAPS.chatsPerHour` ceiling (currently 30). Selection over that cap, including `--full` against the main battery, is rejected before the harness mints a cookie or sends a request. Use `--sample 30`, a <=30-case named contract, or an explicit `--ids` shard; it never rotates subjects to evade the safeguard.
13 changes: 8 additions & 5 deletions eval/qa/README.md
Original file line number Diff line number Diff line change
Expand Up @@ -4,17 +4,20 @@ Measures what the routing evals (`eval/run-routing.mjs`, `eval/agentic/`) can't:
driving this MCP server end-to-end (**search → execute → answer**) produce a **factually
correct, current, non-fabricated answer** to a real Stellar-ecosystem question?

The battery is **owned**: one hand-authored JSON file per case under `eval/qa/corpus/battery/`,
500 cases as of 2026-08-28 (499 as of 2026-08-19), edited directly and reviewed like code. The 2026-08-18 retrieval
audit added five service-semantics cases to the prior 492-case corpus. The 2026-08-19
maintenance change added two broad `scout.hackathonBrief` cases. Provenance is first-class (`truth` block per case), gospel changes are CI-linted at
the moment of change, and the compiled artifacts are generated + byte-pinned. History — the
The battery is owned: one hand-authored JSON file per case under `eval/qa/corpus/battery/`, reviewed like code.
The generated lifecycle registry owns current membership and case counts.
Each case has a `truth` provenance block. CI checks golden changes and byte-pins the generated artifacts.
History — the
vendored-corpus/override era, rubric evolution, and the run archaeology through 2026-07-10 — lives in
[`research/audits/2026-07-qa-history.md`](../../research/audits/2026-07-qa-history.md); the
migration proof is [`reviewed/2026-07-super-corpus-migration.md`](./reviewed/2026-07-super-corpus-migration.md).

## Directory / lane map

Use [Commands](#commands), [Golden lifecycle](#golden-lifecycle), and
[Judging rubric and score comparability](#judging-rubric-and-score-comparability) for current operation.
The dated run sections retain measurement evidence. They do not authorize new paid calls.

```
eval/qa/
corpus/
Expand Down
12 changes: 4 additions & 8 deletions ideas/README.md
Original file line number Diff line number Diff line change
Expand Up @@ -4,12 +4,9 @@ Research notes and possible future work that are not committed product plans.

- [Observability R2 Retention Plan](./observability-r2-retention.md) — optional Logpush/R2 archive
design for investigations that need history beyond the Workers Logs window.
- [Per-user MCP Observability and Future Personalization](./per-user-mcp-observability.md) — current
`/mcp` identity and request tracking, privacy boundaries, and deferred adoption analytics without
turning telemetry into a profile store.
- [Architecture Explorations](./architecture-explorations.md) — ranked A/B and readiness
candidates from the 2026-07-07 first-principles codemode review (per-op null hypothesis,
embedding hybrid, MCP 2026-07-28 spec spike, evidence sidecar, and deferred leftovers).
- [Per-user MCP Observability and Future Personalization](./per-user-mcp-observability.md) — historical identity-design evidence and deferred personalization.
[The usage guide](../usage/README.md) describes the implemented private aggregate reports.
- [Architecture Explorations](./architecture-explorations.md) — remaining questions and completed experiment references from the architecture review.
- [Docs Recency Ranking](./docs-recency-ranking.md) — why modification time should remain
agent-visible metadata or a measured experiment rather than a default ranking boost.
- [Direct Stellar.org Source Coverage](./stellar-org-source-lane.md) — revisit the held
Expand All @@ -21,8 +18,7 @@ Research notes and possible future work that are not committed product plans.
optional user-funded playground inference via ChatGPT/Codex OAuth, with hosted-token custody,
consistency, budget, and upstream-support gates recorded before any spike.
- [Shareable Durable `/playground` Sessions](./shareable-durable-playground-sessions.md) — deferred
durable-session design; the shipped Playground stays stateless while the separate 8,000-character
input change proceeds.
durable-session design; the Playground stays stateless with the existing 8,000-character input limit.
- [Source Delivery: Ranked References](./source-delivery-ranked-references.md) — a deferred
`sources.locate` surface whose measured reopen trigger follows repository-recovery work.
- [Skill Discovery: What Is Still Open After De-vendoring](./skill-discovery-without-bundling.md) —
Expand Down
77 changes: 20 additions & 57 deletions ideas/architecture-explorations.md
Original file line number Diff line number Diff line change
@@ -1,73 +1,36 @@
# Architecture exploration backlog — from the 2026-07-07 first-principles review

Provenance: the 24h ship review (Solo scratchpad `24h-ship-review-2026--551`, archived) ran a
first-principles evaluation of the codemode architecture against this project's goals, with
mid-2026 external evidence. Verdict: the architecture is right and reasonable — two-tool
search+execute economics are ecosystem-validated (Cloudflare's own 2,500-endpoint Code Mode
gateway, Anthropic's 98.7%-reduction guidance), the dossier-runner retirement proved models
out-compose host orchestration, and the cap/artifact-lane decisions are the best-evidenced in
the repo. But the review named unmeasured assumptions and time-sensitive work. This file is the
ranked backlog so it survives the scratchpad's archival. Each item follows the house rule:
conviction or a winning A/B on golden Q→A accuracy.
This backlog retains open questions from the July 7 architecture review.
Completed experiments link to their evidence. They are not queued for another run.
New experiments need the evaluation and authorization gates in [the evaluation map](../eval/EVALS.md).

## Ranked candidates

1. **Per-op MCP tool server vs search+execute — MEASURED NULL / NO SHIP (2026-07-10, todo 903).**
The manifest-derived 50-operation harness ran against the fixed QA-30 and canonical live-10
lanes. After every partial/wrong/flip was reviewed, QA favored search+execute
(20C/9P/1W vs 17C/12P/1W); live favored direct by one verdict (9C/1P vs 10C), within a single
drifting replicate with cross-lane-only order reversal. Direct used lower cache tokens/cost but
more QA turns/calls, weaker plan coverage, and more truncation markers. The large definition
delta was only advertised wire surface under Claude deferred tools, not always-consumed context.
Skills and artifact-read were unavailable in the direct arm. Retain search+execute; do not
rebaseline. Harness, identities, metrics, and row dispositions:
[`eval/qa/reviewed/2026-07-10-per-operation-architecture-ab.md`](../eval/qa/reviewed/2026-07-10-per-operation-architecture-ab.md).
1. **Per-op MCP tool server vs search+execute — MEASURED NULL / NO SHIP.**
Keep the two-tool architecture. The July 10 experiment did not justify a rebaseline or release.
The direct arm lacked skills and artifact reads; its advertised definition size was not consumed-context evidence.
[The reviewed result](../eval/qa/reviewed/2026-07-10-per-operation-architecture-ab.md) preserves the identities, metrics, and row dispositions.

2. **Hybrid lexical+embedding retrieval A/B — MEASURED NO-SHIP (todo 902, 2026-07-10).**
The 2026-07-09 round greenlit the Vectorize frontier spike and hardened the
case: the lumenloop capture class is an entity→family mapping problem in a 66%-mixed-register
query stream, and BOTH lexical levers are now measured at their ceiling — prose guidance is
zero-sum across 7 live agentic runs, and generic catalog cards intercept ≤6.9% of real agent
queries even at the enrichment upper bound (`research/p2-outcome-addendum.md`). What has NOT
been tried: **Cloudflare Vectorize (or committed build-time vectors) with a frontier
embedding model** — round 5f (eval/README.md) tested only offline bge-base and failed all
four modes; that result binds the design (stronger model, determinism preserved: pinned
model + committed generated vectors or a versioned Vectorize index built in refresh, never
at request time), not the idea. Spike design: target exactly the measured class — referee
with the discovery instrument, the mined real-query set (the unavailable historical artifact
was 66% mixed register; the retained reproducible lane is 46.2%), and live
agentic runs at ≥3 runs/arm with per-case family matrices (single runs proven unable to
gate ±2-case movements); docs 100% and scout-medium hold as blocking guardrails. Run in a
fresh worktree after the discovery-redesign merge. **Outcome:** the pinned local
Qwen3-Embedding-0.6B rerank harness and prerequisite discovery lanes landed as eval assets,
but the retrieval mode failed the composite ship gate: mined target top-5 cleared its isolated
lift while top-1 regressed, legacy and extended gates broke, and three vector agent runs failed
docs/scout blockers despite a noisy LumenLoop primary lift. No production
Vectorize/index/runtime path shipped. Full record:
`eval/vectorize/README.md`.
2. **Hybrid lexical+embedding retrieval — MEASURED NO-SHIP.**
The pinned Qwen3-Embedding-0.6B experiment failed the composite release gate.
No production Vectorize binding, index, or retrieval path shipped.
[The Vectorize record](../eval/vectorize/README.md) retains the harness, failures, and reopen requirements.
Any successor must preserve deterministic model/index pins and the Docs and Scout blocking controls.

3. **MCP 2026-07-28 spec readiness spike — RESOLVED 2026-07-30.** Both halves landed. The
instructions-channel question was decided 2026-07-09 (`research/discovery-redesign.md` §4 P1
step 3 — instructions survive in optional `server/discover`; SEARCH_DESCRIPTION is the reliable
carrier). The transport-level compat work then shipped: the handler serves both wire eras —
the 2026-07-28 revision via `server/discover` negotiation, pinned end-to-end by
`test/smoke/mcp-modern-client.test.ts` (which asserts a modern client negotiates and fails
loudly rather than silently falling back), and the 2025 `initialize` lifecycle through the
stateless legacy fallback. See `ARCHITECTURE.md` §1. The one thing this item asked for that is
now merely DUE rather than blocked: the spec date has passed, so re-check real client adoption
of `server/discover` in production logs when convenient — a telemetry question, not a spike.
3. **MCP 2026-07-28 transport readiness — COMPLETE.**
The handler supports modern discovery and the 2025 initialization lifecycle.
[The architecture](../ARCHITECTURE.md#1-a-search-call-end-to-end) and `test/smoke/mcp-modern-client.test.ts` own the current contract.
Checking real client adoption remains an optional telemetry question, not a transport implementation task.

4. **Compact operation-card code-shaped search re-test** — ADR-0001's own named next
experiment: hybrid ranked/code search over op cards with `codemode.search`/`describe`
parity in the search sandbox; rerun the 60-case paired A/B. Win = variant-B answer quality
without the max-turn exhaustion that killed it.

5. **Discovery-only eval instrument** — BUILT 2026-07-09 (`eval/discovery/`, discovery-redesign
round): 43 adjudicated cases over the live MCP HTTP surface, familyHit@3 / usableOp@5,
post-PR-17-fold baseline 32/43 · 25/43. Known scope limit: naive-verbatim single queries
under-represent the mixed register real agents use (66% of mined traffic). Planned extensions:
replay mined real queries (extraction pattern in the 2026-07-09 round ledger) and add the
agent-allowed-≤3-search arm promised by the original Phase 0 plan.
5. **Discovery instruments — COMPLETE.**
The [discovery guide](../eval/discovery/README.md) covers one-shot, bounded-agent, and mined-query replay lanes.
The retained mined sample does not recreate the unavailable July 9 artifact.
Do not carry the original extension plan as unfinished implementation work.

6. **Pre-cap evidence sidecar for QA judging** (runner change). Closes the residual
judge/agent evidence asymmetry at its root: agents read full payloads via
Expand Down
6 changes: 4 additions & 2 deletions ideas/observability-r2-retention.md
Original file line number Diff line number Diff line change
Expand Up @@ -3,9 +3,11 @@
Status: research note only. Do not build this until a real investigation needs history beyond the
current Workers Logs window.

Verified: 2026-07-07 against the current repo and current Cloudflare docs.
Historical research verified on 2026-07-07. Provider limits, prices, and source line numbers below are dated observations.
Use [the architecture](../ARCHITECTURE.md) for current logging and [the usage guide](../usage/README.md) for aggregate retention.
The implemented usage archive is not the raw-log archive proposed here.

## Current state
## Observed state — 2026-07-07

The production worker already emits useful structured observability. `src/observability.ts` is a
thin helper: `logEvent(evt, fields)` writes one flat JSON object to `console.log`. The file's
Expand Down
11 changes: 6 additions & 5 deletions ideas/per-user-mcp-observability.md
Original file line number Diff line number Diff line change
@@ -1,7 +1,8 @@
# Per-user MCP Observability and Future Personalization

Status: privacy-safe request attribution implemented and production-verified 2026-07-13. Product
analytics and personalization remain deliberately deferred.
Status: historical request-attribution design, implemented and production-verified on 2026-07-13.
Personalization remains deferred. [The usage guide](../usage/README.md) describes the later private aggregate-report implementation.
[The architecture](../ARCHITECTURE.md) owns current authentication, logging, and retention behavior.

Recorded: 2026-07-11 for Solo todo
`solo://proj/49/todo/track-users-better-v--889`.
Expand All @@ -14,8 +15,8 @@ We want to answer two related but different questions:
debugging, telemetry, traces, support, and aggregate product analysis?
2. Can that identity eventually support user-controlled memory that tunes answers to an individual?

The implemented local change adds privacy-safe user/client attribution to request logs while
preserving Cloudflare-native invocation joins. Production verification remains deploy-gated. Do
The change added privacy-safe user/client attribution to request logs while
preserving Cloudflare-native invocation joins. Do
not fingerprint users from IP, geo, TLS, or browser characteristics, and do not make operational
telemetry the future personalization database.

Expand All @@ -29,7 +30,7 @@ The target separation is:
These may share an internal user identity, but they need separate schemas, retention, access, and
user controls.

## Current implementation
## Implementation snapshot — 2026-07-13

### Authentication and user identity

Expand Down
Loading
Loading