diff --git a/README.md b/README.md index 05e8781..2e6bb40 100644 --- a/README.md +++ b/README.md @@ -1,83 +1,82 @@ # tallyman-notebooks Spike for the Tallyman London 2026 talk *"The Future of Notebooks in a Claude Code World"*. -The plan and proposal live in `plan.md` / `proposal.md`. This README covers the V0 spike only. +The proposal lives in `proposal.md`. This README covers the V0 spike only. +For the system architecture — subsystem map, on-disk layout, data-flow paths, and an +index of all the docs — start with [docs/architecture.md](docs/architecture.md). ## V0 scope End-to-end: a Claude Code MCP tool that compiles a xorq expression, materializes -a parquet to a content-hashed catalog entry on disk, and pushes a live update +a result to a content-hashed catalog entry on disk, and pushes a live update to a browser companion via SSE. What's working: -- **MCP tools:** +- **MCP tools** (FastMCP over stdio): - Catalog: `catalog_run`, `catalog_load_parquet`, `catalog_create`, `catalog_revise`, `catalog_alias`, `catalog_rename`, `catalog_unalias`, - `catalog_list`, `catalog_diff`. + `catalog_list`, `catalog_diff`, `catalog_chart`, `catalog_recalc`, plus the + summary-stat / post-processing / display-klass authoring tools. - Notebook: `notebook_reorder`, `notebook_remove`, `notebook_edit_markdown`. -- **Companion** (FastAPI on `:7860`): - - `/catalog`, `/catalog/` or `/catalog/` — entry list and detail - with V_n chips, forensic history, and a link to internal lineage. - - `/notebook` — curated narrative: cells anchored on aliases, vertical layout, - inline markdown editor, ↑/↓ reorder, × remove. - - `/lineage` and `/lineage/` — catalog DAG (cross-entry parents derived - from `tracked_expr_from_alias`) and per-entry internal expression DAG. Pure SVG, no - Cytoscape dep. - - `/diff/[//]` — version diff with code diff, schema diff, - per-column stats, key-joined side-by-side, and head() side-by-side. - - `/errors/` — build-failure detail. - - `/api/{entries,aliases,errors,notebook,lineage,catalog_dag}` — JSON. - - `/api/sse` — live updates (`new_entry`, `build_failed`, `alias_changed`, - `notebook_changed`). + - Project: `project_list`, `project_new`, `project_switch`. + + See [docs/architecture.md](docs/architecture.md) for the full tool surface. +- **Companion** (FastAPI on `:7860`) — serves the React SPA + (`packages/app/dist`) as a catch-all and exposes a JSON API + SSE under + `/{project}/api/*`: + - SPA tabs: **Catalog** (entry list + detail with V_n chips and forensic + history), **Notebook** (curated narrative anchored on aliases, drag-reorder, + inline markdown editor, × remove), **Diff** (code diff, schema diff, + per-column stats, key-joined side-by-side, head() side-by-side), **Cache** + (per-entry cache footprint), and **Log** (linear, filterable activity view). + - JSON: `/{project}/api/{entries,entry/,aliases,notebook,errors,log, + data/,diff_data/...,disk_usage,result_cache,staleness}`, plus the + mutation routes (`PATCH notebook`, `PUT code/`, + `PUT markdown/`, `POST reset`, `POST recalc`, + `POST promote_diff/...`). + - `/{project}/api/sse` — live updates (`new_entry`, `build_failed`, + `alias_changed`, `notebook_changed`, `recalc`, `summary_stat_changed`). + - `/internal/notify` — the MCP server's notification hook; fans out to SSE. - **Buckaroo subprocess** — `tallyman run` spawns `python -m buckaroo.server` on `:8700` (falls back to a random port if busy), watches for the `BUCKAROO_PORT=...` handshake, and lazily creates per-entry sessions on first view by POSTing the entry's `xorq_build/` dir to Buckaroo's `/load_expr` endpoint (PR 776) — sort/search push down to the xorq - backend rather than paging over a materialised parquet. The build dir - is expanded into a tmp copy first so `${TALLYMAN_PROJECT_ROOT}` - placeholders are resolved before xorq's loader sees them. Sessions are - persisted under `catalog/buckaroo_sessions.json` and invalidated by - start-time when Buckaroo restarts; tmp dirs are cleaned on - `BuckarooManager.stop()`. Tear-down rides along with the companion. - Disable with `--no-buckaroo`. + backend rather than paging over a materialised parquet. The build dir is + expanded into a stable per-entry path (`.xorq_build_expanded/`, gated by a + `.complete` marker) so `${TALLYMAN_PROJECT_ROOT}` placeholders are resolved + before xorq's loader sees them. Sessions are persisted in a global + `~/.tallyman-notebooks/buckaroo_sessions.json` (keyed by content hash, shared + across projects) and invalidated by start-time when Buckaroo restarts. + Tear-down rides along with the companion. Disable with `--no-buckaroo`. - **Build artifacts are portable.** xorq's absolute filesystem paths are rewritten to `${TALLYMAN_PROJECT_ROOT}` on write and expanded back on load. - **`tallyman serve `** — read-only companion against a project directory that may live anywhere on disk. Mutation routes return 403. -What's NOT yet implemented (see `TICKETS.md` for the full punchlist): +What's NOT yet implemented: -1. SortableJS drag-reorder for the notebook (current ↑/↓ buttons are the - accessibility fallback; drag is the headline UX). -2. Column-level lineage (xorq has the data; current view is op-level only). -3. `tallyman pack` / `tallyman replay` (today's hand-off is `cp -r` / `tar`). -4. ML training pipeline (storyboard beats 7-8). +1. Column-level lineage (xorq has the data; there is no lineage view today). +2. ML training pipeline (storyboard beats 7-8). ## Running the spike +Build the React companion UI once. The FastAPI server serves its `dist/`; Node + +pnpm are install-time prerequisites and the build artifact is not committed: + ```sh -uv sync -uv run tallyman init spike # creates ~/.tallyman/projects/spike/ + fixture -uv run tallyman run --project spike # edit-mode companion on http://127.0.0.1:7860 +cd packages/app && pnpm install && pnpm build # writes packages/app/dist/ ``` -The companion's dataframe embed is a Vite React library in `packages/embed/` -that builds into `src/tallyman_companion/static/buckaroo-embed.{js,css}`. The -build artifact is **not** committed — Node + pnpm are install-time -prerequisites: +Then start the stack: ```sh -cd packages/embed && pnpm install && pnpm build # one-time per checkout -# or, for active embed development: -cd packages/embed && pnpm dev # vite build --watch +uv sync +uv run tallyman init spike # creates ~/.tallyman-notebooks/projects/spike/ + fixture +uv run tallyman run --project spike # edit-mode companion on http://127.0.0.1:7860 ``` -Bump `buckaroo-js-core` in `packages/embed/package.json` to match the -Python `buckaroo` pin in `pyproject.toml` and rebuild whenever either -side moves. - In another terminal, launch Claude Code from this directory; it picks up `.mcp.json` and exposes the `tallyman` MCP server. @@ -96,10 +95,10 @@ forensic history. ### Serving a project as an artifact -Once you've authored a project, hand it off: +Once you've authored a project, pack it and hand it off: ```sh -tar czf my-project.tgz -C ~/.tallyman/projects spike +uv run tallyman pack spike -o my-project.tgz # portable .tgz, ${TALLYMAN_PROJECT_ROOT} preserved # colleague extracts somewhere tar xzf my-project.tgz -C ~/projects/ uv run tallyman serve ~/projects/spike @@ -127,6 +126,6 @@ affordances. Mutation routes return 403. ## Tests ```sh -uv run pytest # 76 tests, ~5s -uv run pytest tests/test_portable.py # the portability proof +uv run pytest # full suite +uv run pytest tests/test_pack.py # the pack / portability proof ``` diff --git a/TICKETS.md b/TICKETS.md deleted file mode 100644 index b1aaccd..0000000 --- a/TICKETS.md +++ /dev/null @@ -1,521 +0,0 @@ -# Tickets - -Generated from a working-system review 2026-05-11. -Each item names the file(s) to touch and a concrete acceptance check. - -Severity legend: - -- **P0** — will hurt the live demo on 2026-06-08 -- **P1** — visible polish; audience-noticeable but not blocking -- **P2** — planned-but-missing V1 work -- **P3** — engineering hygiene -- **P4** — design questions I punted in V0.5 - ---- - -## P0 — demo-blocking - -### ~~T-01 — SSE `new_entry` redirects from any tab~~ ✅ 2026-05-11 - -Fixed: the SSE handler now branches on `window.location.pathname`. On -`/catalog` (no hash) it reloads to refresh the entry list; on -`/catalog/` it focuses the new entry; on other tabs it does -nothing. See `src/tallyman_companion/templates/base.html`. - -### ~~T-02 — Catalog index page doesn't auto-refresh~~ ✅ 2026-05-11 - -Fixed alongside T-01. `/catalog` now reloads on `new_entry`. - -### ~~T-03 — `catalog_load_parquet` only produces scratch~~ ✅ 2026-05-11 - -Added optional `name` parameter; behaves like `catalog_create` when -provided. T-24 (hard-coded project name in synthesized code) fixed at -the same time — the synthesized code now relies on `resolve_project()`. - -### ~~T-04 — Markdown renders as `
`~~ ✅ 2026-05-11
-
-Server-side rendering via the `markdown` package (`fenced_code` +
-`tables` extensions). Raw markdown preserved in a `data-raw` attribute
-so the edit-toggle textarea gets the unrendered source.
-
-### ~~T-05 — `chart_attached` SSE has no client handler~~ ✅ 2026-05-11
-
-Listener added in `base.html`; reloads only when on the affected entry.
-
-### ~~T-06 — No `tallyman pack` command~~ ✅ 2026-05-11
-
-`tallyman pack ` produces `./-.tgz`,
-skipping `__pycache__`, `.DS_Store`, and the stale
-`buckaroo_sessions.json`. Round-trip serve verified by
-`tests/test_pack.py::test_pack_round_trip_serves`. See also T-24
-(also fixed) which would have invalidated builds on a rename.
-
-### ~~T-31 — Buckaroo session pre-hydration~~ ✗ moot under React embed 2026-05-20
-
-Premised on the iframe model where `GET /s/` returned an HTML
-shell that booted a JS bundle. After the iframe → React-embed swap
-(5ee5cc7) the embed mounts directly into the companion's page and
-hydrates over WS — there is no separate `/s/` HTML to
-pre-hydrate. The FOUC story is now "load WS → first frame," not
-"load JS → connect WS → fetch → render"; the right next move there
-is WS-side first-frame size reduction (see T-33's memory baseline)
-rather than HTML pre-hydration.
-
-### ~~T-32 — Iframe pool / multi-session caching~~ ✗ moot 2026-05-20
-
-Same iframe → React-embed swap; there are no iframes left to pool.
-Each `[data-ws-url]` div now mounts a `BuckarooServerView` in place,
-and SPA-lite navigation swaps the host element via the
-`MutationObserver` in `static/buckaroo-embed.js`. T-33 is the live
-guard for the memory shape this changes to.
-
-### ~~T-30 — XorqBuckarooInfiniteWidget over the standalone server~~ ✅ 2026-05-20
-
-**Buckaroo side:** Filed as
-[buckaroo-data/buckaroo#773](https://github.com/buckaroo-data/buckaroo/issues/773);
-implemented as
-[buckaroo-data/buckaroo#776](https://github.com/buckaroo-data/buckaroo/pull/776)
-(`feat/load-expr-server`). Adds `POST /load_expr` + a `backend="xorq"`
-discriminator on the existing `mode="buckaroo"` dispatch arm.
-
-**tallyman-notebooks side:** `BuckarooManager.ensure_session` POSTs to
-`/load_expr` with the entry's `xorq_build/` directory; the dead
-`mode` parameter is gone. One subtlety surfaced during integration:
-buckaroo's `xorq_loading.load_expr_build_dir` calls
-`xorq.api.load_expr` directly, so it does **not** resolve tallyman's
-`${TALLYMAN_PROJECT_ROOT}` placeholders. Without expansion the session
-loads fine (schema flows from YAML) but every paged read returns
-zero rows. ensure_session now `expand_to_tmp`s the build dir before
-POSTing and rmtree's the tmp on `stop()`. Tracked per content_hash
-so concurrent hits don't double-expand.
-
-Verified end-to-end: `GET /catalog/shoe_sales` →
-`buckaroo.server.handlers: load_expr ... rows=4 backend=xorq`.
-
-Files touched: `src/tallyman_companion/buckaroo_lifecycle.py`,
-`tests/test_buckaroo.py` (+5 tests: 4 stress, 1 row-count smoke
-integration). pyproject + uv.lock pin buckaroo to the local PR 776
-editable; switch to a git ref or published version once PR 776 lands.
-
-### ~~T-07 — Buckaroo server is not integrated~~ ✅ done 2026-05-11
-
-`tallyman run` now manages a Buckaroo subprocess (default `:8700`, falls
-back to a random port if busy). Per-entry sessions are lazily created
-on first view via `POST /load`, persisted under
-`catalog/buckaroo_sessions.json` with an envelope shape that records
-Buckaroo's `started` timestamp for restart detection. Iframe in
-`/catalog/` points at `/s/`. Tear-down via stdin
-close (Buckaroo's `--stdio-control` flag) with SIGTERM fallback.
-Disable with `--no-buckaroo` if Buckaroo's runtime deps are missing.
-
-Files added/touched: `src/tallyman_companion/buckaroo_lifecycle.py`,
-`src/tallyman_companion/app.py`, `src/tallyman_companion/templates/entry_detail.html`,
-`src/tallyman_cli/main.py`, `pyproject.toml`. 8 new tests
-(3 unit, 2 integration, 3 companion-level).
-
-Known follow-ups not in scope of this ticket:
-- Buckaroo-managed serve mode for richer hand-off (T-?? to file)
-- WebSocket reconnect handling when the user keeps a tab open across
-  Buckaroo restarts (today the iframe goes dead until reload)
-
----
-
-## P1 — visible polish
-
-### ~~T-08 — `difflib.HtmlDiff` output is ugly~~ ✅ 2026-05-11
-
-Swapped to `difflib.unified_diff` → Pygments DiffLexer + HtmlFormatter
-(monokai style, inline `noclasses=True` so it doesn't require a global
-CSS file). Identical inputs short-circuit to "(identical)" sentinel.
-
-### ~~T-09 — SortableJS drag-reorder~~ ✅ 2026-05-11
-
-SortableJS via jsdelivr CDN. The `⋮⋮` grab handle on each cell's
-header invokes drag; on drop we PATCH the new index. No reload — the
-DOM is already in the dropped position. The ↑/↓ buttons were removed
-since the drag handle covers their case (a11y note: keyboard users can
-still call notebook_reorder via the MCP tool). Drag is suppressed in
-serve mode.
-
-### ~~T-10 — Full-reload on PATCH/PUT~~ ✅ 2026-05-11
-
-- Reorder: drag-end PATCHes the new index; DOM already in correct
-  position, no reload.
-- Remove: PATCH succeeds, `.remove()` on the cell node.
-- Markdown edit: PUT returns `{cell, html}`, client swaps the rendered
-  HTML in place and removes the textarea. Escape cancels without
-  saving.
-
-Side note: I had to revisit the a11y story since the ↑/↓ buttons were
-removed. For now, keyboard users can call notebook_reorder via the MCP
-tool. A keyboard-driven move-cell affordance is a follow-up.
-
-### ~~T-11 — Diff page's compare-pairs link list~~ ✅ 2026-05-11
-
-Capped to adjacent (V_n→V_n+1) pairs plus a V_1→V_n bookend when
-n > 3. For 5 versions: 4 adjacent + 1 bookend = 5 links instead of 20.
-
-### ~~T-12 — Notebook serve-mode placeholder text~~ ✅ 2026-05-11
-
-Fixed in the T-04 template rewrite — placeholder only rendered when
-`not read_only`.
-
-### ~~T-13 — `/catalog` section headers~~ ✅ 2026-05-11
-
-Catalog list now buckets into named / forensic / scratch sections,
-each with a header showing the count. Item rendering factored into
-`_entry_list_item.html` to keep the list-building loop readable.
-
-### ~~T-14 — Raw expr.yaml viewer~~ ✅ 2026-05-11
-
-Entry detail now has a "build artifacts" disclosure that lists every
-file under the entry's `xorq_build/` (expr.yaml, expr_metadata.json,
-build_metadata.json, profiles.yaml). The `${TALLYMAN_PROJECT_ROOT}`
-placeholder is visible in the displayed text — useful for showing the
-portability story to an audience. Test asserts the placeholder is
-present in the rendered page.
-
-### ~~T-15 — Error banner doesn't name the failing tool~~ ✅ 2026-05-11
-
-`record_error` now takes a `tool` field. `_run_and_record` plumbs each
-tool's name through. Catalog banner shows it as a pill; error-detail
-page shows "tool: …". Tests assert both `catalog_run` and
-`catalog_create` failure paths.
-
----
-
-## P2 — V1 / planned-but-missing
-
-### ~~T-16 — Browser-side code edit~~ ✅ 2026-05-11
-
-`PUT /api/code/` builds + revises an alias from edited code. New
-revision lands as V_{n+1}; V_n stays forensic. Errors are recorded via
-`record_error(tool="api_code")` and 400'd back, with the catalog banner
-picking them up via SSE.
-
-UI: `/catalog/` of a named entry now has an "edit + revise"
-button next to the code header (hidden for scratch entries and in
-serve mode). Click swaps the `
` for a textarea + save/cancel
-toolbar; save redirects to the new V_{n+1} on success.
-
-### ~~T-17 — Column-level lineage~~ ✅ 2026-05-11
-
-`tallyman_xorq.column_lineage(project, hash)` loads the entry via the
-portable load path, runs `xorq.common.utils.lineage_utils.build_column_trees`,
-renders each column's tree via `build_tree(...).__str__` (the ASCII
-art form). The `/lineage/` page now surfaces per-column trees
-under a "column lineage" section — one `
` per output column, -first one open. Tests assert all three demo columns appear and that -the trees trace through Filter and Field nodes. - -### T-18 — ML training pipeline - -Beats 7–8 ("train logistic regression... try gradient boosting -instead"). xorq has `xorq.ml` helpers — haven't explored. - -- **Fix:** TBD; needs a spike to understand xorq's ML surface, then - decide whether to wrap as `catalog_train(name, target, features, - model="logistic")` or expect Claude to write the xorq.ml call. -- **Verify:** beats 7–8 of the storyboard execute end-to-end. - -### ~~T-19 — `tallyman replay `~~ ✅ 2026-05-11 - -`tallyman replay ` reads `{project, steps: [{tool, args, -narration?, skip?}]}` and drives the MCP tools in order. `--delay N` -paces between steps; `--continue-on-error` pushes through failures -for stage fallback. `demo/storyboard.json` exercises beats 1, 3, 4, -5, plus a chart attach — a regression test asserts it replays to a -known state under an isolated TALLYMAN_HOME. - -### ~~T-20 — `/api/data/` pagination~~ ✅ 2026-05-11 - -Cursor-style pagination via `?offset=&limit=`. Default limit dropped -from 5000 → 200 (demo-safe; Buckaroo handles the full-data case -in-iframe via T-07). Response shape now includes `offset`, `limit`, -and `total` for clients that need to fetch more. - ---- - -## P3 — engineering hygiene - -### ~~T-21 — xorq cache shared across test runs~~ ✅ 2026-05-11 - -Conftest now sets `XORQ_CACHE_DIR` to a session-scoped `mkdtemp` at -module load (xorq freezes the env var at first import, so the env tweak -has to happen before any test or helper imports xorq). - -### ~~T-22 — Slow stdio tests aren't marked~~ ✅ already done - -`tests/test_mcp_stdio.py` tests are all decorated with -`@pytest.mark.integration`; `pyproject.toml` has -`addopts = "-m 'not integration'"`. Closed on review 2026-05-20. - -### T-33 — Memory-scaling integration test for the React embed - -Once the iframe-to-React-embed switch lands (Option B in the chat: -local esbuild bundle that mounts `BuckarooServerView` from -`buckaroo-js-core` into a `[data-ws-url]` div), we need a test that -catches per-cell leaks before they hit the demo. Each notebook cell -becomes its own `BuckarooServerView` mount with its own WebSocket; an -N-cell notebook = N WS connections + N AG-Grid instances. - -**What to measure, on the same machine, with the same fixture:** - -- Server side: RSS of the Buckaroo subprocess (`buckaroo_lifecycle.py`'s - `BuckarooManager.proc`) sampled before any cells load and after each - cell loads. `psutil.Process(self.proc.pid).memory_info().rss` is the - obvious hook. -- Browser side: `performance.memory.usedJSHeapSize` (Chromium only — - the test driver will need to be Chromium) sampled at the same - checkpoints from the rendered notebook page. - -**Test shape:** - -- Playwright spec under `tests/integration/` (new dir; we don't have - one yet — slot next to `tests/test_mcp_stdio.py`-style integration - files and gate with `@pytest.mark.integration`). -- Fixture: a project with N synthetic catalog entries, one notebook - cell per entry. Parametrise N over `[1, 5, 10, 25]`. -- For each N: launch `tallyman run`, load `/notebook`, wait for all - embeds to reach `initial_state`, sample both metrics, then close the - page and re-sample server RSS to catch sessions that don't release. -- Assert: server RSS growth per cell is sub-linear (i.e. shared - per-server overhead dominates) or at least bounded by a soft cap - (~20 MB/cell?); browser heap growth per cell stays under a soft cap - (~10 MB/cell?). Numbers are placeholders — first run sets the - baseline, future runs guard against regressions. - -**Why this is worth a ticket and not just a follow-up:** - -- The current iframe model isolates each cell in its own browser - context; React-embed collapses them into one. Leaks that were - invisible behind iframe boundaries become visible. -- The talk demo will load a multi-cell notebook live. If the embed - approach leaks per cell, we will only find out when the audience - sees it. -- Bounds the conversation about T-32 ("iframe pool") — once we have - numbers, "should we lazy-mount on scroll?" stops being a guess. - -**Files likely to touch:** - -- `tests/integration/test_embed_memory.py` (new). -- `tests/integration/conftest.py` (new; project + entries fixture). -- `pyproject.toml` — add `playwright` to a `dev` or `integration` - dependency group. -- `.github/workflows/tests.yml` — extend the integration job once the - buckaroo path-source issue (T-26) is resolved. - -Defer until Option B is merged. Should be the first test against the -new embed surface. - -### ~~T-34 — `/internal/notify` should log the event `kind`~~ ✅ 2026-05-20 - -`@app.post("/internal/notify")` now `log.info`s `kind` and `hash` -before publishing to subscribers. Logger is `tallyman.companion`; INFO -level is on by default under uvicorn's `log_level="info"`. - -### ~~T-35 — Companion startup banner should print buckaroo PID~~ ✅ 2026-05-20 - -`tallyman run` banner now includes `pid=` next to the URL. After an -auto-restart in `BuckarooManager._maybe_restart`, the success log -includes both the new pid and the bound port (which can change if the -original port was reclaimed by something else). - -### ~~T-23 — `tallyman init` silently re-runs~~ ✅ 2026-05-11 - -`tallyman init` now errors if the project already exists. `--force` -proceeds (re-initialises dirs that may already be there; preserves -catalog entries and aliases). Tests cover both paths. - -### ~~T-24 — `catalog_load_parquet` hard-codes the project name~~ ✅ 2026-05-11 - -Fixed alongside T-03. The synthesized code now reads `read_project_file(rel_path)` -without an explicit `project=` argument, so a project rename or rehome -no longer breaks the build. - -### ~~T-25 — Dead dep: `sse-starlette`~~ ✗ wrong-headed 2026-05-11 - -The companion's `/api/sse` route returns `EventSourceResponse` from -`sse_starlette.sse`. My V0.5 description was wrong — I don't roll my -own SSE, the dep is in use. Closed without action. - -### ~~T-26 — No CI~~ ✅ 2026-05-24 - -`.github/workflows/tests.yml` runs three sequential jobs on push to -``main``/``spike/**`` and on PRs targeting ``main``: ruff lint, fast -test suite (`-m "not integration"`), and integration tests. Was -provisional until the buckaroo path source was resolved — pinning -``buckaroo==0.14.6`` from PyPI (commit a89c7b4) cleared the blocker. -The embed bundle is not built in CI; Python tests only reference its -URL, never load it. - -### T-36 — `/api/data/` leaf goes Arrow → pandas → records - -`api_data` in `src/tallyman_companion/app.py` currently ends with -`df.to_dict(orient="records")` after the parquet `iter_batches` → -arrow `slice()` → `to_pandas()` chain. The pandas step exists only to -re-emit records that FastAPI then re-serialises to JSON. - -At the actual call volume (vega-embed asks for ≤200 rows of a -pre-aggregated result), this is fine — ~5ms per request. Tracked as -hygiene rather than a real problem: drop pandas from the leaf with -`table.slice(offset, limit).to_pylist()` (arrow → list[dict] -directly), saving one copy and the pandas import on this path. - -Decision deferred — the architecturally pure answer ("send parquet -bytes, decode in browser with hyparquet, vega-embed gets -`{values: rows}` from the decoded objects") is over-engineered for -the chart-data scale we care about. Revisit only if a future caller -needs `/api/data` for larger payloads. - -### ~~T-37 — Embed bundle: esbuild → Vite~~ ✅ 2026-05-24 - -`packages/embed/` migrated from a hand-rolled esbuild script to Vite -library mode. Key issue: `process.env.NODE_ENV` was not replaced at -build time, causing `ReferenceError: process is not defined` at -runtime. Fixed via `define: { "process.env.NODE_ENV": -JSON.stringify("production") }` in `vite.config.ts`. - -Output: `src/tallyman_companion/static/buckaroo-embed.{js,css}` (both -in `.gitignore`; built locally before deploy / before demo). Build -command: `pnpm -C packages/embed build`. - -### ~~T-38 — Charts in notebook view~~ ✅ 2026-05-24 - -`notebook_view` now passes `chart_spec` per cell (via `get_chart`). -Template renders a `.chart-panel.nb-chart` div with `data-spec` and -`data-hash` attributes for any cell that has a chart. The shared -`_vega_mount.html` partial (also used by entry_detail) fetches -`/api/data/`, merges the rows into the Vega-Lite spec, and -calls `vegaEmbed`. CSS: `.chart-panel` uses `display: block` to avoid -collision with vega-embed's own `.vega-embed { display: inline-block }` -rule. - -### ~~T-39 — Post-processing MCP tools~~ ✅ 2026-05-24 - -Three new MCP tools mirror the summary-stats pattern: -`catalog_add_post_processing(name, source)`, -`catalog_remove_post_processing(name)`, -`catalog_list_post_processings()`. Source is validated by -`tallyman_core.post_processing.validate_post_processing_source` (exec in -restricted globals, dry-run against a 3-row memtable; accepts ibis -expr or pandas DataFrame). Scripts land in -`/post_processing/.py`; removal soft-deletes to -`/post_processing/_disabled/`. 15 tests in -`tests/test_post_processing.py`, all passing. - -### ~~T-40 — Large-parquet memory spike~~ ✅ 2026-05-24 - -Viewing a large expression (citibike, 57k rows) spiked companion RSS -to ~7 GB because `catalog_entry` and `api_data` both called -`pf.read()` unconditionally, materialising the full table into memory. - -Fix: `_read_head_rows(path, n)` uses PyArrow `iter_batches`, stopping -once N rows are collected — O(N) memory regardless of file size. -`catalog_entry` now only reads rows when Buckaroo is unavailable -(falls back to the HTML preview); total-row count comes from -`pf.metadata.num_rows` without reading data. `api_data` uses -`_read_head_rows` capped at the requested `limit`. - -### ~~T-41 — Embed size-toggle~~ ✅ 2026-05-24 (CSS layer; grid resize blocked upstream) - -A cycle button on each notebook cell toggles `.nb-buckaroo` between -three heights: 260 px (compact default), 75 vh (`data-size="3q"`), -90 vh (`data-size="max"`). `nbCycleSize(cellId, btn)` drives the -`data-size` attribute; no page reload. CSS transitions on -`.buckaroo-embed` make it smooth. - -Caveat: the AG Grid inside `BuckarooServerView` does not actually grow -when the host div expands, because `domLayout: 'autoHeight'` is not -exposed via `BuckarooServerViewProps` and the viewport-change signal -never reaches the grid. Tracked upstream as -[buckaroo-data/buckaroo#846](https://github.com/buckaroo-data/buckaroo/issues/846). -The CSS toggle is in place and will work once #846 is resolved. - -### T-42 — BuckarooServerView does not resize when host div grows (upstream #846) - -`BuckarooServerView` mounts an AG Grid with a fixed height derived -from the initial container size. When the companion toggles the -`.nb-buckaroo` div from 260 px to 75 vh, the host div grows but the -grid remains 260 px tall — extra rows are hidden. - -Root cause: `domLayout: 'autoHeight'` is available in AG Grid but not -wired into `BuckarooServerViewProps`; even if it were, the component -needs to call `gridApi.sizeColumnsToFit()` or respond to a -`ResizeObserver` on the host element. - -Filed upstream as -[buckaroo-data/buckaroo#846](https://github.com/buckaroo-data/buckaroo/issues/846). - -- **Fix:** Once the upstream PR lands and `buckaroo-js-core` is - published, bump the version in `packages/embed/package.json` and - rebuild the bundle. No companion-side code changes needed. -- **Verify:** Toggle size on a notebook cell with >10 rows; all rows - should be visible after toggle. - ---- - -## P4 — design questions punted in V0.5 - -### T-27 — Markdown re-seed on revise (plan Open Q1) - -Decided "stay with user edits" but never wrote a "re-seed from latest -prompt" affordance. If markdown drifts from the actual code, -demo-time confusion ensues. - -- **Fix:** add a "re-seed markdown" button next to "edit" on each - notebook cell that copies the manifest's `prompt` into the markdown. - -### T-28 — Scratch-parent semantics on revise (plan Open Q2) - -Catalog lineage points at the *hash at build time*, not "latest alias." -This is correct but undocumented. Users will be confused when revising -`shoe_sales` doesn't move its scratch verifications. - -- **Fix:** add a UI note on the catalog DAG explaining the build-time - binding; consider a `--reparent-scratch` flag for `catalog_revise` - that re-runs scratch entries against the new latest. - -### T-43 — Lint ordering non-determinism from Aggregation / JoinChain / Union - -`_nondeterminism_warnings` in `src/tallyman_xorq/build.py` currently covers -value-level non-determinism (`now()`, `random()`, `uuid()`, `sample()`). It -does not flag ops whose output ordering is unspecified: `Aggregation`, -`JoinChain`, `Union`. A consumer that depends on row order (diff, recalc -comparison) can silently see different bytes across cold-cache builds even -though no value changed. - -Design question before implementing: the lint should only fire when no `Sort` -node follows the flagged op — flagging a `union().order_by(...)` would be a -false positive. The check is "does a `Sort` appear strictly above the flagged -op in the tree?" rather than "does any `Sort` exist anywhere?" That walk is -straightforward but needs a test for the sort-above / no-sort-above cases. - -Punted from the reactive-recalc PR (PR #129 context). Implement separately. - -- **Fix:** extend `_NONDETERMINISTIC_OPS` (or a parallel `_ORDERING_OPS` set) - with `Aggregation`, `JoinChain`, `Union`; suppress the warning when a `Sort` - ancestor is present. -- **Files:** `src/tallyman_xorq/build.py`, `tests/test_build.py`. -- **Verify:** `catalog_run` with a bare `union()` emits the lint warning; a - `union().order_by(...)` does not. - -### T-44 — Require an ordering key when a UDF is present - -UDF expressions are non-deterministic in row order (and potentially in value -if the UDF is impure). The recalc / diff machinery relies on a stable row -order to compare builds. When a UDF is present the caller should be required -to supply an `order_by` at the outermost level. - -- **Fix:** in `_nondeterminism_warnings` (or a new `_validate_expr` gate that - hard-errors before execution), detect UDF ops (`ElementWiseVectorizedUDF`, - `ReductionVectorizedUDF`, `AnalyticVectorizedUDF`, `ScalarUDF`) and check - that the root op is a `Sort`. Error (not warn) if not. -- **Files:** `src/tallyman_xorq/build.py`, `tests/test_build.py`. -- **Verify:** `catalog_run` with a UDF and no `order_by` is rejected with a - clear message; adding `.order_by(key)` at the end passes. - -### T-29 — Multi-notebook per project - -V1 = one default notebook per project. V2 needs a tab selector and -per-notebook routing. diff --git a/demo/script.md b/demo/script.md deleted file mode 100644 index 2489d98..0000000 --- a/demo/script.md +++ /dev/null @@ -1,210 +0,0 @@ -# tallyman-notebooks demo script - -**Talk:** *The Future of Notebooks in a Claude Code World* — Tallyman London 2026, 2026-06-08. - -**Stage time:** ~12 minutes live + 1 min hand-off + 1 min closing. - -**Thesis to land:** the notebook is the *story*; the catalog is what *did the work*; the project directory is the *artifact* — content-hashed, reproducible, hand-off-able. - -**Pairing**: this script is the human-readable companion to `demo/storyboard.json`, which `tallyman replay` will execute deterministically if the live demo collapses (see Fallback). - ---- - -## Talk framing (spoken in the first ~90s, before the demo starts) - -Notebooks dominate scientific data work because data is large and computation is expensive. The standard answer to "don't recompute that" is kernel memory — you run the expensive cell once, the result lives in the Python process, and you carefully work around it for the rest of the session. - -That's an adhoc, fragile cache. The failure mode is familiar: you have a dataframe that's right at the edge of your machine's memory. You spent twenty minutes figuring out how to do the join without OOM-killing the kernel. Now it's in memory and you need to keep it there. You won't re-run that cell. You won't restart the kernel. If the session dies, you start over. The notebook isn't reproducible — it's reproducible *except for that one cell*, which is the whole point. - -But the OOM isn't even the worst part. The worst part is what happens after. The kernel dies. You lose everything — not just the cell that crashed, but the five-minute load and the two-minute join you ran before it. So you restart, re-run the load, re-run the join, and now you're staring at the cell that crashed. You don't run it yet. You test it on a sample first. You start navigating the notebook non-linearly — you know which cells are safe to re-run and which ones might kill you. You stop restarting the kernel even when you should, because there are results in memory you can't afford to lose. The notebook has become a minefield. It looks like a reproducible document. It isn't. It's a record of one particular session that you're too scared to replay from the top. - -Marimo addresses a related but different problem: it enforces that the DAG of cell dependencies in your notebook can always be solved, so stale state from out-of-order execution can't hide. That's real progress. But it doesn't help you when the cell you're avoiding is expensive, not stale. Marimo will happily re-run an expensive cell every time its inputs change — which is exactly what you were trying to avoid. - -xorq gives you a better answer: describe the computation, and xorq caches the result by content hash. Re-run the cell — same hash, cache hit, instant. The kernel isn't the cache. The filesystem is. - -That's what this demo shows. - ---- - -## Pre-flight (run 60s before going on stage) - -```sh -# Terminal A — companion + buckaroo subprocess -cd ~/code/tallyman-notebooks -uv run tallyman init spike --force # fresh fixture; idempotent on catalog -uv run tallyman run --project spike # http://127.0.0.1:7860 - -# Terminal B — Claude Code, .mcp.json auto-wires the tallyman MCP server -cd ~/code/tallyman-notebooks -claude . -``` - -Visual: terminal B (left half of screen, Claude Code), browser at http://127.0.0.1:7860/ (right half, full-height). Catalog tab is empty except for the fixture. - -Sanity checks before walking on stage: -- `lsof -iTCP:7860 -sTCP:LISTEN` — companion responding. -- `lsof -iTCP:8700 -sTCP:LISTEN` — buckaroo subprocess up. -- Browser shows empty catalog with the dataset fixture visible under `~/.tallyman/projects/spike/data/orders.parquet`. - ---- - -## Beats - -Each beat = one prompt typed into Claude Code. Watch the browser update live. Presenter cue in *italics*. - -### 1. Load and name the dataset (~60s) - -> Use `catalog_load_parquet` to load `orders.parquet`, name it `orders`. The prompt is "raw orders dataset for the demo". - -*Expected:* `orders` lands as the first named entry. Catalog tab auto-focuses on it. Buckaroo React embed mounts and starts streaming rows over the WS. Notebook tab now has one cell. - -*Presenter:* "The agent picked `catalog_load_parquet` because I gave it a name. If I hadn't, it would've used `catalog_run` and the entry would be unnamed scratch. The tool choice is the legible signal of intent." - -### 2. Free in-table recon (~45s, no prompt) - -*Click a numeric column header in the Buckaroo embed.* - -*Expected:* histogram + null counts + top values appear inline. No round-trip through Claude. - -*Presenter:* "This is the recon tier — three tiers in this model. Inline Buckaroo recon for per-column questions. The next tier is the catalog, which holds every executed expression. The top tier is the notebook, which holds the curated story." - -### 3. LLM-authored summary stat (~75s) - -> Add a summary stat called `boots_pct` that computes the percentage of rows in a column where the value is "boots". Use `catalog_add_summary_stat`. - -*Expected:* the tool writes `~/.tallyman/projects/spike/stats/boots_pct.py`, validates it against a 1-row ibis memtable, returns success. The next time a session is loaded (next beat), the new stat appears in the summary-stats bar above the `category` column. - -*Presenter:* "The stats bar above the Buckaroo table is project-scoped and authored by Claude. The agent writes a `compute(col)` function in a restricted-globals namespace. No real sandbox — this is single-user local — but the validator catches arity / import / non-ibis-return mistakes before the file lands." - -### 4. Filter and name (~60s) - -> Create a named entry `shoe_sales` that filters `orders` to `category == 'boots'` and groups by `region` with total price. - -*Expected:* `shoe_sales` named entry; notebook now has two cells; new buckaroo session; the `boots_pct` summary stat from beat 3 shows up in the new session's stats bar. - -### 5. Ad-hoc validation via a post-processing function (~75s) - -> We want to spot regions with too few transactions. Add a post-processing function `low_volume` that filters to rows where `n < 100`. Use `catalog_add_post_processing`. - -*Expected:* the tool writes `~/.tallyman/projects/spike/post_processing/low_volume.py`, validates it against a small in-memory table (must define `process(expr) -> ibis expression | DataFrame`), returns success. The file appears as a new option in the Buckaroo embed's **post processing** dropdown on the next session load. - -*Click the post-processing dropdown on the `shoe_sales` embed; pick `low_volume`.* Table re-renders to only the low-volume regions. - -*Presenter:* "Post-processing functions are reusable lenses on a result. Same surface as the LLM-authored summary stats from beat 3 — restricted globals, dry-run validator, soft-delete to `_disabled/` — but they transform the *table* rather than compute a per-column scalar. As the session goes on we build up a validation library; one click in the dropdown switches lenses without touching the underlying expression." - -### 6. Revise in place (~75s) - -> Refine shoe_sales to include `boots`, `sneakers`, and `sandals` — all categories of shoe. - -*Expected:* `shoe_sales` V_2 chip; V_1 moves to forensic history; notebook cell updates **in place** (no new cell). Buckaroo session re-mounts on the new content hash. - -*Presenter:* "The cell is the *concept*, not the iteration. Older versions stay in the catalog as forensic artifacts but never clutter the notebook." - -### 7. Forensic diff (~45s) - -*Click `Forensic history` in the entry detail, then click `Diff` next to V_1.* - -*Expected:* `/diff/shoe_sales/1/2` opens — code diff (difflib unified), schema diff, per-column stats, key-joined side-by-side, head() side-by-side. - -*Presenter:* "Every revise leaves an auditable trail. This is the diff a reviewer would want and almost never gets in a Jupyter workflow." - -### 8. Chart attached (~30s) - -> Attach a Vega-Lite bar chart to shoe_sales, region on x, total on y. - -*Expected:* chart panel renders above the table in the entry detail. Vega-embed pulls data from `/api/data/`. - -### 9. Notebook tab — rewrite markdown (~60s) - -*Click the Notebook tab. Click the markdown block above `shoe_sales`. Replace its content (or have Claude do it).* - -> Use `notebook_edit_markdown` to set shoe_sales's markdown to: "Shoes by region — broader bucket (boots, sneakers, sandals)." - -*Expected:* markdown updates live. SSE pushes to any other connected browser. - -### 10. Drag-reorder (~30s) - -*Drag the `shoe_sales` cell above `orders` using the drag handle.* - -*Expected:* SortableJS reorders; the new order persists in `notebooks/default.json`; SSE pushes to peers. - -*Presenter:* "The browser and the agent have parallel surfaces. The agent can call `notebook_reorder`; I can drag. The notebook is curated by whoever is closer to the work at that moment." - ---- - -## Hand-off (~60s) - -*New terminal, in front of the audience:* - -```sh -uv run tallyman pack spike -o /tmp/demo.tgz -mkdir /tmp/demo-handoff && tar -xzf /tmp/demo.tgz -C /tmp/demo-handoff -uv run tallyman serve /tmp/demo-handoff/spike --port 7861 -``` - -*Open http://127.0.0.1:7861/ in a second browser window alongside the first.* - -*Expected:* same catalog, same notebook, same Buckaroo embed, same forensic history. **No drag handles. No × buttons. No contenteditable.** The serve mode hides edit affordances and rejects `PATCH /api/notebook`, `PUT /api/markdown/`, and `POST /internal/notify` with 403. No MCP server. - -*Presenter:* "The project directory *is* the artifact. A colleague with `tallyman` installed sees exactly what I see. Every cell's expression is re-runnable from upstream parquet — content-hashed, deterministic, no kernel state to lose." - ---- - -## Closing line - -> "The notebook is the story. The catalog is what did the work. The project directory is the artifact — and it's reproducible because xorq cached every step by content hash, not by whatever happened to be in memory." - ---- - -## Fallback (if the agent goes off-script or the network drops) - -```sh -uv run tallyman replay demo/storyboard.json --delay 2 -``` - -Runs the same beats deterministically against the MCP tool surface. Pair it with the running companion on :7860 — the browser updates the same way as the live demo. Use `--delay 2` for stage pacing. - -Also rehearse a clean reset in case you need to restart mid-talk: - -```sh -rm -rf ~/.tallyman/projects/spike -uv run tallyman init spike # fresh fixture, fresh catalog -``` - ---- - -## Timing cheatsheet - -| Block | Target | -|---|---| -| Pre-flight | 60s, off-stage | -| Beats 1–3 (load, recon, summary stat) | ~3 min | -| Beats 4–6 (filter, scratch, revise) | ~3 min | -| Beats 7–8 (diff, chart) | ~75s | -| Beats 9–10 (notebook, drag) | ~90s | -| Hand-off | 60s | -| Closing | 30s | -| **Total** | **~11 min** + ~1 min slack | - -If running long: drop beat 8 (chart) — it lands well but isn't load-bearing for the thesis. Never drop the hand-off; that beat is the entire point of "project directory as artifact". - - --- notes -We need to talk about the thesis more. "ntoebooks are used for scientific data because it is large and expensive to compute. In kernel memory (better phrashing for this is needed) is an adhoc buggy cache that many notebook workflows depend on, but what you really want is a way to cache expensive computations. Xorq provides a better way to do that. with xorq you can describe the computations you want, and they will be cached. - - - -more about the buggy cache. I frequently have notebooks where I have done something expensive, normally a download script, reading in a dataframe or some join operation. Sometimes - especially with dataframes that are close to the size of my machine's memory, I will figure out how to just barely get an operation done without OOM killing my kernel, then I'll have a dataframe in memory that I have to carefully operate on. When I'm at that state I'll have this one cell that I need to execute once, then do other adhoc analysis in the notebook without re-running that cell. - -This hurts notebook reproducability, because I'm specifically not re-executing all cells in sequence since that would disrupt my flow. Marimo helps a little bit, but with a different problem, marimo makes sure that the graph of cell dependencies in your notebook can always be solved. for me the only time my notebook goes out of linear dependency order is when I have been changing the order for adhoc experimentation. - -I have done a limited amount of AI assisted notebook coding, but I have found that it is like collaborating with another human on the same notebook. I don't trust the state of the kernel and cells to change beneath me in a coherent way. - - -I joined xorq 3 months ago because I was excited about their core framework that regularlizes and unifies a lot of data engineering tasks into a declarative DSL that addresses cachability, reproducability, verifiable lineage into a coherent dsl. This talk builds heavily on the xorq cache system to build a system for interactive data analysis that bridges the gap between what the tallyman arrow stack is good at, claude does well, and what notebooks are good at. - - -when working in a notebook on a data problem (before buckaroo) I generally start by loading the dataframe and typing `df.head()` then I poke around a bit to get a feel for the data. Buckaroo makes this manual inspection much quicker (it's still very relevant in the claude code world). next I try to perform some operations on the dataframe, some types of transformations to expose a particular aspect or pattern in the data. Claude is very helpful for these parts for writing the python code. - -In addition to transforms I often end up with little transforms that I apply to dataframes (simple cleaning routines). when writing traditional notebooks, these adhoc transforms are messy throw away code, adding them to a library is cumbersome so they just die right there. Claude and buckaroo can help with this too. Buckaroo has a facility called post_processing functions which take a dataframe and return a dataframe, they can be interactively applied. - diff --git a/docs/architecture.md b/docs/architecture.md new file mode 100644 index 0000000..f49bdce --- /dev/null +++ b/docs/architecture.md @@ -0,0 +1,460 @@ +# Tallyman architecture overview + +This is the top-level map of the tallyman codebase. Read it first, then follow +the cross-references into the per-subsystem docs. For a document index with a +currency note on each file, see [Related documentation](#related-documentation) +at the end. + +## What tallyman is + +Tallyman is a deconstructed notebook platform. Instead of a notebook file with +inline cells and outputs, the unit of work is a **catalog entry**: a single +xorq expression, compiled and stored on disk under a content hash. Claude Code +is the author — it drives tallyman through an MCP server, creating and revising +entries as xorq expressions. The entries accumulate in an on-disk, +content-addressed, git-backed catalog. A FastAPI companion server plus a React +SPA visualize that catalog in a browser, and a Buckaroo subprocess provides +interactive data grids over each entry's result. + +Nothing in tallyman is a long-lived application server holding state in memory. +The catalog on disk is the source of truth. The running processes (companion, +Buckaroo, MCP server) are views and editors over it; the MCP server holds no +in-memory catalog state, and the companion holds only its SSE subscriber list. + +### Big-picture flow + +``` + Claude Code + │ (MCP tool calls over stdio) + ▼ + tallyman MCP server ──────────────► on-disk catalog + (compiles xorq exprs, ~/.tallyman-notebooks/projects// + checkpoints to git) (content-addressed entries, aliases, + │ notebook, git history) + │ best-effort HTTP notify ▲ + ▼ │ reads/writes + tallyman companion (FastAPI :7860) ────────┘ + │ REST + SSE + ▼ + React SPA (packages/app) in the browser + │ embeds grids + ▼ + Buckaroo subprocess (:8700) ◄─── companion POSTs xorq_build/ dirs + (interactive dataframe grids, sort/search push down to xorq) +``` + +The typical loop: Claude Code calls an MCP tool to create or revise an entry, +the MCP server compiles the xorq expression and writes a content-addressed +entry plus a git checkpoint, then notifies the companion. The companion emits a +Server-Sent Event (SSE, a push message over an HTTP stream the browser holds +open; see [Live updates over SSE](#live-updates-over-sse)), the SPA refetches +the affected data, and when the user opens +an entry the companion warms a Buckaroo session so the grid loads. See +[expression-lifecycle.md](expression-lifecycle.md) for the full +create-to-view path. + +Three processes run on one machine: `tallyman_mcp` (spawned by Claude Code over +stdio), `tallyman_companion` (the FastAPI app, started with `tallyman run`), and +a `buckaroo` server subprocess that the CLI/companion supervises. + +## Component map + +Tallyman is six subsystems: five Python packages under `src/` and one frontend +workspace under `packages/`. The dependency direction runs core ← xorq ← {mcp, +companion} ← cli; nothing in `tallyman_core` imports the web, MCP, or compute +layers. + +**tallyman_core** (`src/tallyman_core/`) is the native catalog model. It owns +the on-disk representation: versioned entries keyed by content hash, mutable +aliases with version history, the notebook cell list, chart specs, display +configs, project-global post-processing and summary-stat functions, and the +prompt/error/event logs. It manages the git-backed checkpoint transaction +(capture pointers, zip recipes, stage, commit, tag) and the reset-to-revision +operation that rewinds the catalog and reconciles untracked build artifacts +through a holding area called the bullpen. A key invariant lives here: +`assert_catalog_consistent` enforces an allow-listed tracked surface and +verifies that every hash referenced by an alias, chart, or display config has a +durable recipe zip. The call direction is one-way: `catalog_state` calls +`catalog`, never the reverse. Design: [native-catalog-store.md](../plans/native-catalog-store.md). + +**tallyman_xorq** (`src/tallyman_xorq/`) is the xorq integration layer. It +compiles a xorq expression into a content-addressed entry, computes the content +hash from the expression structure (and, in some source-identity modes, the +source file digests), decides whether the entry is expensive enough to bake a +result snapshot, and writes a portable build directory whose absolute paths are +rewritten to `${TALLYMAN_PROJECT_ROOT}` placeholders. It also implements +reactive staleness detection (comparing recorded manifest fields against the +current world) and the recalc cone that recomputes dependents in dependency +order. Reconstruction of an entry's expression from its persisted `expr.py` +happens here, using context variables that pin the entry's recorded source +digests. See [caching.md](caching.md) and [reactive-recalc.md](reactive-recalc.md). + +**tallyman_companion** (`src/tallyman_companion/`) is the FastAPI web server on +port 7860. It surfaces catalog views over REST, pushes live updates over SSE, +manages the Buckaroo subprocess lifecycle, and handles browser-initiated +mutations (code revision, diff promotion, resets). There are no server-side +HTML templates — it serves the compiled React SPA (`packages/app/dist`) as a +catch-all and mounts `/assets` and `/static`. A checkpoint middleware wraps +mutating routes so each authored change lands as one git revision, with an +opt-out denylist for routes that self-checkpoint or should not checkpoint at +all. It builds diff expressions (outer join with membership and per-column +delta/equality sentinel columns) and computes the Buckaroo column-config +overrides that color them. No dedicated subsystem doc yet — the route table is +in `app.py`; the create-to-view path is in [expression-lifecycle.md](expression-lifecycle.md). + +**tallyman_mcp** (`src/tallyman_mcp/`) is the FastMCP server that Claude Code +talks to over stdio. It exposes the catalog, notebook, and project tools +(`catalog_run`, `catalog_create`, `catalog_revise`, `catalog_alias`, +`catalog_diff`, `catalog_recalc`, the chart/display/stat/post-processing tools, +`notebook_*`, and `project_*`). Checkpointing is opt-out: every tool +auto-checkpoints at the dispatch boundary unless it is on the no-checkpoint +list. The active project is session-sticky (seeded on first tool call, surviving +disk changes within the session), and project lifecycle changes are POSTed to +the companion rather than written directly so SSE stays honest. Notifications to +the companion are best-effort and never raise. Every tool, its parameters, and +its side effects (checkpoint, SSE notify, auto-recalc) are documented in +[mcp-server.md](mcp-server.md). + +**tallyman_cli** (`src/tallyman_cli/`) is the Click command-line interface +(entry point `tallyman`). It initializes projects (with synthetic fixture data), +runs the companion and the MCP service, and owns the Buckaroo subprocess via +`BuckarooManager`, which spawns `python -m buckaroo.server` on port 8700 (or a +random free port) and exits when its stdin closes. It also provides `serve` +(read-only companion against a project directory anywhere on disk), `pack` +(portable tarball excluding cache and session state), `reset-to` / `revisions`, +and storyboard `replay` for deterministic rehearsal. See [installing.md](installing.md). + +**Frontend** (`packages/app/`) is the React 18 + Vite SPA. It builds to `dist/` +and is served by FastAPI as a catch-all; it drives refetches off an SSE version +counter rather than polling, defers grid loads until scroll via +`LazyBuckarooEmbed`, and remaps views to new hashes when a recalc event arrives. +Its `BuckarooEmbed` component mounts `BuckarooServerView` from `buckaroo-js-core` +directly and connects over WebSocket. No dedicated frontend doc yet. + +## On-disk catalog layout + +A project lives at `~/.tallyman-notebooks/projects//`. The home root is +`~/.tallyman-notebooks/` by default and is overridable with the `TALLYMAN_HOME` +environment variable (`paths.py:tallyman_home`). The single active project name +is recorded in `~/.tallyman-notebooks/active_project` (one line). The catalog +itself is a git repository at `/artifacts/catalog/`. + +``` +~/.tallyman-notebooks/ + active_project # one line: the active project name + buckaroo_sessions.json # global session map {hash: {session_id, project, started_at}} + projects// + artifacts/ + catalog/ # git repo — the tracked catalog + entries/.zip # tracked recipe zip (expr.py, schema.json, xorq_build/) + entries// # untracked build dir (gitignored, reconciled on reset) + entries.jsonl # pointer list, one {hash} per line + aliases.jsonl # one {alias, latest, history:[V1,V2,...]} per line + notebook.jsonl # one {cell_id, alias, markdown} per cell + compute_cache.jsonl # pointer list of warm cache files + config.json # project settings, e.g. {auto_recalc: bool} + chart_specs/.vl.json # Vega-Lite specs, keyed by content hash + display_configs/.json # {column_config_overrides, diff_provenance} + post_processing/.py # process(expr) functions (_disabled/ = soft-deleted) + stats/.py # compute(col) functions (_disabled/ = soft-deleted) + prompts/.jsonl # append-only per-entry prompt history + bullpen/ # untracked holding area for evicted entries/caches + compute_cache/ # untracked baked result snapshots (xorq) + diff_stat_cache/ # untracked Buckaroo diff-stat caches per entry pair + .gitignore # deny-by-default (entries/*/, bullpen/, caches, *.tmp) + display/.py # display klass files (ColAnalysis subclasses) + errors.jsonl # append-only error log (outside catalog repo) + events.jsonl # append-only activity log (outside catalog repo) + exports/ # marimo .py, screenshots, CSVs + data/ # user input parquets (fixtures) + data/.cas/ # content-addressed source clones (CoW), cas mode only +``` + +Key formats and what's tracked vs untracked: + +- **Recipe zip** (`entries/.zip`) is the durable, deterministic, + content-addressed entry. It contains `expr.py` (the author's literal source, + with portable placeholders), `schema.json` (field names and types), and + `xorq_build/` (the portable expression directory with `expr.yaml` plus deps). + The zip writer only runs inside a checkpoint. +- **Entry build dir** (`entries//`) is ephemeral and gitignored. It holds + `manifest.json` (the build-completeness sentinel), the load-time-expanded + `.xorq_build_expanded/`, and per-entry caches (`.buckaroo_stat_cache/`). A + directory without a `manifest.json` is treated as partial/crashed and is not + zipped. +- **Manifest** (`manifest.json`) carries entry metadata: `content_hash`, + `result_digest`, `sources` (`{rel_path: digest}`), `parents` (DAG edges), + `row_count`, `compile_seconds`, `execute_seconds`, `cache_worthy`, + `cache_bytes`. Written atomically (temp file + `os.replace`). +- **JSONL pointer files** (`entries.jsonl`, `compute_cache.jsonl`) and the + structured logs (`prompts/`, `errors.jsonl`, `events.jsonl`) are + append-oriented and line-delimited. They replaced the single `catalog.yaml` + that older docs reference. +- **Activity logs** (`events.jsonl`, `errors.jsonl`) live in `artifacts/`, + outside the catalog git repo, so they survive reset-to-revision and have no + size cap. The UI filters them on read. +- **No per-entry `result.parquet`.** That layer was removed (#104); an expensive + entry's rows live in the baked `.cache()` snapshot under `compute_cache/`, a + cheap entry keeps no copy and recomputes on read. + +All catalog writers use atomic writes, so a crash mid-write or a checkpoint +firing during a write window leaves a whole file, not a torn one. + +## Core domain concepts + +**Content hash (identity).** Every entry is identified by a hash derived from +its expression structure (xorq's tokenization) and, depending on the +source-identity mode, its source file digests. Identity is structural: two +entries with the same expression and inputs collapse to the same hash, which is +what makes builds idempotent. Because hashes are content-addressed and globally +unique by construction, Buckaroo sessions are keyed globally by content hash, +not per project. The source-identity mode (`off` / `cas` / `salt`, default +`cas`) is decided in [adr-source-identity-content-hash.md](../plans/adr-source-identity-content-hash.md). + +**Result digest.** A second identity axis, recorded for *worthy* (snapshot-baking) +entries only. The content hash keys the expression graph; the `result_digest` +keys the executed *bytes* as a row **multiset**, not a sequence. It is the +SHA-256 of the entry's baked snapshot parquet (`snapshot_file_digest`), which the +bake writes after sorting on a synthetic `original_row_order` and pinning the +parquet write settings, so the bytes are reproducible run-to-run and the file +hash is order-insensitive. Cheap, row-preserving entries record no digest — they +have no snapshot to hash and recompute live. A mismatch when an evicted snapshot +self-heals points at execution nondeterminism (sampling, `now()`, an impure UDF, +source drift), not the unordered-scan row reshuffling the canonical ordering now +absorbs. Design: [adr-result-digest-canonical-ordering.md](../plans/adr-result-digest-canonical-ordering.md). + +**Alias and V_n versions.** An alias is a named, mutable pointer (for example +`sales`) to the latest content hash of a logical entry. Each alias carries an +ordered `history` list of every hash it has pointed at (V1, V2, …, oldest +first). Revising an alias mints a new content hash, advances `latest`, and +appends to `history`; the old versions remain as forensic lineage. +`catalog_diff` resolves version indices (-1 latest, -2 previous) through this +history. + +**Parent edges and following.** At build time each entry records its direct DAG +parents as `{hash, ref, follow}`. `tracked_expr_from_alias('name')` records +`follow=True`: the edge names an alias, and the child goes stale as that alias +advances. `pinned_expr_from_alias('hash')` records `follow=False`: the edge pins +an exact hash and is never disturbed by recalc. Aliases are resolved to hashes +at read time, not baked into the edge. + +**Staleness.** Read-only and side-effect-free. An entry is stale on the alias +axis when a `follow=True` parent's alias head no longer equals the recorded +hash, or on the source axis when a recorded source digest no longer matches the +file's current digest. Computing staleness never executes anything; it only +compares the manifest against the current world and returns reasons. + +**Recalc cone.** When an alias head advances, its dependents form a cone of +entries that may now be stale. The cone is recomputed in topological order +(Kahn's algorithm over intra-cone parent edges) so parents rebuild before +children. Auto-recalc only recomputes the followers of the alias that just +moved; pre-existing ("orphan") staleness is left in place, logged, and +classified against the recorded error store rather than treated as an +unexplained invariant break. See [reactive-recalc.md](reactive-recalc.md) and +[recalc-mechanism.md](../plans/recalc-mechanism.md). + +**Result cache vs source cache (two-axis caching).** Tallyman caches along two +axes. The source axis caches reads of input files. The result axis bakes a +result snapshot for entries judged expensive (those whose expression contains +aggregates, joins, sorts, windows, or UDFs); cheap, row-preserving entries +recompute on every read even when an ancestor is expensive. Baked snapshots +self-heal on read, so a cold read transparently rematerializes. See +[caching.md](caching.md). + +**Portability.** A build directory embeds absolute filesystem paths in +`expr.yaml`. On write these are rewritten to `${TALLYMAN_PROJECT_ROOT}` +placeholders; on load they are expanded into a stable per-entry directory marked +complete by a sentinel, so a project can be copied or packed and run from +anywhere on disk, and the expanded path stays consistent so Buckaroo's +snapshot-cache keys match across restarts. + +**Checkpoint and reset-to-revision.** A checkpoint is an atomic git transaction +under a per-project file lock: it captures pointers, zips pending recipes, +stages all tracked files, commits once, and tags the step. Reset-to-revision +does a hard git reset to a commit and then reconciles untracked build artifacts +back to the recorded pointers, evicting to or restoring from the bullpen without +recompute (a forward reset copies back from the bullpen). Live operations never +read the bullpen. + +**Live updates over SSE.** The companion pushes changes to the browser with +Server-Sent Events (SSE): the SPA opens one long-lived HTTP stream to +`GET /{project}/api/sse` through the browser's native `EventSource`, and the +server writes named event messages down it. This is the inverse of polling. The +browser never asks "anything new?" on a timer; it holds the stream open and the +server speaks when something changes. The SPA registers listeners for these +kinds: `new_entry`, `build_failed`, `notebook_changed`, `chart_attached`, +`post_processing_changed`, `summary_stat_changed`, `recalc`, and +`project_switched` (plus `hello` and `ping`, which open and keep the connection +alive). An event is a signal, not a data payload: every one increments a +monotonic `version` counter held in a React context (`SSEContext.tsx`), and +components key their effects on that counter, so a bump triggers exactly one +refetch of the affected REST resource. That is what "refetches off an SSE +version counter rather than polling" means. Two events also carry state the +refetch cannot derive: `recalc` ships the `{oldHash: newHash}` remap so an open +entry view can follow its entry to the new hash, and `project_switched` ships +the project name to navigate to. If the stream drops, the context flips to +`offline`. Sources: `SSEContext.tsx` on the browser side, the `/{project}/api/sse` +route and the `/internal/notify` fan-out in `app.py` on the server side. + +## Request and data-flow paths + +### catalog_run / catalog_create — author a new entry + +1. Claude Code calls the MCP tool with xorq source. +2. `tallyman_xorq` compiles the expression, computes the content hash, and runs + the build, writing the entry build dir: `expr.py`, `xorq_build/`, + `schema.json`, and finally `manifest.json` (written last, atomically, as the + completeness sentinel). `cache_worthy` entries bake a result snapshot into + `compute_cache/`. +3. The MCP dispatch boundary fires a checkpoint: `tallyman_core` zips the recipe, + stages the tracked surface, commits one git revision, and tags it. +4. The MCP server best-effort POSTs a notification to the companion. +5. The companion emits an SSE `new_entry` event; the SPA bumps its version + counter and refetches the entry list. See [expression-lifecycle.md](expression-lifecycle.md). + +### catalog_revise + auto-recalc — revise an entry and cascade + +1. A revision arrives from `catalog_revise` (MCP) or `PUT /code` (companion). It + mints a new content hash, advances the alias `latest`, and appends the old + hash to `history`. Charts and display configs carry forward from the old hash + to the new one only where the new hash does not already define them. + Self-alias references are rejected — an entry following its own alias would be + permanently stale by design. +2. Auto-recalc, if enabled for the project, walks the recalc cone of the alias's + followers in topological order and rebuilds each, re-pointing aliases before + replaying children. This walk is checkpoint-free. +3. The whole thing lands as one checkpoint, so head advance plus cascade is a + single git revision that reset-to-revision undoes atomically. +4. The companion emits an SSE `recalc` event carrying `{oldHash: newHash}` for + each remapped entry; backgrounded SPA views navigate to the new hash, focused + views stay put. See [reactive-recalc.md](reactive-recalc.md) and + [auto-recalc-on-revise.md](../plans/auto-recalc-on-revise.md). + +### Viewing an entry grid + +1. The SPA entry-detail pane mounts `LazyBuckarooEmbed`, which waits for the grid + to scroll near the viewport (IntersectionObserver) and polls for session + startup. +2. The companion POSTs the entry's `xorq_build/` directory to the Buckaroo + subprocess's `/load_expr`. First the build dir is expanded into a stable + per-entry path with `${TALLYMAN_PROJECT_ROOT}` resolved, so Buckaroo's + snapshot-cache key matches and a cold read of an expensive entry self-heals + its baked snapshot rather than recomputing. +3. Buckaroo creates a session keyed by content hash and streams the grid over + WebSocket; sort and search push down to the xorq backend rather than paging a + materialized parquet. +4. The data tab stays mounted (hidden) across tab switches to keep the WS session + alive. Sessions live only in Buckaroo's RAM; a Buckaroo restart (detected via + a `started_at` timestamp on `/health`) clears the session maps and the next + view re-POSTs. + +### Diffing versions + +1. The SPA diff page resolves the version pair (defaulting to V_{n-1} vs V_n via + the alias history) and requests the diff from the companion. +2. The companion builds a compare expression: an outer join of the two versions + with a membership column (a-only / b-only / both) plus per-column `{col}_eq`, + `{col}_pct_delta`, and `{col}_abs_delta` sentinel columns. The compare + expression is memoized for the process lifetime because the two entry hashes + are immutable. +3. Buckaroo column-config overrides color the diff: categorical coloring for + key/equality columns, numeric coloring for the delta columns. +4. The diff grid loads through the same `/load_expr` path as a normal entry view, + with diff stats cached per entry pair under `diff_stat_cache/`. + +## Related documentation + +Currency notes below reflect a docs-vs-code audit on 2026-06-25. They will drift; +when in doubt, the code wins. + +### Architecture docs (`docs/`) — describe the current system + +- [expression-lifecycle.md](expression-lifecycle.md) — one expression from MCP + ingest to rendered rows, naming every artifact and cache write. **Current.** +- [reactive-recalc.md](reactive-recalc.md) — revise an alias, recompute its + dependents; the cone and the dependency graph. **Current.** +- [caching.md](caching.md) — the caches across the stack and their invalidation. + **Mostly current.** +- [installing.md](installing.md) — install and run the spike. **Mostly current.** +- [mcp-server.md](mcp-server.md) — every MCP tool and prompt Claude Code drives, + with parameters, return shapes, and per-tool side effects. **Current.** + +### Design records / ADRs (`plans/`) + +- [adr-source-identity-content-hash.md](../plans/adr-source-identity-content-hash.md) + — content-addressed source reads so `content_hash` tracks source data. + **Mostly current.** +- [adr-git-subprocess-threading.md](../plans/adr-git-subprocess-threading.md) — + calling git from the multithreaded server (fork-safe `posix_spawn`). + **Mostly current.** +- [adr-result-cache-cost-rubric.md](../plans/adr-result-cache-cost-rubric.md) — + a *proposed* cost-vs-size cache rubric. **Partially stale / not adopted:** the + structural `cache_worthy` admission test it proposes to remove is still the + live gatekeeper, and `ensure_result` it names was removed (#73). +- [adr-result-digest-canonical-ordering.md](../plans/adr-result-digest-canonical-ordering.md) + — `result_digest` as a row multiset via a canonically-ordered snapshot hash, + replacing the per-row Python digest (#137). **Current** (implemented: the + digest is now `snapshot_file_digest`, and `tallyman_read_csv` injects + `original_row_order`). + +### Plans (`plans/`) + +- [native-catalog-store.md](../plans/native-catalog-store.md) — the native + `tallyman_core.catalog` that replaced xorq's catalog package. **Mostly current.** +- [recalc-mechanism.md](../plans/recalc-mechanism.md) — how reactive recalc + works. **Mostly current.** +- [auto-recalc-on-revise.md](../plans/auto-recalc-on-revise.md) — atomic + auto-recalc on revise. **Partially stale:** line numbers and a couple of + function names drifted (`_recalc_walk` → `_replay_cone`), and the "future + Stage C" frontend SSE listener already shipped. +- [remove-ondemand-result-parquet.md](../plans/remove-ondemand-result-parquet.md) + — removing the on-demand `result.parquet` layer (#104). **Current.** +- [project_switcher.md](../plans/project_switcher.md) — the project switcher. + **Mostly current.** +- [89-determinism-prereqs-execution.md](../plans/89-determinism-prereqs-execution.md) + — clearing #89's determinism prerequisites. **Mostly current.** +- [catalog-xorq-integration-tests.md](../plans/catalog-xorq-integration-tests.md) + — coexistence/reset integration tests. **Partially stale:** references the old + `catalog.yaml` / `aliases.json` formats since replaced by JSONL. +- [llm-summary-stats.md](../plans/llm-summary-stats.md) — LLM-authored summary + stats. **Partially stale:** the Buckaroo-side klass pattern it describes + (`_Generated_*` classes) is now a `@stat()` decorator; a few signatures and + the notify `kind` differ. + +### Research notes / experiment logs (`plans/`, `demo/`) — point-in-time records + +The digest investigation behind #137 (current): +[datafusion-scan-order-findings.md](../plans/datafusion-scan-order-findings.md) +— why a parallel datafusion scan emits rows in a different order each run, and +the polars-ingest decision — and +[result-digest-vs-xorq-staleness.md](../plans/result-digest-vs-xorq-staleness.md) +— why `result_digest` can't reuse xorq's content-aware cache staleness. + +Older, historical: +[eda-prompt-research.md](../plans/eda-prompt-research.md), +[eda-codegen-run1.md](../plans/eda-codegen-run1.md), +[haiku-codegen-findings.md](../plans/haiku-codegen-findings.md), +[model-codegen-comparison.md](../plans/model-codegen-comparison.md), +[plotting-testcases.md](../plans/plotting-testcases.md), +[xorq-sklearn-assessment.md](../plans/xorq-sklearn-assessment.md), +[ds-demo-scripts.md](../plans/ds-demo-scripts.md), +[demo/datasets.md](../demo/datasets.md). These are historical; staleness mostly +doesn't apply, except where they assert current system behavior (a few reference +the removed `result.parquet` and the old `~/.tallyman/` path). + +### Root & meta + +- [README.md](../README.md) — V0 spike overview and run instructions. **Current.** +- [proposal.md](../proposal.md) — the talk pitch. **Current** (it's a pitch, not + a spec). + +The original `plan.md` (V0.6 plan) and `TICKETS.md` (V0 punchlist) were removed +as stale cruft — they predated the React SPA migration, the `result.parquet` +removal, and the JSONL catalog format. This doc supersedes them as the +architecture reference. + +> Gaps: there is no dedicated reference for the REST API, the CLI, or the +> frontend SPA architecture. (The MCP tool surface and the authoring extension +> points it exposes — display klasses, summary stats, post-processing — are now +> covered in [mcp-server.md](mcp-server.md).) See the project's gap tracking for +> the current list. diff --git a/docs/installing.md b/docs/installing.md index 96bb88d..95254aa 100644 --- a/docs/installing.md +++ b/docs/installing.md @@ -10,8 +10,8 @@ Install these first: - **[uv](https://docs.astral.sh/uv/)** — manages the Python environment. uv fetches Python 3.13 itself, so you don't need a system Python. -- **Node 18+ and [pnpm](https://pnpm.io/)** — only to build the companion's - dataframe embed (a Vite/React bundle). The build artifact is not committed, +- **Node 18+ and [pnpm](https://pnpm.io/)** — to build the React companion UI + (a Vite SPA the FastAPI server serves). The build artifact is not committed, so this is a one-time per-checkout step. - **git** and **Claude Code**. @@ -34,19 +34,20 @@ This creates `.venv/` in the repo and installs everything from `uv.lock`. You do **not** activate the venv — every command below is prefixed with `uv run`, which resolves the project's venv from the current directory automatically. -## 3. Build the companion embed (one-time) +## 3. Build the companion UI (one-time) -The companion renders dataframes via a bundle that is built, not committed: +The companion serves a React SPA that is built, not committed: ```sh -cd packages/embed && pnpm install && pnpm build +cd packages/app && pnpm install && pnpm build cd ../.. ``` -This writes `src/tallyman_companion/static/buckaroo-embed.{js,css}`. If you're -actively editing the embed, use `pnpm dev` (runs `vite build --watch`) instead. +This writes `packages/app/dist/`, which the FastAPI server mounts as a +catch-all. Without it the server returns a 503 with a build reminder. If you're +actively editing the UI, use `pnpm dev` (Vite dev server) instead. -> Keep `buckaroo-js-core` in `packages/embed/package.json` in sync with the +> Keep `buckaroo-js-core` in `packages/app/package.json` in sync with the > `buckaroo` pin in `pyproject.toml`, and rebuild whenever either side moves. ## 4. Initialize a project @@ -185,8 +186,8 @@ The active project is resolved from `TALLYMAN_PROJECT` first, then the - **`claude mcp list` shows "Pending approval"** — restart `claude` from the repo directory and approve the server. -- **Companion shows no dataframes / embed 404s** — you skipped step 3. Run the - `pnpm build` in `packages/embed`. +- **Companion returns 503 / no UI** — you skipped step 3. Run the `pnpm build` + in `packages/app`. - **`uv run tallyman` fails to resolve the command** — make sure you're in the checkout directory (or a subdirectory). `uv run` picks the venv from the nearest `pyproject.toml`. diff --git a/docs/mcp-server.md b/docs/mcp-server.md new file mode 100644 index 0000000..ddde7e6 --- /dev/null +++ b/docs/mcp-server.md @@ -0,0 +1,502 @@ +# Tallyman MCP server reference + +The MCP server (`src/tallyman_mcp/server.py`) is the surface Claude Code drives. +It is a [FastMCP](https://github.com/jlowin/fastmcp) server that exposes the +catalog as **30 tools and one prompt** over stdio. Claude Code is the only +caller; a human never invokes these directly. Each tool compiles or mutates the +on-disk catalog through `tallyman_core` / `tallyman_xorq`, commits a git +revision, and best-effort notifies the running companion so the browser updates. + +This doc lists every tool, its parameters, its return shape, and — the part that +matters most when reasoning about a session — its **side effects**: whether it +checkpoints, which SSE event it pushes, and whether it cascades a recalc. For +where the MCP server sits in the larger system, see +[architecture.md](architecture.md). + +## How it is launched + +Claude Code reads `.mcp.json` at the repo root, which registers one server named +`tallyman`: + +```json +{ "command": "uv", "args": ["run", "tallyman", "mcp"], + "cwd": "/Users/paddy/tallyman", + "env": { "TALLYMAN_PROJECT": "spike", + "TALLYMAN_COMPANION_URL": "http://127.0.0.1:7860" } } +``` + +So Claude Code spawns `uv run tallyman mcp`. The `tallyman` entry point is +`tallyman_cli.main:cli` (from `pyproject.toml`); the `mcp` subcommand +(`run_mcp` in `src/tallyman_cli/main.py`) calls `tallyman_mcp.main()`, which ends +in `mcp.run()`. With no transport argument FastMCP defaults to **stdio**, so +Claude Code talks to the server over the spawned process's stdin/stdout — one +MCP process per Claude Code session, owned by Claude Code, not by the companion. + +The server reaches the companion through `COMPANION_URL` +(`os.environ.get("TALLYMAN_COMPANION_URL", "http://127.0.0.1:7860")`), firing +best-effort `POST {COMPANION_URL}/internal/notify` calls so the FastAPI companion +refreshes Buckaroo sessions and pushes SSE to open browsers. The companion need +not be up; notifications that fail are logged and dropped. + +## Cross-cutting behavior + +Four mechanisms apply to (almost) every tool, so they are documented once here +rather than repeated per tool. + +**Auto-checkpoint (opt-out).** `mcp.tool` is monkeypatched to +`_checkpointing_tool`, which registers every tool wrapped in `_with_checkpoint`. +After a tool returns, unless its name is in the `_NO_CHECKPOINT` set or the +result carries an `error` key, the wrapper calls +`checkpoint_catalog(project, "tallyman: ")` — **one git revision per +operation**, even when the tool called several `tallyman_core` mutators. A +checkpoint failure is logged, never raised. Opt-out (not opt-in) means a tool +added later is covered automatically; coverage is pinned by +`tests/test_reset_to_revision.py`. The opt-out set is exactly: + +``` +catalog_list, catalog_diff, catalog_scan_staleness, catalog_recalc, +catalog_promote_diff, catalog_list_summary_stats, catalog_list_post_processings, +catalog_run_post_processing, catalog_export_marimo, +project_list, project_switch, project_new +``` + +(`catalog_recalc` and `catalog_promote_diff` are on the list because they +*self*-checkpoint; the rest are read-only or lifecycle ops.) + +**Response tagging.** `_tag_project` wraps every response: it adds a `project` +key (the active project) via `setdefault`; a tool that returns a bare list +(the `catalog_list*` family) is rewrapped as `{project, items: [...]}`; and when +the active project changed since the previous tool call, a `warning` field is +added. The per-tool "Returns" notes below describe the body's shape before this +tagging. + +**Notifications.** `_notify(kind, hash, **extra)` is a 2-second best-effort POST +to the companion's `/internal/notify`, which re-publishes it as an SSE event of +that `kind`. It never raises. Pass extra data as flat kwargs (`remap=...`), not +`extra={...}`, or it nests and is silently dropped. The "Notifies" line per tool +lists the kinds it emits. + +**Sticky active project.** `_resolve_active_project()` returns the in-process +`_mcp_active_project` (set by `project_switch` / `project_new`) ahead of the +on-disk `active_project` file, so switching once is sticky for the rest of the +session regardless of what another session writes to disk. The first tool call +seeds it. `SESSION_ID` (an 8-hex uuid) tags this process in `events.jsonl`. + +**Auto-recalc.** `catalog_revise` and `catalog_promote_diff` advance an existing +alias head, which can make followers stale. Both route through +`_auto_recalc_after_head_advance` (when the project's `auto_recalc` switch is on): +a checkpoint-free cascade rebuilds the followers in dependency order and leaves +them in the working tree, so the head advance and the whole cascade land as one +git revision. A non-empty cascade emits a `recalc` SSE event. + +## Side-effect matrix + +| Tool | Checkpoints | SSE notify | Auto-recalc | +|---|---|---|---| +| `catalog_run` | yes | `new_entry`, `build_failed` | no | +| `catalog_load_parquet` | yes | `new_entry`, `build_failed`, `notebook_changed`¹ | no | +| `catalog_create` | yes | `new_entry`, `build_failed`, `notebook_changed` | no | +| `catalog_revise` | yes | `new_entry`, `build_failed`, `recalc` | yes | +| `catalog_alias` | yes | `alias_changed`, `notebook_changed` | no | +| `catalog_rename` | yes | `alias_renamed`, `notebook_changed` | no | +| `catalog_unalias` | yes | `alias_changed`, `notebook_changed`² | no | +| `notebook_reorder` | yes | `notebook_changed` | no | +| `notebook_remove` | yes | `notebook_changed` | no | +| `notebook_edit_markdown` | yes | `notebook_changed` | no | +| `catalog_chart` | yes | `chart_attached` | no | +| `catalog_diff` | no | — | no | +| `catalog_promote_diff` | self³ | `entry_added`, `recalc`⁴ | yes | +| `catalog_scan_staleness` | no | — | no | +| `catalog_recalc` | self³ | `recalc`⁴ | yes | +| `catalog_add_summary_stat` | yes | `summary_stat_changed` | no | +| `catalog_remove_summary_stat` | yes | `summary_stat_changed` | no | +| `catalog_list_summary_stats` | no | — | no | +| `catalog_add_display_klass` | yes | `display_changed` | no | +| `catalog_list_display_klasses` | yes ⚠ | — | no | +| `catalog_remove_display_klass` | yes | `display_changed` | no | +| `catalog_add_post_processing` | yes | `post_processing_changed` | no | +| `catalog_remove_post_processing` | yes | `post_processing_changed` | no | +| `catalog_list_post_processings` | no | — | no | +| `catalog_run_post_processing` | no | — | no | +| `catalog_export_marimo` | no | — | no | +| `catalog_list` | no | — | no | +| `project_list` | no | — | no | +| `project_switch` | no | — ⁵ | no | +| `project_new` | no | — ⁵ | no | + +¹ `notebook_changed` only when `name` is supplied. ² only when a cell was +actually removed. ³ self-checkpoints exactly one revision; on the opt-out list so +the dispatch boundary does not double-commit. ⁴ only on a committing run that +produced a non-empty remap. ⁵ the *companion* broadcasts `project_switched`; the +MCP tool itself sends no `_notify`. ⚠ see [Known quirks](#known-quirks). + +Any tool whose result carries an `error` key skips its checkpoint and sends no +notify; failures come back inline in the response (most as `{error, error_id}` +for build failures, `{error}` for validation/precondition failures), not as +raised exceptions. + +--- + +## Authoring tools + +The compile-and-persist family. The `code` argument is a self-contained Python +script that binds a top-level `expr` to a xorq/ibis expression; +`catalog_run`'s docstring is the canonical cookbook for writing it (namespaces, +the datafusion-only backend, data sourcing, and the `xorq.ml` MODELING section). +Data sourcing inside `code` uses four helpers from `tallyman_xorq.io`: +`tracked_expr_from_alias` (a catalog entry, recorded as a lineage parent), +`pinned_expr_from_alias` (a catalog entry by alias or hash, no following), +`read_project_file` (a raw parquet under `data/`), and `tallyman_read_csv` (CSV +ingest — injects an `original_row_order` column so the baked snapshot is +byte-stable across builds; use it for all CSV reads instead of +`xo.deferred_read_csv`, #137). + +### `catalog_run(code, prompt="") -> dict` +Execute an expression and persist it as an **unnamed (scratch)** entry. Claude's +default authoring tool for a one-off run. +- **Params:** `code` (required) — script binding `expr`; `prompt` — optional + intent string, recorded on the entry. +- **Returns:** `{hash, row_count, execute_seconds, schema, entry_path, url}`, + plus `lint_warnings` when the nondeterminism lint fired. `{error, error_id}` + on a `BuildError`. +- **Writes:** the entry build dir under `catalog/entries//`; a `build_ok` + or `build_error` event to `events.jsonl`. +- **Promote** a scratch entry to a name afterward with `catalog_alias`. + +### `catalog_load_parquet(rel_path, prompt="", name="") -> dict` +Register a parquet under `/data/` as an entry by synthesizing a +`read_project_file(rel_path)` recipe — the no-code "load this file" path. With +`name`, behaves like `catalog_create` (names the entry, appends a notebook cell). +- **Params:** `rel_path` (required) — path under `data/`; `prompt` — optional + intent / cell markdown; `name` — optional alias. +- **Returns:** same as `catalog_run`, plus `alias` and `version` when named. + `{error}` if the alias already exists; `{error, error_id}` on a missing file. +- Unlike `catalog_run`, it does **not** emit a `build_ok` event; success shows + only via the notifies. + +### `catalog_create(name, code, prompt="") -> dict` +Execute and persist as a **named** entry (alias) and append a notebook cell. +Called when the user names a concept ("name this `shoe_sales`"). +- **Params:** `name` (required) — new alias, must not exist; `code` (required); + `prompt` — optional, recorded on the `alias_set` event and used as cell + markdown. +- **Returns:** `catalog_run` shape plus `alias`, `version`. `{error}` if the + alias exists (steers to `catalog_revise`); `{error, error_id}` on build + failure. + +### `catalog_revise(name, code, prompt="") -> dict` +Persist a **new version** of an existing alias: build, repoint the alias head, +carry chart/display config forward, and cascade-recompute the alias's stale +followers. The canonical head-advance path. +- **Params:** `name` (required) — existing alias; `code` (required) — a + self-contained recipe that **must not reference its own alias by name** + (rejected, #135 — inline the source or pin the previous version by hash); + `prompt` — optional. +- **Returns:** `catalog_run` shape plus `alias`, `version`; `carried_over` + (subset of `chart` / `display_config`) when non-empty; and a `recalc` + sub-report when the project's `auto_recalc` is on. `{error}` if the alias is + absent; `{error, error_id}` on build failure or a self-alias rejection. +- The head advance and the entire follower cascade commit as **one** git + revision; the `recalc` notify fires only when the cascade produced a non-empty + remap. + +## Alias tools + +None recompute anything; they only move alias pointers and re-anchor notebook +cells. + +### `catalog_alias(hash, name) -> dict` +Promote a scratch entry (by content hash) to a named alias and append a notebook +cell. Used after a `catalog_run` when a scratch entry earns a permanent name. +- **Params:** `hash` (required) — must name an on-disk entry; `name` (required) + — must be free. +- **Returns:** `{hash, alias, version, url}`. `{error}` if the hash has no entry + or the name is taken. + +### `catalog_rename(old_name, new_name) -> dict` +Rename an alias, preserving its full version history and notebook position. +- **Params:** `old_name` (must exist), `new_name` (must be free). +- **Returns:** `{old, new, hash}`. `{error}` if `old_name` is absent or + `new_name` is taken. + +### `catalog_unalias(name) -> dict` +Drop an alias (its entries revert to scratch; the entry dirs are untouched) and +remove any notebook cells anchored on it. +- **Params:** `name` (must exist). +- **Returns:** `{removed_alias, notebook_cells_removed}`. Emits + `notebook_changed` only when a cell was actually removed. + +## Notebook tools + +All three act on the project's single `default` notebook +(`catalog/notebook.jsonl`); `cell_id` is the 12-hex id created when an alias is +added to the notebook. A missing cell returns +`{error: "no cell with id ''"}` and skips the checkpoint. + +### `notebook_reorder(cell_id, new_index) -> dict` +Move a cell to a 0-based position. `new_index` is clamped to range, but the +response echoes the raw argument, so an out-of-range value reports verbatim while +the cell lands at the end. +- **Returns:** `{cell, new_index}`. + +### `notebook_remove(cell_id) -> dict` +Remove a cell; the underlying catalog entry is untouched. Removing the last cell +deletes `notebook.jsonl` outright. +- **Returns:** `{removed: }`. + +### `notebook_edit_markdown(cell_id, markdown) -> dict` +Replace the markdown shown above a cell, wholesale (not a merge). An empty +string clears it. +- **Returns:** `{cell}` (the updated cell). + +## Chart and diff tools + +### `catalog_chart(hash_or_alias, vega_spec) -> dict` +Attach a Vega-Lite v5 spec to an entry so the companion renders a chart above the +table. The target resolves as a hash first, then as an alias (to its latest +hash). +- **Params:** `hash_or_alias` (required); `vega_spec` (required) — dict or JSON + string, stored as-is with **no** server-side schema validation (a malformed- + but-JSON spec only fails in the browser). +- **Returns:** `{hash, spec_path}`. `{error}` for an unknown target or invalid + JSON. +- **Writes:** `catalog/chart_specs/.vl.json`. + +### `catalog_diff(name, va=-2, vb=-1) -> dict` +**Read-only** diff of two versions of an alias (default previous vs latest). +Each side's expression comes from `cached_result_expr`, so it does not depend on +a materialized result existing. +- **Params:** `name` (required); `va`, `vb` — version indices, 1-based or + negative-from-end (`-1` latest, `-2` penultimate). +- **Returns:** `{alias, before:{version,hash}, after:{version,hash}, schema, + stats, keyed_summary}`. The heavyweight `table_html` is stripped; `code` and + `head` diffs are computed but not returned. `{error}` for no history / + out-of-range / missing-on-disk. +- On the opt-out list: never checkpoints, never notifies. For the visual, + promotable form, use `catalog_promote_diff`. + +### `catalog_promote_diff(name, va=-2, vb=-1, alias=None) -> dict` +Promote a version-over-version diff into a **first-class catalog entry** whose +expression is the keyed full-outer-join comparison (with Buckaroo diff coloring). +Unlike `catalog_diff`, the result is post-processable, chartable, and +marimo-exportable. +- **Params:** `name` (required); `va`, `vb` — version indices; `alias` — name for + the new entry (default `diff_{name}_v{a}_v{b}`); if it already exists it is + re-pointed (which can stale its followers and trigger the cascade). +- **Returns:** `{alias, hash, source_alias, va, vb, a_hash, b_hash, keys, + row_count}`, plus a `recalc` sub-report only when an existing alias was + re-pointed and `auto_recalc` is on. `{error}` for no history / out-of-range / + no stable join key / build failure. +- **Self-checkpoints**: the checkpoint-free cascade runs before its own + `checkpoint_catalog`, so the promote and cascade land in one revision (hence + its place on the opt-out list). Always emits `entry_added`; emits `recalc` only + on the re-point-with-cascade branch. + +## Staleness and recalc tools + +### `catalog_scan_staleness() -> dict` +**Read-only** scan of every live entry for staleness against its recorded inputs. +Purely diagnostic — never rebuilds, repoints, or checkpoints. Call it first to +see what is stale. +- **Returns:** `{stale, transitively_stale, entries, orphan_stale}`. `stale` is + the default root set `catalog_recalc` uses. `orphan_stale` populates only under + auto-recalc mode. + +### `catalog_recalc(roots=None, dry_run=True) -> dict` +Recompute stale entries and their dependents in dependency order. Defaults to a +non-committing **dry-run preview**; call again with `dry_run=False` to commit. +- **Params:** `roots` — content hashes to recompute with their descendant cone; + `None` defaults to every directly-stale entry. Roots naming no live entry are + silently dropped. `dry_run` — preview when `True` (default). +- **Returns:** the `RecalcReport` (`project, roots, cone, entries, dry_run, + status, remap, checkpoint_step, error`). Action vocabulary differs by mode + (`rebuild`/`cascade`/`unchanged` on dry-run; `rebuilt`/`noop`/`failed`/ + `skipped` on a real run). A nothing-stale fast path returns a short + `{status:'ok', ..., note:'nothing stale to recompute'}`. +- **Self-checkpoints arg-aware:** a dry-run takes no checkpoint; a committing run + takes exactly one for the whole walk, so the recompute is a single revision + `reset-to` can undo atomically. A committing run with a non-empty remap emits + `recalc`. Build failures are encoded in `status`/`error`/per-entry `error` + (the walk halts, the rebuilt prefix stays committed); a dependency cycle yields + `status=='cycle'` rather than raising. + +## Summary-stat tools + +Project-authored per-column statistics. Source runs in a restricted-globals +sandbox; errors come back inline rather than later in the Buckaroo log. There is +no separate update tool — re-adding the same `name` overwrites. + +### `catalog_add_summary_stat(name, source) -> dict` +Validate a `compute(col)` function against a 1-row ibis memtable, write it to +`/stats/.py`, and hot-reload it into open Buckaroo sessions. +- **Params:** `name` (required) — a valid Python identifier; `source` (required) + — must define `compute(col)` taking one ibis column and returning an ibis + scalar. +- **Returns:** `{name, path}`. `{error}` on any validation failure (bad + identifier, syntax error, exec error, missing/ wrong-arity `compute`, + non-ibis return). +- To surface a stat as a pinned row in the main table (not just the summary + panel), follow up with `catalog_add_display_klass`. + +### `catalog_remove_summary_stat(name) -> dict` +Soft-delete by moving `stats/.py` to `stats/_disabled/.py` (the `_` +prefix is what Buckaroo's scanner skips). Idempotent. +- **Returns:** `{name, moved_to}`. `{error}` if no such active stat. + +### `catalog_list_summary_stats() -> list` +List every stat, active first then disabled, each with `{name, path, source, +disabled}`. Read-only (on the opt-out list). The returned `source` lets Claude +read/edit before overwriting. + +## Display-klass tools + +Project-authored `ColAnalysis` subclasses (with a string `df_display_name`) that +control column rendering and pinned rows. Source is exec'd with the Buckaroo +styling base classes already in scope; validated in a restricted sandbox before +the file lands. + +### `catalog_add_display_klass(name, source) -> dict` +Validate and persist a display klass to `display/.py`, then hot-reload it +into open sessions (`display_changed`). The primary use is extending +`DefaultMainStyling` (`df_display_name='main'`) or `DefaultSummaryStatsStyling` +(`'summary'`) with a `pinned_rows` entry so a stat shows as a frozen row. +- **Params:** `name` (required) — valid identifier; `source` (required) — must + define a `ColAnalysis` subclass carrying a string `df_display_name`. +- **Returns:** `{name, path}`. `{error}` on any validation failure. + +### `catalog_list_display_klasses() -> list` +List every display klass (active first, then disabled), each with `{name, path, +source, disabled}`. **See [Known quirks](#known-quirks):** unlike the other +`*_list_*` tools this one is *not* on the opt-out list, so it commits an empty +git revision on each call. + +### `catalog_remove_display_klass(name) -> dict` +Soft-delete by moving `display/.py` to `display/_disabled/`. Re-adding +restores it. Emits `display_changed`. +- **Returns:** `{name, moved_to}`. `{error}` if no such active klass. + +## Post-processing tools + +Project-authored `process(expr)` functions that transform a whole result table, +selectable in the Buckaroo embed's "post processing" dropdown. Same restricted +sandbox as summary stats (third-party imports like numpy/sklearn raise +`ImportError`; fit models via the `xorq.ml` path on `catalog_create` instead). + +### `catalog_add_post_processing(name, source) -> dict` +Validate a `process(expr)` function and write it to +`/post_processing/.py`. +- **Params:** `name` (required) — valid identifier, becomes the dropdown label; + `source` (required) — defines `process(expr)` returning an ibis expression or a + pandas DataFrame. +- **Returns:** `{name, path}`. `{error}` on validation failure. +- **Validator gotcha:** the dry-run table has only columns `{a, b}`, so a + `process` that references real column names is rejected here even when + `catalog_run_post_processing` previewed it fine — guard with + `if 'col' not in expr.columns: return expr`. The new option appears only on the + next session load (V1 does not hot-swap loaded sessions). + +### `catalog_remove_post_processing(name) -> dict` +Soft-delete to `post_processing/_disabled/`. Still shows in the list with +`disabled=true`. Emits `post_processing_changed`. +- **Returns:** `{name, moved_to}`. `{error}` if no such active function. + +### `catalog_list_post_processings() -> list` +Read-only inventory: `{name, path, source, disabled}` per function, active first. +On the opt-out list. + +### `catalog_run_post_processing(code, entry) -> dict` +**Preview** a `process(expr)` against a real entry's result without persisting +anything — the iterate-before-commit tool. Note the arg is `code` here, not +`source`. +- **Params:** `code` (required) — `process(expr)` source; `entry` (required) — + alias or content hash to test against. +- **Returns:** `{entry, row_count, columns, preview}` (first 20 rows). `{error}` + on any run failure. +- Reads the entry's actual result via `cached_result_expr` (a cheap entry + recomputes; an expensive one reads its baked snapshot, self-healing if + evicted), so the preview reflects real data — wider than `add`'s `{a, b}` + dry-run. Persists nothing; on the opt-out list, no notify. + +## Export and listing tools + +### `catalog_export_marimo() -> dict` +Export the default notebook to a runnable Marimo file at +`/notebook_marimo.py` (also downloadable via the companion's +`/export/marimo` route). Writes an artifact, not a revision (on the opt-out +list). +- **Returns:** `{path, cells}`. `cells` is the **total** number of cells in the + default notebook — cells with an unresolved alias or a missing `expr.py` are + counted here but silently skipped during export, so the file may contain fewer. + `{error}` wraps any render/write failure (including a corrupt `notebook.jsonl`, + which is caught, and a `FileNotFoundError` from a promoted-diff cell whose + source `expr.py` is missing). + +### `catalog_list() -> list` +Enumerate the active project's catalog, most-recent-first. Each item is the +entry's full `manifest.json` plus `alias` and `version` (when the hash is an +alias head, else `None`) and `columns` (a compact `"name:type, ..."` string). +Read-only (opt-out). Includes scratch entries. Per its docstring, Claude should +call this before writing an expression against an entry it did not just build, so +it references exact column names rather than guessing. + +## Project-lifecycle tools + +These require the companion (`tallyman run`) to be active: the **companion** is +the sole writer of the `active_project` file and the genesis baseline, reached +via `_companion_post`. All three are on the opt-out list. On success the +companion broadcasts a `project_switched` SSE event itself (the MCP tool sends no +`_notify`). Each updates the in-process sticky `_mcp_active_project`. + +### `project_list() -> dict` +List projects on disk and report the active one. **Safe even when the companion +is down** — it reads disk directly. +- **Returns:** `{active, available}`. Never errors (`available` is `[]` if no + projects root). Call before `project_switch` to learn the legal `name` values. + +### `project_switch(name) -> dict` +Retarget all subsequent `catalog_*` / `notebook_*` calls at another existing +project, by POSTing to the companion. Pins the sticky project on success. +- **Params:** `name` (required) — must exist and pass the companion's name + validation. +- **Returns:** `{previous, active}`. On failure `{error}` (sticky unchanged): + e.g. `companion not reachable at : . Project lifecycle tools require + 'tallyman run' to be active.`, or `companion at returned : ...` + (400 invalid/reserved name, 404 not found, 403 serve mode). + +### `project_new(name, with_fixture=False) -> dict` +Create a new project on disk (the companion runs `ensure_project` + genesis +baseline) and switch to it. +- **Params:** `name` (required) — `^[a-z0-9][a-z0-9_-]{0,31}$`, not reserved, not + colliding; `with_fixture` — when `True`, the companion seeds the shoe-orders + demo parquet into `data/`. +- **Returns:** `{name, active}`. On failure `{error}` (sticky unchanged): same + shapes as `project_switch`, with 409 on a name collision. + +## The MCP prompt + +### `ds_modeling_workflow(dataset, target="") -> str` +An `@mcp.prompt()` (not a tool) that returns a static guidance string scaffolding +a modeling workflow: load → engineer features → split → fit via `xorq.ml` → +score → chart. Pure function — no side effects, no I/O, not routed through the +checkpoint or tagging wrappers. +- **Params:** `dataset` (required) — a parquet under `data/`, interpolated into + the text; `target` — column to predict (empty renders as `` for + unsupervised work). +- It encodes the hard constraints the model must follow: fit the model *as a + catalog entry* with `xorq.ml` (never in a post-processing sandbox, which blocks + sklearn/scipy/numpy); copy exact fit/predict signatures from the MODELING + section of `catalog_run`'s docstring; score in pure ibis (`deferred_sklearn_metric` + does not survive a catalog build); `import xorq.vendor.ibis as ibis` and + `import xorq.api as xo`; datafusion is the only backend; check columns with + `catalog_list` first; no `project=` kwargs. + +## Known quirks + +- **`catalog_list_display_klasses` checkpoints.** It is the one read-only `*_list*` + tool missing from the `_NO_CHECKPOINT` set (`server.py`), so `_with_checkpoint` + runs `checkpoint_catalog`, which commits with `--allow-empty` and advances the + step tag. Each call therefore appends an **empty** catalog revision — harmless + to correctness but it inflates the git history and the step count. Adding it to + `_NO_CHECKPOINT` (next to `catalog_list_summary_stats` / + `catalog_list_post_processings`) would bring it in line with the other list + tools. diff --git a/plan.md b/plan.md deleted file mode 100644 index 1aeebea..0000000 --- a/plan.md +++ /dev/null @@ -1,340 +0,0 @@ -# tallyman-notebooks — plan - -The xorq MCP app that backs the Tallyman London 2026 talk *"The Future of Notebooks in a Claude Code World"* (see `proposal.md`). This document is the result of a grilling pass; it supersedes earlier drafts. Decisions baked in from the grilling are stated; remaining open questions are listed at the end. - -> **V0.6 implementation status (2026-05-24).** Working end-to-end on branch -> `spike/catalog-run-loop`: -> -> - 16 MCP tools — the original 13 (`catalog_*`, three `notebook_*`) plus -> the LLM-summary-stats trio (`catalog_add_summary_stat`, -> `catalog_remove_summary_stat`, `catalog_list_summary_stats`) backed by -> `/stats/.py` files and the `project_root` field on -> `/load_expr`. Design lives in `plans/llm-summary-stats.md`. -> - Catalog tab (named/forensic/scratch ordering, V_n chips, forensic history, -> build-failure banner with detail page). -> - Notebook tab (cells anchored on aliases, drag-reorder via SortableJS -> with ↑/↓ button fallback for accessibility, click-to-edit markdown, -> × remove; auto-append on `catalog_create`/`catalog_alias`). -> - Lineage tab (catalog DAG via `tracked_expr_from_alias` parent edges, plus per-entry -> internal expression DAG; pure SVG, no Cytoscape dep yet). -> - Diff tab (`/diff/[//]`): code diff (difflib unified), -> schema diff (added/removed/changed-type), per-column stats, key-joined -> side-by-side, head() side-by-side. All five flavors shown together. -> - Vega-Lite charts attached to entries via `catalog_chart`; entry-detail -> embeds vega-embed and pulls data via `/api/data/`. -> - Buckaroo lifecycle: companion-managed Tornado subprocess on :8700, -> `/load_expr` POST per entry (buckaroo PR 776 has landed), React embed -> (`static/buckaroo-embed.js`) mounted into entry-detail and talking -> to the buckaroo WS for paged rows + sort/search push-down. -> `buckaroo==0.14.6` and `xorq==0.3.26` pinned from PyPI (no editable -> sources). Search regression in 0.14.4/0.14.5 reported as -> buckaroo-data/buckaroo#838, fixed in 0.14.6. -> - **`tallyman serve `** read-only artifact mode and -> **`tallyman pack []`** tar-up command (portable `.tgz`, -> `${TALLYMAN_PROJECT_ROOT}` placeholders preserved, optional -> `--exclude-cache`). -> - **`tallyman replay `** rehearsal mode that drives the -> MCP tool surface deterministically (the open question #10 fallback). -> - MCP-over-stdio proven by `tests/test_mcp_stdio.py` (spawns the CLI as a -> subprocess and round-trips real tool calls). -> -> **Not yet built (V1):** column-level lineage; ML training pipeline -> (beats 7-8); `catalog_prune` MCP tool. - ---- - -## What we're building - -A local app that demonstrates the **deconstructed notebook**: terminal Claude writes intent, xorq+Ibis on Arrow handles compute, a live browser companion is the work surface. The companion has two faces: - -- **Catalog view** (default tab) — every executed expression appears here, with live Buckaroo for search/sort/filter/recon. Named entries (concepts) sit alongside scratch entries (footnotes); scratch nests under the named entry it verifies. -- **Notebook view** (second tab) — a curated, ordered, editable narrative built from named catalog entries. Reorderable, markdown-editable, exportable. - -You drive the demo from a Claude Code terminal on the left half of your laptop screen; the companion lives in a browser on the right half, full-height. June 8th 2026. - ---- - -## Mental model - -Three tiers of state — the load-bearing reframe from grilling: - -1. **Named catalog entries (aliased)** — `shoe_sales`, `weekday_by_hour`. The user explicitly names them ("name this X" in a prompt). Become notebook cells. Each alias names a *concept*; the latest version of the alias is the current best answer to that concept; older versions stay in the catalog as forensic artifacts but are not in the notebook view. - -2. **Unnamed catalog entries (scratch)** — verifications, dives, "show your work" runs. Persisted in the catalog, content-addressed, navigable in the catalog tab. Nested as collapsed footnotes under their parent named entry (lineage-derived; agent can override). Not in the notebook unless promoted. - -3. **Inline Buckaroo recon** — null counts, dtype distributions, top-K values, histograms. Free, no expression needed. Buckaroo handles per-column questions inside the table widget; eliminates a class of "tell me about column X" prompts entirely. - -Three layers of curation: **disk-of-runs → catalog → notebook view**. The catalog is curated (wrong explanations get pruned); the notebook is curated further (only named entries, ordered, with markdown). Honesty over completeness; nothing is forced to live forever. - -Identity is by **content hash**, never by timestamp. Aliases are mutable handles on hashes. - ---- - -## Process topology - -``` -┌────────────────────────┐ MCP stdio ┌──────────────────────────┐ -│ Claude Code terminal │ ──────────────► │ tallyman_mcp server │ -│ (left half of screen) │ │ (FastMCP, ~14 tools) │ -└────────────────────────┘ └────────┬─────────────────┘ - │ writes catalog - │ entries to disk - ▼ - ┌──────────────────────────┐ - │ catalog store on disk │ - │ ~/.tallyman/projects/ │ - │ /catalog/... │ - └────────┬─────────────────┘ - │ reads - ▼ -┌────────────────────────┐ HTTP + SSE ┌─────────────────────────┐ -│ Browser │ ◄────────── │ tallyman_companion │ -│ (right half of │ │ (FastAPI + Jinja2) │ -│ screen, full-height) │ ── click ──► │ port 7860 │ -│ │ └────────┬────────────────┘ -│ ┌──────────────────┐ │ │ spawns + supervises -│ │ React embed │ │ WebSocket (paged ▼ -│ │ (buckaroo-embed) │◄─┼──rows, sort, search)─►┌─────────────────────┐ -│ │ in entry detail │ │ :8700 │ buckaroo server │ -│ └──────────────────┘ │ │ (subprocess, 8700) │ -└────────────────────────┘ └─────────────────────┘ -``` - -Three processes on one machine — `tallyman_mcp` (spawned by Claude Code), `tallyman_companion` (the FastAPI app you start manually with `tallyman run`), and a `buckaroo` server subprocess managed by the companion. - -The companion notifies the browser of changes via SSE. The MCP server notifies the companion via `POST /internal/notify` after each tool call that mutates the catalog. - ---- - -## On-disk layout - -A "project" is a directory that owns one catalog and one or more notebooks. V1 has a single default notebook per project. - -``` -~/.tallyman/projects// -├── catalog/ -│ ├── entries// -│ │ ├── manifest.json # name?, version?, parent_hash?, prompt, code_path, created_at, parent_alias? -│ │ ├── expr.py # source script (self-contained, runnable) -│ │ ├── expr.yaml # decompiled DAG (from xorq build) -│ │ ├── result.parquet # cached result (Buckaroo loads this) -│ │ ├── schema.json # arrow schema + row count -│ │ └── lineage.json # node + edge list, includes upstream catalog hashes -│ ├── aliases.json # {alias: latest_hash} -│ ├── alias_history.json # {alias: [hash_v1, hash_v2, ...]} -│ ├── chart_specs/ # optional Vega-Lite specs keyed by hash -│ │ └── .vl.json -│ └── buckaroo_sessions.json # {hash: buckaroo_session_id} (regenerated on startup) -├── notebooks/ -│ └── default.json # cell list, markdown blocks, ordering -└── tallyman.toml # project config (name, default notebook, settings) -``` - -Append-only-ish: catalog entries are immutable (content-hashed), but `aliases.json`, `alias_history.json`, and `notebooks/*.json` are mutable indexes maintained by the system. Pruning a catalog entry deletes its directory and removes references; the catalog reflects what you want to keep. - ---- - -## Pieces - -Seven Python packages in one monorepo. Boundary discipline matters because the talk-quality bar is "this works end-to-end without surprises." - -| Package | Owns | Depends on | -|---|---|---| -| `tallyman_core` | Disk layout, manifest schema, alias bookkeeping, notebook JSON model. No HTTP, no MCP, no compute. | (stdlib + pydantic) | -| `tallyman_xorq` | Compute layer. Wraps xorq's `xo.build_expr`, expr.yaml decompile, lineage extraction. Takes Python code → catalog entry on disk. | `xorq`, `tallyman_core` | -| `tallyman_mcp` | The MCP server. Exposes ~13 tools. Calls into `tallyman_xorq` and `tallyman_core`. Posts notifications to companion's `/internal/notify`. | `fastmcp`, `tallyman_xorq`, `tallyman_core` | -| `tallyman_companion` | FastAPI app. Reads catalog from disk. Manages buckaroo subprocess. Serves Jinja2 templates + SSE. Accepts notebook edits from browser. | `fastapi`, `tallyman_core`, `buckaroo` | -| `tallyman_companion/static/` | Static assets: Buckaroo bundles (copied from `~/buckaroo` build output), Vega-Lite, Cytoscape, custom JS for notebook drag/edit. | (build artifact) | -| `tallyman_companion/templates/` | Jinja2 templates: `base.html`, `catalog.html`, `notebook.html`, `lineage.html`, `diff.html`, fragments for SSE updates. | (templates) | -| `tallyman_cli` | The `tallyman` console script. `tallyman init ` / `tallyman run` / `tallyman mcp` (the latter is what Claude Code spawns) / `tallyman serve ` (read-only companion against a handed-off project) / `tallyman pack []` (portable `.tgz`) / `tallyman replay ` (deterministic rehearsal). | `click`, `tallyman_companion`, `tallyman_mcp` | - -Single `pyproject.toml` at the repo root, src-layout, uv-managed. Python 3.13. - ---- - -## MCP tool surface - -16 tools implemented as of V0.6 (13 catalog/notebook + 3 summary-stats); names use `catalog_*`, `notebook_*`, `viewer_*` prefixes for legibility. The plan-original `viewer_*` row is still V1 because the demo so far does not need it (the agent's tool choice + SSE auto-focus covers the cases). - -**Catalog mutation:** -| Tool | Behavior | -|---|---| -| `catalog_run(code)` | Execute, persist as scratch (no alias). Returns content hash. Companion auto-focuses on the new entry in the catalog tab. ✅ | -| `catalog_load_parquet(rel_path)` | Convenience: load `/data/` as a scratch entry without writing xorq dialect. ✅ | -| `catalog_create(name, code)` | Execute, persist with alias. Auto-appends to active notebook. Errors if alias exists. ✅ | -| `catalog_revise(name, code)` | Execute, persist as new version of an existing alias. Cell updates in place; older versions become forensic. Errors if alias doesn't exist. ✅ | -| `catalog_alias(hash, name)` | Promote a scratch entry to a named entry post-hoc. Errors if alias exists. ✅ | -| `catalog_rename(old_name, new_name)` | Change an alias name. Notebook follows. ✅ | -| `catalog_unalias(name)` | Drop an alias; entries remain, notebook cell is removed. ✅ | -| `catalog_chart(hash_or_alias, vega_spec)` | Attach a Vega-Lite chart spec to an entry. Companion renders it above the table. ✅ | -| `catalog_add_summary_stat(name, source)` | Write `/stats/.py` (`def compute(col): ...`); validated against a 1-row ibis memtable in a restricted-globals namespace before persisting. Picked up by buckaroo on next session-load via `project_root`. ✅ | -| `catalog_remove_summary_stat(name)` | Soft-disable by moving the file to `stats/_disabled/` (buckaroo skips the `_` prefix). ✅ | -| `catalog_list_summary_stats()` | List enabled summary-stat files in the project. ✅ | -| `catalog_prune(hash_or_alias)` | Remove an entry from the catalog. Removes from notebook if present. *(V1 — not yet implemented)* | - -**Notebook mutation:** -| Tool | Behavior | -|---|---| -| `notebook_reorder(cell_id, new_index)` | Move a cell. Mirrors browser drag-and-drop. ✅ | -| `notebook_remove(cell_id)` | Remove a cell from the notebook. Catalog entry untouched. ✅ | -| `notebook_edit_markdown(cell_id, markdown)` | Replace the markdown above a cell. ✅ | - -**Inspection / control:** -| Tool | Behavior | -|---|---| -| `catalog_list()` | List entries (named first, then forensic, then scratch). Each annotated with `alias` and `version` when applicable. ✅ | -| `catalog_diff(name, va=-2, vb=-1)` | Forensic diff between two versions of an alias. Returns code/schema/stats/keyed-summary; the companion's `/diff/` page renders the full HTML. ✅ | -| `catalog_show(hash_or_alias)` | Manifest + schema + head() preview. *(V1 — `/api/data/` + `/catalog/` cover this for now)* | -| `catalog_lineage(hash_or_alias)` | Lineage graph as JSON. *(V1 — `/api/lineage/` and `/api/catalog_dag` cover this from the companion)* | -| `viewer_focus(hash_or_alias)` | Switch the catalog tab's focus to this entry. *(V1 — currently handled by the SSE `new_entry` auto-focus)* | -| `viewer_open_notebook()` | Switch to the notebook tab. *(V1 — same justification)* | - -The agent's choice between `catalog_run`, `catalog_create`, and `catalog_revise` is the visible signal of its intent — communicated and overrideable. If the agent picks `catalog_run` when you wanted a named entry, you (or the agent on a follow-up) call `catalog_alias` to promote. - ---- - -## Companion: routes and tabs - -FastAPI app, Jinja2 templates, vanilla JS for interactions. Server-Sent Events for live updates. No SPA framework — keep moving parts low. - -The companion has two modes. **Edit mode** (`tallyman run`) is the authoring surface used during a session: full routes, drag-and-drop, markdown editing, MCP notifications. **Serve mode** (`tallyman serve `) is the read-only artifact view: a colleague (or future-you) runs it against a handed-off project directory and sees exactly the catalog and notebook the author last saved. Serve mode hides drag handles, ×, and contenteditable affordances; rejects `PATCH /api/notebook`, `PUT /api/markdown/`, and `POST /internal/notify` with 403; and does not start the MCP server. The project directory plus a `tallyman` install is the artifact — there is no `.ipynb`, no static export, no other deliverable. - -| Route | Purpose | -|---|---| -| `/` | Redirect to `/catalog` | -| `/catalog` | Default tab. Two-column layout *inside the tab*: left = entry list (named first, sortable, searchable input filtering by name+prompt), right = entry detail (Buckaroo React embed + tabs: Data \| Schema \| Code \| Prompt \| Lineage \| Forensic-history). | -| `/catalog/` | Deep link to an entry's detail view. | -| `/notebook` | Notebook tab. Vertical scroll of cells; markdown editor above each code cell (contenteditable + serialized to markdown on save); drag-handle for reorder; × for remove; chart panel above table when present. Reorder uses [SortableJS](https://sortablejs.github.io/Sortable/). | -| `/lineage` | Cytoscape DAG view of the catalog. Click a node = focus that entry in the catalog tab. | -| `/diff` | Forensic V_n vs V_n-1 (or arbitrary pair). Code diff (Pygments), schema diff, data-sample diff. | -| `/api/sse` | EventSource stream of `{kind, ...}` events. | -| `/api/notebook` | `PATCH` for reorder/remove/markdown edits from the browser. | -| `/api/markdown/` | `PUT` for markdown edits. | -| `/internal/notify` | Internal POST endpoint. Only `tallyman_mcp` calls this. Fans out to SSE clients. | - -**Tab semantics:** clicking a notebook cell in the `/notebook` tab auto-syncs `/catalog` to that cell's entry, so when you switch back to catalog the entry is already focused. - -**Markdown editor:** uses a small textarea or Monaco-lite for V1; Markdown rendered with `markdown-it` on save. No WYSIWYG — keep it predictable for the live demo. - ---- - -## Buckaroo integration - -Buckaroo runs as a **live Tornado server** managed by the companion subprocess group. This enables search, sort, filter, and column-level recon (null counts, distributions) directly in the table widget — the in-line recon tier of the three-tier model. - -**Lifecycle:** -1. `tallyman run` starts: companion spawns `python -m buckaroo.server --port 8700 --no-browser --stdio-control` as a subprocess. Companion holds the subprocess handle; cleans up on shutdown. -2. When a new catalog entry lands (companion is notified via `/internal/notify`), the companion expands the entry's `xorq_build/` dir into a tmp copy with `${TALLYMAN_PROJECT_ROOT}` placeholders resolved, then POSTs that path to buckaroo's `/load_expr` endpoint (added in upstream PR 776). Buckaroo loads the xorq expression and returns a `session_id`. Stored in `buckaroo_sessions.json` keyed by content hash. -3. The entry-detail page renders a `
` placeholder; the React embed (`static/buckaroo-embed.js`, built from `packages/embed/`) mounts `BuckarooServerView` into it and pages rows over the WS. Sort/search push-down lands on the underlying backend rather than re-materialising the parquet. -4. On companion startup, `buckaroo_sessions.json` is loaded but treated as a *naming hint* — buckaroo itself is fresh, so sessions are lazily re-created on first view (and the cache is invalidated when buckaroo's reported `started` timestamp doesn't match what was last persisted). - -**What goes through Buckaroo vs not:** -- Tabular results → React embed (`BuckarooServerView`) mounted into the entry-detail page, talking to Buckaroo's WS. -- Chart (Vega-Lite spec attached) → renders above the embed in the entry detail panel. -- Schema, code, prompt, lineage, forensic history → Jinja2-rendered HTML in the entry detail tabs (NOT in Buckaroo). - -**Why not the Buckaroo MCP server?** Different pattern: that one opens a sibling browser tab. We want Buckaroo embedded inside our companion's catalog tab, not as a sibling. We use the standalone Tornado server directly. - -**Why React embed and not iframe?** The original design used `