HeliosDB-Nano as an optional read store (a fork to poke at, not a review ask)

[danimoya]

Hey Nate —

I built a database — HeliosDB-Nano (Apache-2.0, OSS) — and went looking for a real workload to stress it. Yours was the one I actually wanted to run, so I deployed token-dashboard against my own ~/.claude/projects/ and lived on it for weeks. The streaming-snapshot dedup by message.id is the right call — that's the part most people get wrong, and it's why your totals actually match what the API billed. Read-only over the JSONL, stdlib-only, no build step: I kept all of that intact. I didn't have to fight your code to extend it, which is rarer than it should be.

This PR adds HeliosDB-Nano as an optional primary read store (TD_PRIMARY_STORE=helios). It's off by default; SQLite stays the ingest source of truth and nothing changes if the flag is unset. I deliberately did not want to drop a giant review burden on you — it's a big diff and your project is rightly stdlib-only — so treat this as a fork to poke at if you're curious, not a review ask. Merge nothing, merge a slice, or just read the numbers. I want to be straight about both halves of it, because the wins are real and so are the warts.

Wins. Vector/semantic search over prompts is the thing SQLite can't do at all — median 13 ms across search/clusters/alerts/audit in mirror mode. Once the materialized views were correct, read latency flipped: /api/overview 3.2 s → 0.010 s, /api/projects 9.3 s → 0.246 s, /api/sessions 7.8 s → 0.087 s, every tab sub-second. And it stayed correct: 6/6 read functions match SQLite to the exact integer (cache_read 17,420,235,587 = 17,420,235,587), and the match held after a live scan added ~210 messages.

Warts, no spin. Those MVs returned wrong aggregates for weeks — COUNT(DISTINCT) ~60× off — while pg_mv_staleness() falsely reported FRESH. That parked me. Fixed upstream now, but it was the blocker. "Embedded mode" isn't in-process either — it pipes over a subprocess REPL, 715 ms vs sqlite3's 1.7 ms on a 433k-row COUNT(*). Both are filed upstream with repros, not complaints.

I think the integration is the receipts, not a pitch — local vector + relational in one engine is a real, mostly-empty gap, and I'm building toward it. If any of this is interesting to you, I'd rather build it with you than at you: poke at the fork, tear it apart, or just compare notes. No pressure either way.

— danimoya

---

Technical detail — architecture, measured impact, honest caveats, how to try it (click to expand)

What it adds (opt-in, zero impact when off)

• A db_helios.py read layer mirroring every read function in db.py, selected at startup only when TD_PRIMARY_STORE=helios. With the flag unset, none of this code is on the hot path and the dashboard behaves exactly as it does today.
• A write/mirror path (helios_writer.py) that pushes the SQLite corpus into HeliosDB after each scan, preserving the (session_id, message_id) streaming-snapshot eviction so totals stay equal to SQLite.
• Six single-table GROUP BY materialized views (td_overview, td_overview_daily, td_project_summary, td_session_summary, td_model_breakdown, td_skill_breakdown) backing the aggregate routes.
• Vector-search features with no SQLite equivalent (search, semantic clustering, alerts, audit, pulse) + an Insights/MCP UI tab — the original reason to integrate.

Architecture

~/.claude/projects/*.jsonl
        | scanner (unchanged: incremental mtime + byte offset)
        v
   SQLite  <-- source of truth, ingest target, rollback target
        | helios_writer (post-scan mirror, ON CONFLICT DO UPDATE)
        v
  HeliosDB-Nano (RocksDB-backed)
        | 6 materialized views (single-table GROUP BY)
        v
   /api/* reads when TD_PRIMARY_STORE=helios

Reads an MV can't serve correctly (the expensive_prompts self-join, the mcp_* rollups) delegate back to SQLite rather than return wrong numbers. Rollback = run without the flag.

Measured impact

Validation corpus: 422,982 messages · 194,140 tool_calls · 214 sessions · 59 projects · 339 MB in SQLite.

Route	Live-SQL (helios)	MV-backed (helios)	SQLite baseline
/api/overview	3.2 s	0.010 s	0.6 s
/api/projects	9.3 s	0.246 s	0.5 s
/api/sessions	7.8 s	0.087 s	0.4 s
/api/mcp/summary	18 s	0.250 s	—
/api/prompts	>60 s → 502	0.689 s (SQLite-delegated self-join)	—
/api/daily	—	0.007 s	—
/api/by-model	—	0.001 s	—
/api/skills	—	0.001 s	—

Honest framing: the SQLite baselines are not an apples-to-apples loss for SQLite — the helios numbers benefit from precomputed MVs, and SQLite could be given the same treatment. What's genuinely SQLite-impossible is the vector search, not the aggregate speed. Container cold-start ~6 s. Full bulk migration: 470 s for 427,339 msgs / 80,750 tool_calls / 116,869 tool_results, 0 failed.

How it stays correct

• 6/6 read functions match SQLite to the exact integer (scripts/_cutover_validate.py), re-passed after a live scan.
• Caught + fixed a ~2% mirror drift with a clean DROP + re-migrate (drift was 453,387 vs 444,911 msgs).
• RocksDB durability survived kill -9.

How to try it

export TD_PRIMARY_STORE=helios   # default unset -> SQLite reads, unchanged
python3 cli.py dashboard

Leave it unset and nothing changes. Keep HOST=127.0.0.1.

The one dependency, honestly

The default path stays stdlib-only, no build step, no pip install — unchanged. When (and only when) the flag is on, this talks to HeliosDB over the Postgres wire via pg8000 (pure-Python). I'm not bundling a native extension or asking you to install anything to run the dashboard as it ships. (FWIW the published heliosdb_sqlite "embedded" wheel is not in-process — it shells out to a subprocess REPL, 715 ms vs sqlite3's 1.7 ms on 433k rows — so I use the PG wire, not that wheel.)

Honest caveats / limitations

Originally measured on HeliosDB-Nano v3.33.0, re-run on v3.34.0, and re-verified through v3.37.0 (2026-06-04, same ~448k-message corpus). Update: every item I'd flagged as open has since been fixed upstream — the list below now reads mostly as a changelog. Most were upstream HeliosDB issues; the rest are worked around client-side.

Fixed in v3.34.0–v3.37.0

• DELETE/DROP speed — used to hang (needed kill -9), then merely slow (~140 s for a real 200-row DELETE on v3.33.0). v3.34.0: 1.05 s (~133×); v3.36.1: 0.713 s, and the per-statement floor that forced the batch-DELETE-per-scan workaround is gone (no-op DELETE ~0.6 s → ~0 s).
• Live-JOIN latency — the one route served by a live HeliosDB JOIN (/api/tools) was 0.586 s on v3.33.0. v3.34.0: 0.009 s (~65×) (~0.011 s on v3.36.1). (The MV-backed routes were already sub-second on both.)
• MCP routes now mounted on --http-port — v3.37.0: GET /mcp/info returns real capabilities and POST /mcp tools/list returns a real catalog, so the MCP tab finally has an endpoint. And CODE_EMBED(text) is now a real SQL function (384-dim vectors), so semantic search / clusters / dup-retrieval run on real embeddings instead of the BLAKE2b hash fallback.
• COUNT(*) over a materialized view + MV-column aggregate return type — fixed v3.35.0 (MV COUNT(*) returns the right count; aggregates come back as ints, consistent with base tables).
• CTEs with $param binding — fixed v3.36.0 (a wire-routing bug; WITH … SELECT was misrouted to the command path). db_helios.py no longer needs to stay CTE-free.
• SHOW BRANCHES empty after CREATE BRANCH — fixed v3.36.2 (now lists named branches).
• CREATE SCHEMA — fixed v3.36.0 (accepted as a flat-namespace no-op).
• Cold-start regression (~11 s → 27 s on v3.34.0) — fixed v3.35.0 (~9 s).

Fixed earlier

• MVs were the critical blocker — wrong aggregates for weeks (COUNT(DISTINCT) ~60× off, COUNT/SUM ~1000× off) while pg_mv_staleness() reported FRESH. Fixed v3.32.2+.

Still open

• The published heliosdb_sqlite "embedded" wheel is not in-process — it shells out to a subprocess REPL (715 ms vs sqlite3's 1.7 ms on a 433k-row COUNT(*)), so I use the PG wire (pg8000), not that wheel. This is the one item from the original list that hasn't moved.

Worked around client-side (version-independent)

• Aggregate return types — pg8000 returns base-table SUM/COUNT as ints but MV-column aggregates as strings (v3.33.0) / NULL (v3.34.0); coerced by column name so the cost math doesn't 502.
• Concurrency — a shared connection isn't thread-safe under ThreadingHTTPServer; fixed with per-thread connections.

Credit where due: HeliosDB upstream closed 8 of 11 dashboard-migration bugs I filed by v3.26.0, the ON CONFLICT DO UPDATE path went 0.4 → 690 ops/sec at 424k rows (~1700×) in v3.30, v3.34.0 took the two biggest operational warts (DELETE, live JOIN) off the board, and v3.35.0–v3.37.0 closed all 7 issues I'd filed against the cutover — including MCP-over-HTTP and a real CODE_EMBED SQL function in v3.37.0.

Scope / size

Large diff — 32 files, +5,933 / −62 (biggest new modules: db_helios.py +733, helios_writer.py +641, mcp.py +442, db.py +423). Happy to split this — e.g. land the read-only mirror + vector features first and the MV primary-store cutover second, or feature-gate the modules so they only load when the flag is set. Tell me the shape you'd prefer.

Testing done

• All 68 Python unit tests pass; the default (flag-off) path is unchanged.
• scripts/_cutover_validate.py — 6/6 exact-integer match, re-passed after a live scan.
• Full bulk migration validated end-to-end (470 s, 0 failures), re-run cleanly to eliminate drift.
• All 14 API routes exercised under TD_PRIMARY_STORE=helios; latencies above are measured.

[danimoya]

Quick update, Nate — since I opened this on v3.34.0, the HeliosDB-Nano side closed out every rough edge I'd flagged, so the "Still open" list in the description is now mostly historical (I've refreshed it in the PR body too). Re-verified against live builds through v3.37.0 (2026-06-04), same ~448k-message corpus:

All 7 issues I filed are fixed:

• MCP-over-HTTP is now mounted on --http-port (/mcp, /mcp/info, tools/list return real catalogs) — v3.37.0. The MCP tab finally has an endpoint to talk to.
• CODE_EMBED(text) is a real SQL function returning a 384-dim vector — v3.37.0. Semantic search / clusters / dup-retrieval now run on real embeddings instead of the BLAKE2b hash fallback.
• CTE + $param → v3.36.0; SHOW BRANCHES lists names → v3.36.2; CREATE SCHEMA → v3.36.0; MV COUNT(*) + MV-column SUM type → v3.35.0; cold-start regression → v3.35.0 (~27 s back to ~9 s).

Perf that moved (same corpus, real HTTP path):

• Real 200-row DELETE: 140 s (v3.33.0) → 1.05 s (v3.34.0) → 0.713 s (v3.36.1).
• Live-JOIN /api/tools: 0.586 s → ~0.01 s.
• MV-backed reads unchanged and still sub-second (/api/overview 0.010 s, /api/projects 0.246 s, /api/sessions 0.087 s); data exactness held at 6/6 exact-integer across the bumps.

Only real remaining gap I'd still flag: the published heliosdb_sqlite "embedded" wheel still shells out to a subprocess REPL (715 ms vs sqlite3's 1.7 ms), so I use the PG wire, not that wheel. Same offer stands — poke at the fork or just compare notes.

— danimoya