CLI-doc

[sebschopf]

How to give LLMs the full SurrealDB documentation — a build script + CLI

TL;DR: The SurrealDB docs repo contains 1 096 .mdx pages — everything an AI
agent needs to write correct SurrealQL. I wrote two small shell scripts: one that
clones the official surrealdb/docs.surrealdb.com repo and converts those pages
to clean markdown, and a lightweight CLI that navigates them with minimal token
waste. If you have a better way, I'll happily drop mine.

Table of contents

• The problem
• The build script
• The CLI (surreal-docs)
• Integration with agent-skills
• Two scripts, one workflow
• Why this matters

The problem

LLMs routinely produce incorrect SurrealQL because they guess from outdated or
fragmented training data. The surrealdb/agent-skills repo is a great starting
point, but it doesn't cover every statement, every function, every CLI flag.

Meanwhile, the SurrealDB documentation website (Astro + MDX) contains all the
answers — but it's not directly consumable by an agent: the pages have JSX
components (<Tabs>, <RailroadDiagram>), Astro frontmatter, and a site-specific
folder structure.

The build script

build-surrealdb-docs.sh (gist):

1. Clones surrealdb/docs.surrealdb.com (shallow, blobless — fast and light).
2. Converts every .mdx file under src/content/ to plain .md:
   • Strips JSX components and imports.
   • Extracts the title and description from the frontmatter.
   • Keeps all code blocks, tables, and markdown intact.
3. Generates an index.json tree of all 1 096 pages.

The output lands in ~/.cache/surrealdb-docs/ — about 8.5 MB of clean markdown.

# Initial build
./scripts/build-surrealdb-docs.sh

# Refresh (re-clone + re-convert)
./scripts/build-surrealdb-docs.sh --update

The CLI (surreal-docs)

After the build, the agent has 8.5 MB of markdown. The naive approach — dump a
whole page to stdout — burns hundreds of tokens on content it may not need.
surreal-docs
solves this: it resolves a topic to a file path (~3 tokens), then the agent
decides whether to read a summary or the full page.

# ❌ Naive: dumps 2,000+ tokens
grep -rl "DEFINE INDEX" ~/.cache/surrealdb-docs/content/ | xargs cat | head -50

# ✅ Token-optimized: returns a path (~3 tokens)
surreal-docs indexes
→ /home/.../define/indexes.md
→ agent calls read_file() on that path

Five modes, each for a specific agent workflow:

Command	Token cost	What it returns
surreal-docs select	~3	Just the file path — agent calls read_file
surreal-docs --search HNSW	~50	List of matching paths + titles
surreal-docs --summary indexes	~200	Title + first 20 lines (decide if useful)
surreal-docs --list statements	~50	Section file list
surreal-docs --show indexes	Full page	For humans in the terminal

Resolution is case-insensitive and matches partial names: surreal-docs indexes
finds define/indexes.md automatically.

Known limitation: the docs cache is a static snapshot. There is no auto-update
mechanism — you must re-run build-surrealdb-docs.sh --update manually to pull the
latest documentation. We did not build a scheduled refresh or a background watcher.

Integration with agent-skills

I keep the official surrealdb/agent-skills loaded as always-on "skills" (they
cover the 80 % use-case), and fall back to the cached markdown tree when the
agent needs the full reference. The two layers work together:

Layer	Loaded	Token cost	Covers
Agent skills (6 topics)	Always	~30K total	SurrealQL, functions, perf, vectors, CLI, SurrealKit
Cached docs (1 096 pages)	On-demand	~0 until read	Everything — every statement, every function, every flag

Two scripts, one workflow

Script	Role	Gist
build-surrealdb-docs.sh	Clones + converts 1 096 .mdx → .md	gist
surreal-docs	Resolves topics → file paths or summaries	gist

Run both once, then the agent has the entire SurrealDB documentation available
on-demand, without wasting context tokens on pages it doesn't need.

Why this matters

I'm building a VTT (Virtual Tabletop) on top of SurrealDB with an AI coding
agent. Every wrong query costs me real debugging time. Having the entire
documentation available to the agent without guessing is a game-changer.

Every coding agent (Cline, Claude Code, Copilot, Cursor) faces the same
constraint: a finite context window. Throwing 8.5 MB of documentation at it
at once is impossible. But letting it navigate that documentation —
resolving topics to paths, reading only what's relevant — makes the entire
knowledge base usable, one page at a time.

If the SurrealDB team could ship an official markdown export (or even an
llms.txt / docs.json endpoint), this whole workflow would be trivial for
everyone. Until then, 150 lines of bash do the job.