Skip to content

Commit 315e422

Browse files
steveyeggeclaude
andcommitted
docs: update dolt-storage.md to match reality
Schema section now reflects actual beads schema (v6) instead of aspirational mail/channels tables. Three Data Planes updated to show JSONL git-push as the live ledger transport. DoltHub remotes marked as in-progress (Julian Knutsen / Wasteland). File layout corrected with actual databases (sky, wyvern) and daemon/ paths. All ~/.dolt-data paths fixed to ~/gt/.dolt-data. Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
1 parent 8e68922 commit 315e422

1 file changed

Lines changed: 89 additions & 47 deletions

File tree

docs/design/dolt-storage.md

Lines changed: 89 additions & 47 deletions
Original file line numberDiff line numberDiff line change
@@ -1,7 +1,7 @@
11
# Dolt Storage Architecture
22

33
> **Status**: Current reference for Gas Town agents
4-
> **Updated**: 2026-02-16
4+
> **Updated**: 2026-02-28
55
> **Context**: Dolt is the sole storage backend for Beads and Gas Town
66
77
---
@@ -11,7 +11,9 @@
1111
Gas Town uses [Dolt](https://github.com/dolthub/dolt), an open-source
1212
SQL database with Git-like versioning (Apache 2.0). One Dolt SQL server
1313
per town serves all databases via MySQL protocol on port 3307. There is
14-
no embedded mode, no SQLite, and no JSONL.
14+
no embedded mode and no SQLite. JSONL is used only for disaster-recovery
15+
backups (the JSONL Dog exports scrubbed snapshots every 15 minutes to a
16+
git-backed archive), not as a primary storage format.
1517

1618
The `gt daemon` manages the server lifecycle (auto-start, health checks
1719
every 30s, crash restart with exponential backoff).
@@ -23,10 +25,11 @@ Dolt SQL Server (one per town, port 3307)
2325
├── hq/ town-level beads (hq-* prefix)
2426
├── gastown/ rig beads (gt-* prefix)
2527
├── beads/ rig beads (bd-* prefix)
26-
└── ... additional rigs
28+
├── wyvern/ rig beads (wy-* prefix)
29+
└── sky/ rig beads (sky-* prefix)
2730
```
2831

29-
**Data directory**: `~/.dolt-data/` — each subdirectory is a database
32+
**Data directory**: `~/gt/.dolt-data/` — each subdirectory is a database
3033
accessible via `USE <name>` in SQL.
3134

3235
**Connection**: `root@tcp(127.0.0.1:3307)/<database>` (no password for
@@ -74,43 +77,77 @@ transaction to maintain atomicity.
7477

7578
## Schema
7679

80+
Schema version 6. The full schema lives in `beads/.../storage/dolt/schema.go`.
81+
Key tables shown below; see source for indexes and full column lists.
82+
7783
```sql
84+
-- Core: every bead is a row in issues (tasks, messages, agents, gates, etc.)
7885
CREATE TABLE issues (
79-
id VARCHAR(64) PRIMARY KEY,
80-
type VARCHAR(32),
81-
title TEXT,
82-
description TEXT,
83-
status VARCHAR(32),
84-
priority INT,
85-
owner VARCHAR(255),
86+
id VARCHAR(255) PRIMARY KEY,
87+
title VARCHAR(500) NOT NULL,
88+
description TEXT NOT NULL,
89+
status VARCHAR(32) NOT NULL DEFAULT 'open',
90+
priority INT NOT NULL DEFAULT 2,
91+
issue_type VARCHAR(32) NOT NULL DEFAULT 'task',
8692
assignee VARCHAR(255),
87-
labels JSON,
88-
parent VARCHAR(64),
89-
created_at TIMESTAMP,
90-
updated_at TIMESTAMP,
91-
closed_at TIMESTAMP
93+
owner VARCHAR(255) DEFAULT '',
94+
sender VARCHAR(255) DEFAULT '', -- messaging
95+
mol_type VARCHAR(32) DEFAULT '', -- molecule type
96+
work_type VARCHAR(32) DEFAULT 'mutex', -- mutex vs open_competition
97+
hook_bead VARCHAR(255) DEFAULT '', -- agent hook
98+
role_bead VARCHAR(255) DEFAULT '', -- agent role
99+
agent_state VARCHAR(32) DEFAULT '', -- agent lifecycle
100+
wisp_type VARCHAR(32) DEFAULT '', -- TTL-based compaction class
101+
metadata JSON DEFAULT (JSON_OBJECT()), -- extensible metadata
102+
created_at DATETIME, updated_at DATETIME, closed_at DATETIME
103+
-- ... plus ~20 more columns (see schema.go)
92104
);
93105

94-
CREATE TABLE mail (
95-
id VARCHAR(64) PRIMARY KEY,
96-
thread_id VARCHAR(64),
97-
from_addr VARCHAR(255),
98-
to_addrs JSON,
99-
subject TEXT,
100-
body TEXT,
101-
sent_at TIMESTAMP,
102-
read_at TIMESTAMP
106+
-- Relationships between beads
107+
CREATE TABLE dependencies (
108+
issue_id VARCHAR(255) NOT NULL,
109+
depends_on_id VARCHAR(255) NOT NULL,
110+
type VARCHAR(32) NOT NULL DEFAULT 'blocks', -- blocks, parent-child, thread
111+
PRIMARY KEY (issue_id, depends_on_id)
103112
);
104113

105-
CREATE TABLE channels (
106-
id VARCHAR(64) PRIMARY KEY,
107-
name VARCHAR(255),
108-
type VARCHAR(32),
109-
config JSON,
110-
created_at TIMESTAMP
114+
-- Labels (many-to-many)
115+
CREATE TABLE labels (
116+
issue_id VARCHAR(255) NOT NULL,
117+
label VARCHAR(255) NOT NULL,
118+
PRIMARY KEY (issue_id, label)
111119
);
120+
121+
-- Audit trail
122+
CREATE TABLE comments (id BIGINT AUTO_INCREMENT PRIMARY KEY, issue_id, author, text, created_at);
123+
CREATE TABLE events (id BIGINT AUTO_INCREMENT PRIMARY KEY, issue_id, event_type, actor, old_value, new_value, created_at);
124+
125+
-- Agent interaction log
126+
CREATE TABLE interactions (id, kind, actor, issue_id, model, prompt, response, created_at);
127+
128+
-- Infrastructure
129+
CREATE TABLE config (key PRIMARY KEY, value); -- runtime config knobs
130+
CREATE TABLE metadata (key PRIMARY KEY, value); -- schema version, etc.
131+
CREATE TABLE routes (prefix PRIMARY KEY, path); -- prefix→database routing
132+
CREATE TABLE issue_counter (prefix PRIMARY KEY, last_id); -- sequential ID generation
133+
CREATE TABLE child_counters (parent_id PRIMARY KEY, last_child);
134+
CREATE TABLE federation_peers (name PRIMARY KEY, remote_url, sovereignty, last_sync);
135+
136+
-- Compaction
137+
CREATE TABLE issue_snapshots (id, issue_id, compaction_level, original_content, ...);
138+
CREATE TABLE compaction_snapshots (id, issue_id, compaction_level, snapshot_json, ...);
139+
CREATE TABLE repo_mtimes (repo_path PRIMARY KEY, mtime_ns, last_checked);
112140
```
113141

142+
**Wisps** (ephemeral patrol data) reuse the same `issues` table with
143+
`wisp_type` set. They are Dolt-ignored (`dolt_ignore` table) so wisp
144+
mutations don't generate Dolt commits — only structural changes to the
145+
ignore config itself are committed.
146+
147+
**Mail** is implemented as beads with `issue_type='message'` in the
148+
issues table — there is no separate mail table. The `sender` field and
149+
`dependencies` (type='thread') provide threading.
150+
114151
## Dolt-Specific Capabilities
115152

116153
These are available to agents via SQL and used throughout Gas Town:
@@ -135,16 +172,17 @@ Arrays (labels): `union` merge. Counters: `max`.
135172

136173
Beads data falls into three planes with different characteristics:
137174

138-
| Plane | What | Mutation | Durability | Transport |
139-
|-------|------|----------|------------|-----------|
140-
| **Operational** | Work in progress, status, assignments, heartbeats | High (seconds) | Days–weeks | Dolt SQL server (local) |
141-
| **Ledger** | Completed work, permanent record, skill vectors | Low (completion boundaries) | Permanent | DoltHub remotes + federation |
142-
| **Design** | Epics, RFCs, specs — ideas not yet claimed | Conversational | Until crystallized | DoltHub commons (shared) |
175+
| Plane | What | Mutation | Durability | Transport | Status |
176+
|-------|------|----------|------------|-----------|--------|
177+
| **Operational** | Work in progress, status, assignments, heartbeats | High (seconds) | Days–weeks | Dolt SQL server (local) | **Live** |
178+
| **Ledger** | Completed work, permanent record | Low (completion boundaries) | Permanent | JSONL export → git push to GitHub | **Live** |
179+
| **Design** | Epics, RFCs, specs — ideas not yet claimed | Conversational | Until crystallized | DoltHub commons (shared) | **Planned** |
143180

144181
The operational plane lives entirely in the local Dolt server. The ledger
145-
and design planes federate via DoltHub using the Highway Operations
146-
Protocol — Gas Town's public federation layer built on Dolt's native
147-
push/pull remotes.
182+
plane is currently served by the JSONL Dog, which exports scrubbed snapshots
183+
to a git-backed archive every 15 minutes — this is the durable record that
184+
survives disasters (proven in Clown Show #13). The design plane will
185+
federate via DoltHub as part of the Wasteland commons (in progress).
148186

149187
## Data Lifecycle: Think Git, Not SQL (CRITICAL)
150188

@@ -226,7 +264,7 @@ CALL DOLT_BRANCH('-d', 'compact-temp');
226264
**Monthly flatten** (nuclear option from Tim Sehn):
227265
```bash
228266
# Squash ALL history to a single commit
229-
cd ~/.dolt-data/<db>
267+
cd ~/gt/.dolt-data/<db>
230268
dolt checkout -b fresh-start
231269
dolt reset --soft $(dolt log --oneline | tail -1 | awk '{print $1}')
232270
dolt commit -m "Flatten: squash history to single commit"
@@ -245,7 +283,7 @@ first, gc second.
245283
```bash
246284
# Manual gc (stop server first for exclusive access)
247285
gt dolt stop
248-
cd ~/.dolt-data/<db> && dolt gc
286+
cd ~/gt/.dolt-data/<db> && dolt gc
249287
gt dolt start
250288
```
251289

@@ -300,7 +338,7 @@ protocol.
300338

301339
### Git Remote Cache
302340

303-
Dolt maintains a cache at `.dolt-data/<db>/.dolt/git-remote-cache/` that
341+
Dolt maintains a cache at `~/gt/.dolt-data/<db>/.dolt/git-remote-cache/` that
304342
stores git objects built from Dolt's internal format. Per the Dolt team
305343
(Dustin Brown, 2026-02-26):
306344

@@ -336,11 +374,14 @@ remote with local state. Subsequent pushes should work without `--force`.
336374
- **Server downtime**: Push requires exclusive access to the data directory,
337375
so the server must be stopped during push. This creates a maintenance window.
338376

339-
### Future: DoltHub Remotes
377+
### DoltHub Remotes (In Progress)
340378

341379
DoltHub's native protocol (`https://doltremoteapi.dolthub.com/...`) avoids
342-
the git-remote-cache entirely and is much faster. Migration requires DoltHub
343-
accounts and reconfiguring remotes with `dolt remote set-url`.
380+
the git-remote-cache entirely and is much faster. Julian Knutsen is
381+
implementing DoltHub-based federation as part of the Wasteland commons —
382+
this will replace git-protocol remotes for the design and ledger planes.
383+
Migration requires DoltHub accounts and reconfiguring remotes with
384+
`dolt remote set-url`.
344385

345386
## File Layout
346387

@@ -350,10 +391,11 @@ accounts and reconfiguring remotes with `dolt remote set-url`.
350391
│ ├── hq/ Town beads (hq-*)
351392
│ ├── gastown/ Gastown rig (gt-*)
352393
│ ├── beads/ Beads rig (bd-*)
353-
│ └── wyvern/ Wyvern rig (wy-*)
394+
│ ├── wyvern/ Wyvern rig (wy-*)
395+
│ └── sky/ Sky rig (sky-*)
354396
├── daemon/
355397
│ ├── dolt.pid Server PID (daemon-managed)
356-
│ ├── dolt-server.log Server log
398+
│ ├── dolt.log Server log
357399
│ └── dolt-state.json Server state
358400
└── mayor/
359401
└── daemon.json Daemon config (dolt_server section)

0 commit comments

Comments
 (0)