docs(upgrade): legacy JSONL state bridge for v2.10+ (#1625)

* docs: add legacy state upgrade guide

* docs: simplify legacy upgrade commands

* docs: clarify legacy state version boundaries

* docs: use tilde paths in legacy upgrade guide

* docs: add copyable legacy migration prompt

* docs: streamline legacy migration steps

* docs: present legacy migration as agent or manual paths

* docs: refine legacy migration agent prompt

* docs: restore migrated state to default path

Author: Jakob-Strobl

documentation/.vitepress/config.ts

@@ -44,6 +44,7 @@ export default defineConfig({
           items: [
             { text: 'Overview', link: '/guide/' },
             { text: 'Installation', link: '/guide/installation' },
+            { text: 'Legacy State Upgrade', link: '/guide/legacy-state-upgrade' },
             { text: 'First Workflow', link: '/guide/first-workflow' },
           ],
         },

documentation/architecture/design-rationale.md

@@ -18,17 +18,17 @@ MCP also enables input validation. When an agent calls `exarchos_workflow({ acti
 
 Trade-off: MCP adds operational overhead. The server needs to start up, establish stdio communication, and initialize its state. This adds latency to the first tool call. For simple tasks that don't need structured workflows, a few markdown files in the context are lighter. The MCP approach pays off when workflows are complex enough that losing state mid-session would cost more than the server overhead.
 
-## Why event sourcing over a database
+## Why event sourcing
 
-Agent workflows have a specific access pattern: events happen in order, most writes are appends, and the most common query is "give me the current state." This is a good fit for event sourcing with JSONL files.
+Agent workflows have a specific access pattern: events happen in order, most writes are appends, and the most common query is "give me the current state." This is a good fit for event sourcing.
 
-JSONL is simple. Each event is a line of JSON appended to a file. No schema migrations, no connection pooling, no query language. You can debug a workflow by opening the file in a text editor. You can back it up by copying a file. You can move it to another machine by copying a directory.
+Early Exarchos releases used JSONL files because they were simple: one event per line, easy to copy, and easy to inspect in a text editor. Current releases use SQLite as the required local event-store substrate so concurrent agents, idempotency claims, stream queries, and projections share one durable source of truth.
 
 Events are the audit trail. With a traditional database, you'd need a separate record-keeping system to answer "what happened during this workflow?" With event sourcing, the events *are* the history. Every transition, guard failure, task assignment, and review result is recorded with timestamps and context.
 
-Exarchos does use SQLite as an optional acceleration layer. The SQLite backend caches queries and sequence lookups for better performance on large event streams. But JSONL is always the source of truth. If the SQLite database corrupts, the server deletes it and rebuilds from JSONL on the next startup. No data loss.
+Because SQLite is local, the operational model is still file-based: back up or move the workflow store by copying the state directory while agents are stopped. If you have a pre-v2.9.0 JSONL-only directory, bridge it through v2.9.x before starting v2.10.0 or later; see [Legacy State Upgrade](/guide/legacy-state-upgrade).
 
-Trade-off: Query flexibility is limited. You can't write arbitrary SQL against a JSONL file. The solution is CQRS (Command Query Responsibility Segregation) materialized views, where the `exarchos_view` tool provides pre-built projections like pipeline status, task details, and convergence metrics. This adds code complexity, but it cleanly separates the write path (append events) from the read path (query views).
+Trade-off: Query flexibility is intentionally mediated. Even though events live in SQLite, callers should use CQRS (Command Query Responsibility Segregation) materialized views rather than coupling directly to tables. The `exarchos_view` tool provides pre-built projections like pipeline status, task details, and convergence metrics. This adds code complexity, but it cleanly separates the write path (append events) from the read path (query views).
 
 ## Why typed agents over a single general-purpose agent
 

documentation/architecture/event-sourcing.md

@@ -8,15 +8,15 @@ outline: deep
 
 Agent sessions are fragile. They end when context windows fill up and compact, when the user closes their laptop, when the process crashes, or when the network drops. Any of these can happen mid-operation.
 
-With mutable state, a crash mid-write can leave a half-updated JSON file. You can't tell what happened. Did the task complete? Did the review pass? The state says one thing, but the state might be wrong.
+With mutable state, a crash mid-write can leave a half-updated record. You can't tell what happened. Did the task complete? Did the review pass? The state says one thing, but the state might be wrong.
 
 Event sourcing sidesteps this. Every action is recorded as an immutable event, appended to a log. State is computed from events, not stored directly. If state gets corrupted, you replay the events and rebuild it. The events themselves are the truth.
 
 This gives you crash recovery, a full audit trail, and reconciliation. If a session dies between writing an event and updating state, the next session reconciles automatically. You can answer "what happened during this workflow?" by reading the event log, since every transition, guard failure, and task assignment is recorded with timestamps and context. And if state gets out of sync from a bug, a concurrent write, or a corrupted file, you rebuild it from events. This is not hypothetical; it happens in practice when hook subprocesses write events while the main server is restarting.
 
 ## How it works
 
-Each workflow gets its own JSONL file: `{featureId}.events.jsonl`. One event per line, append-only. A typical event looks like this:
+Each workflow gets its own stream in the local SQLite event store. A typical event looks like this:
 
 ```json
 {
@@ -36,9 +36,9 @@ Events have:
 - `timestamp` -- ISO 8601, used for time-based queries
 - `idempotencyKey` (optional) -- deduplication key for retry safety
 
-State is a projection: a JSON object computed by reading events from sequence 0. In practice, state is cached in a `{featureId}.json` file and only new events (those with sequence numbers higher than the state's `_eventSequence`) are applied. This means state reads are fast (just read the JSON file) while still being rebuildable from events.
+State is a projection computed by reading events from sequence 0. In practice, projected state and CQRS views are cached so reads are fast while remaining rebuildable from events.
 
-The event store uses a `.seq` cache file alongside each JSONL stream for O(1) sequence lookup. On startup, it cross-validates the cached sequence against the actual JSONL line count and falls through to a full scan if they disagree.
+The event store keeps stream metadata, high-water marks, idempotency claims, projected state, and materialized-view snapshots in SQLite. Pre-v2.9.0 JSONL-only state directories need the [legacy state upgrade](/guide/legacy-state-upgrade) bridge before v2.10.0 or later can open them.
 
 ## Reconciliation
 
@@ -52,22 +52,22 @@ This reads the event store, compares sequence numbers against the state's `_even
 
 Reconciliation handles several real-world scenarios:
 
-- Crash recovery. Hook subprocesses write events directly to JSONL via sidecar files. On the next MCP server startup, sidecar events are merged into the main stream, and reconciliation brings state up to date.
-- State corruption. If the JSON state file is deleted or truncated, reconciliation rebuilds it entirely from events.
-- Sequence corruption. If events in the JSONL file have non-monotonic sequence numbers (from a bug or disk corruption), the event store detects this during initialization and re-sequences the entire stream.
+- Crash recovery. If a session ends after an event write but before a projected-state refresh, reconciliation brings state up to date on the next read.
+- State corruption. If projected state is missing or stale, reconciliation rebuilds it entirely from events.
+- Sequence conflicts. If another writer appends to a stream between read and write, optimistic concurrency reports the mismatch instead of losing an update.
 
 ## Concurrency control
 
 The event store uses optimistic concurrency via `expectedSequence`. A caller can pass the sequence number it last read; if another write happened in between, the append fails with a `SequenceConflictError`. This prevents lost updates when multiple processes try to write events to the same stream.
 
-Within a single process, a per-stream promise-chain lock serializes writes. Multiple event store instances sharing the same directory are prevented by a PID lock file that detects stale locks from crashed processes.
+Within a single process, a per-stream promise-chain lock serializes writes. Across processes, SQLite WAL and bounded busy handling coordinate concurrent access to the same state directory.
 
 ## Trade-offs vs. mutable state
 
 Event sourcing is not free:
 
-- Storage. Events accumulate, but JSONL is compact and workflows are finite. A complex feature workflow might produce a few hundred events over its lifetime, a few kilobytes of text.
-- Query complexity. You can't just read a field from the event log. You need projections (the cached state file) or materialized views (the CQRS views in `exarchos_view`). This adds code, but it also cleanly separates write and read concerns.
-- In-memory event log cap. The internal event log in state is capped at 100 entries (configurable via `EVENT_LOG_MAX`) to prevent unbounded memory growth. This means old events are still in JSONL but not in the in-memory state `_events` array. Materialized views query the store directly when they need historical data.
+- Storage. Events accumulate, but workflows are finite. A complex feature workflow usually produces a few hundred events.
+- Query complexity. You still should not treat the event table as mutable application state. Use projections or materialized views through `exarchos_workflow` and `exarchos_view`; this adds code, but it cleanly separates write and read concerns.
+- Operational dependency. Current releases require a working SQLite driver. If neither the bundled runtime nor the Node SQLite driver can load, the server fails fast instead of falling back to JSONL-only mode.
 
 The benefits (crash recovery, audit trails, reconciliation) matter more for agent workflows than for typical applications because agent sessions are inherently unreliable. When your process can vanish at any moment, immutable event logs are cheap insurance.

documentation/architecture/index.md

@@ -23,7 +23,7 @@ The MCP server (`servers/exarchos-mcp/`) handles all workflow logic. It exposes
 
 A fifth tool (`exarchos_sync`) exists for future remote synchronization but is hidden from agents.
 
-The event store persists everything to JSONL files (one per workflow) with JSON state files derived from events. JSONL is always the source of truth. An optional SQLite backend accelerates queries but self-heals from JSONL if the database corrupts.
+The event store persists workflow events to a local SQLite database with projected workflow state derived from those events. Pre-v2.9.0 JSONL-only state directories must be bridged through v2.9.x before v2.10.0 or later can open them; see [Legacy State Upgrade](/guide/legacy-state-upgrade).
 
 Lifecycle hooks intercept Claude Code events (session start, pre-compact, task completion, teammate idle) and trigger MCP operations. Hooks run as lightweight CLI subcommands with tight timeouts (5-30 seconds), skipping heavy initialization to stay fast.
 
@@ -35,7 +35,7 @@ Validation scripts are deterministic bash programs that replace prose checklists
 
 Agent-first. Every tool accepts structured JSON input, validates it with Zod schemas, and returns structured JSON output with clear error messages. When a guard blocks a transition, the error includes the expected state shape and a suggested fix (the exact tool call to resolve it). This is designed for LLM consumption, not human CLI usage.
 
-Event-sourced. Every workflow action produces an immutable event appended to a JSONL stream. State is derived from events, never mutated directly. If state and events diverge, `reconcile` rebuilds state from the event log. This matters because agent sessions end abruptly: context compaction, crashes, laptop lids closing. Mutable state can corrupt silently. Events don't.
+Event-sourced. Every workflow action produces an immutable event appended to a stream. State is derived from events, never mutated directly. If state and events diverge, `reconcile` rebuilds state from the event log. This matters because agent sessions end abruptly: context compaction, crashes, laptop lids closing. Mutable state can corrupt silently. Events don't.
 
 Token-efficient. LLM context windows are finite, and Exarchos is infrastructure. Every token it consumes is a token unavailable for actual coding. Lazy schema registration keeps MCP startup under 500 tokens. Field projection on state queries cuts response size by roughly 90%. Artifact references store file paths instead of inlining content. Every design choice accounts for context window cost.
 

documentation/architecture/platform-portability.md

@@ -74,8 +74,8 @@ This split is the reason portability works:
 
 **Platform-agnostic (runtime):**
 - MCP server -- stdio JSON-RPC, works with any client
-- Event store -- JSONL append-only log on the filesystem
-- State store -- JSON files derived from events
+- Event store -- local SQLite append-only streams on the filesystem
+- State store -- projections derived from events
 - Dispatch engine -- TypeScript handlers, zero Claude Code dependency
 - CLI -- Commander-based, zero Claude Code dependency
 

documentation/guide/installation.md

@@ -224,6 +224,8 @@ Workflow state at `~/.exarchos/state/` is forward-compatible — the v2.9 binary
 
 If the plugin update lands before the binary install, MCP server registration and the eight lifecycle hooks fail with `exarchos: command not found` until you complete step 2 and restart. Order matters here.
 
+Upgrading pre-v2.9.0 JSONL-only state directly to v2.10.0 or later requires an extra bridge step through v2.9.x. See [Legacy State Upgrade](/guide/legacy-state-upgrade) before pointing v2.10.0+ at a state directory that contains `*.events.jsonl` files and no `exarchos.db`.
+
 ## Update
 
 The bootstrap installers are idempotent — re-run the same one-liner and the new binary atomically replaces the old one. SHA-512 verification guards against partial writes.