ai-assistant-demo

A personal, Jarvis-style AI assistant — voice and an interactive UI as equal peers, built security-first: the permission gate, token broker, and audit log exist before anything they could fail to protect.

Status: Phase 5 of 6 (voice) — done

Voice in and out, fully local audio: wake word ("hey jarvis" via openWakeWord) → faster-whisper STT → the same web API the browser uses → Piper (or macOS say) TTS back. A fourth client of an already-working system, not a new subsystem — the voice client needs nothing but the assistant's URL, so it can run on a different machine than the orchestrator.

Phase	Builds	Status
1 — Security spine	Orchestrator loop, model adapter, JWT-verified fake tool, CLI confirmation, audit log	✅
2 — Real tools via MCP	MCP client, tool fingerprint registry (TOFU + drift detection), tiered tokens	✅
3 — Memory	pgvector fact store (RLS), persona loader, embedding adapter	✅
4 — UI	Web chat, confirmation cards, read-only persona/memory browsers	✅
5 — Voice	Wake word → local STT → orchestrator → local TTS, spoken confirmations	✅
6 — Deploy	Split to target topology; dev is single-machine by design	—

How the gate works

Three import-isolated packages form a real trust boundary:

• src/assistant — the orchestrator side: hand-controlled tool-calling loop (the single enforcement point), model adapter, CLI confirmation, fingerprint registry, and a broker that mints Ed25519-signed JWT permission slips tiered by risk: destructive calls get a single-use, argument-bound, seconds-lived token minted only after the user confirms; read-only calls ride a longer-lived tool-scoped token. Every step is audited with what triggered it (user_request vs tool_output — tool output is untrusted and re-enters the same gate).
• src/toolwrapper — a verifying HTTP proxy that spawns and owns one stdio MCP server as a child process, holding only the public key. There is no path to the tool that skips it: discovery and every call go through the wrapper, which verifies signature, expiry, tier, and tool binding — plus argument binding and atomic single-use jti consumption at the high tier. Each boundary enforces its own configured tier, so a low-tier token can never authorize a destructive call regardless of what the orchestrator minted. It can verify but structurally cannot mint; the orchestrator can mint but never verifies its own tokens.
• src/mcpservers — the actual stdio MCP servers (notes: read-only; files: destructive, sandboxed to a root dir), spawned with an explicit env allowlist rather than inheriting the wrapper's environment.

Tool definitions are trusted on first use: each tool's name + description + input schema is fingerprinted and requires one-time approval. On every reconnect the fingerprint is re-checked; a changed definition (the MCP "rug pull") is logged old-vs-new, requires explicit re-approval, and stays confirmation-forced for the session even if re-approved.

Memory

Memory is a direct integration (src/assistant/memory/), not an MCP tool — it's internal and has no separate trust boundary. Two tiers:

• Fact-recall — a Postgres + pgvector store keyed by user_id. Row-level security is enforced, not just declared: memory connects as a dedicated non-superuser role and each operation binds app.current_user_id, so the RLS policy limits every query to that user's rows in the database itself, even if the SQL omits a WHERE. Relevant facts are auto-recalled by semantic similarity each turn and injected into the system prompt; the model persists new facts with a remember_fact tool (an internal low-risk write — audited, but no confirmation or token).
• Persona/profile — hand-authored markdown in a folder (the dev stand-in for a synced vault), concatenated into the system prompt to shape tone.

Embeddings are an adapter chosen by config: local uses a small local sentence-transformers model (private, no API key; uv sync --extra local-embeddings), hashing is a zero-dependency fallback. Both produce 384-dim vectors, so the schema is backend-independent.

Voice

src/voiceclient is import-isolated from everything else and reaches the orchestrator only over HTTP — the same /api/chat the browser uses. All audio stays on-device: openWakeWord scores the wake word locally, faster-whisper transcribes locally, Piper synthesizes locally (say is the zero-setup macOS fallback). Backends are config, not architecture — each is a swappable adapter.

Confirmation is modality-agnostic by construction: a destructive tool call lands on the shared decision queue, where browser buttons and the voice client poll the same pending list — the voice client reads the request aloud and listens for yes/no, and whichever surface answers first wins. Ambiguous spoken answers are re-asked once, then denied; unanswered requests time out to deny. The gate always fails closed.

uv sync --extra voice
uv run assistant-web        # the voice client talks to this
uv run assistant-voice      # dev defaults: push-to-talk + `say`
# target stack: WAKE_BACKEND=openwakeword, TTS_BACKEND=piper (see .env.example)

Setup

Requires uv and Docker.

uv sync                                  # add --extra local-embeddings for the local embedding model
uv run python scripts/generate_keys.py   # dev Ed25519 keypair → keys/ (gitignored)
cp .env.example .env                     # then set ANTHROPIC_API_KEY and ASSISTANT_APP_PASSWORD
docker compose up -d                     # Postgres on host port 5433, schema auto-applied

Set ASSISTANT_APP_PASSWORD in .env before docker compose up — Postgres init uses it to create the assistant_app role and refuses to start without it. Use the same value in MEMORY_DATABASE_URL.

The default EMBEDDING_BACKEND=local needs the local-embeddings extra; set it to hashing to run without extra dependencies.

Run

uv run tool-wrapper notes   # terminal 1: read-only notes server (low tier)
uv run tool-wrapper files   # terminal 2: destructive files server (high tier)
uv run assistant            # terminal 3: CLI chat — or:
uv run assistant-web        # web UI on http://127.0.0.1:8080 (localhost only)

First run prompts one-time approval per discovered tool (TOFU). Destructive tool calls require confirmation before a token is minted — Allow? [y/N] in the CLI, an Allow/Deny card in the web UI; reads don't. In the web UI, unanswered confirmations time out to deny.

To try the web UI without an API key: uv run python scripts/preview_ui.py serves it on port 8090 with a scripted model and real wrappers/memory.

Tests

uv run pytest

tests/test_bypass.py proves the destructive tool won't run without a confirmed, single-use, unexpired token — seven bypass attempts against a live wrapper fronting the real MCP server (no token, forged signature, expired, replayed jti, tampered arguments, wrong-tool token, low-tier token at the high boundary), each asserting nothing reached the filesystem. tests/test_registry.py covers TOFU and rug-pull drift; tests/test_tiering.py covers both tiers' token rules; tests/test_orchestrator.py runs the loop end-to-end. tests/test_memory.py covers semantic recall and RLS user isolation (including that a raw un-filtered SELECT still can't cross users); tests/test_memory_orchestrator.py covers persona injection, auto-recall, and the remember_fact path. tests/test_webui.py covers the decision queue's fail-closed semantics and the allow/deny confirmation flows end-to-end over HTTP. tests/test_voice.py drives the voice loop against the same gate with scripted audio backends — a spoken yes executes, a spoken no doesn't, and ambiguous answers fail closed. Skips if Postgres isn't running.

Acknowledgements

Built with assistance from Claude Code (Anthropic) for code generation and project scaffolding. All architecture, security design, and threat-model decisions are my own.