Voice Copilot: voice layer over a complex web app (CRM demo)

Two-tier LLM (Gemini 2.5 Flash reflexes + 2.5 Pro vectorless recall),
agent-as-retriever memory (no embeddings), LiveKit RPC capability
registry (navigate/execute/set_field/get_state), human-in-the-loop
confirm, and live voice form-fill. Includes architecture doc, demo
script, social content, and a screen-record helper.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>

Author: hatimanees

.gitignore

@@ -0,0 +1,32 @@
+# secrets — NEVER commit
+.secrets/
+*.json.key
+**/vertex.json
+.env
+.env.local
+agent/.env
+web/.env
+
+# python
+__pycache__/
+*.pyc
+.venv/
+venv/
+*.egg-info/
+
+# node
+node_modules/
+dist/
+.vite/
+*.log
+
+# demo recordings (large binaries)
+demo/
+*.mp4
+*.mov
+
+# os / editor
+.DS_Store
+Thumbs.db
+.idea/
+.vscode/

ARCHITECTURE.md

@@ -0,0 +1,78 @@
+# Architecture
+
+Voice Copilot is a **voice layer you drop onto a complex web app**. It turns
+"click through 6 screens / raise a ticket / ping the ops team" into "click the mic
+and say what you want." The victim app here is a real-estate CRM, but the pattern
+is app-agnostic.
+
+## The whole loop
+
+```mermaid
+flowchart LR
+  U([🎙️ User speaks]) -->|WebRTC audio| LK[LiveKit room]
+  LK --> STT[Deepgram STT]
+  STT --> FLASH["Voice brain<br/>Gemini 2.5 Flash<br/>(turn-taking + tool routing)"]
+  FLASH --> TTS[ElevenLabs TTS]
+  TTS -->|audio track| LK --> U
+
+  FLASH -.->|recall query| PRO["Vectorless memory<br/>Gemini 2.5 Pro<br/>(reasons over knowledge tree)"]
+  PRO -.->|resolved facts| FLASH
+
+  FLASH ==>|"navigate / execute / set_field / get_state<br/>(LiveKit RPC)"| WEB[React CRM app]
+  WEB ==>|state + confirm result| FLASH
+  WEB -.->|live event feed| SIDE[Debug sidebar]
+
+  subgraph Knowledge["Vectorless knowledge layer (docs-ready)"]
+    PRO
+    TREE[("CRM data + user prefs<br/>+ session context<br/>— structured tree, no embeddings")]
+    PRO --- TREE
+  end
+```
+
+## Three ideas doing the heavy lifting
+
+### 1. Two-tier LLM — reflexes vs. reasoning
+A single model is either too slow for natural turn-taking or too dumb to resolve
+"the villa lead who went quiet after the price." So we split it:
+
+- **Gemini 2.5 Flash** is the *voice brain*: sub-second turn-taking, decides which
+  tool to call. It deliberately knows **nothing** about the CRM data.
+- **Gemini 2.5 Pro** sits **behind** the `recall` tool and does the slow reasoning.
+  Latency there is fine because it only fires when Flash lacks context.
+
+### 2. Vectorless memory (agent-as-retriever) — no embeddings, no vector DB
+`recall(query)` hands the whole knowledge tree — CRM records + user preferences +
+session context — to Pro as structured text and asks it to *reason* to the answer
+(PageIndex / vectorless-RAG style, same shape as how Claude Code / Cursor retrieve).
+
+Why it matters for the roadmap: the tree is just **readable structured data**. Drop
+in product docs, portal how-tos, or an SOP file and the same `recall` path answers
+"how do I issue a refund on Exotel" — support work, no re-architecture.
+
+### 3. The app publishes its own capabilities — no DOM scraping, no vision
+The agent never guesses the UI. The React app **registers RPC methods** the agent
+calls:
+
+| Tool | What the app does | Confirm? |
+|---|---|---|
+| `navigate(target, filters)` | route to dashboard / leads / pipeline / new-lead / a specific lead | no |
+| `get_state()` | return current view + form values + which fields are filled | no |
+| `set_field(field, value)` | mirror a spoken value into the New-Lead form, live | no |
+| `execute(action, params)` | mutate data **behind a visual Confirm card** | **yes** |
+| `recall(query)` | (agent-internal) vectorless lookup via Pro | no |
+
+This is why it's reliable enough to demo live: navigation/mutation are typed,
+app-owned actions — not screenshot guesses. Any mutating action is gated by a
+human-in-the-loop Confirm card + spoken confirmation.
+
+## Request lifecycle — "add a new lead" (voice form-fill)
+1. Flash calls `navigate("new_lead")` → form opens.
+2. Flash calls `get_state()` → sees which fields the user already typed.
+3. For each *empty* field, Flash asks conversationally; after each answer it calls
+   `set_field(...)` and the field **pulses and fills on screen**.
+4. Flash calls `execute("create_lead", {...})`; the app **merges** its params with
+   the on-screen draft and shows the Confirm card. User confirms → lead created.
+
+## Stack (all free-tier)
+LiveKit Agents (Python, VAD + turn detection) · Deepgram STT · Gemini 2.5 Flash +
+2.5 Pro on Vertex · ElevenLabs TTS · React + Vite + livekit-client.

GOAL.md

@@ -0,0 +1,82 @@
+# Voice Copilot — Project Goal
+
+## One-liner
+A voice copilot you drop into any complex web app. User clicks 🎙️, speaks in plain
+language, and the copilot **understands → navigates → executes** — confirming and
+collecting missing inputs by voice. The reusable, impressive artifact is the *voice
+layer*; the Mini-CRM is just the "victim app" that proves it works on something
+genuinely painful to click through.
+
+**Purpose:** portfolio piece to showcase applied voice-AI skill (intent understanding,
+app-state grounding, safe agentic action).
+
+## The 3 capabilities → 3 agent tools
+| Capability | Tool | Behavior |
+|---|---|---|
+| Understand what user *really* wants | `recall(query)` | Vectorless memory search via heavier LLM; resolves entity refs + prefs |
+| Navigate | `navigate(target, filters?)` | Frontend routes to page/section, highlights |
+| Execute | `execute(action, params)` + `collect(field)` | Runs action behind confirmation gate; voice slot-fills missing params |
+
+## Architecture (agent-as-retriever + two-tier LLM)
+```
+🎙️ User ──▶ LiveKit Agent ── Gemini 2.5 Flash (reflexes: turn-taking, tool routing)
+                 ├─▶ recall(query)   ──▶ Gemini 2.5 Pro reasons over memory tree ──▶ answer
+                 ├─▶ navigate(target) ─RPC─▶ React Mini-CRM (routes + highlights)
+                 ├─▶ collect(field)   ─RPC─▶ React (voice slot-fills a form field)
+                 └─▶ execute(action)  ─RPC─▶ React (Confirm card + 🔊/🔇 + voice confirm)
+```
+- **Frontend publishes its own capability registry** (routes + typed actions) to the agent.
+  Agent never scrapes DOM or guesses — it calls back via **LiveKit RPC**. Reliable + safe.
+- **Flash** = voice loop only. **2.5 Pro** = lives *behind* `recall`, does vectorless
+  reasoning. Latency there is acceptable (only fires when Flash lacks context).
+
+## Memory = vectorless (grounded in 2025/26 SOTA)
+No embeddings, no vector DB. Structured, human-readable tree (JSON/markdown):
+session context + CRM data + persistent user prefs. `recall` hands the relevant slice
+to Pro and asks it to reason. Debuggable, zero infra.
+- Ref: PageIndex / vectorless RAG (VectifyAI, Sep 2025); agent-as-retriever (Claude Code/Cursor).
+- Decision: **session context + vectorless persistent prefs** for v1.
+
+## Stack (all free-tier)
+| Layer | Choice |
+|---|---|
+| Orchestration | LiveKit Agents (Python) — free VAD, barge-in, turn detection |
+| STT | Deepgram streaming |
+| Voice brain | Gemini 2.5 Flash (Vertex) |
+| Recall brain | Gemini 2.5 Pro (Vertex, behind `recall`) |
+| TTS | ElevenLabs (fallback: Deepgram Aura) |
+| Frontend | React + LiveKit client SDK (owns routing, forms, Confirm card + mute) |
+
+## Confirmation UX
+Visual Confirm/Cancel card **+** spoken "…confirm?" **+** mute (🔇) toggle.
+Navigation/read = no confirm; mutating actions = confirm.
+
+## v1 hero flows (the 90-sec demo)
+1. **Navigate** — "Show me Dubai leads that went cold." → filter + route, no confirm.
+2. **Recall + navigate** — "Open the one I talked to yesterday about the villa." → `recall` resolves entity → opens.
+3. **Execute + confirm** — "Move this to Won." → Confirm card + voice → executes.
+4. **Execute + slot-fill** — "Schedule a follow-up." → `collect` voice-fills date/note → confirm → done.
+
+## Latency target
+< 1.5s voice round-trip for navigate/execute. `recall` may take longer (acceptable).
+
+## Milestones
+- **M0** — Repo scaffold: React Mini-CRM (seeded leads) + LiveKit room + agent skeleton.
+- **M1** — Capability registry + `navigate` RPC working end-to-end (flow 1).
+- **M2** — `recall` tool + vectorless memory tree + Gemini 2.5 Pro (flow 2).
+- **M3** — `execute` + Confirm card + mute + voice confirm (flow 3).
+- **M4** — `collect` slot-filling (flow 4). Record demo video.
+
+## Open items
+- Gemini Vertex service-account JSON — user to provide (drop at `C:\voice-copilot\.secrets\vertex.json`, gitignored).
+- Deepgram + ElevenLabs API keys — free-tier signup.
+
+## Status
+- Goal locked 2026-07-12.
+- **M0 scaffold DONE** (2026-07-12): repo structure, agent skeleton (main.py + 4 tools),
+  vectorless memory.py, token_server.py, React Mini-CRM (App.tsx + Confirm/collect cards),
+  seed data, README, .gitignore. Creds verified: Vertex Flash+Pro reachable; vectorless
+  `recall` resolves fuzzy refs end-to-end.
+- **Blocking to run:** LiveKit Cloud creds (URL/key/secret) in `agent/.env`; then
+  `pip install -r agent/requirements.txt` + `npm install` in `web/`.
+- **Next: M1** — wire `navigate` RPC live (flow 1) once LiveKit creds land.

README.md

@@ -0,0 +1,104 @@
+<h1 align="center">🎙️ Voice Copilot</h1>
+
+<p align="center">
+  <b>A voice layer you drop onto any complex web app.</b><br/>
+  Click the mic, say what you want — it <b>navigates</b>, <b>fills forms</b>, and
+  <b>acts</b> (with confirmation), instead of you clicking through 6 screens or
+  raising a ticket.
+</p>
+
+<p align="center">
+  <i>Demo app: a real-estate CRM. The reusable part is the voice layer.</i>
+</p>
+
+---
+
+## Why
+
+So much software makes you **connect with a team to do a basic thing**. Government
+portals. Telephony consoles like **Exotel / Acefone / Twilio**. Internal admin
+tools. You know *what* you want — "move this lead to Won", "add a new lead", "show
+me the cold Dubai deals" — but the UI makes you hunt, and the deeper actions mean a
+ticket and a wait.
+
+**Voice Copilot collapses that.** You talk; it does the clicking. And because the
+knowledge layer is [vectorless and docs-ready](./ARCHITECTURE.md), the same thing
+that resolves *"the villa lead who went quiet after the price"* can later answer
+*"how do I issue a refund on this portal"* — turning it into a support copilot too.
+
+> ⚠️ This is a **rough prototype to showcase the pattern**, not a polished product.
+
+## What it does (demo flows)
+
+| You say | It does |
+|---|---|
+| *"Show me Dubai leads that went cold"* | filters the table live |
+| *"Open the villa lead who went quiet after the price"* | **vectorless recall** resolves the fuzzy reference → opens the lead |
+| *"Move Meera to Won"* | shows a **Confirm card** (+ spoken confirm) → updates the pipeline |
+| *"Add a new lead"* | walks you through it by voice; **each field fills in live**; confirm → created |
+
+A live **debug sidebar** shows the whole pipeline as it runs: 🎧 STT → 🧠 LLM → 🔧 tool calls.
+
+## How it works
+
+Two-tier LLM (fast **Gemini 2.5 Flash** for the voice loop + **2.5 Pro** behind a
+`recall` tool for reasoning), a **vectorless** knowledge layer (no embeddings, no
+vector DB), and an app that **publishes its own capabilities over LiveKit RPC** so
+the agent never scrapes the DOM. Every mutating action is gated by a human confirm.
+
+👉 **[Full architecture + diagram →](./ARCHITECTURE.md)**
+
+```
+🎙️ User ─▶ LiveKit ─▶ Gemini 2.5 Flash (voice reflexes, tool routing)
+                          ├─ recall(query)   ─▶ Gemini 2.5 Pro over vectorless tree
+                          ├─ navigate / get_state / set_field ─RPC▶ React CRM
+                          └─ execute(...)    ─RPC▶ Confirm card + voice confirm
+```
+
+**Stack (all free-tier):** LiveKit Agents · Deepgram STT · Gemini 2.5 Flash + Pro (Vertex) · ElevenLabs TTS · React + Vite.
+
+## Run it locally
+
+**Prereqs:** Python 3.11+, Node 18+, and accounts for LiveKit Cloud, Deepgram,
+ElevenLabs, and Google Vertex (all have free tiers).
+
+1. **Secrets** — create `.secrets/` with:
+   - `vertex.json` — a Vertex service-account key
+   - `.env` — `DEEPGRAM_API_KEY`, `ELEVENLABS_API_KEY`, and your LiveKit
+     `LIVEKIT_URL` / `LIVEKIT_API_KEY` / `LIVEKIT_API_SECRET`
+
+   (`.secrets/` is gitignored — nothing sensitive is committed.)
+
+2. **Agent** (two terminals):
+   ```bash
+   cd agent && python -m venv .venv && . .venv/Scripts/activate
+   pip install -r requirements.txt
+   python token_server.py     # terminal 1 — LiveKit tokens on :8787
+   python main.py dev          # terminal 2 — voice worker
+   ```
+
+3. **Web**:
+   ```bash
+   cd web && npm install && npm run dev   # http://localhost:5173
+   ```
+
+Open the app → **Talk to Copilot** → try the flows above.
+
+## Repo layout
+```
+agent/        Python LiveKit agent — voice loop + tools (recall/navigate/execute/set_field/get_state)
+agent/memory.py   vectorless recall via Gemini 2.5 Pro (no embeddings)
+web/          React + Vite CRM — RPC handlers, 4 views, Confirm card, debug sidebar
+data/         seed CRM data (also the memory source of truth)
+content/      demo script + social posts
+scripts/      screen-recording helper + smoke tests
+```
+
+## Roadmap
+- **Docs-aware support:** ingest portal how-tos/SOPs into the vectorless tree → answer "how do I do X" and do it.
+- Risk-tiered confirmation (auto-run reads, confirm mutations).
+- Persistent, learned user preferences.
+- Drop-in SDK so any React app can register its capability registry.
+
+## License
+MIT. Built as a portfolio showcase — feedback and ideas welcome.

agent/.env.example

@@ -0,0 +1,19 @@
+# Copy to ../.secrets/.env is NOT used by the agent; the agent reads THIS file's
+# real counterpart at agent/.env  OR the values below via the shell.
+# Fill LiveKit values from LiveKit Cloud (livekit.io -> project -> Settings -> Keys).
+
+# --- LiveKit (REQUIRED — the one remaining gap) ---
+LIVEKIT_URL=wss://YOUR-PROJECT.livekit.cloud
+LIVEKIT_API_KEY=
+LIVEKIT_API_SECRET=
+
+# --- Vertex / Gemini (already provisioned) ---
+GOOGLE_APPLICATION_CREDENTIALS=C:\voice-copilot\.secrets\vertex.json
+VERTEX_PROJECT_ID=your-gcp-project-id
+VERTEX_LOCATION=us-central1
+VOICE_LLM_MODEL=gemini-2.5-flash
+RECALL_LLM_MODEL=gemini-2.5-pro
+
+# --- STT / TTS (loaded from ../.secrets/.env by config.py) ---
+# DEEPGRAM_API_KEY and ELEVENLABS_API_KEY live in C:\voice-copilot\.secrets\.env
+ELEVENLABS_VOICE_ID=

agent/config.py

@@ -0,0 +1,43 @@
+"""Central config. Loads secrets from C:\\voice-copilot\\.secrets\\.env and the
+agent's own .env, and exposes typed settings for the voice pipeline + Vertex."""
+from __future__ import annotations
+import os
+from pathlib import Path
+from dotenv import load_dotenv
+
+ROOT = Path(__file__).resolve().parent.parent          # C:\voice-copilot
+SECRETS = ROOT / ".secrets"
+
+# Load .secrets/.env first (Deepgram/ElevenLabs keys), then agent/.env (overrides).
+load_dotenv(SECRETS / ".env")
+load_dotenv(Path(__file__).resolve().parent / ".env")
+
+
+def _req(name: str) -> str:
+    v = os.getenv(name)
+    if not v:
+        raise RuntimeError(f"Missing required env var: {name}")
+    return v
+
+
+# LiveKit — the transport/room layer
+LIVEKIT_URL = os.getenv("LIVEKIT_URL", "")
+LIVEKIT_API_KEY = os.getenv("LIVEKIT_API_KEY", "")
+LIVEKIT_API_SECRET = os.getenv("LIVEKIT_API_SECRET", "")
+
+# Vertex / Gemini
+GOOGLE_APPLICATION_CREDENTIALS = os.getenv(
+    "GOOGLE_APPLICATION_CREDENTIALS", str(SECRETS / "vertex.json"))
+os.environ["GOOGLE_APPLICATION_CREDENTIALS"] = GOOGLE_APPLICATION_CREDENTIALS
+VERTEX_PROJECT_ID = os.getenv("VERTEX_PROJECT_ID", "your-gcp-project-id")
+VERTEX_LOCATION = os.getenv("VERTEX_LOCATION", "us-central1")
+VOICE_LLM_MODEL = os.getenv("VOICE_LLM_MODEL", "gemini-2.5-flash")
+RECALL_LLM_MODEL = os.getenv("RECALL_LLM_MODEL", "gemini-2.5-pro")
+
+# STT / TTS
+DEEPGRAM_API_KEY = os.getenv("DEEPGRAM_API_KEY", "")
+ELEVENLABS_API_KEY = os.getenv("ELEVENLABS_API_KEY", "")
+ELEVENLABS_VOICE_ID = os.getenv("ELEVENLABS_VOICE_ID", "iWNf11sz1GrUE4ppxTOL")
+
+DATA_DIR = ROOT / "data"
+LEADS_FILE = DATA_DIR / "leads.json"