🎙️ Voice Copilot

An expert that lives inside your software.
Click the mic and ask in plain words — "what am I looking at?", "what does this graph mean?", "what should I do next?" — and it explains the tool, grounded in the product's own knowledge, pointing at the exact part of the screen. When you want, it also acts (navigates, updates records, fills forms) — with confirmation.

Demo app: a basic real-estate CRM — a stage to show the possibility.

---

The problem

Companies build powerful, complex tools — CRMs, analytics dashboards, telephony consoles (Exotel / Acefone / Twilio), government portals, internal admin — and then hand them to people who are left staring at the screen:

• What does this graph mean? What is this metric?
• What's this feature even for?
• What's the right way to actually use this?

Today that gap is filled by training sessions, support tickets, docs nobody reads, or "let me connect you with the team" — and it burns an enormous amount of time.

The idea

Voice Copilot is an embedded expert that closes the comprehension gap in plain language — and can act when you ask. It knows what tool you're in, what you're looking at, and what the product's knowledge says, so it can teach the tool while you use it.

The mini-CRM here is deliberately basic — the point isn't the CRM, it's the possibility. This is where we start. The real asset is the voice + knowledge layer, which is vectorless and docs-ready: drop in a product's real docs/SOPs and the copilot becomes an expert in that tool — and can help with support work, not just clicks.

⚠️ A rough prototype to showcase the pattern, not a polished product.

What it does

Understand (the point):

You ask	It does
"What am I looking at?" / "What does the leads-by-stage chart mean?"	explains it in plain words and pulses the chart on screen
"What does open pipeline mean — should I worry it's bigger than won?"	grounded explanation: no, that's healthy — here's why
"What should I do next?"	best-practice guidance specific to this tool

Act (when you want it to):

You say	It does
"Open the villa lead who went quiet after the price"	vectorless recall resolves the fuzzy reference → opens the lead
"Move Meera to Won"	Confirm card (+ spoken confirm) → updates the pipeline
"Add a new lead"	voice form-fill — each field fills in live — then confirm

A live sidebar shows the whole pipeline as it runs: 🎧 STT → 🧠 LLM → 🔧 tool calls.

How it works

The copilot's explain and recall tools reason over a vectorless knowledge tree (no embeddings, no vector DB) with Gemini 2.5 Pro, while a fast Gemini 2.5 Flash runs the voice loop. The app publishes its own capabilities over LiveKit RPC, so the agent never scrapes the DOM. Every mutating action is gated by a human confirm.

👉 Full architecture + diagram →

🎙️ User ─▶ LiveKit ─▶ Gemini 2.5 Flash (voice reflexes, tool routing)
              ├─ explain(q)  ─▶ Gemini 2.5 Pro over PRODUCT-KNOWLEDGE tree  (+highlight UI)
              ├─ recall(q)   ─▶ Gemini 2.5 Pro over CRM-DATA tree
              ├─ navigate / get_state / set_field ─RPC▶ React app
              └─ execute(...) ─RPC▶ Confirm card + voice confirm

Stack (all free-tier): LiveKit Agents · Deepgram STT · Gemini 2.5 Flash + Pro (Vertex) · ElevenLabs TTS · React + Vite.

Run it locally

Prereqs: Python 3.11+, Node 18+, accounts for LiveKit Cloud, Deepgram, ElevenLabs, Google Vertex (all free tiers).

1. Secrets — create .secrets/ with vertex.json (Vertex service-account key) and .env (DEEPGRAM_API_KEY, ELEVENLABS_API_KEY, LiveKit LIVEKIT_URL / LIVEKIT_API_KEY / LIVEKIT_API_SECRET, and VERTEX_PROJECT_ID). .secrets/ is gitignored — nothing sensitive is committed.
2. Agent (two terminals):

   cd agent && python -m venv .venv && . .venv/Scripts/activate
   pip install -r requirements.txt
   python token_server.py     # terminal 1 — LiveKit tokens on :8787
   python main.py dev          # terminal 2 — voice worker

3. Web: cd web && npm install && npm run dev → http://localhost:5173

Repo layout

agent/knowledge.py   PRODUCT knowledge — what views/metrics/graphs mean + best practice (the "expert")
agent/memory.py      vectorless recall over CRM data (fuzzy reference resolution)
agent/main.py        voice loop + tools (explain/recall/navigate/execute/set_field/get_state)
web/                 React + Vite CRM — RPC handlers, 4 views, highlight, Confirm card, debug sidebar
data/                seed CRM data

Roadmap

• Real docs ingestion: point the vectorless layer at a product's actual docs/SOPs → instant expert + support copilot for any tool.
• Drop-in SDK so any app can register its capability registry + knowledge.
• Risk-tiered confirmation; persistent learned preferences.

License

MIT. Built as a portfolio showcase — feedback and ideas welcome.