Voice-agent: live retrieval web UI + region metadata filtering

moss-live-labs/examples/voice-agent/.env.example

@@ -7,7 +7,10 @@ LIVEKIT_API_SECRET=secret
 MOSS_PROJECT_ID=Your moss project id here
 MOSS_PROJECT_KEY=Your moss project key here
 
-# AI Provider Keys
+# Initial region (US or EU) — the UI region picker overrides this live per query.
+MOSS_REGION=US
+
+# AI Provider Keys (OpenAI = LLM, Deepgram = STT, Cartesia = TTS)
 OPENAI_API_KEY=Your openai api key here
 DEEPGRAM_API_KEY=Your deepgram api key here
 CARTESIA_API_KEY=Your cartesia api key here

moss-live-labs/examples/voice-agent/DEMO_SCRIPT_METADATA.md

@@ -0,0 +1,49 @@
+# Demo Script — Metadata Filtering (region-scoped answers)
+
+~90s. Shows the **same question** returning **different, correct answers** by region,
+because Moss filters retrieval on `metadata.region` — one shared index, scoped per line.
+
+## Before you record
+- [ ] **Re-seed** (metadata changed): delete the old `demo-customer_faqs` index in the Moss
+      portal, then `python seed_index.py`.
+- [ ] `python agent.py dev` · `livekit-server --dev` · web UI at localhost:3000
+- [ ] Region is chosen with the **US / EU picker** in the panel (starts on US) — no restart.
+
+## 1 · Frame it (0:00–0:12)
+> "Same knowledge base, one index. This support panel has a region picker — right now it's
+> set to the US, and Moss only retrieves policies that apply to that region. Watch."
+
+**[Connect. Panel shows the picker on US and `filter · region ∈ [US, all]`.]**
+
+## 2 · US answer (0:12–0:35)
+> "What's your return window?"
+
+**[Point at panel: the filter line + the retrieved US chunk.]**
+> "Thirty days. The panel shows the filter — region is US or global — and it pulled the
+> US returns policy. The EU policy is in the same index, but it was filtered out."
+
+## 3 · Switch region (0:35–1:05)
+**[Click EU in the picker — the filter line flips to `region ∈ [EU, all]` instantly.]**
+> "Now I switch the same agent to the EU — no restart, just the filter. Same question."
+
+> "What's your return window?"
+
+> "Fourteen days — the EU right of withdrawal. Same index, same question, different answer,
+> because the metadata filter scoped retrieval to the right region."
+
+## 4 · Why it matters (1:05–1:25)
+> "That's one line of filter in the query. It's how you serve region-specific policies,
+> or isolate tenants, or gate content by plan — all from a single Moss index, evaluated
+> locally in milliseconds. Open source at github.com/usemoss/moss."
+
+---
+
+## The filter (for reference)
+```python
+QueryOptions(top_k=5, alpha=0.8,
+    filter={"field": "region", "condition": {"$in": ["EU", "all"]}})
+```
+
+## Try these too
+- "How much is shipping?" → US quotes dollars, EU quotes euros (with VAT).
+- A global question ("I forgot my password") → same answer in both regions (region = `all`).

moss-live-labs/examples/voice-agent/README.md

@@ -1,58 +1,73 @@
 # Voice Agent
 
-A voice assistant that answers questions using your Moss knowledge base. You talk to it, it searches your data, and responds out loud.
+A customer-support **voice agent** grounded in a Moss knowledge base, with a brand-styled
+web UI that shows Moss doing the retrieval live. You talk to it, it searches your data,
+and a side panel surfaces the exact chunks Moss returned (with match scores and query
+latency) for every turn.
+
+```
+Browser (web/)  ⟷  LiveKit room  ⟷  agent.py (STT → LLM → TTS)  ⟷  Moss (RAG)
+```
 
 ## What you need
 
-- A [Moss](https://moss.dev) account with a project and an index loaded with your data
-- A [LiveKit](https://livekit.io) account (or run it locally)
-- An [OpenAI](https://platform.openai.com) API key (for the AI responses)
-- A [Deepgram](https://deepgram.com) API key (for speech-to-text)
-- A [Cartesia](https://play.cartesia.ai) API key (for text-to-speech)
+- A [Moss](https://moss.dev) account (project ID + key)
+- [LiveKit](https://livekit.io) running locally (`livekit-server --dev`)
+- [OpenAI](https://platform.openai.com) (LLM), [Deepgram](https://deepgram.com)
+  (speech-to-text), and [Cartesia](https://cartesia.ai) (text-to-speech) API keys
+- Python 3.14+ (`uv`) and Node 18.18+ (`npm`) for the web UI
 
 ## Setup
 
-1. Install dependencies:
+```bash
+uv sync
+cp .env.example .env          # fill in your Moss + provider keys
+python agent.py download-files
+```
 
-   ```bash
-   uv sync
-   ```
+`.env` keys: `MOSS_PROJECT_ID`, `MOSS_PROJECT_KEY`, `OPENAI_API_KEY`, `DEEPGRAM_API_KEY`, `CARTESIA_API_KEY`.
+For local LiveKit, leave the `LIVEKIT_*` values as-is; for LiveKit Cloud, set them to your
+project's URL/key/secret and copy the same three into `web/.env.local`. The index name
+defaults to `demo-customer_faqs` (override with `MOSS_INDEX_NAME`).
 
-2. Copy the env file and fill in your keys:
+## 1. Seed the knowledge base
 
-   ```bash
-   cp .env.example .env
-   ```
+Loads the sample support FAQs in `data/faqs.json` into a Moss index:
 
-   Open `.env` and add:
+```bash
+python seed_index.py
+```
 
-   ```env
-   LIVEKIT_URL=ws://localhost:7880
-   LIVEKIT_API_KEY=devkey
-   LIVEKIT_API_SECRET=secret
+## 2. Run it
 
-   MOSS_PROJECT_ID=your-project-id
-   MOSS_PROJECT_KEY=your-project-key
-   MOSS_INDEX_NAME=your-index-name
+Open three terminals:
 
-   OPENAI_API_KEY=...
-   DEEPGRAM_API_KEY=...
-   CARTESIA_API_KEY=...
-   ```
+```bash
+# a) LiveKit server (local dev — uses devkey/secret)
+livekit-server --dev
 
-3. Download the required model files:
+# b) the agent (joins rooms automatically)
+python agent.py dev
 
-   ```bash
-   python agent.py download-files
-   ```
+# c) the web UI
+cd web
+npm install
+cp .env.local.example .env.local
+npm run dev            # → http://localhost:3000
+```
 
-## Run
+Open http://localhost:3000, click **Start the demo**, and talk. The right-hand panel
+shows what Moss retrieves on each turn.
 
-```bash
-python agent.py console
-```
+> Prefer no UI? `python agent.py console` still works for a mic-only, terminal session.
+
+## How the retrieval panel works
+
+On each user turn, `agent.py` queries Moss and publishes the results to the LiveKit room
+on the `moss.retrieval` data channel (`{query, docs:[{text, score}], took_ms}`). The web
+UI listens on that channel and renders them. The voice pipeline is otherwise untouched.
 
-Speak into your microphone and the agent responds out loud.
+See [`DEMO_SCRIPT_METADATA.md`](./DEMO_SCRIPT_METADATA.md) for a ready-to-record walkthrough.
 
 ## Resources
 

moss-live-labs/examples/voice-agent/agent.py

@@ -1,7 +1,11 @@
+import json
 import logging
 import os
+import time
 from dotenv import load_dotenv
+from livekit import rtc
 from livekit.plugins import openai, deepgram, silero, cartesia
+from livekit.plugins.turn_detector.multilingual import MultilingualModel
 from livekit.agents import (
     JobContext,
     WorkerOptions,
@@ -22,49 +26,110 @@
 MOSS_PROJECT_ID = os.getenv("MOSS_PROJECT_ID")
 MOSS_PROJECT_KEY = os.getenv("MOSS_PROJECT_KEY")
 INDEX_NAME = os.getenv("MOSS_INDEX_NAME", "demo-customer_faqs")
+# This support line serves one region. Metadata filtering scopes retrieval to
+# region-specific policies + global ("all") docs. Set MOSS_REGION=EU to compare.
+ALLOWED_REGIONS = {"US", "EU"}
+REGION = os.getenv("MOSS_REGION", "US")
+if REGION not in ALLOWED_REGIONS:
+    REGION = "US"
 
 logging.basicConfig(level=logging.INFO)
 logger = logging.getLogger("moss-agent")
 
 class MossSemanticRetrievalAgent(Agent):
 
-    def __init__(self, moss_client: MossClient):
+    def __init__(self, moss_client: MossClient, room: rtc.Room):
         super().__init__(
             instructions="""
-                You are a helpful customer support voice assistant.
-                You have access to a knowledge base which will be provided to you as context.
-                Always answer the user's question based on the provided context.
-                If the context doesn't contain the answer, politely say you don't know.
+                You are Northwind's customer support voice assistant, speaking directly to
+                one customer on a call. Answer naturally and concisely using ONLY the
+                knowledge-base context provided for the current question.
+
+                - Present policies as simply "our policy" — the customer's own. Never mention
+                  regions, "other regions", that policies vary by location, the knowledge
+                  base, filters, or how you look answers up.
+                - The context can change between questions; do not reuse facts or numbers from
+                  earlier in the conversation if they are not in the current context.
+                - If the current context doesn't answer the question, say you don't know and
+                  offer to help with something else.
+
+                Keep replies to a sentence or two, warm and clear for voice.
             """
         )
         self.moss = moss_client
+        self.room = room
+        self.region = REGION  # live-updated from the UI region picker
+
+    async def _publish_retrieval(self, query: str, results, fallback_ms: float) -> None:
+        """Send the retrieved chunks to the web UI over a LiveKit data channel."""
+        # Use Moss's own server-reported search time; fall back to wall-clock.
+        server_ms = getattr(results, "time_taken_ms", None)
+        took_ms = float(server_ms) if server_ms is not None else fallback_ms
+        payload = {
+            "query": query,
+            "docs": [
+                {
+                    "id": getattr(d, "id", None),
+                    "text": d.text,
+                    "score": float(getattr(d, "score", 0.0)),
+                }
+                for d in (results.docs if results and results.docs else [])
+            ],
+            "took_ms": round(took_ms, 2),
+            "region": self.region,
+        }
+        try:
+            await self.room.local_participant.publish_data(
+                payload=json.dumps(payload).encode("utf-8"),
+                reliable=True,
+                topic="moss.retrieval",
+            )
+        except Exception as e:
+            logger.warning(f"Failed to publish retrieval data: {e}")
 
     async def on_user_turn_completed(self, turn_ctx: ChatContext, new_message: ChatMessage) -> None:
         """
         Intercept user message -> Search Moss -> Inject Context -> Continue
         """
         user_query = new_message.text_content
+        if not user_query or not user_query.strip():
+            # ignore empty/interim transcription artifacts
+            await super().on_user_turn_completed(turn_ctx, new_message)
+            return
         logger.info(f"User asked: {user_query}")
 
         try:
-            # 1. Automatic Search 
+            # 1. Automatic Search — metadata-filtered to this region + global docs
+            region_filter = {"field": "region", "condition": {"$in": [self.region, "all"]}}
+            t0 = time.perf_counter()
             results = await self.moss.query(
                 INDEX_NAME,
                 user_query,
-                QueryOptions(top_k=5, alpha=0.8)
+                QueryOptions(top_k=5, alpha=0.8, filter=region_filter),
             )
-
-            # 2. Context Injection
+            took_ms = (time.perf_counter() - t0) * 1000.0
+
+            # 2. Stream the retrieval to the web UI (the Moss knowledge-base panel)
+            await self._publish_retrieval(user_query, results, took_ms)
+
+            # 3. Context Injection
             if results.docs:
                 context_str = "\n".join([f"- {d.text}" for d in results.docs])
-                injection = f"Relevant context from knowledge base:\n{context_str}\n\nUse this to answer the user."
-
-                # Insert into chat history as a system message
+                injection = (
+                    f"Relevant information:\n{context_str}\n\n"
+                    "Answer using only this. Do not mention these notes or where they came from."
+                )
                 turn_ctx.add_message(role="system", content=injection)
-                logger.info(f"Injected context: {context_str[:100]}...")  # Log first 100 chars
+                logger.info(f"Injected context ({took_ms:.1f}ms): {context_str[:100]}...")
             else:
+                # No match: keep the agent from inventing an answer.
+                turn_ctx.add_message(
+                    role="system",
+                    content="No relevant information was found. Say you don't have that detail "
+                    "and offer to connect them with a person. Do not make up specifics.",
+                )
                 logger.info("No relevant context found in Moss index")
-                
+
         except Exception as e:
             logger.error(f"Moss search failed: {e}", exc_info=True)
 
@@ -73,32 +138,68 @@ async def on_user_turn_completed(self, turn_ctx: ChatContext, new_message: ChatM
 
 
 async def entrypoint(ctx: JobContext):
+    if not MOSS_PROJECT_ID or not MOSS_PROJECT_KEY:
+        raise SystemExit(
+            "Missing MOSS_PROJECT_ID / MOSS_PROJECT_KEY. Copy .env.example to .env and fill them in."
+        )
     await ctx.connect()
 
     # Initialize Moss
     moss_client = MossClient(project_id=MOSS_PROJECT_ID, project_key=MOSS_PROJECT_KEY)
-
-    # Pre-load index
+
+    # Pre-load the index locally. This is required: region metadata filtering is
+    # only applied to locally loaded indexes (a cloud-fallback query silently
+    # ignores the filter), so a failed load must be fatal rather than a warning.
     try:
         await moss_client.load_index(INDEX_NAME)
         logger.info(f"Successfully loaded index: {INDEX_NAME}")
     except Exception as e:
-        logger.warning(f"Index not found or failed to load: {e}")
-        logger.warning("Moss queries will fail until the index is created. Run upload.py first.")
+        raise SystemExit(
+            f"Failed to load index '{INDEX_NAME}': {e}. Run seed_index.py first "
+            "(region metadata filtering needs the index loaded locally)."
+        )
 
     # Create Session
     session = AgentSession(
-        stt=deepgram.STT(),
-        llm=openai.LLM(model="gpt-4o"),
-        tts=cartesia.TTS(model="sonic-3-2026-01-12"),
-        vad=silero.VAD.load(),
-        turn_handling={"interruption": {"mode": "vad"}},
+        stt=deepgram.STT(model="nova-2", language="en-US"),
+        llm=openai.LLM(model="gpt-4o-mini"),
+        # sonic-turbo = Cartesia's lowest-latency model; "Jacqueline" voice.
+        # Swap the id for any voice from play.cartesia.ai.
+        tts=cartesia.TTS(model="sonic-turbo", voice="9626c31c-bec5-4cca-baa8-f8ba9e84c8bc"),
+        # activation_threshold above the 0.5 default + a short silence window
+        # cuts false triggers so the agent doesn't talk over the caller.
+        vad=silero.VAD.load(min_silence_duration=0.5, activation_threshold=0.6),
+        # A real turn-detection model + endpointing delays make turn-taking
+        # feel crisp instead of guessing on raw VAD.
+        turn_handling={
+            "turn_detection": MultilingualModel(),
+            "endpointing": {"min_delay": 0.5, "max_delay": 1.5},
+        },
     )
 
+    agent = MossSemanticRetrievalAgent(moss_client, ctx.room)
+
+    # The UI region picker publishes { "region": "US" | "EU" } on this topic.
+    @ctx.room.on("data_received")
+    def _on_data(pkt: rtc.DataPacket):
+        if pkt.topic == "moss.region":
+            try:
+                r = json.loads(bytes(pkt.data).decode("utf-8")).get("region")
+                if r in ALLOWED_REGIONS:
+                    agent.region = r
+                    logger.info(f"Region filter set to {r}")
+                else:
+                    logger.warning(f"Ignoring unknown region {r!r} (allowed: {sorted(ALLOWED_REGIONS)})")
+            except Exception as e:
+                logger.warning(f"Bad region packet: {e}")
+
     # Start the session with our custom MossSemanticRetrievalAgent
-    await session.start(
-        agent=MossSemanticRetrievalAgent(moss_client),
-        room=ctx.room,
+    await session.start(agent=agent, room=ctx.room)
+
+    # Speak first, instantly — a fixed opener via say() skips the LLM round-trip.
+    await session.say(
+        "Thanks for calling Northwind support. How can I help you today?",
+        allow_interruptions=True,
     )
 
 if __name__ == "__main__":