|
| 1 | +# Voice Assistant with Moss (TEN Framework) |
| 2 | + |
| 3 | +A real-time voice agent built on the [TEN Framework](https://github.com/ten-framework/ten-framework) that grounds its answers in a [Moss](https://moss.dev) session. On every final ASR transcript the control extension asks Moss for session-scoped context (~1–10ms, in-process) and injects it into the LLM prompt before the model responds — so answers reflect your knowledge base with no perceptible added latency. |
| 4 | + |
| 5 | +The Moss integration lives in the `main_python` control extension and is powered by the [`ten-moss`](../../packages/ten-moss) package (`MossSessionManager`). |
| 6 | + |
| 7 | +## How it works |
| 8 | + |
| 9 | +``` |
| 10 | +mic ─▶ agora_rtc ─▶ streamid_adapter ─▶ stt (deepgram) ─┐ |
| 11 | + │ data: asr_result (final) |
| 12 | + ▼ |
| 13 | + main_control (main_python) |
| 14 | + │ MossSessionManager.query_context(text) |
| 15 | + ▼ ── Moss session query <10ms ──▶ index |
| 16 | + │ ◀── grounding ── |
| 17 | + queue_llm_input("{context}\n\n[Current User Question]\n{text}") |
| 18 | + ▼ |
| 19 | + llm (openai) ─▶ tts (elevenlabs) ─▶ agora_rtc ─▶ speaker |
| 20 | +``` |
| 21 | + |
| 22 | +The Moss delta over the stock TEN voice assistant is small and lives in three places in `main_python`: |
| 23 | + |
| 24 | +- `config.py` — `MainControlConfig` inherits `MossSessionConfig` (the `moss_*` properties). |
| 25 | +- `extension.py` `on_init` — opens the Moss session (`MossSessionManager.from_config(...).open()`), best-effort. |
| 26 | +- `extension.py` `_on_asr_result` — `query_context(text)` and prepends the grounding to the user's turn. |
| 27 | + |
| 28 | +## Provenance |
| 29 | + |
| 30 | +The `tenapp/` baseline (graph, `main_python` control extension, agent runtime, scripts) is vendored from the TEN Framework `voice-assistant` example at commit |
| 31 | +[`c385d27`](https://github.com/ten-framework/ten-framework/tree/c385d2724a1f3e6ac4ee0b81fcc7dada8346c0e0/ai_agents/agents/examples/voice-assistant), |
| 32 | +licensed under **Apache-2.0** (headers preserved). Only the Moss delta described above is Moss-authored. |
| 33 | + |
| 34 | +Two small correctness patches were applied on top of the vendored baseline: |
| 35 | +`agent/decorators.py` fixes the `agent_event_handler` annotation to `type[AgentEvent]`, |
| 36 | +and `extension.py` parses `session_id` defensively so a non-numeric value can't crash the |
| 37 | +ASR handler. |
| 38 | + |
| 39 | +## Prerequisites |
| 40 | + |
| 41 | +- A **TEN Framework checkout**. This example references shared TEN extensions via relative paths (`../../../ten_packages/extension/...`) and runs with TEN's own tooling, so it lives **inside** a TEN Framework repo. It ships the TEN app (`tenapp/`) — not the repo-level run harness (playground / server / Taskfile / Dockerfile), which the TEN Framework provides. |
| 42 | +- A **Moss** project (`MOSS_PROJECT_ID` / `MOSS_PROJECT_KEY`) — [moss.dev](https://moss.dev). |
| 43 | +- Provider keys: **Agora** (transport), **Deepgram** (STT), **OpenAI** (LLM), **ElevenLabs** (TTS). |
| 44 | + |
| 45 | +## Run |
| 46 | + |
| 47 | +1. **Build the demo knowledge index** (from this directory — needs only the Moss SDK): |
| 48 | + ```bash |
| 49 | + cp .env.example .env # fill in MOSS_PROJECT_ID / MOSS_PROJECT_KEY / MOSS_INDEX_NAME |
| 50 | + python create_index.py # reads data/knowledge.jsonl, creates MOSS_INDEX_NAME |
| 51 | + ``` |
| 52 | + |
| 53 | +2. **Drop the app into a TEN checkout.** Copy `tenapp/` to |
| 54 | + `ten-framework/ai_agents/agents/examples/voice-assistant-with-moss/tenapp/`, alongside |
| 55 | + the sibling `voice-assistant` example whose `Taskfile`/`playground`/`server` harness you |
| 56 | + reuse. `main_python` depends on [`ten-moss`](https://pypi.org/project/ten-moss/) (listed |
| 57 | + in `main_python/requirements.txt`), so `task install` installs it from PyPI automatically — |
| 58 | + no manual step needed. |
| 59 | + |
| 60 | +3. **Run with TEN's tooling** from that example dir (`task install && task run`, per the TEN |
| 61 | + docs), providing the same env vars as step 1. Then open the TEN playground |
| 62 | + (http://localhost:3000) and ask something covered by `data/knowledge.jsonl` — e.g. |
| 63 | + *"how long do refunds take?"* — to hear grounded answers. |
| 64 | + |
| 65 | +## See the difference Moss makes (no voice stack needed) |
| 66 | + |
| 67 | +`compare.py` answers the same questions with the same LLM twice — once **without** Moss and |
| 68 | +once **with** the Moss grounding the agent injects — so you can see the improvement without |
| 69 | +standing up the full voice pipeline. It needs only your Moss + OpenAI keys (no Agora/STT/TTS): |
| 70 | + |
| 71 | +```bash |
| 72 | +cp .env.example .env # MOSS_* + OPENAI_API_KEY (+ optional OPENAI_MODEL) |
| 73 | +python create_index.py # build the index once |
| 74 | +pip install ten-moss openai python-dotenv |
| 75 | +python compare.py # or: python compare.py "your own question?" |
| 76 | +``` |
| 77 | + |
| 78 | +Sample run (`gpt-4o-mini` over `data/knowledge.jsonl`): |
| 79 | + |
| 80 | +| Question | Without Moss | With Moss | |
| 81 | +| --- | --- | --- | |
| 82 | +| How long do refunds take? | "5–10 business days" ❌ | "3–5 business days once approved" ✅ | |
| 83 | +| Can I cancel my order? | "within a specific timeframe… check our policy" | "within 1 hour of placement" ✅ | |
| 84 | +| Which payment methods? | misses American Express | "Visa, Mastercard, Amex, PayPal, Apple Pay" ✅ | |
| 85 | +| Do you offer price matching? | "provide competitor details" | "authorized retailers within 14 days" ✅ | |
| 86 | +| How fast is express shipping? | "1–3 business days" ❌ | "1–2 business days" ✅ | |
| 87 | + |
| 88 | +Without grounding the model confidently invents plausible-but-wrong specifics; with Moss it |
| 89 | +answers from your knowledge base. This is the exact delta the live voice agent applies per |
| 90 | +turn — flip `enable_moss` in `property.json` to A/B the same thing in the playground. |
| 91 | + |
| 92 | +## Showcase the speed (live, in the agent) |
| 93 | + |
| 94 | +Every turn, the control extension logs the retrieval cost using the SDK's own |
| 95 | +`SearchResult.time_taken_ms` (surfaced by `ten-moss` as `last_time_taken_ms`), with the |
| 96 | +wall-clock alongside for reference: |
| 97 | + |
| 98 | +``` |
| 99 | +[retrieval-latency] backend=moss(in-process) time_taken_ms=2 (wall_clock=64ms) |
| 100 | +``` |
| 101 | + |
| 102 | +And in the playground transcript you see, per turn, **what Moss retrieved + the SDK |
| 103 | +`time_taken_ms`**, followed by the **LLM's answer**: |
| 104 | + |
| 105 | +``` |
| 106 | +🔎 Moss · retrieved in 2 ms (SDK time_taken_ms) |
| 107 | + Relevant knowledge from Moss: [1] Refunds are processed within 3-5 business days… |
| 108 | +<the assistant's spoken answer> |
| 109 | +``` |
| 110 | + |
| 111 | +It also emits a **per-turn latency breakdown** — a grep-able log line *and* a note in the |
| 112 | +transcript — so you can see where the turn's time goes across the pipeline: |
| 113 | + |
| 114 | +``` |
| 115 | +[latency-breakdown] turn=3 moss_retrieval_ms=2 llm_ttft_ms=480 llm_total_ms=1150 turn_total_ms=1160 |
| 116 | +``` |
| 117 | + |
| 118 | +- **moss_retrieval_ms** — the SDK's `SearchResult.time_taken_ms` (in-process retrieval engine time). |
| 119 | +- **llm_ttft_ms** — time to the LLM's first token after dispatch. |
| 120 | +- **llm_total_ms** — full LLM generation for the turn. |
| 121 | +- **turn_total_ms** — ASR-final → LLM-final (the whole control-side turn). |
| 122 | + |
| 123 | +ASR timing appears in the Deepgram STT extension logs and TTS audio-out in the ElevenLabs |
| 124 | +TTS logs (both per turn in the worker log) — so between those and the line above you get the |
| 125 | +full component-by-component breakdown. |
| 126 | + |
| 127 | +**TEN default retrieval vs Moss (real numbers).** TEN's shipped memory/RAG backends |
| 128 | +(memU, OceanBase PowerRAG, EverMemOS) are all remote services — every turn is a network |
| 129 | +round trip. To compare TEN's default (memU) against Moss with real logged latency from |
| 130 | +both agents, follow **[`BENCHMARK.md`](BENCHMARK.md)**: it adds the same one‑line |
| 131 | +latency log to TEN's shipped `voice-assistant-with-memU` example, then you run both agents |
| 132 | +and read `[retrieval-latency]` from each (memU: hundreds of ms; Moss: single‑digit ms). |
| 133 | + |
| 134 | +**Quick single‑agent approximation (no memU key).** If you just want to hear the effect in |
| 135 | +this one agent, set `moss_simulate_remote_ms` on the `main_control` node in |
| 136 | +`tenapp/property.json` to a remote‑like latency and re‑run `task run`: |
| 137 | + |
| 138 | +- `0` → Moss in‑process (~2 ms) — the agent replies immediately. |
| 139 | +- `400` → the same agent, same answer, but audibly **pauses ~400 ms before every reply**. |
| 140 | + |
| 141 | +## Configuration |
| 142 | + |
| 143 | +Moss is configured on the `main_control` node in `tenapp/property.json` (env-substituted): |
| 144 | +`moss_project_id`, `moss_project_key`, `moss_index_name`, `moss_model_id`, |
| 145 | +`moss_top_k`, `moss_alpha`, `moss_context_header`, `enable_moss`, |
| 146 | +`moss_simulate_remote_ms`. Set `enable_moss` to `false` to run the plain voice |
| 147 | +assistant with no grounding; set `moss_simulate_remote_ms` to imitate a slow remote store. |
| 148 | + |
| 149 | +## Testing status |
| 150 | + |
| 151 | +The `ten-moss` package is covered by offline unit tests (`packages/ten-moss/tests/`). |
| 152 | +This end-to-end app is **not** run in CI — it requires the TEN toolchain plus paid |
| 153 | +Agora/Deepgram/OpenAI/ElevenLabs credentials, so it is validated manually via the |
| 154 | +steps above. |
0 commit comments