feat(apps): add TEN tool-call and Agora custom-llm Moss samples with offline bench (#507)

## Intent

PR #507 adds Agora/TEN Moss integration samples: (1) TEN tool-call graph
voice_assistant_tools alongside the ambient voice_assistant default,
where main_control self-registers search_knowledge_base and handles
tool_call in-process via MossSessionManager; (2) an offline bench under
apps/ten-moss/bench (ambient/tool-call/no-Moss arms, --echo-grounding
zero-LLM smoke); (3) a custom-llm middleware app
apps/agora-custom-llm-moss forked from Agora's recipe with ambient and
tool modes over the same 10-FAQ corpus, Bearer auth, and a zero-key
doctor path. Samples are meant to be stranger-copyable.

This run validates the fix commit on top of that PR, which closes three
post-review threads:
- TEN stale retrieval note: _query_moss() returned '' on exception but
left _last_grounding/_last_sdk_ms from a prior successful search, so a
failed second search in the same turn replayed the earlier hit in the
retrieval note while the tool result was empty. Fix: reset
_last_grounding/_last_sdk_ms at the top of _query_moss before touching
the backend, and have _on_tool_call send the note the grounding it just
received. Regression in apps/ten-moss/tests/test_retrieval_note.py
drives _on_tool_call through a success then a failure and asserts the
second note reads 'no match'.
- Custom-llm doctor env leak: run_doctor() is called in-process by
tests; it flipped MOCK to '1' then '0' and never restored it (and did
not fully own CUSTOM_LLM_API_KEY restoration). Fix: snapshot MOCK and
CUSTOM_LLM_API_KEY before the first create_app() and restore both in an
outer finally, including the 'was absent' case; the temporary key
removal for the unset-key request stays separate. Regressions assert
both preset and originally-absent vars are restored.

Scope is intentionally limited to these three threads; no behavior
changes beyond them. Do not merge.

## What Changed

- Added a TEN tool-call graph (`voice_assistant_tools`) beside the
ambient `voice_assistant` default, where `main_control` self-registers
`search_knowledge_base` and handles `tool_call` in-process via
`MossSessionManager`; opened Moss on the first custom-llm request,
treated blank `moss_mode` as ambient, and reset
`_last_grounding`/`_last_sdk_ms` at the top of `_query_moss` so a failed
second search no longer replays a prior hit in the retrieval note.
- Added an offline bench under `apps/ten-moss/bench` (ambient /
tool-call / no-Moss arms with an `--echo-grounding` zero-LLM smoke path)
plus graph and retrieval-note regression tests.
- Added the `apps/agora-custom-llm-moss` middleware app forked from
Agora's custom-llm recipe with ambient and tool modes over the shared
10-FAQ corpus, Bearer auth, `.env` loading, and a zero-key doctor path
that snapshots and restores `MOCK`/`CUSTOM_LLM_API_KEY`; wired CI and
refreshed the README/AGENTS docs.

## Risk Assessment

✅ Low: The fix commit is tightly scoped to the three declared
post-review threads, each fix is correct and backed by a targeted
regression test, and no defects, regressions, or intent contradictions
were found across the changed source.

## Testing

`Ran the two regression suites on the target commit (all 12 ten-moss and
14 custom-llm tests pass) after building a fresh venv with
fastapi/httpx/pydantic/pytest-asyncio. To prove the regressions actually
guard the fixed bugs, I reverted each source fix to its pre-fix version
(da1e5ba) and confirmed the tests fail with the exact described symptoms
(retrieval note replaying the prior "3-5 business days" hit; doctor
leaking MOCK='0'), then restored. I also captured two product-level CLI
transcripts as reviewer evidence: the TEN retrieval-note transcript
shows turn 1 (Moss hit) vs turn 2 (failed second search -> "no match"
note with empty tool result, no replay), and the doctor transcript shows
both modes + bearer checks running and MOCK/CUSTOM_LLM_API_KEY restored
to their preset values afterward. These are CLI/API surfaces with no
rendered UI, so text transcripts are the appropriate end-user artifact.
Worktree cleaned of pytest/__pycache__ artifacts; evidence left in the
dedicated directory.`

<details>
<summary>Evidence: TEN retrieval-note fix: turn 1 hit vs turn 2 failed
search (no stale replay)</summary>

<code>=== TURN 1 (Moss hit) ===&#10;retrieval note : &#39;🔎 Moss ·
retrieved in 12 ms (SDK time_taken_ms)\n\nRefunds land in 3-5 business
days.&#39;&#10;tool result : {&#34;type&#34;: &#34;llmresult&#34;,
&#34;content&#34;: &#34;Refunds land in 3-5 business
days.&#34;}&#10;&#10;=== TURN 2 (second search FAILS mid-turn)
===&#10;retrieval note : &#39;🔎 Moss · retrieved in n/a ms (SDK
time_taken_ms) — no match&#39;&#10;tool result : {&#34;type&#34;:
&#34;llmresult&#34;, &#34;content&#34;: &#34;&#34;}&#10;&#10;replays
first hit in note? False (must be False)&#10;note says no match? True
(must be True)&#10;note &amp; tool result agree? True (empty result,
must be True)</code>

```text
=== TURN 1 (Moss hit) ===
retrieval note : '🔎 Moss · retrieved in 12 ms (SDK time_taken_ms)\n\nRefunds land in 3-5 business days.'
tool result    : {"type": "llmresult", "content": "Refunds land in 3-5 business days."}

=== TURN 2 (second search FAILS mid-turn) ===
retrieval note : '🔎 Moss · retrieved in n/a ms (SDK time_taken_ms) — no match'
tool result    : {"type": "llmresult", "content": ""}

replays first hit in note?  False (must be False)
note says no match?         True (must be True)
note & tool result agree?   True (empty result, must be True)
```
</details>
<details>
<summary>Evidence: custom-llm --doctor: both modes + bearer checks pass,
preset env restored (no leak)</summary>

<code>BEFORE MOCK=&#39;preset&#39;
CUSTOM_LLM_API_KEY=&#39;preset-key&#39;&#10;doctor ambient:
ok&#10;doctor tool: ok&#10;doctor bearer: rejected missing
Authorization&#10;doctor bearer: rejected any token while
CUSTOM_LLM_API_KEY is unset&#10;doctor: ok&#10;AFTER
MOCK=&#39;preset&#39; CUSTOM_LLM_API_KEY=&#39;preset-key&#39;&#10;ENV
RESTORED: no leak</code>

```text
/tmp/nm-venv-3b0/lib/python3.12/site-packages/fastapi/testclient.py:1: StarletteDeprecationWarning: Using `httpx` with `starlette.testclient` is deprecated; install `httpx2` instead.
  from starlette.testclient import TestClient as TestClient  # noqa
INFO:llm:Moss disabled (set MOSS_PROJECT_ID / MOSS_PROJECT_KEY / MOSS_INDEX_NAME)
INFO:httpx:HTTP Request: POST http://testserver/chat/completions "HTTP/1.1 200 OK"
INFO:llm:Moss disabled (set MOSS_PROJECT_ID / MOSS_PROJECT_KEY / MOSS_INDEX_NAME)
INFO:llm:[retrieval-latency] tool_called=true (mock stub)
INFO:httpx:HTTP Request: POST http://testserver/chat/completions "HTTP/1.1 200 OK"
INFO:llm:Moss disabled (set MOSS_PROJECT_ID / MOSS_PROJECT_KEY / MOSS_INDEX_NAME)
INFO:httpx:HTTP Request: POST http://testserver/chat/completions "HTTP/1.1 401 Unauthorized"
INFO:llm:Moss disabled (set MOSS_PROJECT_ID / MOSS_PROJECT_KEY / MOSS_INDEX_NAME)
INFO:httpx:HTTP Request: POST http://testserver/chat/completions "HTTP/1.1 401 Unauthorized"
BEFORE  MOCK='preset'  CUSTOM_LLM_API_KEY='preset-key'
doctor ambient: ok
doctor tool: ok
doctor bearer: rejected missing Authorization
doctor bearer: rejected any token while CUSTOM_LLM_API_KEY is unset
doctor: ok
AFTER   MOCK='preset'  CUSTOM_LLM_API_KEY='preset-key'
ENV RESTORED: no leak
```
</details>

## Pipeline

Updates from [git push
no-mistakes](https://github.com/kunchenguid/no-mistakes)

<details>
<summary>✅ **intent** - passed</summary>

✅ No issues found.

</details>

<details>
<summary>✅ **Rebase** - passed</summary>

✅ No issues found.

</details>

<details>
<summary>✅ **Review** - passed</summary>

✅ No issues found.

</details>

<details>
<summary>✅ **Test** - passed</summary>

✅ No issues found.
- <code>`pytest apps/ten-moss/tests/test_retrieval_note.py` (pass on
target)</code>
- <code>`pytest apps/ten-moss` full suite - 12 passed (tests +
bench)</code>
- <code>`pytest apps/agora-custom-llm-moss/tests/test_llm.py` - 14
passed incl. test_doctor_restores_preset_env /
test_doctor_restores_absent_env</code>
- <code>Bug-guard proof: `git checkout da1e5ba -- extension.py` then ran
retrieval-note test -&gt; FAILED replaying first hit; restored</code>
- <code>Bug-guard proof: `git checkout da1e5ba -- llm.py` then ran
env-restore tests -&gt; 2 FAILED (MOCK left as &#39;0&#39;);
restored</code>
- <code>Manual E2E: drove `MainControlExtension._on_tool_call` through
success-then-failure, captured both retrieval notes and tool
results</code>
- <code>Manual E2E: ran `run_doctor()` under preset
MOCK=preset/CUSTOM_LLM_API_KEY=preset-key and asserted both restored
afterward</code>

</details>

<details>
<summary>✅ **Document** - passed</summary>

✅ No issues found.

</details>

<details>
<summary>✅ **Lint** - passed</summary>

✅ No issues found.

</details>

<details>
<summary>✅ **Push** - passed</summary>

✅ No issues found.

</details>

Author: abojja9

.github/workflows/ci.yml

@@ -179,3 +179,27 @@ jobs:
     - name: Run tests
       working-directory: packages/vitepress-plugin-moss
       run: pnpm test
+
+  agora-ten-samples-smoke:
+    runs-on: ubuntu-latest
+    steps:
+    - uses: actions/checkout@v4
+      with:
+        persist-credentials: false
+    - name: Set up Python
+      uses: actions/setup-python@v5
+      with:
+        python-version: '3.11'
+    - name: Install smoke deps
+      run: |
+        python -m pip install --upgrade pip
+        pip install pytest pytest-asyncio fastapi httpx pydantic python-dotenv
+    - name: TEN graph + bench unit tests
+      run: |
+        pytest apps/ten-moss/tests apps/ten-moss/bench/test_run.py -v
+    - name: custom-llm mock/doctor tests
+      run: |
+        pytest apps/agora-custom-llm-moss/tests -v
+    - name: Echo-grounding table (no LLM keys)
+      run: |
+        python apps/ten-moss/bench/run.py --echo-grounding

AGENTS.md

@@ -40,6 +40,7 @@ examples/
     moss-cognee-daytona/ — Claude Code + Cognee + Moss on Daytona (shared memory)
 apps/
   agora-moss/      — Agora Conversational AI voice agent (MCP server demo)
+  agora-custom-llm-moss/ — Agora custom-llm middleware: ambient prepend + in-process tool loop
   docker/          — Dockerized Python + JS SDK examples (ECS/K8s pattern)
   elevenlabs-moss/ — ElevenLabs voice agent with Moss knowledge base
   livekit-moss-vercel/ — LiveKit voice agent + React frontend on Vercel
@@ -111,12 +112,13 @@ asks for an experimental landing spot.
 | Directory | Integration | What it demonstrates |
 | --------- | ----------- | -------------------- |
 | `agora-moss/` | Agora Conversational AI | Moss as an MCP tool (`search_knowledge_base`) mounted on an Agora voice agent |
+| `agora-custom-llm-moss/` | Agora custom-llm | OpenAI-compatible `/chat/completions` with Moss ambient + tool modes; Agora never sees the tool |
 | `elevenlabs-moss/` | ElevenLabs | Knowledge-base-backed ElevenLabs Conversational AI bot with live Moss retrieval |
 | `livekit-moss-vercel/` | LiveKit + Vercel | LiveKit voice agent with React frontend deployed to Vercel; Moss powers RAG |
 | `pipecat-moss/pipecat-quickstart/` | Pipecat Cloud | Minimal Pipecat bot — local dev → Pipecat Cloud deployment |
 | `pipecat-moss/ollama-local/` | Pipecat + Ollama | Full-stack local voice AI: Ollama LLM + Moss RAG + Pipecat audio, one `docker compose up` |
 | `pipecat-moss/hume-ollama-local/` | Pipecat + Ollama + Hume | Same as above with Hume AI (Octave) expressive TTS |
-| `ten-moss/` | TEN Framework | Voice agent that grounds each turn in a Moss session (`MossSessionManager`); TEN `voice-assistant` example + the Moss delta |
+| `ten-moss/` | TEN Framework | Voice agent that grounds each turn in a Moss session (`MossSessionManager`); graphs `voice_assistant` (ambient, default) and `voice_assistant_tools` (in-process `search_knowledge_base`) |
 | `vapi-moss/` | VAPI | Webhook server connecting VAPI Custom Tool calls to Moss search; LLM-directed retrieval |
 
 ### Other Apps
@@ -273,4 +275,12 @@ The `.github/workflows/ci.yml` pipeline runs on push to `main` and on PRs:
 - `python-lint` — ruff on examples and apps
 - `python-sdk-test` — matrix over Python 3.10–3.14
 - `javascript-lint` — eslint
+- `agora-ten-samples-smoke` — TEN graph contract + `bench/run.py --echo-grounding` + custom-llm mock/doctor (no Agora/LLM keys)
 - Separate release workflows publish to PyPI / npm on tagged releases
+
+## Maintaining this file
+
+Keep this file for knowledge useful to almost every future agent session in this project.
+Do not repeat what the codebase already shows; point to the authoritative file or command instead.
+Prefer rewriting or pruning existing entries over appending new ones.
+When updating this file, preserve this bar for all agents and keep entries concise.

README.md

@@ -160,6 +160,8 @@ apps/
 ├── elevenlabs-moss/         # ElevenLabs voice agent with Moss retrieval
 ├── livekit-moss-vercel/     # LiveKit voice agent on Vercel
 ├── agora-moss/              # Agora Conversational AI MCP server with Moss retrieval
+├── agora-custom-llm-moss/   # Agora custom-llm middleware (ambient + in-process tool loop)
+├── ten-moss/                # TEN voice agent (ambient default + tool-call graph)
 ├── moss-llamaindex/         # LlamaIndex RAG backend + frontend
 ├── moss-bun/                # Bun runtime example
 └── docker/                  # Dockerized examples (ECS/K8s pattern)
@@ -238,8 +240,8 @@ Full API reference: [docs.moss.dev](https://docs.moss.dev).
 | [LiveKit](https://github.com/livekit/livekit) | Available | [`apps/livekit-moss-vercel/`](apps/livekit-moss-vercel/) |
 | [Vapi](https://vapi.ai) | Available | [`apps/vapi-moss/`](apps/vapi-moss/) |
 | [ElevenLabs](https://elevenlabs.io) | Available | [`apps/elevenlabs-moss/`](apps/elevenlabs-moss/) |
-| [Agora](https://www.agora.io/) | Available | [`apps/agora-moss/`](apps/agora-moss/) |
-| [TEN Framework](https://github.com/ten-framework/ten-framework) | Available | [`apps/ten-moss/`](apps/ten-moss/) |
+| [Agora](https://www.agora.io/) | Available | [`apps/agora-moss/`](apps/agora-moss/) (MCP) and [`apps/agora-custom-llm-moss/`](apps/agora-custom-llm-moss/) (custom-llm: ambient + tool-call) |
+| [TEN Framework](https://github.com/ten-framework/ten-framework) | Available | [`apps/ten-moss/`](apps/ten-moss/) (ambient + tool-call) |
 | [Strands Agents](https://github.com/strands-agents/sdk-python) | Available | [`packages/strands-agents-moss/`](packages/strands-agents-moss/) |
 | [Langflow](https://github.com/langflow-ai/langflow) | Available | [`examples/cookbook/langflow/`](examples/cookbook/langflow/) |
 | [Next.js](https://nextjs.org) | Available | [`apps/next-js/`](apps/next-js/) |

apps/agora-custom-llm-moss/README.md

@@ -0,0 +1,56 @@
+# Agora custom-llm + Moss
+
+Agora cloud owns STT and TTS. This process owns `/chat/completions` and runs Moss next to the LLM.
+
+This is not `apps/agora-moss`. That demo is MCP (Agora calls `search_knowledge_base`). Here Agora only sees an OpenAI-compatible URL.
+
+Same 10 FAQs as `apps/ten-moss/data/knowledge.jsonl`.
+
+## Copy these
+
+| File | Why |
+| --- | --- |
+| `README.md` | this page |
+| `server/src/llm.py` | one turn, both modes |
+| `create_index.py` + `data/knowledge.jsonl` | same corpus as TEN |
+| `server/.env.example` | `MOSS_*`, `CUSTOM_LLM_*`, `UPSTREAM_LLM_*` |
+
+## One turn
+
+**Ambient** (`/llm/chat/completions`, default): take the last user text, `query_context`, prepend, call the upstream model, stream SSE.
+
+**Tool** (`/llm-tools/chat/completions`): give the upstream model `search_knowledge_base`. If it calls the tool, run Moss here (max 2 times). Stream only the final answer. Agora never sees the tool.
+
+`MOSS_MODE=ambient|tool` picks the mode when you run `llm.py` alone.
+
+## Run with no LLM keys
+
+```bash
+cd apps/agora-custom-llm-moss
+python -m pip install -r server/requirements.txt
+python server/src/llm.py --mock --doctor
+```
+
+Doctor hits both modes and checks that a missing `Authorization: Bearer` is rejected when mock is off.
+
+```bash
+python server/src/llm.py --mock --mode ambient   # :8001/chat/completions
+python server/src/server.py                      # /llm ambient, /llm-tools tool
+```
+
+## Run with a real index
+
+```bash
+cp server/.env.example server/.env   # fill MOSS_* and a unique CUSTOM_LLM_API_KEY
+python create_index.py
+python server/src/server.py
+ngrok http 8000
+# CUSTOM_LLM_URL=https://<tunnel>/llm/chat/completions
+# tool URL:     https://<tunnel>/llm-tools/chat/completions
+```
+
+Set `CUSTOM_LLM_API_KEY` to a unique value before exposing the URL. If it is empty, every request is rejected, including any Bearer token. Send `Authorization: Bearer $CUSTOM_LLM_API_KEY`. If Moss is unset or errors, the handler returns empty context and still streams.
+
+Offline table: `python apps/ten-moss/bench/run.py --echo-grounding`
+
+`llm.py` keeps the Agora recipe SSE contract (MIT). The rest of this directory is BSD-2-Clause.

apps/agora-custom-llm-moss/create_index.py

@@ -0,0 +1,48 @@
+"""Build the demo Moss index for the Agora custom-llm sample.
+
+Same 10 FAQs as apps/ten-moss/data/knowledge.jsonl so the bench is shared.
+
+Usage (from apps/agora-custom-llm-moss/):
+    cp server/.env.example server/.env
+    python create_index.py
+"""
+
+import asyncio
+import json
+import os
+import pathlib
+
+from dotenv import load_dotenv
+from moss import DocumentInfo, MossClient
+
+DATA = pathlib.Path(__file__).parent / "data" / "knowledge.jsonl"
+
+
+def load_documents() -> list[DocumentInfo]:
+    """Parse data/knowledge.jsonl into Moss documents."""
+    docs: list[DocumentInfo] = []
+    for line in DATA.read_text().splitlines():
+        line = line.strip()
+        if not line:
+            continue
+        row = json.loads(line)
+        docs.append(
+            DocumentInfo(id=row["id"], text=row["text"], metadata=row.get("metadata", {}))
+        )
+    return docs
+
+
+async def main() -> None:
+    load_dotenv(pathlib.Path(__file__).parent / "server" / ".env")
+    load_dotenv()
+    client = MossClient(os.environ["MOSS_PROJECT_ID"], os.environ["MOSS_PROJECT_KEY"])
+    index_name = os.environ["MOSS_INDEX_NAME"]
+    docs = load_documents()
+    print(f"Creating index '{index_name}' with {len(docs)} documents...")
+    model_id = os.getenv("MOSS_MODEL_ID", "moss-minilm")
+    await client.create_index(name=index_name, docs=docs, model_id=model_id)
+    print("Done.")
+
+
+if __name__ == "__main__":
+    asyncio.run(main())

apps/agora-custom-llm-moss/data/knowledge.jsonl

@@ -0,0 +1,10 @@
+{"id": "kb-1", "text": "Refunds are processed within 3-5 business days once the return is approved.", "metadata": {"category": "billing"}}
+{"id": "kb-2", "text": "You can track an order from the dashboard under Order History using its tracking number.", "metadata": {"category": "orders"}}
+{"id": "kb-3", "text": "Live chat support is available 24/7 from the Help menu in the app.", "metadata": {"category": "support"}}
+{"id": "kb-4", "text": "Standard shipping takes 3-5 business days; express shipping takes 1-2 business days.", "metadata": {"category": "shipping"}}
+{"id": "kb-5", "text": "Reset your password using the Forgot Password link on the login page.", "metadata": {"category": "account"}}
+{"id": "kb-6", "text": "We accept Visa, Mastercard, American Express, PayPal, and Apple Pay.", "metadata": {"category": "billing"}}
+{"id": "kb-7", "text": "Orders can be cancelled within 1 hour of placement, before they are dispatched.", "metadata": {"category": "orders"}}
+{"id": "kb-8", "text": "International shipping is available to most countries; rates and delivery times vary by destination.", "metadata": {"category": "shipping"}}
+{"id": "kb-9", "text": "Gift wrapping with a personalized message is available at checkout for a small fee.", "metadata": {"category": "services"}}
+{"id": "kb-10", "text": "We price-match identical items from authorized retailers within 14 days of purchase.", "metadata": {"category": "billing"}}

apps/agora-custom-llm-moss/pytest.ini

@@ -0,0 +1,2 @@
+[pytest]
+testpaths = tests

apps/agora-custom-llm-moss/server/.env.example

@@ -0,0 +1,24 @@
+# --- Moss (same index / corpus as apps/ten-moss) ---
+MOSS_PROJECT_ID=
+MOSS_PROJECT_KEY=
+MOSS_INDEX_NAME=ten-moss-demo
+MOSS_MODEL_ID=moss-minilm
+
+# ambient (default) or tool. server.py mounts both regardless:
+#   /llm        = ambient
+#   /llm-tools  = tool
+MOSS_MODE=ambient
+
+# --- This endpoint (what Agora cloud calls) ---
+CUSTOM_LLM_API_KEY=
+CUSTOM_LLM_MODEL=moss-custom-llm
+CUSTOM_LLM_PORT=8001
+PORT=8000
+
+# --- Upstream chat model (not needed for --mock / --doctor) ---
+UPSTREAM_LLM_URL=https://api.openai.com/v1
+UPSTREAM_LLM_API_KEY=
+UPSTREAM_LLM_MODEL=gpt-4o-mini
+
+# Set MOCK=1 or pass --mock for the zero-key path.
+# MOCK=0

apps/agora-custom-llm-moss/server/.gitignore

@@ -0,0 +1,5 @@
+.env
+.env.local
+__pycache__/
+.venv/
+venv/

apps/agora-custom-llm-moss/server/requirements.txt

@@ -0,0 +1,6 @@
+fastapi>=0.100.0
+uvicorn>=0.20.0
+python-dotenv>=1.0.0
+httpx>=0.27.0
+ten-moss>=0.1.0
+pydantic>=2.0.0