docs(examples): add semantic-cache example (cache LLM responses by meaning)

A tiny semantic cache built on Moss: embed each question, look up the nearest
one already answered, and return the stored answer if it clears a similarity
threshold — skipping the model call entirely. Two phrasings of the same
question ('what are your hours?' / 'when do you open?') hit the same cache
entry, where an exact-text cache would miss and pay the model twice.

Includes semantic_cache.py (the SemanticCache class + a runnable demo), README,
and config. Retrieval is on-device and sub-10ms, far cheaper than the call it
avoids.

Author: HarshaNalluru

moss-live-labs/examples/semantic-cache/.env.example

@@ -0,0 +1,6 @@
+# Moss
+MOSS_PROJECT_ID=Your moss project id here
+MOSS_PROJECT_KEY=Your moss project key here
+
+# OpenAI — the model whose responses we cache
+OPENAI_API_KEY=Your openai api key here

moss-live-labs/examples/semantic-cache/.python-version

@@ -0,0 +1 @@
+3.12

moss-live-labs/examples/semantic-cache/README.md

@@ -0,0 +1,62 @@
+# Semantic cache for LLM responses
+
+Cache LLM answers by **meaning**, not by exact text.
+
+A normal cache keys on the literal request string, so two phrasings of the same
+question miss:
+
+```
+"what are your hours?"   ->  MISS  ->  calls the model
+"when do you open?"      ->  MISS  ->  calls the model again   (paid twice)
+```
+
+A semantic cache embeds the question and looks up the nearest one it has already
+answered. If it's close enough, it returns the stored answer with no model call.
+Repeat questions come back in single-digit-millisecond retrieval, on-device.
+
+```
+"what are your hours?"   ->  MISS  ->  calls the model, stores the answer
+"when do you open?"      ->  HIT   ->  returns the stored answer, no model call
+```
+
+## How it works
+
+The whole thing is `SemanticCache` in [`semantic_cache.py`](./semantic_cache.py):
+
+```python
+async def ask(self, question):
+    hit = await self.index.query(question, QueryOptions(top_k=1))
+    if hit.docs and hit.docs[0].score >= THRESHOLD:   # close enough in meaning?
+        return hit.docs[0].metadata["answer"]           # cache hit — no LLM call
+    answer = await call_the_model(question)             # miss — ask once
+    await self.index.add_docs(
+        [DocumentInfo(id=question, text=question, metadata={"answer": answer})])
+    return answer
+```
+
+Moss keys the cache on the question's embedding and serves the nearest match
+in <10 ms locally, so the lookup is far cheaper than the model call it avoids.
+The one knob that matters is `THRESHOLD` (cosine similarity): too low and you
+answer questions people didn't quite ask; too high and you miss obvious matches.
+
+## What you need
+
+- A [Moss](https://moss.dev) account (`MOSS_PROJECT_ID` / `MOSS_PROJECT_KEY`)
+- An OpenAI key (the example uses `gpt-4o-mini` as the model being cached)
+- Python 3.10+
+
+## Run
+
+```bash
+uv sync                       # or: pip install moss openai python-dotenv
+cp .env.example .env          # fill in your keys
+python semantic_cache.py
+```
+
+Expected: the first question is a `MISS` (calls the model), the paraphrased
+second question is a `HIT` (returns instantly, no model call).
+
+## Resources
+
+- [Docs](https://docs.moss.dev)
+- [GitHub](https://github.com/usemoss/moss)

moss-live-labs/examples/semantic-cache/pyproject.toml

@@ -0,0 +1,10 @@
+[project]
+name = "moss-semantic-cache"
+version = "0.1.0"
+description = "Semantic cache for LLM responses, built on Moss"
+requires-python = ">=3.10"
+dependencies = [
+    "moss",
+    "openai>=1.0",
+    "python-dotenv>=1.0",
+]

moss-live-labs/examples/semantic-cache/semantic_cache.py

@@ -0,0 +1,81 @@
+"""Semantic cache for LLM responses, built on Moss.
+
+A normal cache keys on the exact text of a request, so two ways of asking the
+same thing ("what are your hours?" / "when do you open?") miss and you pay the
+model twice. A *semantic* cache keys on meaning: embed the question, look up the
+nearest one you've already answered, and if it's close enough, return the stored
+answer without calling the model.
+
+The whole thing is the SemanticCache class below. The one knob that matters is
+the similarity threshold.
+"""
+
+import asyncio
+import os
+import time
+
+from dotenv import load_dotenv
+from openai import AsyncOpenAI
+
+from moss import MossClient, DocumentInfo, QueryOptions
+
+load_dotenv()
+
+# cosine similarity above which a cached answer is "close enough" to reuse.
+# too low -> you answer questions people didn't quite ask; too high -> you miss.
+THRESHOLD = 0.92
+
+moss = MossClient(
+    project_id=os.getenv("MOSS_PROJECT_ID"),
+    project_key=os.getenv("MOSS_PROJECT_KEY"),
+)
+llm = AsyncOpenAI()
+
+
+class SemanticCache:
+    """A tiny vector index of past questions -> answers, queried by meaning."""
+
+    def __init__(self, index):
+        self.index = index
+
+    async def ask(self, question: str) -> tuple[str, bool]:
+        # 1. look for the closest question we've already answered
+        hit = await self.index.query(question, QueryOptions(top_k=1))
+        if hit.docs and hit.docs[0].score >= THRESHOLD:
+            return hit.docs[0].metadata["answer"], True  # cache hit — no LLM call
+
+        # 2. miss: ask the model once
+        resp = await llm.chat.completions.create(
+            model="gpt-4o-mini",
+            messages=[{"role": "user", "content": question}],
+        )
+        answer = resp.choices[0].message.content
+
+        # 3. remember it so any wording of it is instant next time
+        await self.index.add_docs(
+            [DocumentInfo(id=question, text=question, metadata={"answer": answer})]
+        )
+        return answer, False
+
+
+async def main():
+    # a fresh in-memory session index acts as the cache for this run
+    index = await moss.session("qa-cache")
+    cache = SemanticCache(index)
+
+    # the 2nd question means the same as the 1st, phrased differently -> cache hit
+    questions = [
+        "What are your opening hours?",
+        "when do you open?",
+        "How do I reset my password?",
+    ]
+    for q in questions:
+        t = time.perf_counter()
+        answer, hit = await cache.ask(q)
+        ms = (time.perf_counter() - t) * 1000
+        tag = "HIT " if hit else "MISS"
+        print(f"[{tag} {ms:7.1f} ms]  {q}\n   -> {answer.strip()[:90]}\n")
+
+
+if __name__ == "__main__":
+    asyncio.run(main())