Add semantic-cache example (cache LLM responses by meaning)

[HarshaNalluru]

What

A small, runnable example under moss-live-labs/examples/semantic-cache that caches LLM responses by meaning instead of exact text.

A normal cache keys on the literal request string, so two phrasings of the same question miss and you pay the model twice:

"what are your hours?"  -> MISS -> calls the model
"when do you open?"     -> MISS -> calls the model again

This example embeds each question, looks up the nearest one already answered, and returns the stored answer if it clears a similarity threshold — no model call:

"what are your hours?"  -> MISS -> calls the model, stores the answer
"when do you open?"     -> HIT  -> returns the stored answer, no model call

The whole thing

async def ask(self, question):
    hit = await self.index.query(question, QueryOptions(top_k=1))
    if hit.docs and hit.docs[0].score >= THRESHOLD:   # close enough in meaning?
        return hit.docs[0].metadata["answer"]           # cache hit — no LLM call
    answer = await call_the_model(question)             # miss — ask once
    await self.index.add_docs(
        [DocumentInfo(id=question, text=question, metadata={"answer": answer})])
    return answer

Moss serves the nearest-match lookup on-device in <10 ms, far cheaper than the model call it avoids. The one knob that matters is THRESHOLD (cosine similarity).

Contents

• semantic_cache.py — SemanticCache class + a runnable demo (first question misses, the paraphrased second hits)
• README.md, .env.example, pyproject.toml

Run

uv sync && cp .env.example .env   # Moss + OpenAI keys
python semantic_cache.py

[CLAassistant]

All committers have signed the CLA.

[Copilot]

Pull request overview

Adds a new runnable moss-live-labs example demonstrating a semantic cache pattern: reuse prior LLM answers when a new question is sufficiently similar in embedding space, avoiding repeated model calls.

Changes:

• Introduces SemanticCache demo script that queries a vector index for nearest prior question and reuses its stored answer above a similarity threshold.
• Adds usage docs and environment template for Moss + OpenAI credentials.
• Adds a minimal pyproject.toml and Python version hint for running the example.

Reviewed changes

Copilot reviewed 5 out of 5 changed files in this pull request and generated 5 comments.

Show a summary per file

File	Description
moss-live-labs/examples/semantic-cache/semantic_cache.py	Implements the semantic-cache demo logic and runnable CLI loop.
moss-live-labs/examples/semantic-cache/README.md	Explains the semantic-cache concept and how to run the example.
moss-live-labs/examples/semantic-cache/pyproject.toml	Declares minimal dependencies for running the example.
moss-live-labs/examples/semantic-cache/.python-version	Pins a local Python version for convenience.
moss-live-labs/examples/semantic-cache/.env.example	Provides env var template for Moss and OpenAI keys.

---

💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.

[Copilot]

Pull request overview

Copilot reviewed 5 out of 5 changed files in this pull request and generated 2 comments.

[github-actions]

Codex review

No issues found.

[HarshaNalluru]

Thanks for the review — addressed the actionable points:

• Env validation: the script now fails fast with a clear message if MOSS_PROJECT_ID / MOSS_PROJECT_KEY / OPENAI_API_KEY are missing.
• Version pin: pinned moss>=1.1.1 to match the other moss-live-labs examples.
• metadata["answer"]: now read defensively with .get("answer") and skipped if absent.
• Determinism: clarified that the session holds entries in memory for the run (so the demo is a clean MISS → HIT), and documented push_index() for cross-run persistence.
• README: reworded to note the shown snippet is simplified vs the runnable ask().

On the session() comments (MossClient has no session): that's from the SDK source vendored in this repo, which is older than the published moss package. client.session(index_name=...) is the current, documented API — see https://docs.moss.dev/docs/reference/python/sessions — and pip install moss (pinned >=1.1.1) provides it, so the example runs as written. Happy to switch to the create_index/load_index index APIs instead if we'd rather the example not depend on the sessions API.

[Copilot]

Pull request overview

Copilot reviewed 5 out of 5 changed files in this pull request and generated 3 comments.

[HarshaNalluru]

Follow-up review addressed:

• Query now uses alpha=1.0 (pure semantic), so the score reflects meaning and matches the "semantic similarity" threshold wording.
• Guarded optional metadata ((metadata or {}).get(...)) and nullable OpenAI message.content.
• The demo now uses a unique session name per run so it's deterministic (clean MISS → HIT); documented the stable-name + push_index() path for real cross-run persistence.

The remaining session() notes are from the older SDK vendored in this repo; client.session(index_name=...) is the current published API (docs: https://docs.moss.dev/docs/reference/python/sessions). Resolving these threads.

[Copilot]

Pull request overview

Copilot reviewed 5 out of 5 changed files in this pull request and generated 2 comments.

[Copilot]

Pull request overview

Copilot reviewed 5 out of 5 changed files in this pull request and generated 4 comments.

[Copilot]

Pull request overview

Copilot reviewed 5 out of 6 changed files in this pull request and generated 1 comment.

[HarshaNalluru]

Note: the recurring session() / SessionIndex flags come from the older SDK vendored in this repo. client.session(index_name=...) + push_index() are the current published API (docs: https://docs.moss.dev/docs/reference/python/sessions), and the example pins moss>=1.1.1 with a committed uv.lock, so it installs and runs as written. Resolving.

[Copilot]

Pull request overview

Copilot reviewed 5 out of 6 changed files in this pull request and generated no new comments.

[Copilot]

Pull request overview

Copilot reviewed 5 out of 6 changed files in this pull request and generated no new comments.