11-semantic-cache

11 — Semantic Response Cache

Status: ⚠️ Investigated, not worth building. Real customer traffic has near-zero repetition; the apparent 71% repetition was an artifact of one developer's test chat.

Original hypothesis

Plumbing intake traffic is repetitive (FAQ-like). A semantic cache (embed query → return cached answer if cosine sim ≥ 0.92) could absorb 30–50% of traffic at near-zero latency. Design in the original draft below.

What the data actually shows

Measured real conversation history (measure_hitrate.py):

Apparent repetition (misleading)

• 295 user messages total, 71% exact duplicates
• Top messages: "hello" (44×), "(photo)" (33×), "Hello" (24×), "test" (16×)

The artifact

The dev/test chat alone has 185 turns of repetitive testing ("hello", "test", repeated photo uploads). This single conversation accounts for nearly all the duplication.

Real customer traffic (dev chat excluded)

Hi have a small leak in my faucet
my sink is clogged
my sink is dripping
my toilet is running
305-315-6628 and I prefer text

Zero duplicates. Every customer query is unique — customers describe their situation in their own words. Plumbing intake is naturally varied.

Why the cache doesn't fit

1. No repetition to exploit. Real traffic has ~0% exact-match and the semantic near-matches ("my sink is clogged" vs "my sink is dripping") need different replies — caching would serve wrong answers.
2. State-dependent. Most messages are multi-turn intake ("My name is Stefano", "Any time is fine", "yes"). "yes" means something different at each point. A cache keyed on message text alone is unsafe.
3. Replies aren't deterministic across model versions. The same "hello" got "Something went wrong" during the broken-prod era and a real greeting after. A cache would freeze stale replies.

When to revisit

Only worth building if:

• Traffic scales to hundreds of unique users/day AND
• A clear FAQ pattern emerges ("what are your hours", "do you install water heaters") that's genuinely stateless AND repeated
• The bot adds a dedicated FAQ mode where canned answers are appropriate

Until then, the model + prompt cache (#13, already working at 9× speedup) handles latency adequately.

What would actually help latency

The real latency win is already in place: llama.cpp's prompt cache caches the 538-token system prompt (350ms cold → 38ms warm). That's the dominant cost per request, and it's solved. Adding a semantic cache on top would complicate the system for no measurable gain on real traffic.

---

Original design (kept for reference, not implemented)

Draft architecture (superseded by the data above)

user query → embed(query) → cosine vs cache → HIT (return) / MISS (model)

• Embedding model: bge-small-en (130 MB, 384-dim)
• Vector store: numpy matrix + SQLite (≤10k entries, brute-force cosine is sub-ms)
• Admission: only cache validated good-path, single-turn, stateless replies
• Threshold: cosine ≥ 0.92
• Composes with cascade: cache → router → distill/E2B → judge → store

Files

File	Purpose
README.md	This document (investigation result)
measure_hitrate.py	Repetition/semantic-similarity analysis on real DB traffic
hitrate.log	Output of the analysis