|
| 1 | +# PartsLine — Voice Agent for Auto Parts Counters |
| 2 | + |
| 3 | +## What it is |
| 4 | +A browser-based voice agent that answers the highest-volume call an auto |
| 5 | +parts counter gets: "do you have [part] for [vehicle]?" It searches a |
| 6 | +parts catalog with vehicle details as metadata filters, checks stock, |
| 7 | +quotes price, sets parts aside by name, and transfers anything complex |
| 8 | +to a human — with the vehicle/part context already captured. |
| 9 | + |
| 10 | +Built on Moss because the problem is retrieval-shaped: catalogs are too |
| 11 | +large to prompt-stuff, one question triggers multiple lookups |
| 12 | +(part → fitment → stock), and on a live call those lookups must resolve |
| 13 | +mid-sentence (~5-15ms in-process vs ~200-600ms over a cloud DB round trip). |
| 14 | + |
| 15 | +## Who it's for |
| 16 | +Demo audience now (incl. as an open-source reference app); architecture |
| 17 | +kept honest so a real shop's catalog/POS data could swap in later. |
| 18 | + |
| 19 | +## Guardrails (the "trained new employee" rules) |
| 20 | +- Exact-match-or-ask: never quote fitment on an ambiguous vehicle spec; |
| 21 | + ask the disambiguating question ("2.5 or 3.6?") |
| 22 | +- Grounded-only: every fitment/price/stock claim comes from a retrieval |
| 23 | + result; no answers from LLM world-knowledge |
| 24 | +- Hedged stock language ("we're showing 2 in stock") |
| 25 | +- Immediate cheerful transfer for: modifications, interchange questions, |
| 26 | + returns/warranty, fleet/commercial pricing |
| 27 | + |
| 28 | +## MVP scope |
| 29 | +Caller: part lookup w/ disambiguation → price + stock quote → |
| 30 | +set-aside by first name → or transfer-with-context. |
| 31 | +Shop: minimal call-log screen (time, vehicle, part, outcome). |
| 32 | + |
| 33 | +## Explicitly NOT building (v1) |
| 34 | +Ordering/payments · order status · returns handling · multi-part calls · |
| 35 | +phone telephony (browser voice only) · callback-number capture · |
| 36 | +real catalog/POS integration · CRM/analytics · auth/multi-tenancy · |
| 37 | +public deployment (laptop-demoed) |
| 38 | + |
| 39 | +## Data model |
| 40 | +- PART (part_number, name, category, brand, description, price) |
| 41 | +- VEHICLE (year, make, model, engine, trim?) |
| 42 | +- FITMENT (part↔vehicle join + condition notes; THE core domain) |
| 43 | +- STOCK (separate from PART by design — the future POS-sync seam) |
| 44 | +- CALL_LOG (append-only: time, vehicle, parts, outcome, set-aside name) |
| 45 | + |
| 46 | +Implementation note: FITMENT is flattened into one Moss document per |
| 47 | +(part × vehicle) combination, with vehicle attributes (make/model/year/ |
| 48 | +engine) as string metadata fields on each document. Confirmed working |
| 49 | +via live test (see Retrieval Findings below). |
| 50 | + |
| 51 | +Synthetic catalog: 3-4 categories (belts, brakes, filters, batteries), |
| 52 | +~200-500 parts, ~50-100 vehicles, deliberately messy fitment: dual-engine |
| 53 | +vehicles, mid-year splits, superseded part numbers, universal-fit parts, |
| 54 | +vehicles absent from catalog. |
| 55 | + |
| 56 | +## Retrieval Findings (from live Moss test — moss-test/seed.py + query.py) |
| 57 | + |
| 58 | +A 12-document trap-laden mini-catalog was pushed to a real Moss index |
| 59 | +and queried live. Results, and what they mean for the build: |
| 60 | + |
| 61 | +**CONFIRMED WORKING — dual-engine disambiguation.** |
| 62 | +Unfiltered query ("serpentine belt" for 2014 Outback) returned BOTH the |
| 63 | +2.5L and 3.6L belts with close scores (1.000 / 0.969) — this is the |
| 64 | +visible-ambiguity signal the agent needs to trigger "2.5 or 3.6?". |
| 65 | +Adding `engine=2.5` to the filter cleanly isolated the correct part. |
| 66 | +→ Build implication: when a filtered-by-known-attributes query returns |
| 67 | + >1 result, the agent must ask a disambiguating question before |
| 68 | + answering. This is a hard rule, not a nicety. |
| 69 | + |
| 70 | +**CRITICAL FINDING — semantic score is NOT a safe "we don't carry this" |
| 71 | +signal.** An unfiltered query for "brake pads for a 2019 RAV4" (a vehicle |
| 72 | +NOT in the catalog at all) returned Civic/Camry brake pads scoring |
| 73 | +0.909-0.994 — indistinguishable from genuine matches. There is no score |
| 74 | +threshold that separates a real match from a confidently-wrong one. |
| 75 | +By contrast, the SAME query WITH a vehicle filter (`model=RAV4`) |
| 76 | +correctly returned zero results. |
| 77 | +→ Build implication (HARD REQUIREMENT): the agent must NEVER accept a |
| 78 | + fitment answer from an unfiltered/semantic-only query. Vehicle |
| 79 | + identity (make/model/year/engine) must always be applied as a metadata |
| 80 | + filter before a part is quoted to a caller. An empty filtered result |
| 81 | + is the ONLY trustworthy "we don't carry that" signal. This changes the |
| 82 | + guardrail from "prefer filtering" to "filtering is mandatory before |
| 83 | + any fitment claim." |
| 84 | + |
| 85 | +**FINDING — discontinued parts can outrank their replacement.** |
| 86 | +A discontinued part (A-100, 0 stock, superseded_by="A-100B") scored |
| 87 | +HIGHER (0.994) than its current replacement (A-100B, 0.975) on a plain |
| 88 | +semantic query. Text similarity has no concept of "this part is dead." |
| 89 | +→ Build implication: retrieval logic must explicitly exclude or |
| 90 | + deprioritize documents where `superseded_by` is set or `stock="0"`, |
| 91 | + rather than trusting raw top-result ranking. Options to evaluate in |
| 92 | + build: (a) filter out `stock=0` at query time when alternatives exist, |
| 93 | + (b) always chase `superseded_by` chains before presenting a result, |
| 94 | + (c) surface both with the agent explicitly saying "that one's been |
| 95 | + replaced by X." Needs a decision during Phase 2 build, not left to |
| 96 | + the LLM's judgment. |
| 97 | + |
| 98 | +**UNRESOLVED — production-date / mid-year-split filtering not actually |
| 99 | +tested.** The test used discrete string tags (`prod_cutoff: |
| 100 | +"before-2014-03"` / `"from-2014-03"`) rather than a real date value with |
| 101 | +$lte/$gte comparison. Filtering on year=2014 alone correctly returned |
| 102 | +BOTH pad versions (expected, since prod_cutoff wasn't filtered on) — but |
| 103 | +this does NOT confirm whether Moss's `$lte`/`$gte` operators work |
| 104 | +cleanly against real date metadata for mid-year splits. |
| 105 | +→ Build implication: re-test with an actual date field before assuming |
| 106 | + this pattern works. If date-range filtering proves awkward, fallback |
| 107 | + is precomputing the split into a boolean/enum field like `prod_cutoff` |
| 108 | + (as tested) and having the agent ask "was this bought before or after |
| 109 | + March 2014?" — a viable path either way, but confirm which. |
| 110 | + |
| 111 | +## Evaluation discipline |
| 112 | +15-25 held-out caller scenarios written before build, kept OUT of the |
| 113 | +repo (and away from coding agents) during development; run manually |
| 114 | +against the finished agent; added to the repo as documented evals after |
| 115 | +v1. MUST include explicit cases for: dual-engine disambiguation, |
| 116 | +superseded-part handling, and an absent-vehicle call (RAV4-style) to |
| 117 | +confirm the agent refuses to guess rather than offering a wrong part. |
| 118 | + |
| 119 | +## Presentation |
| 120 | +- Demo page: one "talk" button, live transcript, inline lookup-chips |
| 121 | + showing each retrieval as it fires (the proof layer) |
| 122 | +- Call-log page: single newest-first list |
| 123 | +- Open source (MIT), README-first, reproducible via seed script |
| 124 | + |
| 125 | +## Stack |
| 126 | +- Voice/transport: LiveKit Agents (Python) — browser WebRTC native; |
| 127 | + Moss's own reference integration path |
| 128 | +- Retrieval: Moss — catalog index (semantic + $eq/$and metadata |
| 129 | + filters), retrieval exposed as LLM function tools; per-call session |
| 130 | + optional. Confirmed live: `create_index`, `load_index`, filtered and |
| 131 | + unfiltered `query` all work as documented against a real project. |
| 132 | +- STT: Deepgram · LLM: GPT-4o (Groq as TTFT upgrade) · TTS: Cartesia |
| 133 | +- Frontend: Next.js (demo page + /calls log route) |
| 134 | +- DB: SQLite (CALL_LOG only); catalog source-of-truth = JSON in repo, |
| 135 | + seeded to Moss via script (see moss-test/seed.py as the working |
| 136 | + reference implementation) |
| 137 | +- Deploy: none for v1 (local); later Vercel + LiveKit Cloud/Fly.io |
| 138 | + |
| 139 | +## Architecture sketch |
| 140 | +Browser (Next.js, LiveKit client) |
| 141 | + ⇄ LiveKit room ⇄ Agent worker (Python: Deepgram → GPT-4o ⇄ Moss |
| 142 | + function tools → Cartesia) → SQLite CALL_LOG → /calls page |
| 143 | + |
| 144 | +Moss function-tool logic (per the findings above) MUST: |
| 145 | +1. Always resolve vehicle identity (make/model/year/engine) before |
| 146 | + querying for a part — never answer from an unfiltered query. |
| 147 | +2. If a filtered query returns >1 result, ask a disambiguating question |
| 148 | + instead of picking one. |
| 149 | +3. If a filtered query returns 0 results, tell the caller "we don't |
| 150 | + carry a match for that vehicle" — do not fall back to unfiltered |
| 151 | + semantic search to find a "close enough" answer. |
| 152 | +4. Exclude/deprioritize discontinued (`stock=0` or `superseded_by`-set) |
| 153 | + parts rather than trusting raw ranking; resolve to the current |
| 154 | + replacement when one exists. |
| 155 | + |
| 156 | +## Security-sensitive areas |
| 157 | +- All keys in .env (gitignored) + committed .env.example; clean from |
| 158 | + first commit (repo is public from day one) |
| 159 | +- Spend caps + alerts on all provider accounts (Moss, LiveKit, Deepgram, |
| 160 | + OpenAI/Groq, Cartesia), day one |
| 161 | +- Post-v1 public deploy adds: rate limiting on the agent endpoint, |
| 162 | + abuse pass (strangers can burn LLM/TTS credits via the talk button) |
| 163 | + |
| 164 | +## Build sequencing (mock-first) |
| 165 | +1. ~~Next.js page + fake transcript (clickable skeleton)~~ |
| 166 | +2. ~~Catalog seed script + Moss index + a CLI that answers text |
| 167 | + queries~~ — DONE: moss-test/seed.py + query.py, findings above |
| 168 | + incorporated. Re-test production-date filtering ($lte/$gte) before |
| 169 | + moving on if mid-year-split parts matter for the demo story. |
| 170 | +3. LiveKit voice loop with a dumb echo agent (voice proven separately) |
| 171 | +4. Wire 2+3: function tools enforcing the four hard rules above, |
| 172 | + disambiguation loop, superseded-part handling |
| 173 | +5. Lookup-chips in transcript; CALL_LOG + /calls page LAST |
| 174 | +6. Run held-out evals (incl. RAV4-style absent-vehicle case); fix; |
| 175 | + publish repo |
| 176 | + |
| 177 | +## Open questions carried forward |
| 178 | +1. Real-world validation: has anyone confirmed with an actual parts |
| 179 | + counter that this is the call volume/pain they'd want solved? |
| 180 | + (Deliberately parked, not resolved.) |
| 181 | +2. STT disambiguation tolerance: how many "sorry, which one was that?" |
| 182 | + re-asks is acceptable before it feels broken? (Resolved as: graceful |
| 183 | + re-asking is acceptable, expected behavior — not a defect to eliminate.) |
| 184 | +3. Production-date filtering: does Moss's $lte/$gte work cleanly on a |
| 185 | + real date field, or does the mid-year-split need to stay a |
| 186 | + precomputed boolean tag? RE-TEST BEFORE BUILDING THE FEATURE. |
| 187 | +4. Superseded-part resolution strategy: pick (a), (b), or (c) from the |
| 188 | + Retrieval Findings section above during Phase 2 build — do not leave |
| 189 | + this to implicit LLM judgment, it's a guardrail decision. |
0 commit comments