Skip to content

handoff.md

Latest commit

 

History

History
194 lines (151 loc) · 10.7 KB

handoff.md

File metadata and controls

194 lines (151 loc) · 10.7 KB
Raw

Handoff — OnyxAI Waves 1–7 Complete

Date: 2026-05-14
Branch: main (uncommitted)
Last commit: f5b3c4c — syntax-highlight-formatting
Session: Waves 1–7 implementation + testing

Goal

OnyxAI is a native Android AI chat app (Expo SDK 55 + React Native). The product thesis is:

  • Model switching — pick from free/cheap frontier models per message
  • Persistent memory — facts extracted and recalled across conversations
  • Pay-as-you-go credits — transparent per-message INR pricing
  • Proper formatting — dark-themed markdown + syntax highlighting
  • Internet search — Tavily-powered web search with memory cross-reference
  • Semantic retrieval — vector search over old messages (closes the context gap)
  • Interactive artifacts — roadmap trees, flowcharts, bar charts rendered in chat
  • PDF export — anti-slop styled PDFs via expo-print
  • Deep reading — Firecrawl URL extraction for full-page ingestion

📍 Current State

Wave Status

Wave Feature Status
Wave 0 Markdown Rendering + Syntax Highlighting ✅ Complete
Wave 1 Context Window Management ✅ Complete
Wave 2 Streaming Response (SSE) ✅ Complete
Wave 3 Image & File Uploads ✅ Complete
Wave 4 Semantic Memory & Retrieval ✅ Complete
Wave 5 Tavily Search & Internet Access ✅ Complete
Wave 6 Document RAG ⏭️ Skipped (folded into Wave 7)
Wave 7 Interactive Artifacts + PDF Export + Firecrawl ✅ Complete

Infrastructure Health

Layer Status
DB Schema ✅ All migrations through 0007_wave5_search.sql deployed
Worker ✅ 9 endpoints live: /chat, /chat/summarize, /embed, /memory/extract, /upload/analyze, /search, /payments/*
Mobile ✅ Both mobile (npx tsc --noEmit) and worker (npx tsc --noEmit) compile cleanly
Storage ✅ chat-images (public) + chat-files (private) buckets with RLS policies
Search ✅ Tavily API integrated (key: tvly-dev-...) — test-verified with 5 real-time queries
Semantic Retrieval ✅ Layer 3 live — match_messages RPC returns relevant old messages, threshold 0.55
Tests ✅ Manual: memory test (20-fact recall), search test (5 Tavily queries), semantic retrieval (4/4 recall at 0.55)
Commits ⬜ All changes are uncommitted

✈️ Files in Flight (Active Modifications)

New Files (Waves 2–5)

File Purpose Wave
supabase/migrations/0005_wave3_uploads.sql uploads table + RLS 3
supabase/migrations/0006_wave4_embeddings.sql embedding columns + IVFFlat indexes + match_memory_facts RPC 4
supabase/migrations/0007_wave5_search.sql search_results table + search_enabled column 5
supabase/storage_policies.sql Storage bucket RLS policies (run via SQL Editor) 3
worker/src/upload.ts POST /upload/analyze — vision model image description 3
worker/src/embed.ts POST /embed — OpenRouter embeddings API proxy 4
worker/src/search.ts POST /search — Tavily search with memory cross-reference 5
lib/uploads.ts Client: pickImage, pickDocument, uploadToStorage, analyzeUpload 3
lib/memory.ts Client: getEmbedding, getMemorySystemPrompt, extractMemoryFacts 4
lib/search.ts Client: searchWeb helper for Tavily 5
components/search/SourceCard.tsx Search result card with domain/title/content 5
components/search/SearchMemoryBanner.tsx "Based on what you've told OnyxAI" banner 5
scripts/test-memory.ts 20-fact memory test (summarization verification) 1
scripts/test-waves-4-5.ts Combined search + semantic retrieval test 4–5
scripts/retest-semantic.ts Quick semantic retrieval re-test on existing conversation 4
~/.commandcode/plans/waves-4-5-memory-search.md Implementation plan for Waves 4–5 4–5
~/.commandcode/plans/wave-7-artifacts.md Implementation plan for Wave 7 7

Modified Files (Waves 2–5)

File What Changed
worker/src/chat.ts Streaming SSE (Wave 2) — raw fetch to OpenRouter, parse SSE. Layer 3 retrieval (Wave 4) — embed query + match_messages. Tavily search integration (Wave 5) — auto/force search modes. Summarization model changed to Gemini Flash Lite. match_threshold lowered to 0.55.
worker/src/memory.ts Rewrote from scaffold — real fact extraction via Gemini, embedding generation, deduplicated insert into memory_facts
worker/src/supabase.ts Added matchMemoryFacts RPC wrapper. Fixed matchMessages param bug (conversation_id → conv_id).
worker/src/index.ts Registered /upload/analyze, /embed, /search routes
worker/src/types.ts Added TAVILY_API_KEY to Env
worker/.dev.vars Added TAVILY_API_KEY=tvly-dev-...
worker/.dev.vars.example Added TAVILY_API_KEY= placeholder
hooks/useChat.ts Streaming: real token streaming with AbortController. Embeddings: embed user messages after sending (background). Memory: trigger extraction every 15 messages. Search: passes enableSearch/forceSearch params.
lib/openrouter.ts New streamChatFromWorker() with SSE parsing, AbortController, image_url content building. Added enableSearch/forceSearch params.
types/index.ts Added remoteUrl, mimeType, sizeBytes to Attachment. Added attachments to Message.
components/chat/InputBar.tsx Camera + attach buttons made pressable. Added search mode toggle (🌐 auto/force/off).
components/chat/MessageBubble.tsx Renders image thumbnails for attachments
components/chat/AttachmentPreview.tsx Shows image previews in composer
app/(tabs)/index.tsx Attachment state management. Camera/gallery/document picker flow. Upload to Supabase Storage. Search mode state + toggle.

🏗️ Architecture

Expo App (React Native)
├── Supabase (Auth + DB + Storage + Vector Search)
│   ├── conversation_summaries  ← Wave 1
│   ├── match_messages RPC      ← Wave 1
│   ├── match_memory_facts RPC  ← Wave 4
│   ├── uploads                 ← Wave 3
│   ├── search_results          ← Wave 5
│   ├── messages.embedding      ← Wave 4
│   └── storage buckets         ← Wave 3
└── Cloudflare Worker
    ├── POST /chat              ← Streaming SSE (Wave 2) + Layer 3 retrieval (Wave 4) + Tavily search (Wave 5)
    ├── POST /chat/summarize    ← Wave 1 (Gemini Flash Lite)
    ├── POST /embed             ← Wave 4 (all-MiniLM-L6-v2, 384-dim)
    ├── POST /memory/extract    ← Wave 4 (fact extraction + embedding + dedup insert)
    ├── POST /upload/analyze    ← Wave 3 (vision model image description)
    ├── POST /search            ← Wave 5 (Tavily + topic extraction + memory cross-ref)
    └── POST /payments/...

🔑 Key Numbers

Constant Value Location Meaning
EPHEMERAL_WINDOW 8 hooks/useChat.ts Messages kept verbatim
SUMMARIZE_CHUNK_SIZE 10 worker/src/chat.ts Messages per summary block
SUMMARIZE_EVERY 10 hooks/useChat.ts Trigger summarization every N msgs
MEMORY_EXTRACT_EVERY 15 hooks/useChat.ts Trigger fact extraction every N msgs
MATCH_THRESHOLD 0.55 worker/src/chat.ts Cosine similarity floor for retrieval
MATCH_COUNT 3 worker/src/chat.ts Old messages to retrieve
Vector dim 384 migrations all-MiniLM-L6-v2 (free, fast)
Embedding model sentence-transformers/all-MiniLM-L6-v2 worker Free via OpenRouter
Summarization model google/gemini-2.5-flash-lite worker Reliable JSON output
Vision model google/gemini-2.5-flash-lite worker Image description
Fact extraction model google/gemini-2.5-flash-lite worker Structured JSON facts
Search API Tavily worker search_depth: advanced, include_answer: true

🧪 Test Results

Wave 1 — Context Window

  • Summaries created after 20 messages ✅
  • Facts from summarized zone recalled ✅
  • 2-message gap identified (summarized range vs ephemeral window mismatch)

Wave 2 — Streaming

  • SSE streaming works with Gemini Flash Lite ✅
  • Stop button aborts via AbortController ✅

Wave 3 — Uploads

  • Image upload to Supabase Storage works ✅
  • Image thumbnails render in MessageBubble ✅

Wave 4 — Semantic Retrieval

  • Embeddings stored: 12/24 messages in test conversation ✅
  • RPC works: match_messages returns correct old messages ✅
  • Recall at 0.55 threshold: 4/4 facts recalled (blood type, beta fish, chai, startup event) ✅
  • Bug fixed: RPC param conversation_id → conv_id (PostgREST mismatch)

Wave 5 — Tavily Search

  • 5/5 real-time queries returned accurate data with citations ✅
  • Sensex (75,398.72), PM (Modi), React 19 features, Bangalore weather, cricket score ✅

📝 Notes for Next Developer

  1. The 2-message gap — SUMMARIZE_CHUNK_SIZE (10) ≠ EPHEMERAL_WINDOW (8). Change endIdx in handleSummarize() from messageCount - SUMMARIZE_CHUNK_SIZE to messageCount - EPHEMERAL_WINDOW to close the gap.

  2. Semantic retrieval is live — handleChat calls getEmbedding() + match_messages() on every request with conversationId. Threshold is 0.55. Works end-to-end. Lowering it further catches more but adds noise.

  3. Embeddings are generated client-side — useChat.ts calls /embed after each user message and PATCHes the message row. This is fire-and-forget — if it fails, the message won't be retrievable. Consider moving embedding generation to the worker.

  4. Memory extraction is fire-and-forget — triggers every 15 messages. Errors are silently swallowed. Facts use the UNIQUE(user_id, content) constraint for dedup — 409 is normal.

  5. Tavily API key is in worker/.dev.vars. Free tier: 1000 searches/month. To test search: toggle 🌐 to force mode in InputBar.

  6. Storage RLS — policies are in supabase/storage_policies.sql. Must be run via Supabase SQL Editor (not auto-applied by db push).

  7. Worker uses raw fetch for chat — bypassed the OpenRouter SDK for /chat to avoid type validation issues with mixed content (string + image arrays). SDK still used for summarize, memory extract, and upload analyze.

  8. All migrations are deployed — 0001 through 0007 applied. Next migration is 0008 for Wave 6 or 7.

  9. TypeScript compiles clean — both npx tsc --noEmit in root and worker pass.

  10. Plans exist — ~/.commandcode/plans/waves-4-5-memory-search.md and ~/.commandcode/plans/wave-7-artifacts.md.

End of handoff.