On-device document question answering: ask questions about a local document by voice and hear the answer spoken back. It runs as a private, local RAG pipeline, fully on-device with no cloud and no data leaving the device.
picoLLM handles both retrieval and generation. It embeds the document, retrieves the passages most relevant to your question, and generates an answer grounded in them. Cheetah Streaming Speech-to-Text transcribes the spoken question, and Orca Streaming Text-to-Speech speaks the answer as it is generated.
- Chunk the document into overlapping passages.
- Embed each chunk with picoLLM and cache the embeddings on disk, so later runs skip re-embedding.
- Transcribe the spoken question with Cheetah, embed it, and retrieve the most similar passages by semantic search.
- Generate the answer with picoLLM, grounded only in the retrieved passages. If the answer is not in the document, it says so instead of guessing.
- Stream the answer to Orca and speak it sentence by sentence as it is produced.
Does it run fully on-device? Yes. Indexing, retrieval, transcription, answer generation, and speech all run on-device, with no network calls and no data leaving the device.
How does it avoid making up answers? It answers only from the passages retrieved from your document. If the answer is not in them, it tells you it does not know rather than guessing.
Does it re-read the whole document for every question? No. The document is chunked and embedded once, then the embeddings are cached on disk and reused across questions and runs.
Can I ask questions by voice? Yes. You ask out loud, Cheetah transcribes the question, and Orca speaks the grounded answer back as it streams.
More FAQs can be found on Picovoice website.