VaultStack RAG System

A document-grounded RAG system with verification for high-stakes domains (legal, medical).

Features

• Flexible LLM provider: Uses OpenAI-compatible APIs or a local Hugging Face model
• Local embeddings: Runs BGE-small-en-v1.5 locally (no GPU required)
• Document-grounded: Answers ONLY from provided documents
• Verification checklist: Each answer is verified against source documents independently
• Citation tracking: Every claim includes [Document, Section] citations
• No hallucination: Multiple guardrails prevent factual fabrication
• Rest API + Frontend: FastAPI backend with a vanilla JS dashboard UI

Architecture

Documents → Loader → Chunker → Embeddings (local) → Qdrant
                                                      ↓
Query → Retriever → Generator (LLM) → Verifier (LLM) → Response + Checklist

Frontend (vanilla HTML/CSS/JS)  ←→  FastAPI Server  ←→  VaultStack Engine  ←→  Qdrant

Prerequisites

1. LLM provider — OpenAI-compatible endpoint, or a local Hugging Face model
2. Qdrant running locally

Start Qdrant

# Download binary from https://github.com/qdrant/qdrant/releases
./qdrant.exe

# Or Docker
docker run -d -p 6333:6333 qdrant/qdrant

Setup

cd vaultstack
pip install -r requirements.txt

Copy .env.example to .env and choose one LLM mode.

OpenAI-compatible mode:

LLM_PROVIDER=openai
LLM_API_KEY=gsk_your_groq_key_here
LLM_BASE_URL=https://api.groq.com/openai/v1
LLM_MODEL=llama-3.1-8b-instant
GROQ_COOLDOWN=65.0

Local Hugging Face Llama 8B mode:

LLM_PROVIDER=huggingface
HF_MODEL_ID=meta-llama/Llama-3.1-8B-Instruct
HF_TOKEN=hf_your_token_if_model_is_gated
HF_DEVICE_MAP=auto
HF_TORCH_DTYPE=auto
HF_LOAD_IN_4BIT=false
GROQ_COOLDOWN=0
QDRANT_HOST=localhost
QDRANT_PORT=6333
DOCUMENTS_DIR=./documents

For Meta Llama models on Hugging Face, accept the model license on Hugging Face and create a read token before first startup. The first run downloads the model, so it can take a while and needs enough disk/RAM/VRAM for an 8B model.

Usage

Start the server

python server.py

Open http://localhost:8000 in your browser.

Workflow

1. Upload documents — drag-and-drop PDF, DOCX, TXT files
2. Upload a checklist — CSV, JSON, or TXT with one question per line
3. Click "Run Analysis" — the system processes each question against the documents
4. Review results — supported, conflict, insufficient, or pending status per item
5. Check citations — each answer cites the source documents and sections
6. Export — download results as CSV

API Endpoints

Method	Path	Description
GET	/api/config	System configuration
GET	/api/documents	List uploaded documents (scope: ?client_id&case_id)
POST	/api/documents/upload	Upload documents (form fields client_id, case_id)
POST	/api/checklist/upload	Upload a checklist
GET	/api/checklist	Get checklist items
POST	/api/analysis/run	Trigger analysis (background job → poll status)
GET	/api/analysis/stream	Live SSE feed of results as they complete
GET	/api/analysis/status	Get analysis results
GET	/api/analysis/summary	Get summary statistics
POST	/api/query	Single question query (client_id, case_id in body)
POST	/api/reset	Drop a scope's collection (?client_id&case_id)

/api/analysis/run is REST-style: it returns {"status":"running"} immediately and runs in the background — poll /api/analysis/status for results, or open /api/analysis/stream for a live SSE feed (the dashboard uses the stream).

Tenant isolation (client / contract)

Documents and embeddings are isolated per (client_id, case_id). Each scope is stored in its own Qdrant collection (vaultstack__<client>__<case>) and its own documents subdirectory, so retrieval for one client/contract can never surface another's embeddings. The default scope (default/default) keeps the bare vaultstack collection and the root documents/ directory for backward compatibility. Pass client_id/case_id on upload, query, and analysis to target a specific scope.

API Usage Example

# Scoped query — only ever answered from client "acme", contract "msa-2024"
curl -X POST http://localhost:8000/api/query \
  -H "Content-Type: application/json" \
  -d '{"question": "What is the termination clause in the MSA?",
       "client_id": "acme", "case_id": "msa-2024"}'

How It Works

1. Document Loading: PDFs and text files are parsed and chunked
2. Embedding: Chunks are embedded locally using BGE-small-en-v1.5
3. Retrieval: Top-k relevant chunks are retrieved from Qdrant
4. Generation: Configured LLM generates answer with inline citations
5. Verification (two-tier):
   • Self-consistency: the same documents shown to the generator are re-checked independently, blind to the generated answer — catches hallucinated or misread citations. Failing this produces Unverified Citation, not Conflict — it means "don't trust this without manual review," distinct from an actual contradiction.
   • Contradiction check: a small pool of other documents is checked for content that actively disagrees with the answer (a different number, date, or term for the same fact). Only this can produce Conflict — a source simply not mentioning the topic is not a conflict.
6. Checklist Output: Supported / Conflict / Unverified Citation / Insufficient per item

Evaluation

eval/eval_set.json has 65 ground-truth questions (factual, cross-document, and negative/refusal cases) derived from TEST_QUESTIONS.md, covering every document in documents/.

python -m eval.runner                    # retrieval metrics only — no LLM calls, fast
python -m eval.runner --full             # + generation/citation metrics — costs LLM calls
python -m eval.runner --client acme --case msa-2024   # eval a specific scope

Retrieval-only mode reports Hit Rate, Precision, Recall, MRR, and nDCG at several cutoffs (default k=1, TOP_K, 10) — this answers "are we finding the right documents at all," independent of the LLM. --full additionally reports answer accuracy, citation precision/recall, hallucination rate (a Supported result whose answer doesn't match ground truth — meaning the verifier itself was fooled), and refusal accuracy on the negative cases.