Vault RAG Pipeline

A small, explicit RAG pipeline for turning a private Obsidian vault into a searchable corpus for an AI agent.

The design assumes the vault contains operational history, incident notes, configuration notes, and other private material. Redaction happens before embedding. Originals stay untouched. The public repo contains only the pipeline code, not a vault, staging directory, reports, or environment file.

Architecture

1. Copy private vault text into a local staging/ directory outside the vault.
2. Run scripts/redact.py against staging/.
3. Chunk the redacted Markdown/text files with scripts/ingest.py.
4. Batch embed chunks through Ollama using nomic-embed-text or another configured model.
5. Upsert points into Qdrant with deterministic UUID5 IDs based on source_path and chunk_index.
6. Query the collection with scripts/vault_search.py and cite source_path in agent answers.

The ingest path is idempotent: the same source path and chunk index produce the same point ID, so reruns update existing points instead of creating random duplicates.

Redaction design

The redaction step is mandatory before embedding private operational notes.

• RFC1918 private IPv4 addresses are replaced with [HOST-*] tokens based on the final octet. This preserves enough retrievability to find related notes without embedding literal private addresses.
• Secret scrubbing is driven by gitleaks detect --no-git against the staging tree, then exact matched secret strings are replaced with [REDACTED-<rule>] markers.
• The original vault is not modified. Redaction only touches files under staging/ when run with --apply.
• Reports are written under reports/, which is ignored from commit one.

This is intentionally conservative. It favors preserving useful incident/configuration context while removing direct network addresses and credential material before anything enters an embedding model or vector database.

Requirements

• Python 3.11+
• gitleaks available in PATH
• Ollama with an embedding model such as nomic-embed-text
• Qdrant

The scripts use only Python's standard library. No Python package install is required.

Setup

Clone the repo and create a local .env from the example:

cp .env.example .env

Set these variables for your environment:

• PIPELINE_ROOT — local pipeline working directory containing staging/ and reports/
• VAULT_SOURCE_DIR — private Obsidian vault source directory
• OLLAMA_URL — Ollama base URL, for example http://ollama-host:11434
• OLLAMA_MODEL — embedding model, default nomic-embed-text
• QDRANT_URL — Qdrant base URL, for example http://qdrant-host:6333
• QDRANT_COLLECTION — Qdrant collection name
• EMBED_BATCH and UPSERT_BATCH — optional batch tuning knobs

Load the environment in your shell before running commands:

set -a
. ./.env
set +a

Create local working directories when needed:

mkdir -p "$PIPELINE_ROOT/staging" "$PIPELINE_ROOT/reports"

Copy vault content into staging/ using your preferred local copy method. Do not point redaction directly at the live vault and do not commit staging output.

Usage

Dry-run redaction:

python3 scripts/redact.py --dry-run

Apply redaction to staged files:

python3 scripts/redact.py --apply

Ingest redacted staged files into Qdrant:

python3 scripts/ingest.py

Search the corpus:

python3 scripts/vault_search.py "how did we fix the backup restore issue"

Tune retrieval strictness:

python3 scripts/vault_search.py "reverse proxy migration" --limit 8 --min-score 0.55

If no result clears the score threshold, the search script prints:

NO_RESULTS

Agent integration rule: before answering questions about past incidents, fixes, configurations, or homelab history, run vault_search.py with a concise query, cite source_path, and say when retrieval returns NO_RESULTS. Use live checks for current system state.

Lessons learned

• CPU embedding is slow enough to matter. Expect throughput to be dominated by embedding latency, not Qdrant upsert time.
• Batch sizing is a tradeoff. Larger embedding batches improve throughput until the embedding host saturates or request timeouts become annoying. Start small, then raise gradually.
• gitleaks detect --no-git scans the working tree. That is what you want for staged corpus redaction.
• Git-history scans are a separate problem. Use normal gitleaks detect with git log options when auditing whether secrets ever existed in repository history.
• Private IP redaction needs to handle punctuation and sentence boundaries cleanly. A redactor that misses an address because it is followed by a period is not a redactor. It is decorative regex.
• Redacted host tokens preserve retrieval quality better than deleting addresses entirely.

Safety notes

• Never write generated staging files, reports, or environment files into the original vault.
• Never commit .env, staging/, reports/, or exclude-paths.txt.
• Run a final secret scan and private-address scan before the first public push.