feat(rag): add fleischwolf-rag — pluggable RAG subsystem - #23
Merged
Conversation
Add a new workspace crate that builds a Retrieval-Augmented-Generation layer on top of the document converter: source → convert to Markdown → chunk → embed → vector store → retrieve → (optional) LLM answer synthesis. Every external dependency is a swappable trait: - Chunking: Markdown-aware, configurable size (300) + overlap (5%), heading context, section-bounded sliding window. - Embedders: Ollama/bge-m3 (default, 1024-d), Gemini, local ONNX (feature onnx-embed), and a deterministic hashing embedder for offline tests/eval. - Vector stores: SQLite (default, bundled), PostgreSQL + pgvector (feature postgres), in-memory. Documents and chunks live in separate tables. - Retrieval: dense vector, sparse BM25, Hybrid (RRF), Multi-Query fusion, HyDE. - LLM: OpenRouter (default model DeepSeek-V3). - Sources: local folder (default), FTP/SFTP (feature remote-sources). - Queues: in-process (default), RabbitMQ (feature rabbitmq), Redis pub/sub (feature redis). Includes an evaluation harness that sweeps a matrix of chunk sizes / overlaps / retrieval modes over a labelled dataset and ranks configs by recall@k / MRR / nDCG@k / latency, a clap-based CLI (init-db, ingest, query, eval, stats), and config loaded from .env (see .env.example). The default feature set is fully offline and unit-tested (33 tests); the heavier backends are real client code, compile-checked but exercised only against live services. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01PxznqztpBUiRN4FH6pKGun
Replace the brute-force cosine scan in the SQLite store with the sqlite-vec extension, statically compiled into the bundled SQLite and registered via sqlite3_auto_extension so every pooled connection sees it. Embeddings move out of a BLOB column on chunks into a vec0 virtual table (chunks_vec, cosine metric, dimension fixed at migrate time) sharing the chunks rowid, and vector_search becomes a KNN `embedding MATCH ? AND k = ?` query joined back to chunks. The store now takes the embedding dimension at connect time and validates it on insert and search. Adds vec0-backed unit tests (in-memory SQLite KNN, dimension mismatch); CLI smoke test returns identical cosine scores to the previous brute-force implementation. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01PxznqztpBUiRN4FH6pKGun
Time the three ingestion phases (parsing via fleischwolf, chunking, embedding) per document and store the results in the documents table's JSON metadata column under "metrics", so new metrics can be added later without a schema migration. Recorded: source file size, page count where the format has pages (PDF via pdfium page_count, PPTX slides / XLSX sheets via zip entry names), Markdown word count, chunk count, embedded word count, and per-phase seconds + words/sec — plus pages/sec for parsing when the page count is known. Adds a metrics module (ProcessingMetrics/PhaseMetrics/Timings + count_pages), a list_documents method on VectorStore (memory/sqlite/postgres), a per-document metrics table in the `stats` CLI subcommand, an info-level tracing line per ingested document, and unit + integration tests over the stored JSON shape. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01PxznqztpBUiRN4FH6pKGun
Add `fleischwolf-rag serve` (axum): GET /health (public), and under
API-key auth GET /api/stats, /api/documents, /api/documents/{id}, and
GET/POST /api/search with mode (vector|bm25|hybrid|multi-query|hyde),
top_k, and an optional answer=true flag for LLM-grounded answers. Keys
come from RAG_API_KEYS (comma-separated, in .env) and are accepted as
X-Api-Key or Authorization: Bearer; auth is fail-closed — the server
refuses to start with an empty key list. `serve --ingest` indexes the
configured source before serving; RAG_HTTP_ADDR / --addr control the bind.
Docs: add a from-zero quickstart to the crate README (index any documents
folder offline with the hash embedder, then switch to Ollama/bge-m3), a
REST API endpoint table with curl examples, and .env.example entries.
Tests: offline HTTP integration test over an ephemeral port (auth 401s,
Bearer + X-Api-Key, documents with metrics, search in all offline modes,
POST body, 400s for bad mode / LLM mode without key, empty-key refusal).
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01PxznqztpBUiRN4FH6pKGun
Fixes the `fmt · clippy (pinned)` CI job on #23 — the new crate had never been run through cargo fmt. No functional changes. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01PxznqztpBUiRN4FH6pKGun
…rag-system-3knaso
Master released v0.11.0 (workspace-wide version bump); fleischwolf-rag's path dependencies still required ^0.10.0, so the PR merge ref failed to resolve. Merged origin/master and aligned the pins. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01PxznqztpBUiRN4FH6pKGun
…naso' into claude/fleischwolf-rag-system-3knaso
…m warnings RAG_DOCUMENTS_OUTPUT (added to .env.example in 0dca270) is now implemented: every ingested document's parsed Markdown is mirrored into that folder with the source's directory structure preserved and `.md` appended to the original file name (report.pdf -> report.pdf.md; original .md inputs also get the suffix since conversion may reformat them). SourceRef carries a rel_path for this (folder/FTP/SFTP). Writes are best-effort and never fail ingestion; path components are sanitized so a hostile rel_path cannot escape the root. Fix the "corrupt deflate stream" WARN flood when ingesting real-world PDFs: the warnings come from lopdf's per-stream zlib decode (it skips such streams and continues; fleischwolf's PDF parse is unaffected) and surfaced only because the RAG CLI's tracing subscriber bridges `log` records. The default env filter is now "info,lopdf=error"; RUST_LOG still overrides. Verified on rag/data: one report emitted 486 warnings unfiltered, 0 filtered, and still ingested 131 pages / 400 chunks with a correct markdown dump. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01PxznqztpBUiRN4FH6pKGun
…rors Ingestion now runs as three overlapped stages joined by bounded channels: fleischwolf's convert_streaming emits Markdown as produced (per page for PDF) on a parser thread, a StreamingChunker cuts completed sections incrementally (at ATX headings outside code fences — chunks never cross headings, so batch and streaming chunking are byte-identical; property-tested at every split granularity), and an embed/insert worker embeds 64-chunk batches concurrently with parsing. Phase metrics record busy time per stage, so words/sec stays meaningful under overlap. A failed ingest now rolls back its document row and partial chunks (new VectorStore::delete_document on memory/sqlite/postgres) instead of leaving a hash that future runs would skip. Adds an info log when a document starts processing (uri, name, bytes) in addition to the completion log with metrics. `query` now synthesizes an LLM answer by default when OPENROUTER_API_KEY is set (sources listed below the answer); --chunks restores the plain list, and a hint is printed when no key is configured. LLM HTTP errors now include the provider's response body plus a hint for the common 401 (a native DeepSeek sk-... key needs OPENROUTER_BASE_URL=https://api.deepseek.com and RAG_LLM_MODEL=deepseek-chat; OpenRouter keys start with sk-or-). Documented in .env.example and the README. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01PxznqztpBUiRN4FH6pKGun
The per-document stats table now uses a fixed 65-char title column (ellipsized with "..." when longer, char-safe) and right-aligned numeric columns, so rows line up in the terminal regardless of title length. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01PxznqztpBUiRN4FH6pKGun
…h-safe ingest
Evaluation additions:
- `eval --from-md-dir DIR --questions FILE` assembles the dataset from the
RAG_DOCUMENTS_OUTPUT markdown mirror plus a questions file, so the sweep
runs on the real ingested corpus. Questions accept both the crate's
{query, relevant} shape and the QA-benchmark {text, kind} shape; entries
without `relevant` labels are skipped with a note.
- `eval --sizes 200,300 --overlaps 0,0.05` exposes the chunk-config matrix
(cross product); the default remains the three built-in pairs.
- New `answers` subcommand: runs an unlabelled {text, kind} benchmark through
the full RAG + LLM loop against the configured store and prints each answer
with source count and latency (Markdown or --json). Automates the
rag/questions.json performance check.
Interrupted-ingest hygiene (explains "duplicate" doc rows with no metrics):
a document row used to be inserted with its real content hash before
processing, so a killed run left a row that dedup then skipped forever.
Now the initial upsert carries a "pending:" sentinel hash — the real hash is
written only on success — and re-ingesting a source first deletes any stale
rows for the same URI (new VectorStore::delete_documents_by_source), which
also replaces old versions of changed files instead of accumulating them.
New `prune` subcommand removes incomplete rows left by older builds.
Documented eval/answers/prune usage, dataset format, and metric meanings in
the crate README.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01PxznqztpBUiRN4FH6pKGun
…rag-system-3knaso
Master's release automation bumps every sibling crate's exact version pin on each release (v0.11.0, v0.12.1 while this PR has been open), which repeatedly broke the PR merge-ref build: fleischwolf-rag required the previous exact version. Merge latest master and switch the fleischwolf / fleischwolf-pdf requirements to ">=0.12", which tolerates future release bumps; after merge, release.sh rewrites them to exact pins automatically (its sed matches any quoted version on `path = "../fleischwolf…"` lines). Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01PxznqztpBUiRN4FH6pKGun
…json
The questions loader now accepts `question` as an alias (alongside `query` /
`text`) and ignores extra fields, so `answers --json` output is directly
reusable as a questions file. The "no relevant labels" error now explains how
to annotate entries, and the README documents the intended round-trip:
answers -> review + add `relevant` snippets -> eval.
Annotate rag/questions.json with verified `relevant` labels (each snippet
grep-confirmed against the parsed markdown of the corresponding PDF):
Tradition "operating margin of 9.9%", Yellow Pages "Normal Course Issuer Bid"/
"871,135", Holley "Baer Brakes"/"Empower Ltd", Mercia "Enterprise Ventures
Group" — the latter shows the recorded LLM answer ("no mergers mentioned") was
a retrieval miss, exactly what eval is meant to catch.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01PxznqztpBUiRN4FH6pKGun
The 10-K incorporates executive compensation by reference to the 2023 Proxy
Statement — the dollar figures are not in the document, so the recorded LLM
answer ('cannot answer') was correct. The Item 11 passage citing the proxy
statement is the right retrieval evidence; '2023 Proxy Statement' is unique to
the CrossFirst document in this corpus. All 5 questions now score in eval.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01PxznqztpBUiRN4FH6pKGun
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Add a new workspace crate that builds a Retrieval-Augmented-Generation layer
on top of the document converter: source → convert to Markdown → chunk → embed
→ vector store → retrieve → (optional) LLM answer synthesis.
Every external dependency is a swappable trait:
context, section-bounded sliding window.
onnx-embed), and a deterministic hashing embedder for offline tests/eval.
postgres), in-memory. Documents and chunks live in separate tables.
(feature redis).
Includes an evaluation harness that sweeps a matrix of chunk sizes / overlaps /
retrieval modes over a labelled dataset and ranks configs by recall@k / MRR /
nDCG@k / latency, a clap-based CLI (init-db, ingest, query, eval, stats), and
config loaded from .env (see .env.example). The default feature set is fully
offline and unit-tested (33 tests); the heavier backends are real client code,
compile-checked but exercised only against live services.
Co-Authored-By: Claude Opus 4.8 noreply@anthropic.com
Claude-Session: https://claude.ai/code/session_01PxznqztpBUiRN4FH6pKGun