Skip to content

rag-guide.md

Latest commit

 

History

History
117 lines (95 loc) · 4.59 KB

rag-guide.md

File metadata and controls

117 lines (95 loc) · 4.59 KB
Raw
title Retrieval-augmented generation (RAG)

Build a semantic knowledge base and answer questions grounded in it, using the ai/embed connector, a pgvector table, and ai/completion. Two example workflows tie it together:

  • rag-ingest — embed a document and store it in the vector store.
  • rag-ask — embed a question, find the nearest documents by cosine distance, and have an LLM answer grounded in them (with citations).

This is semantic retrieval — matches are by meaning, not keywords. For a simpler keyword-only variant with no extensions, see the office-assistant MVP, which uses Postgres full-text search.

Prerequisites

  • Credentials: an openai credential (referenced as openai) for ai/embed and ai/completion, and a postgres credential (referenced as kb-db) for the vector store.
  • A Postgres database with the pgvector extension. Apply the schema from rag-kb-schema.sql:
psql "$KB_DATABASE_URL" -f packages/site/src/content/examples/rag-kb-schema.sql

Apply both workflows:

mantle apply packages/site/src/content/examples/rag-ingest.yaml
mantle apply packages/site/src/content/examples/rag-ask.yaml

How it works

ai/embed returns each embedding as a pgvector text literal (output.vector, e.g. "[0.1,0.2,...]"). The kb/upsert and kb/query connectors take that literal and handle the pgvector SQL for you — the ::vector casts, the cosine-distance operator, ORDER BY/LIMIT, JSONB metadata, and multi-row inserts:

# ingest — store content + embedding + metadata
- action: kb/upsert
  credential: kb-db
  params:
    table: kb_documents
    content: "{{ inputs.content }}"
    vector: "{{ steps['embed'].output.vector }}"
    metadata: { title: "{{ inputs.title }}", source: "{{ inputs.source }}" }
    conflict_target: dedupe_key   # idempotent re-ingest

# ask — nearest-neighbour search
- action: kb/query
  credential: kb-db
  params:
    table: kb_documents
    vector: "{{ steps['embed-question'].output.vector }}"
    columns: [content, metadata]
    top_k: 5

kb/query returns the requested columns plus a distance field (lower = closer; for cosine, similarity = 1 - distance). The retrieved rows go to ai/completion as JSON context, and the model answers using only those passages.

kb/* are thin sugar — they run against a pgvector database you provide (the step credential) and don't manage schema, so you still create the table yourself. If you'd rather write the SQL directly, postgres/query works too (that's what these connectors generate).

Ingesting documents

mantle run rag-ingest \
  --input title="Client C integration" \
  --input source="confluence/12345" \
  --input content="Team A connects to Client C via the X integration; Team B uses the direct API."

rag-ingest splits the document into overlapping chunks with text/chunk, embeds them all in one ai/embed call, and stores them in one kb/upsert. A single metadata object is broadcast to every chunk. Re-ingesting identical content is a no-op (the schema dedupes on md5(content)).

Asking questions

mantle run rag-ask \
  --input question="Who works with Client C and how?" \
  --output json

The answer step's output.text is the grounded response (the CLI prints step outputs with --output json or -v).

Notes and limitations

  • Embedding dimension must match the model. The schema uses vector(1536) for text-embedding-3-small; use vector(3072) for text-embedding-3-large. Embed queries and documents with the same model.
  • Provider support: ai/embed supports openai (and OpenAI-compatible endpoints like Azure OpenAI or local servers via base_url) and bedrock with the Amazon Titan text-embedding models (amazon.titan-embed-text-v2:0, amazon.titan-embed-text-v1). To use Bedrock, set provider: bedrock, a region, and an aws credential; adjust the schema's vector(...) dimension to match (Titan v2 defaults to 1024). Cohere Bedrock models are a follow-up.
  • You provide the vector store. kb/* run against a pgvector database you create and point a credential at; they don't provision the extension or table.
  • Chunking is fixed-size. text/chunk splits by a sliding window over characters or words with overlap. Semantic/recursive (separator-aware or token-accurate) chunking and metadata filtering on kb/query are tracked by #153.