RAG System with OpenWebUI and Ollama

A complete Retrieval-Augmented Generation (RAG) system using Docker Compose with OpenWebUI, Ollama, and a custom RAG pipeline for document question answering. This system was developed as a summer research project under Professor Michael O'Leary at Towson University for use with his Case Studies class textbook. The RAG system is specifically designed to help students in the cybersecurity track interact with course materials through intelligent question-answering capabilities. The included FAISS index file is built from Professor O'Leary's textbook and optimized for cybersecurity case study content. While the system includes tools to generate custom indexes from any PDF document, the primary use case is educational support for students enrolled in the Case Studies course. We encourage supporting Professor O'Leary's work by purchasing his textbook through official channels. Please note that the FAISS index file is provided solely for educational purposes within the course context. Any attempts to reverse engineer the index file with the intent to reconstruct or distribute the underlying textbook content would be inappropriate and potentially violate intellectual property rights. The system is designed to enhance learning while respecting the author's work.

Quick Start
Project Structure
Setup Instructions
Configuration
Daily Usage
Generating FAISS Index from PDF
Advanced Configuration
Support

Quick Start

# Clone or create project directory
cd pipeline_container
# remove existing volume 
docker volume rm pipeline_container_data
# Start the system
docker compose up -d

# Wait for Ollama to start (about 30 seconds)
# Then pull the required model
docker exec ollama ollama pull llama3.2:3b

# Access the web interface
# Open http://localhost:3000 in your browser
# Click signup if avalible and make account info(Reamber this)
# If account is required use the info below
# username: zathras@askzathras.com
# password: Password1!

Project Structure

rag-system/
├── pipelines/
│   └── rag_pipeline.py          # Custom RAG pipeline
├── data/                        # OpenWebUI data (auto-created)
├── faiss_index_/               # Your FAISS vector index
│   ├── index.faiss
│   ├── index.pkl
│   └── (other index files)
├── content/                     # Source documents for indexing
│   └── book.pdf                 # Your PDF document
├── requirements.txt             # Python dependencies
├── docker compose.yml          # Docker services configuration
├── create_index.py             # Script to generate FAISS index
└── README.md                   # This file

Setup Instructions

Prerequisites

Docker Desktop installed and running
Your FAISS index files ready (or PDF to generate index from)

Step 1: Start the System

# Start all containers
docker compose up -d

# Check that all containers are running
docker compose ps

Step 2: Setup Ollama Model

# Wait for Ollama to be ready (about 30 seconds)
# Then pull the language model
docker exec ollama ollama pull llama3.2:3b

# Verify the model is installed
docker exec ollama ollama list

Step 3: Access the Interface

Open your browser and go to http://localhost:3000
Sign in with zathras account in OpenWebUI
- zathras@askzathras.com
- Password1!
Select "RAG Pipeline" from the model dropdown
Start asking questions about your documents!

Configuration

Pipeline Settings

You can modify the RAG pipeline settings in rag_pipeline.py or in the pipeline section of the admin panel:

embedding_model_name: The sentence transformer model for embeddings
faiss_index_path: Path to your FAISS index (default: /app/faiss_index_)
ollama_model: The Ollama model to use (default: llama3.2:3b)
retrieval_k: Number of documents to retrieve (default: 4)
score_threshold: Minimum similarity score (default: 0.7)

Docker Services

OpenWebUI: http://localhost:3000 - Web interface
Pipelines: http://localhost:9099 - RAG pipeline API
Ollama: http://localhost:11434 - Language model API

Daily Usage

Starting the System

docker compose up -d

Stopping the System

docker compose down

Restarting (keeps all data)

docker compose restart

Checking Status

# View running containers
docker compose ps

# View logs
docker compose logs pipelines
docker compose logs ollama

Generating FAISS Index from PDF

The container system includes a pre-generated FAISS index for demonstration purposes. However, you can create your own index from any PDF document.

Prerequisites for Index Generation

Python 3.8+ installed locally
Required Python packages (see requirements below)

Step 1: Install Dependencies

    pip install -r requirements.txt

pip install langchain-ollama langchain langchain-community langchain-huggingface faiss-cpu sentence-transformers pypdf

Step 2: Prepare Your Document

Create a content/ directory in your project folder
Place your PDF file in the directory (e.g., content/your-document.pdf)
Update the file path in the index generation script

Step 3: Modify the Index Generation Script

Update the document path in create_index.py:

# Change this line to point to your PDF file
loader = PyPDFLoader("./content/your-document.pdf")

Step 4: Generate the Index

# Run the index generation script
python create_index.py

The script will:

Load and parse your PDF document
Split the text into chunks
Generate embeddings for each chunk
Create and save the FAISS index to faiss_index_/
Run diagnostic tests to verify the index

Changing Document Source

To use a different document:

Replace the PDF: Place your new PDF in the content/ directory
Update the path: Modify the file path in create_index.py
Regenerate index: Run python create_index.py
Restart containers: Run docker compose restart pipelines

Important Note: The current system includes a pre-generated index for demonstration. You must replace this with your own document and regenerate the index, as the original document is not publicly available.

GPU Acceleration (Linux Only)

To use CUDA for faster embedding generation on Linux systems:

Install CUDA-enabled packages:

pip uninstall faiss-cpu
pip install faiss-gpu

Update device setting in create_index.py:

# Change this line
model_kwargs = {"device": "cuda"}  # was "cpu"

Ensure NVIDIA Docker support is enabled in your Docker setup

Customizing Index Generation

You can modify several parameters in create_index.py:

# Text splitting parameters
text_splitter = RecursiveCharacterTextSplitter(
    chunk_size=400,        # Adjust chunk size
    chunk_overlap=50,      # Adjust overlap between chunks
    separators=["\n\n", "\n", " ", ""]
)

# Embedding model selection
embedding_model_name = "sentence-transformers/all-MiniLM-L6-v2"  # Change model

# Retrieval parameters
retriever = vectorstore.as_retriever(
    search_type="similarity",
    search_kwargs={"k": 4}  # Number of chunks to retrieve
)

Testing Your Generated Index

The index generation script includes comprehensive diagnostics:

Document parsing verification: Checks if PDF loaded correctly
Text splitting analysis: Verifies chunk sizes and content
Embedding creation: Tests vector store functionality
Retrieval testing: Runs sample queries against your content

Common Issues

Empty pages detected: Some PDFs have formatting that creates empty pages - this is usually normal
Short chunks: Very short chunks might indicate formatting issues in the source PDF
CUDA errors: Ensure you have compatible NVIDIA drivers and CUDA toolkit installed
Memory issues: Large documents may require more RAM - consider reducing chunk_size

Advanced Configuration

Using Different Models

To use a different Ollama model:

Pull the model: docker exec ollama ollama pull <model-name>
Update ollama_model in rag_pipeline.py or admin panel
Restart: docker compose restart pipelines

GPU Support for Ollama

To enable GPU support for Ollama, uncomment the GPU section in docker compose.yml:

deploy:
  resources:
    reservations:
      devices:
        - driver: nvidia
          count: 1
          capabilities: [gpu]

Proxy Setup

Some networks force traffic through a proxy. When that happens, the system may fail with errors like:

Ollama network problem
DNS lookup failure for: ollama
RAG pipeline returning HTML / Proxy Error / 502

This means the proxy is intercepting container-to-container traffic. Use the steps below.

1. Use Your Host IP Instead of `ollama` / `pipelines`

In OpenWebUI → Settings → Connections:

Ollama API Base URL:
http://YOUR-HOST-IP:11434
OpenAI API Base URL (pipelines):
http://YOUR-HOST-IP:9099

Example:
http://172.16.202.47:11434
http://172.16.202.47:9099

Using IP avoids proxy DNS hijacking and will allow traffic to be routed.

2. Fix the RAG Pipeline (rag_pipeline.py)

The pipeline defaults to http://ollama:11434, which breaks behind proxies.

Edit pipelines/rag_pipeline.py:

import os

OLLAMA_BASE_URL = os.getenv(
    "OLLAMA_BASE_URL",
    "http://YOUR-HOST-IP:11434"  # set your IP here
)

llm = ChatOllama(  # or OllamaChatCompletion
    model="llama3.2:3b",
    base_url=OLLAMA_BASE_URL,
)

3.Add NO_PROXY Rules to docker-compose.yml (Optional)

add to docker-compose.yml for each service

environment:
 - OLLAMA_BASE_URL=http://YOUR-HOST-IP:11434
 - NO_PROXY=ollama,pipelines,open-webui,localhost,127.0.0.1,YOUR-HOST-IP
 - no_proxy=ollama,pipelines,open-webui,localhost,127.0.0.1,YOUR-HOST-IP

Support

Logs and Debugging

# View all logs
docker compose logs

# View specific service logs
docker compose logs pipelines
docker compose logs ollama
docker compose logs open-webui

# Follow logs in real-time
docker compose logs -f pipelines

No Ollama models in OpenWebUI

docker exec ollama ollama list
docker exec -it open-webui bash -lc 'curl -s http://YOUR-HOST-IP:11434/api/tags'

Pipelines Not Detected

docker exec -it pipelines bash -lc 'ls /app/pipelines'
curl -s -H "Authorization: Bearer 0p3n-w3bu!" http://YOUR-HOST-IP:9099/pipelines
docker compose logs --tail=200 pipelines

Clean Reset

# Stop and remove everything (keeps volumes)
docker compose down

# Remove volumes too (complete reset)
docker compose down -v

# Rebuild and start fresh
docker compose up -d

FilesExpand file tree

README.md

Latest commit

History