Educational project — not intended for production use. See the License & Disclaimer section for full details.
A benchmarking suite for evaluating and comparing different Retrieval-Augmented Generation (RAG) architectures against the FinanceBench dataset. The project spans a spectrum from a simple one-shot vector-search RAG all the way to a multi-step agentic RAG with reflection and playbook-guided retrieval, all backed by Neo4j as the graph/vector store and LangGraph for agentic orchestration.
- Project Overview
- Architecture
- Prerequisites
- Installation
- Configuration
- Dataset Setup
- Running Benchmarks
- Result Files
- Running Tests
- Docker
- Project Layout
- Notes on Benchmark Results
- License & Disclaimer
This repository was created to explore how different RAG strategies perform on a real-world financial Q&A benchmark. The research question driving it: does adding agentic reasoning, structured document representation, and iterative retrieval measurably improve answer quality compared to a plain vector-search baseline?
The included results (under data/financebench/results/) capture several rounds of experiments with progressively more sophisticated architectures, all evaluated against the same set of questions.
All implementations share the same interface defined in BasicRag (rag/basic_rag.py) and are instantiated through a factory:
from rag.rag_builder import rag_builder, RagType
rag = rag_builder(RagType.LEARNING_AGENTIC_RAG, config)RagType |
Class | Description |
|---|---|---|
SIMPLE_RAG |
SimpleRag |
Plain text splitting + vector similarity search |
STRUCTURED_RAG |
StructuredRag |
Docling-powered hierarchy/table/image extraction → Neo4j |
SEMI_STRUCTURED_RAG |
SemiStructuredRag |
Enhanced Docling chunking with OCR and document structure |
AGENTIC_RAG |
AgenticRag |
SemiStructuredRAG + LangGraph QuestionAnswering agent |
LEARNING_AGENTIC_RAG |
LearningAgenticRag |
Agentic RAG with reflection loops and playbook-guided retrieval |
Controlled via the RetrievalStrategy enum in rag/basic_rag.py:
JUST_TEXT— return only the matched chunk textTEXT_WITH_CONTEXT— include surrounding context (parent chunks, document metadata)TEXT_WITH_CONTEXT_EXTENDED— wider context windowTEXT_WITHOUT_CONTEXT_EXTENDED— extended retrieval without contextual expansion
The LearningAgenticRAG uses a LangGraph multi-step workflow:
Question → Retrieval Phase (with iterations) → Evaluation → Answering Phase (with iterations) → Answer
QuestionAnswering— main orchestrator that drives the retrieval → evaluation → answer loopPlaybookCurator— learns structured retrieval strategies across questions and stores them in Neo4j for reuse in subsequent queries
- Python ≥ 3.10, < 3.14
- Poetry for dependency management
- A running Neo4j instance (Community or Enterprise; version ≥ 5.x recommended) with the APOC plugin if you intend to use full-text indexes
- An LLM provider account — the default configuration uses Azure OpenAI but OpenAI, Anthropic, and Ollama are also supported
- (Optional) Docker, if you prefer running everything in a container
# Clone the repository
git clone <repo-url>
cd agenticrag-benchmark
# Install Poetry if you don't have it
pip install poetry
# Install all Python dependencies
poetry installCopy the provided example and fill in your values:
cp .env.example .envThen edit .env:
# ── LLM provider (Azure OpenAI default) ──────────────────────────────────────
AZURE_OPENAI_API_KEY="your-api-key"
AZURE_OPENAI_ENDPOINT="https://your-resource.cognitiveservices.azure.com/"
AZURE_OPENAI_DEPLOYMENT="gpt-4o-mini" # deployment / model name
AZURE_OPENAI_API_VERSION="2024-02-15-preview"
MODEL_NAME="azure" # azure | openai | anthropic | ollama
# ── Neo4j ─────────────────────────────────────────────────────────────────────
NEO4J_URI="neo4j://localhost:7687"
NEO4J_USER="neo4j"
NEO4J_PASSWORD="your-password"
NEO4J_DATABASE="neo4j" # optional, defaults to neo4j
NEO4J_ENCRYPTED="false"Alternative LLM providers — if you prefer OpenAI, Anthropic, or a local Ollama instance, update MODEL_NAME accordingly and supply the corresponding API key. See llms/models.py for the full provider logic.
The FinanceBench dataset files must be placed under data/financebench/:
data/financebench/
├── financebench_open_source.jsonl # Open-source Q&A pairs (~150 questions)
├── financebench_document_information.jsonl # Document metadata
├── financebench_extended_50questions.jsonl # Additional 50 curated questions
└── pdfs/ # PDF source documents (374 files)
The open-source JSONL files are available from the FinanceBench repository. The PDF files can be downloaded from the same source; place them all under data/financebench/pdfs/.
Note on the PDF set: The
pdfs/directory in this repository contains the full set of 374 documents referenced by FinanceBench. However, the benchmark runs reported indata/financebench/results/were executed only against the subset of documents referenced by the questions that were evaluated at the time. This means that for one-shot (non-agentic) RAG runs, having extra PDFs in the directory that were not specifically targeted can marginally affect retrieval precision — additional documents introduce more noise for vector similarity search. For the Agentic RAG variants, no measurable quality difference was observed when running with the full document set.
This is the primary entry point used to generate all the results stored in data/financebench/results/. It:
- Loads FinanceBench questions (open-source set + the extended 50 questions)
- Optionally builds Neo4j storage by parsing PDFs with Docling
- Runs every question through the configured RAG architecture
- Evaluates each answer with an LLM judge (comparing model answer vs. gold answer)
- Writes per-question results and an aggregated summary to CSV
Step 1 — Build the database (once)
The first time you run against a new Neo4j database you must parse the PDFs, chunk them, and populate the vector index. Set build_first: True in the configuration block:
architecture_config = {
"name": "My Experiment",
"rag_type": RagType.LEARNING_AGENTIC_RAG, # choose RAG type
"provider": "azure",
"model_name": "gpt-5-mini",
"eval_model_name": "gpt-5-mini",
"embedding_model": "bge-m3",
"database_name": "financebench.ablation", # Neo4j database name
"chunk_size": 8192,
"retrieval_strategy": RetrievalStrategy.TEXT_WITH_CONTEXT,
"build_first": True, # parse PDFs → chunk → index in Neo4j
"extract_metacognition": True,
"continue_previous_run": False,
}poetry run python evaluate_rag_implementation.pyThis step is slow — Docling parses every PDF and creates embeddings for all chunks. Once it completes you will not need to run it again unless you want to rebuild with different chunking parameters or a different embedding model.
Step 2 — Run the benchmark
After the database is built, switch build_first to False (and optionally enable continue_previous_run to resume an interrupted run):
"build_first": False,
"continue_previous_run": True, # resume from last results file if interruptedpoetry run python evaluate_rag_implementation.pyThe continue_previous_run flag makes the script find the most recent results CSV in the output directory and skip questions that already have an answer recorded, so you can safely stop and restart a long benchmark run.
An earlier, standalone version of the agentic benchmark loop. Useful for quick experiments with a single agentic configuration without the extra infrastructure of evaluate_rag_implementation.py.
poetry run python evaluate_agentic_rag.pyAll results land in data/financebench/results/, organised by test run:
results/
├── test_1_basic_rag/ # Initial one-shot RAG experiments
├── test_1_agentic_rag/ # Initial agentic RAG experiments
├── test_2_basic_rag/
├── test_2_agentic_rag/
├── test_3_agentic_rag/
├── test_4_agentic_rag/
├── test_5_agentic_rag_ablation/ # Ablation studies
└── ...
Each directory contains:
| File pattern | Content |
|---|---|
evaluation_results_<timestamp>.csv |
One row per question: id, question, gold answer, model answer, is_correct, retrieval/answering iterations |
evaluation_summary_<timestamp>.csv |
Aggregate stats: accuracy, average iteration counts, broken down by question type |
Integration tests require a running Neo4j instance pre-populated with graph data:
# Run the StructuredRAG integration test
poetry run pytest -m integration tests/test_structured_rag_answer.py -s
# Run the full test suite
poetry run pytestdocker build -t agenticrag-benchmark .
docker run -it --rm -p 8000:80 \
--env-file .env \
--name agenticrag-benchmark \
agenticrag-benchmarkTo run a benchmark inside the container you will need to mount the data directory and ensure the container can reach your Neo4j instance.
agenticrag-benchmark/
├── agent/ # LangGraph agentic components
│ ├── question_answering.py # Main QA orchestrator
│ ├── question_answering_graphs.py
│ ├── question_answering_nodes.py
│ ├── retrieval_nodes.py
│ ├── retriever_tools.py
│ └── playbook_curator/ # Playbook-based retrieval learning
├── app/ # FastAPI application
├── data/financebench/ # Dataset, PDFs and benchmark results
├── llms/
│ └── models.py # Multi-provider LLM factory
├── rag/
│ ├── basic_rag.py # Abstract base + RetrievalStrategy enum
│ ├── simple_rag.py
│ ├── structured_rag.py
│ ├── semi_structured_rag.py
│ ├── agentic_rag.py
│ ├── learning_agentic_rag.py
│ └── rag_builder.py # Factory (RagType enum + rag_builder())
├── schemas/ # Pydantic / dataclass schemas
├── services/
│ ├── data_loader.py # FinanceBench dataset loader
│ ├── rag_evaluator.py # LLM-based answer grader
│ ├── neo4j_service.py # Neo4j driver wrapper
│ ├── context_graph.py # Q&A step logging to Neo4j
│ └── playbook.py # Playbook generation service
├── tests/
├── utils/
│ ├── config.py
│ └── logger.py
├── evaluate_rag_implementation.py # Primary benchmark entry point
├── evaluate_agentic_rag.py # Standalone agentic benchmark
├── compare_results.py # Utility to compare CSV result files
├── pyproject.toml
├── Dockerfile
└── .env.example
- All results under
data/financebench/results/were produced with Azure OpenAI (gpt-5-mini) as both the answering and evaluation model. - The Neo4j vector index was built with
bge-m3embeddings at a chunk size of 8192 tokens. - Agentic runs use a
LEARNING_AGENTIC_RAGconfiguration with playbook curation enabled. The playbook is accumulated within a single run; it is not seeded from prior runs unless the same Neo4j database is reused. - The
is_correctcolumn is determined by an LLM judge, not exact string matching, so results carry the inherent variability of LLM evaluation. - For reproducible comparisons across runs use the same
database_nameand avoid rebuilding the index between runs.
MIT License — with additional disclaimers
Copyright (c) 2025 GraphAware
Permission is hereby granted, free of charge, to any person obtaining a copy
of this software and associated documentation files (the "Software"), to deal
in the Software without restriction, including without limitation the rights
to use, copy, modify, merge, publish, distribute, sublicense, and/or sell
copies of the Software, and to permit persons to whom the Software is
furnished to do so, subject to the following conditions:
The above copyright notice and this permission notice shall be included in all
copies or substantial portions of the Software.
THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR
IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,
FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE
AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER
LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM,
OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE
SOFTWARE.
Educational and research use only. This project is released exclusively for educational and research purposes. It is a benchmark and experimentation harness, not a production-ready system. It has not been hardened for security, has not undergone production-grade testing, and must not be deployed in any environment where reliability, availability, or data integrity are required.
No production use. The authors expressly disclaim any responsibility for damages, data loss, security incidents, financial loss, or any other harm arising from the use or misuse of this software in a production or commercial setting.
LLM costs. Running the full benchmark suite makes a significant number of LLM API calls. You are solely responsible for any costs incurred with your LLM provider.
Third-party data. The FinanceBench dataset is subject to its own license terms. Refer to the FinanceBench repository for details. The PDF documents included in or referenced by the dataset are subject to the copyright of their respective owners.
No guarantees on accuracy. Benchmark results reflect the configuration and data used at the time of the run. They should not be interpreted as definitive claims about the accuracy or capability of any RAG architecture or LLM.