Upload documents. Ask questions. Watch 4 AI agents think in parallel, debate each other's findings, and synthesize a final answer — all streamed live.
Every RAG system works the same way: embed query → find top chunks → stuff them into one LLM call → get an answer. That one LLM call has to juggle everything at once — it doesn't deeply reason about each source, it averages them.
AgentRAG does it differently.
Each retrieved chunk gets its own dedicated AI agent with an isolated context window. Those agents think in parallel, cross-examine each other's findings, and a synthesis agent compiles everything into a final answer. It's the difference between asking one person to read four documents and asking four domain experts to each study one document, then convene.
Your Query
│
├─ Is it a greeting? ──► Direct friendly reply (no council overhead)
│
└─ Is it a follow-up? ──► Query expansion rewrites "tell me more"
into a self-contained question
│
▼
gemini-embedding-2-preview
(1536-dim semantic search)
│
▼
ChromaDB vector store
(top 4 chunks retrieved)
│
┌───────────────────┼───────────────────┐
▼ ▼ ▼
Agent Alpha Agent Beta Agent Gamma ... Agent Delta
[Chunk #1] [Chunk #2] [Chunk #3] [Chunk #4]
│ │ │ │
└───────── all fire SIMULTANEOUSLY ─────────────────┘
answers stream as each finishes
│
Orchestrator
(cross-examination round,
also parallel)
│
Synthesis Agent
(streams final answer live,
with citations + history context)
Everything streams to a live Council Panel in the UI as it happens.
Upload a document → ask a question → watch the council deliberate in real-time
Council Panel shows each agent responding as they finish — not in order, but as the fastest one completes first. You can watch the orchestrator question them, run cross-examination, and hand off to synthesis.
Prerequisites: Python 3.11+, Node 18+, Google AI Studio API key (free tier works)
git clone https://github.com/nileshpatil6/multiagent-council-rag.git
cd multiagent-council-rag
echo "GOOGLE_API_KEY=your_key_here" > .env
bash scripts/start.shgit clone https://github.com/nileshpatil6/multiagent-council-rag.git
cd multiagent-council-rag
echo GOOGLE_API_KEY=your_key_here > .env
scripts\start.batExpand for step-by-step
Backend
cd backend
python -m venv .venv
# Windows
.venv\Scripts\Activate.ps1
# macOS / Linux
source .venv/bin/activate
pip install -r requirements.txt
uvicorn main:app --port 8001 --reloadFrontend (new terminal)
cd frontend
npm install
npm run devNo LangChain agents, no LlamaIndex, no CrewAI, no AutoGen. Every "agent" is a plain Python class holding a raw Gemini chat session. The "council" is structured prompting + asyncio.as_completed. Frameworks would add abstraction without adding value here — and would make the SSE streaming much harder to control precisely.
All 4 agents receive their question simultaneously. Total latency = slowest single agent, not the sum. Same for cross-examination rounds. This uses asyncio.as_completed so responses appear in the UI as each finishes.
Most RAG systems pass history only to generation. AgentRAG rewrites the query itself before embedding — so "tell me more about that" becomes "tell me more about backpropagation in neural network training" before hitting the vector store. This fixes the root cause of follow-up failure.
Each agent only sees its own chunk. It cannot hallucinate about content in other chunks. The orchestrator synthesizes across agents — no single model sees everything at once.
| Feature | Description |
|---|---|
| Multi-agent council | Each chunk gets its own Gemini chat session |
| Parallel execution | All agents fire simultaneously, fastest appears first |
| Real-time SSE streaming | Council events stream live to a dedicated panel |
| History-aware retrieval | Follow-up queries rewritten before embedding |
| Casual bypass | Greetings skip the council entirely |
| Document management | Upload PDF, DOCX, TXT with drag-and-drop |
| Persistent storage | ChromaDB survives restarts, no re-uploading needed |
| Source citations | Final answer links back to source chunks |
| Professional UI | Light theme, 3-panel layout, agent color coding |
Create .env in the project root (copy from backend/.env.example):
| Variable | Default | Description |
|---|---|---|
GOOGLE_API_KEY |
— | Required. Free at aistudio.google.com |
CHAT_MODEL |
gemini-flash-latest |
Gemini model for all agents |
EMBEDDING_MODEL |
gemini-embedding-2-preview |
Embedding model |
EMBEDDING_DIMENSIONS |
1536 |
Vector size (128–3072) |
TOP_K_CHUNKS |
4 |
Chunks retrieved per query = number of agents |
CHUNK_SIZE |
800 |
Characters per chunk |
CHUNK_OVERLAP |
150 |
Overlap between adjacent chunks |
MAX_COUNCIL_ROUNDS |
2 |
1 = direct answers only · 2 = + cross-examination |
BACKEND_PORT |
8001 |
FastAPI port |
longmemory-multiagent/
├── backend/
│ ├── main.py # FastAPI app + lifespan
│ ├── config.py # .env → typed settings
│ ├── requirements.txt
│ ├── agents/
│ │ ├── chunk_agent.py # Isolated Gemini chat per chunk
│ │ ├── orchestrator_agent.py # Parallel council + query expansion
│ │ └── synthesis_agent.py # Streaming final answer
│ ├── api/
│ │ ├── chat.py # SSE endpoint + casual bypass
│ │ └── documents.py # Upload / list / delete
│ ├── services/
│ │ ├── embedding_service.py # Async Gemini embedding wrapper
│ │ ├── vector_store.py # ChromaDB CRUD
│ │ └── document_parser.py # PDF / DOCX / TXT → chunks
│ └── models/
│ └── schemas.py # Pydantic models + SSE event types
│
├── frontend/src/
│ ├── stores/appStore.ts # Zustand global state
│ ├── api/{chat,documents}.ts # Typed fetch + SSE consumer
│ └── components/
│ ├── layout/Sidebar.tsx # Upload zone + document list
│ ├── chat/ # Chat window + message bubbles
│ └── council/ # Live agent feed panel
│
├── scripts/
│ ├── start.sh # One-command startup (macOS/Linux)
│ └── start.bat # One-command startup (Windows)
│
├── .github/
│ ├── workflows/ci.yml # CI: backend imports + TS type check
│ └── ISSUE_TEMPLATE/
└── .env.example
The /api/chat/query endpoint returns a stream of typed JSON events:
council_open
orchestrator_speaks ← "All 4 agents now thinking simultaneously..."
agent_speaks (×4) ← appear as each agent finishes, any order
orchestrator_speaks ← cross-examination intro
agent_speaks (×N) ← cross-exam responses, also parallel
orchestrator_speaks ← handing off to synthesis
synthesis_start
synthesis_token (×many) ← streamed word by word
synthesis_complete ← includes citations array
council_close
Each event is a standard SSE data: line with a JSON payload. The type field discriminates the event shape.
GET /health
GET /api/documents/list
POST /api/documents/upload multipart/form-data, field: file
DEL /api/documents/{id}
POST /api/chat/query { query: string, history: [{role, text}] }
Factual, structured documents work best. Free downloads from Project Gutenberg:
- On the Origin of Species — Darwin — dense argument structure, great for follow-ups
- The Wealth of Nations — Adam Smith — very long, rich in economic concepts
- War and Peace — Tolstoy — 400+ chunks, stress-tests parallel retrieval
- Support more file types (EPUB, Markdown, HTML)
- Configurable number of agents (currently fixed at TOP_K_CHUNKS)
- Agent memory across sessions (long-term memory layer)
- Docker Compose setup
- Evaluation metrics for council answer quality
- Support for image/diagram understanding (gemini-embedding-2-preview is multimodal)
Pull requests are welcome. For significant changes, open an issue first to discuss.
# Type check
cd frontend && npx tsc --noEmit
# CI runs automatically on pushSee CONTRIBUTING.md for more details.
MIT © 2026 nileshpatil6
If AgentRAG was useful, consider giving it a star — it helps others find the project.