first commit

Author: Meetmendapara09

.gitignore

@@ -0,0 +1,21 @@
+# Python-generated files
+__pycache__/
+*.py[oc]
+build/
+dist/
+wheels/
+*.egg-info
+
+# Virtual environments
+.venv
+
+# Keys and secrets
+.env
+
+# Data files
+data/
+
+# Frontend
+frontend/node_modules/
+frontend/dist/
+frontend/.vite/

.python-version

@@ -0,0 +1 @@
+3.10

CLAUDE.md

@@ -0,0 +1,166 @@
+# CLAUDE.md - Technical Notes for LLM Council
+
+This file contains technical details, architectural decisions, and important implementation notes for future development sessions.
+
+## Project Overview
+
+LLM Council is a 3-stage deliberation system where multiple LLMs collaboratively answer user questions. The key innovation is anonymized peer review in Stage 2, preventing models from playing favorites.
+
+## Architecture
+
+### Backend Structure (`backend/`)
+
+**`config.py`**
+- Contains `COUNCIL_MODELS` (list of OpenRouter model identifiers)
+- Contains `CHAIRMAN_MODEL` (model that synthesizes final answer)
+- Uses environment variable `OPENROUTER_API_KEY` from `.env`
+- Backend runs on **port 8001** (NOT 8000 - user had another app on 8000)
+
+**`openrouter.py`**
+- `query_model()`: Single async model query
+- `query_models_parallel()`: Parallel queries using `asyncio.gather()`
+- Returns dict with 'content' and optional 'reasoning_details'
+- Graceful degradation: returns None on failure, continues with successful responses
+
+**`council.py`** - The Core Logic
+- `stage1_collect_responses()`: Parallel queries to all council models
+- `stage2_collect_rankings()`:
+  - Anonymizes responses as "Response A, B, C, etc."
+  - Creates `label_to_model` mapping for de-anonymization
+  - Prompts models to evaluate and rank (with strict format requirements)
+  - Returns tuple: (rankings_list, label_to_model_dict)
+  - Each ranking includes both raw text and `parsed_ranking` list
+- `stage3_synthesize_final()`: Chairman synthesizes from all responses + rankings
+- `parse_ranking_from_text()`: Extracts "FINAL RANKING:" section, handles both numbered lists and plain format
+- `calculate_aggregate_rankings()`: Computes average rank position across all peer evaluations
+
+**`storage.py`**
+- JSON-based conversation storage in `data/conversations/`
+- Each conversation: `{id, created_at, messages[]}`
+- Assistant messages contain: `{role, stage1, stage2, stage3}`
+- Note: metadata (label_to_model, aggregate_rankings) is NOT persisted to storage, only returned via API
+
+**`main.py`**
+- FastAPI app with CORS enabled for localhost:5173 and localhost:3000
+- POST `/api/conversations/{id}/message` returns metadata in addition to stages
+- Metadata includes: label_to_model mapping and aggregate_rankings
+
+### Frontend Structure (`frontend/src/`)
+
+**`App.jsx`**
+- Main orchestration: manages conversations list and current conversation
+- Handles message sending and metadata storage
+- Important: metadata is stored in the UI state for display but not persisted to backend JSON
+
+**`components/ChatInterface.jsx`**
+- Multiline textarea (3 rows, resizable)
+- Enter to send, Shift+Enter for new line
+- User messages wrapped in markdown-content class for padding
+
+**`components/Stage1.jsx`**
+- Tab view of individual model responses
+- ReactMarkdown rendering with markdown-content wrapper
+
+**`components/Stage2.jsx`**
+- **Critical Feature**: Tab view showing RAW evaluation text from each model
+- De-anonymization happens CLIENT-SIDE for display (models receive anonymous labels)
+- Shows "Extracted Ranking" below each evaluation so users can validate parsing
+- Aggregate rankings shown with average position and vote count
+- Explanatory text clarifies that boldface model names are for readability only
+
+**`components/Stage3.jsx`**
+- Final synthesized answer from chairman
+- Green-tinted background (#f0fff0) to highlight conclusion
+
+**Styling (`*.css`)**
+- Light mode theme (not dark mode)
+- Primary color: #4a90e2 (blue)
+- Global markdown styling in `index.css` with `.markdown-content` class
+- 12px padding on all markdown content to prevent cluttered appearance
+
+## Key Design Decisions
+
+### Stage 2 Prompt Format
+The Stage 2 prompt is very specific to ensure parseable output:
+```
+1. Evaluate each response individually first
+2. Provide "FINAL RANKING:" header
+3. Numbered list format: "1. Response C", "2. Response A", etc.
+4. No additional text after ranking section
+```
+
+This strict format allows reliable parsing while still getting thoughtful evaluations.
+
+### De-anonymization Strategy
+- Models receive: "Response A", "Response B", etc.
+- Backend creates mapping: `{"Response A": "openai/gpt-5.1", ...}`
+- Frontend displays model names in **bold** for readability
+- Users see explanation that original evaluation used anonymous labels
+- This prevents bias while maintaining transparency
+
+### Error Handling Philosophy
+- Continue with successful responses if some models fail (graceful degradation)
+- Never fail the entire request due to single model failure
+- Log errors but don't expose to user unless all models fail
+
+### UI/UX Transparency
+- All raw outputs are inspectable via tabs
+- Parsed rankings shown below raw text for validation
+- Users can verify system's interpretation of model outputs
+- This builds trust and allows debugging of edge cases
+
+## Important Implementation Details
+
+### Relative Imports
+All backend modules use relative imports (e.g., `from .config import ...`) not absolute imports. This is critical for Python's module system to work correctly when running as `python -m backend.main`.
+
+### Port Configuration
+- Backend: 8001 (changed from 8000 to avoid conflict)
+- Frontend: 5173 (Vite default)
+- Update both `backend/main.py` and `frontend/src/api.js` if changing
+
+### Markdown Rendering
+All ReactMarkdown components must be wrapped in `<div className="markdown-content">` for proper spacing. This class is defined globally in `index.css`.
+
+### Model Configuration
+Models are hardcoded in `backend/config.py`. Chairman can be same or different from council members. The current default is Gemini as chairman per user preference.
+
+## Common Gotchas
+
+1. **Module Import Errors**: Always run backend as `python -m backend.main` from project root, not from backend directory
+2. **CORS Issues**: Frontend must match allowed origins in `main.py` CORS middleware
+3. **Ranking Parse Failures**: If models don't follow format, fallback regex extracts any "Response X" patterns in order
+4. **Missing Metadata**: Metadata is ephemeral (not persisted), only available in API responses
+
+## Future Enhancement Ideas
+
+- Configurable council/chairman via UI instead of config file
+- Streaming responses instead of batch loading
+- Export conversations to markdown/PDF
+- Model performance analytics over time
+- Custom ranking criteria (not just accuracy/insight)
+- Support for reasoning models (o1, etc.) with special handling
+
+## Testing Notes
+
+Use `test_openrouter.py` to verify API connectivity and test different model identifiers before adding to council. The script tests both streaming and non-streaming modes.
+
+## Data Flow Summary
+
+```
+User Query
+    ↓
+Stage 1: Parallel queries → [individual responses]
+    ↓
+Stage 2: Anonymize → Parallel ranking queries → [evaluations + parsed rankings]
+    ↓
+Aggregate Rankings Calculation → [sorted by avg position]
+    ↓
+Stage 3: Chairman synthesis with full context
+    ↓
+Return: {stage1, stage2, stage3, metadata}
+    ↓
+Frontend: Display with tabs + validation UI
+```
+
+The entire flow is async/parallel where possible to minimize latency.

README.md

@@ -0,0 +1,87 @@
+# LLM Council
+
+![llmcouncil](header.jpg)
+
+The idea of this repo is that instead of asking a question to your favorite LLM provider (e.g. OpenAI GPT 5.1, Google Gemini 3.0 Pro, Anthropic Claude Sonnet 4.5, xAI Grok 4, eg.c), you can group them into your "LLM Council". This repo is a simple, local web app that essentially looks like ChatGPT except it uses OpenRouter to send your query to multiple LLMs, it then asks them to review and rank each other's work, and finally a Chairman LLM produces the final response.
+
+In a bit more detail, here is what happens when you submit a query:
+
+1. **Stage 1: First opinions**. The user query is given to all LLMs individually, and the responses are collected. The individual responses are shown in a "tab view", so that the user can inspect them all one by one.
+2. **Stage 2: Review**. Each individual LLM is given the responses of the other LLMs. Under the hood, the LLM identities are anonymized so that the LLM can't play favorites when judging their outputs. The LLM is asked to rank them in accuracy and insight.
+3. **Stage 3: Final response**. The designated Chairman of the LLM Council takes all of the model's responses and compiles them into a single final answer that is presented to the user.
+
+## Vibe Code Alert
+
+This project was 99% vibe coded as a fun Saturday hack because I wanted to explore and evaluate a number of LLMs side by side in the process of [reading books together with LLMs](https://x.com/karpathy/status/1990577951671509438). It's nice and useful to see multiple responses side by side, and also the cross-opinions of all LLMs on each other's outputs. I'm not going to support it in any way, it's provided here as is for other people's inspiration and I don't intend to improve it. Code is ephemeral now and libraries are over, ask your LLM to change it in whatever way you like.
+
+## Setup
+
+### 1. Install Dependencies
+
+The project uses [uv](https://docs.astral.sh/uv/) for project management.
+
+**Backend:**
+```bash
+uv sync
+```
+
+**Frontend:**
+```bash
+cd frontend
+npm install
+cd ..
+```
+
+### 2. Configure API Key
+
+Create a `.env` file in the project root:
+
+```bash
+OPENROUTER_API_KEY=sk-or-v1-...
+```
+
+Get your API key at [openrouter.ai](https://openrouter.ai/). Make sure to purchase the credits you need, or sign up for automatic top up.
+
+### 3. Configure Models (Optional)
+
+Edit `backend/config.py` to customize the council:
+
+```python
+COUNCIL_MODELS = [
+    "openai/gpt-5.1",
+    "google/gemini-3-pro-preview",
+    "anthropic/claude-sonnet-4.5",
+    "x-ai/grok-4",
+]
+
+CHAIRMAN_MODEL = "google/gemini-3-pro-preview"
+```
+
+## Running the Application
+
+**Option 1: Use the start script**
+```bash
+./start.sh
+```
+
+**Option 2: Run manually**
+
+Terminal 1 (Backend):
+```bash
+uv run python -m backend.main
+```
+
+Terminal 2 (Frontend):
+```bash
+cd frontend
+npm run dev
+```
+
+Then open http://localhost:5173 in your browser.
+
+## Tech Stack
+
+- **Backend:** FastAPI (Python 3.10+), async httpx, OpenRouter API
+- **Frontend:** React + Vite, react-markdown for rendering
+- **Storage:** JSON files in `data/conversations/`
+- **Package Management:** uv for Python, npm for JavaScript

backend/__init__.py

@@ -0,0 +1 @@
+"""LLM Council backend package."""

backend/config.py

@@ -0,0 +1,31 @@
+"""Configuration for the LLM Council."""
+
+import os
+from dotenv import load_dotenv
+
+load_dotenv()
+
+# OpenRouter API key
+OPENROUTER_API_KEY = os.getenv("OPENROUTER_API_KEY")
+
+# Council members - list of OpenRouter model identifiers
+COUNCIL_MODELS = [
+    "mistralai/devstral-2512:free",
+    "xiaomi/mimo-v2-flash:free",
+    "kwaipilot/kat-coder-pro:free",
+    "tngtech/deepseek-r1t2-chimera:free",
+    # "z-ai/glm-4.5-air:free",
+    "nvidia/nemotron-3-nano-30b-a3b:free",
+    # "qwen/qwen3-coder:free",
+    "deepseek/deepseek-r1-0528:free",
+    "meta-llama/llama-3.3-70b-instruct:free"
+]
+
+# Chairman model - synthesizes final response
+CHAIRMAN_MODEL = "openai/gpt-oss-120b:free"
+
+# OpenRouter API endpoint
+OPENROUTER_API_URL = "https://openrouter.ai/api/v1/chat/completions"
+
+# Data directory for conversation storage
+DATA_DIR = "data/conversations"