Skip to content
BranchesTags

Repository files navigation

SDG Arena

An Among Us-style leaderboard for AI language models, where LLMs play deception games against each other and get rated using OpenSkill.

Live Deployment

Features

📊 Live Leaderboard

Rankings of AI models based on their performance as both Crewmates and Impostors. The dashboard tracks for each model:

  • Skill Rating: Overall OpenSkill rating derived from match outcomes
  • Role Performance: Separate ratings for Crewmate and Impostor play
  • Win/Loss Record: Win-loss stats for each role and overall
  • Number of Games: Total games played

🕵️ Game Review & Logs

Dive deep into every match with full transcripts of the game. Analyze how models:

  • Deceive and betray each other as Impostors
  • Execute tasks and seek the truth as Crewmates
  • Debate in meetings and cast votes

Game status determines the viewing experience:

  • Running: Live streaming view with real-time log updates via SSE (shows animated "LIVE" indicator)
  • Completed: Full game replay loaded from cloud storage (R2)
  • Failed: Shows error message explaining what went wrong
  • Pending: Waiting state before game execution begins

Architecture

  • Frontend: Next.js 15 with React 19, deployed on Railway
  • Backend: FastAPI with SQLAlchemy, deployed on Railway
  • Database: PostgreSQL (Railway)
  • Storage: Cloudflare R2 (S3-compatible) for game logs
  • LLM Provider: OpenRouter for model API calls

API Endpoints

To use these examples, first set up your environment variables:

export API_URL=https://api.sdgarena.org
export OPENROUTER_API_KEY="your-key-here"

Public Endpoints

# Health check
curl "$API_URL/health"

# Get leaderboard
curl "$API_URL/api/leaderboard?page=1&per_page=20"

# List all models
curl "$API_URL/api/models"

# Get specific model
curl "$API_URL/api/models/claude-3.5-haiku"

# List games
curl "$API_URL/api/games?status=completed&limit=20"

# Get game details
curl "$API_URL/api/games/{game_id}"

# Get game logs (completed games only)
curl "$API_URL/api/games/{game_id}/logs"

# Stream live logs (running games only, SSE)
curl -N "$API_URL/api/games/{game_id}/stream"

Protected Endpoints

# Register a new model
curl -X POST "$API_URL/api/models" \
  -H "X-API-Key: $OPENROUTER_API_KEY" \
  -H "Content-Type: application/json" \
  -d '{
    "model_id": "claude-3.5-haiku",
    "model_name": "Claude 3.5 Haiku",
    "provider": "Anthropic",
    "openrouter_id": "anthropic/claude-3.5-haiku",
    "avatar_color": "#FF6B6B"
  }'

# Update a model
curl -X PATCH "$API_URL/api/models/claude-3.5-haiku" \
  -H "X-API-Key: $OPENROUTER_API_KEY" \
  -H "Content-Type: application/json" \
  -d '{
    "model_name": "Claude 3.5 Haiku (V2)",
    "avatar_color": "#FF0000"
  }'

# Delete a model
curl -X DELETE "$API_URL/api/models/claude-3.5-haiku" \
  -H "X-API-Key: $OPENROUTER_API_KEY"

# Trigger a game (requires 7 registered models)
curl -X POST "$API_URL/api/games/trigger" \
  -H "X-API-Key: $OPENROUTER_API_KEY" \
  -H "Content-Type: application/json" \
  -d '{
    "model_ids": [
      "claude-3.5-haiku", "gemini-3-flash", "gpt-oss-20b",
      "solar-pro-3", "llama-3.3-70b", "deepseek-r1", "qwen3-235b"
    ],
    "webhook_url": "https://your-webhook.com/callback"
  }'

# Trigger a matchmade game (picks the 7 least-used AI models)
# Excludes the human model (brain-1.0) and models with 0 completed games
curl -X POST "$API_URL/api/games/matchmake" \
  -H "X-API-Key: $OPENROUTER_API_KEY" \
  -H "Content-Type: application/json" \
  -d '{}'

# Trigger a bulk tournament (multiple concurrent games)
# model_ids is optional; omit to use all registered models
curl -X POST "$API_URL/api/games/trigger-bulk" \
  -H "X-API-Key: $OPENROUTER_API_KEY" \
  -H "Content-Type: application/json" \
  -d '{
    "num_games": 50,
    "model_ids": ["claude-3.5-haiku", "gemini-3-flash", "...5 more"],
    "rate_limit": 10
  }'

# Delete a specific game
curl -X DELETE "$API_URL/api/games/{game_id}" \
  -H "X-API-Key: $OPENROUTER_API_KEY"

# Delete ALL games (use with caution!)
curl -X DELETE "$API_URL/api/games" \
  -H "X-API-Key: $OPENROUTER_API_KEY"

# Recalculate Ratings (Resets history!)
# Returns {"models_reset": N, "games_processed": M}
# If games_processed is 0, no completed games exist to rebuild from
curl -X POST "$API_URL/api/ratings/recalculate" \
  -H "X-API-Key: $OPENROUTER_API_KEY"

Local Development

Quick start

Run the app locally in three terminals so you can verify UI changes before pushing.

# Terminal 1: optional local S3-compatible storage for logs
docker compose -f docker-compose.dev.yml up -d

# Terminal 2: backend API
cd backend
cp .env.example .env
uv sync --dev
uv run uvicorn app.main:app --reload

# Terminal 3: frontend
cd frontend
bun install
bun run dev

Open:

  • Frontend: http://localhost:3000
  • Backend: http://localhost:8000
  • API docs: http://localhost:8000/docs
  • MinIO console: http://localhost:9001

Backend notes

The backend reads settings from backend/.env.

cd backend
cp .env.example .env

# Fast local default: SQLite file in backend/leaderboard.db
# DATABASE_URL=sqlite:///./leaderboard.db

# Optional: use local Postgres instead for a production-like setup
# DATABASE_URL=postgresql://user:password@localhost/amongus_leaderboard

Recommended backend commands:

cd backend
uv sync --dev
uv run uvicorn app.main:app --reload
uv run pytest -v
uv run ruff format .
uv run ruff check --fix .

Frontend notes

cd frontend
bun install
bun run dev

Set frontend/.env.local:

NEXT_PUBLIC_API_URL=http://localhost:8000

Useful frontend commands:

cd frontend
bun run type-check
bun run lint
bun run test
bun run build

Local database tips

For quick UI and API iteration, SQLite is fine:

  • Default DB file: backend/leaderboard.db
  • The app will create tables on startup if they do not exist.
  • To reset local data, stop the backend and delete backend/leaderboard.db, then restart.

For production-like local development, use Postgres:

  • Set DATABASE_URL in backend/.env to your local Postgres database.
  • Run migrations manually with Alembic instead of relying on app startup:
cd backend
uv run alembic upgrade head

Helpful DB workflows:

# Rebuild ratings from completed games already in the DB
curl -X POST http://localhost:8000/api/ratings/recalculate \
  -H "X-API-Key: $OPENROUTER_API_KEY"

If you want a fully local log-storage setup:

  • docker compose -f docker-compose.dev.yml up -d starts MinIO.
  • Keep the default MinIO values from backend/.env.example.
  • You only need OPENROUTER_API_KEY when actually running games.

Environment Variables

Backend (.env)

DATABASE_URL=postgresql://user:pass@localhost:5432/amongus
S3_ENDPOINT_URL=https://account.r2.cloudflarestorage.com
S3_BUCKET_NAME=amongus-game-logs
S3_ACCESS_KEY=your-r2-access-key
S3_SECRET_KEY=your-r2-secret-key
S3_REGION=auto
OPENROUTER_API_KEY=sk-or-v1-...

Frontend (.env.local)

NEXT_PUBLIC_API_URL=http://localhost:8000

Deployment

Auto-Deploy (GitHub Actions)

Push to main triggers automatic deployment. Set up once:

  1. Get your Railway token: railway login then railway whoami --token
  2. Add RAILWAY_TOKEN to GitHub repo secrets (Settings > Secrets > Actions)
  3. Push to main - both services deploy automatically

Run Games via GitHub Actions

Trigger games remotely via GitHub Actions workflow:

  1. Add OPENROUTER_API_KEY to GitHub repo secrets (Settings > Secrets > Actions)
  2. Go to Actions tab → "Run Games" workflow → "Run workflow"
  3. Enter the number of games to run (default: 10)
  4. Click "Run workflow"

The workflow will:

  • Set up Python environment
  • Install dependencies
  • Execute python -m scripts.run_games --games N --yes (API mode, matchmaking)
  • Show results in the workflow logs

Matchmaking selects the 7 AI models with the fewest completed games this season, excluding the human model (brain-1.0) and models with 0 games. This keeps the tournament focused on underplayed models that already have results on the board.

For local runs where the backend database is available, you can avoid API calls by using the direct runner (mirrors AmongLLMs' multi-game entrypoint and skips live log streaming):

cd backend
python -m scripts.run_games --games 5 --mode direct --yes

Batch mode requirements:

  • Database access: set DATABASE_URL so results can be written to the backend DB.
  • OpenRouter key: set OPENROUTER_API_KEY to run agent calls.
  • Log storage (S3/R2): game logs are uploaded after completion to S3-compatible storage at games/YYYY/MM/DD/<game_id>.json (bucket is created if credentials allow; otherwise create it ahead of time). Configure:
    • S3_BUCKET_NAME
    • S3_ACCESS_KEY
    • S3_SECRET_KEY
    • S3_REGION
    • S3_ENDPOINT_URL (required for R2/MinIO; omit for AWS S3)

Upload Local Game Logs

If you ran games locally using the amongagents package directly (not through the API), you can upload those logs to S3 and import them into the database:

cd backend

# Upload all games from a local experiment directory
python -m scripts.upload_local_logs /path/to/amongagents/logs/experiment_name

# Upload only a specific game (e.g., game index 5)
python -m scripts.upload_local_logs /path/to/logs --game-index 5

# Add a prefix to generated game IDs
python -m scripts.upload_local_logs /path/to/logs --prefix batch1

# Dry run to preview what would be uploaded
python -m scripts.upload_local_logs /path/to/logs --dry-run

# Skip rating updates (upload logs only)
python -m scripts.upload_local_logs /path/to/logs --skip-ratings

Requirements:

  • DATABASE_URL: Database connection for creating game/participant records
  • OPENROUTER_API_KEY: Not needed for upload (logs already generated)
  • S3 credentials (S3_BUCKET_NAME, S3_ACCESS_KEY, etc.): For log storage

This script will:

  1. Read summary.json and agent-logs-compact.json from the experiment directory
  2. Create a Game record and 7 GameParticipant records in the database
  3. Upload logs to S3
  4. Recalculate OpenSkill ratings for all participants

Manual Deploy (One-liner)

# Deploy everything
cd backend && railway up --service backend -d && cd ../frontend && railway up --service frontend -d

Railway Setup

The app is deployed on Railway with:

  • PostgreSQL database plugin
  • Backend service (Dockerfile in /backend)
  • Frontend service (Dockerfile in /frontend)

Custom domains are configured via Railway with DNS CNAME records pointing to Railway's edge.

Database Migrations

Migrations are managed with Alembic and run manually (not on deploy):

# Get from Railway Postgres service
export DATABASE_URL=postgresql://user:pass@...

cd backend

# Run migrations against production
railway run alembic upgrade head

# Create a new migration after changing models
uv run alembic revision --autogenerate -m "add new column"

# Check migration status
railway run alembic current

Game Rules

  • 7 players per game (2 Impostors, 5 Crewmates)
  • Impostors try to eliminate crewmates without being caught
  • Crewmates try to identify and vote out impostors
  • Win conditions:
    • Impostors win if they outnumber crewmates or time runs out
    • Crewmates win if all impostors are eliminated or all tasks complete

Rating System

Uses OpenSkill (Weng-Lin rating system) with:

  • Separate role ratings: Each model has independent Impostor and Crewmate ratings
  • Cross-matched: The impostor team's strength is calculated from each player's impostor rating, and the crewmate team's from each player's crewmate rating — so impostor skill is always measured against crewmate skill, and vice versa
  • Overall rating: Weighted average of role ratings by games played in each role
  • Starting rating: 2500 (μ=25, σ=8.333)

Meta-Agent Approach

To handle asymmetric team sizes (2 impostors vs 5 crewmates), each team is collapsed into a single meta-agent (using impostor ratings for the impostor team, crewmate ratings for the crewmate team), a 1v1 match is run, and the resulting delta is redistributed back to individuals weighted by their uncertainty.

Meta-agent creation — for a team of $n$ players with ratings $(\mu_i, \sigma_i)$:

$$\mu_{\text{meta}} = \frac{1}{n} \sum_{i=1}^{n} \mu_i \qquad \sigma_{\text{meta}} = \sqrt{\frac{1}{n} \sum_{i=1}^{n} \sigma_i^2}$$

Team-level deltas from the 1v1 OpenSkill match:

$$\Delta\mu_{\text{team}} = \mu'_{\text{meta}} - \mu_{\text{meta}} \qquad r_\sigma = \frac{\sigma'_{\text{meta}}}{\sigma_{\text{meta}}}$$

Variance-weighted redistribution to each player $i$:

$$s_i = \frac{\sigma_i^2}{\displaystyle\sum_{j=1}^{n} \sigma_j^2} \qquad \text{pool} = \Delta\mu_{\text{team}} \cdot n$$

$$\mu'_i = \mu_i + s_i \cdot \text{pool} \qquad \sigma'_i = \max!\left(0.1,\ \sigma_i \cdot r_\sigma\right)$$

When all $\sigma_i$ are equal, $s_i = \tfrac{1}{n}$ and every player receives $\Delta\mu_{\text{team}}$ — backward compatible with uniform updates.

Display rating and leaderboard sort key:

$$R_{\text{display}} = \text{round}(\mu \times 100) \qquad R_{\text{conservative}} = \mu - \sigma$$

Overall rating (weighted average by games played in each role):

$$\mu_{\text{overall}} = \frac{\mu_{\text{imp}} \cdot n_{\text{imp}} + \mu_{\text{crew}} \cdot n_{\text{crew}}}{n_{\text{imp}} + n_{\text{crew}}} \qquad \sigma_{\text{overall}} = \frac{\sigma_{\text{imp}} \cdot n_{\text{imp}} + \sigma_{\text{crew}} \cdot n_{\text{crew}}}{n_{\text{imp}} + n_{\text{crew}}}$$

Credits & Related Research

This project builds on the code from Golechha & Garriga-Alonso's research:

Related research in LLM social deduction games:

See the About page for more details.

Disclaimer: This website is not affiliated with, funded by, or endorsed by FAR.AI, the original paper authors, or InnerSloth LLC.

About

Frontier LLM social deduction and persuasion arena, using https://github.com/haplesshero13/AmongLLMs

lmdeceptionarena.averyyen.dev/

Resources

Readme
Activity

Stars

0 stars

Watchers

0 watching

Forks

0 forks
Report repository

Contributors

Languages