Skip to content

README.md

Latest commit

 

History

History
231 lines (183 loc) · 7.77 KB

README.md

File metadata and controls

231 lines (183 loc) · 7.77 KB
Raw

Judge Service

Backend judge for AI Deep Dive.

  • The browser runs public tests with Pyodide.
  • The server runs hidden tests and torch-required problems.
  • Jobs are queued in Redis Streams and handled by light and torch workers.
  • Results are stored in SQLite on a single VM.

Layout

judge/
  src/judge/
    api.py         FastAPI endpoints
    queue.py       Redis Streams wrapper
    worker.py      Worker loop
    runner.py      Execution-plan runner
    results.py     SQLite results store
    problems.py    Canonical problem loader + compiler
    models.py      Pydantic models
  problems/        Canonical problem data (server only)
  deploy/          VM deploy templates + scripts

Quick start (local)

Install uv first if it is not already on your machine.

cd judge
uv venv .venv
source .venv/bin/activate
uv pip install -e .

Start Redis locally:

redis-server

Build the local runtime corpus:

PYTHONPATH=src python scripts/build_runtime_problem_corpus.py

Start the API:

uvicorn judge.api:app --reload

Frontend environment variable:

NEXT_PUBLIC_JUDGE_API_URL=http://localhost:8000

Set JUDGE_ALLOWED_ORIGINS when the web app runs on a different origin. Use a comma-separated list.

Start workers:

python -m judge.worker --stream queue:light --group workers-light --consumer light-1
python -m judge.worker --stream queue:torch --group workers-torch --consumer torch-1

CPU-only PyTorch is required to run torch problems locally. Install it in the same venv using the official PyTorch install selector.

Run the warm-fork security probe:

python judge/scripts/warm_fork_security_probe.py --skip-bench

Build runtime problem corpus

PYTHONPATH=src python scripts/build_runtime_problem_corpus.py

This builds a runtime-ready corpus under judge/data/runtime-problems/current. Authored source problems stay in judge/problems; generated hidden tests live only in the runtime corpus.

For torch-backed challenges, use the same Python environment used by judge workers when building the runtime corpus. This keeps numeric expected values aligned with runtime behavior:

/opt/ai-deep-dive/judge/.venv/bin/python scripts/build_runtime_problem_corpus.py

Submit a sample job

curl -X POST http://localhost:8000/submit \
  -H "Content-Type: application/json" \
  -d '{
    "problem_id": "build-gpt/01-from-text-to-bytes/01-encoder",
    "code": "def encode_string(text):\n    return list(text.encode(\"utf-8\"))\n"
  }'

Run custom public-style cases:

curl -X POST http://localhost:8000/run \
  -H "Content-Type: application/json" \
  -d '{
    "problem_id": "build-gpt/01-from-text-to-bytes/01-encoder",
    "code": "def encode_string(text):\n    return list(text.encode(\"utf-8\"))\n",
    "cases": [
      {
        "id": "case1",
        "inputs": { "text": "\"A\"" },
        "expected_literal": "[65]"
      }
    ]
  }'

Configuration

Environment variables:

  • JUDGE_REDIS_URL (default: redis://localhost:6379/0)
  • JUDGE_RESULTS_DB (default: judge/data/judge.db)
  • JUDGE_PROBLEMS_ROOT (default: judge/data/runtime-problems/current)
  • JUDGE_MAX_OUTPUT_CHARS (default: 2000)
  • JUDGE_JOB_CLAIM_IDLE_MS (default: 30000)
  • JUDGE_JOB_CLAIM_COUNT (default: 10)
  • JUDGE_API_WORKERS (default: 1)
  • JUDGE_JOB_RETENTION_DAYS (default: 7)
  • JUDGE_QUEUE_MAXLEN (default: 10000, per-stream backlog admission limit; 0 disables)
  • JUDGE_BACKUP_DIR (default: judge/data/backups)
  • JUDGE_BACKUP_RETENTION_DAYS (default: 7)
  • JUDGE_ALLOWED_ORIGINS (comma-separated)
  • JUDGE_ISOLATE_BIN (default: /usr/bin/isolate)
  • JUDGE_ISOLATE_USE_CGROUPS (default: 1)
  • JUDGE_ISOLATE_PROCESSES (default: 64)
  • JUDGE_ISOLATE_WALL_TIME_EXTRA_S (default: 2)
  • JUDGE_ISOLATE_TIMEOUT_GRACE_S (default: 5)
  • JUDGE_ISOLATE_FSIZE_KB (default: 1024)
  • JUDGE_PYTHON_BIN (default: worker interpreter path)
  • JUDGE_TORCH_EXECUTION_MODE (default: warm_fork, options: isolate, warm_fork)
  • JUDGE_WARM_FORK_ENABLE_NO_NEW_PRIVS (default: 1)
  • JUDGE_WARM_FORK_ENABLE_SECCOMP (default: 1)
  • JUDGE_WARM_FORK_SECCOMP_FAIL_CLOSED (default: 1)
  • JUDGE_WARM_FORK_CLEAR_ENV (default: 1)
  • JUDGE_WARM_FORK_DENY_FILESYSTEM (default: 1)
  • JUDGE_WARM_FORK_ALLOW_ROOT (default: 0, keep disabled in production)
  • JUDGE_WARM_FORK_CHILD_NOFILE (default: 64)
  • PROMETHEUS_MULTIPROC_DIR (optional, for API + worker metric aggregation)

Notes

  • light profile always executes inside isolate.
  • torch profile uses warm fork execution by default.
  • Set JUDGE_TORCH_EXECUTION_MODE=isolate to force per-job isolate execution.
  • VM deployment scripts enforce CPU-only PyTorch wheels for judge workers.
  • Warm fork hardening defaults: no_new_privs=on, seccomp=on, clear_env=on.
  • Worker consumer names must follow the template services (light-%i, torch-%i).
  • /health is a liveness endpoint only and returns {"status":"ok"} when the API process is up.
  • /ready is a readiness endpoint and checks Redis, SQLite, and canonical problem files. It returns HTTP 200 with status=ready when all checks pass, and HTTP 503 with status=not_ready plus per-dependency check details otherwise.
  • Hidden tests are not publicly served. On submit, the first failing executed testcase is returned for debugging.
  • /result/{job_id} includes error_kind (user or internal) so clients can render user-facing failures without string parsing.
  • Production setup lives in judge/deploy/. See the deploy README for steps.
  • Daily timers run on the VM to prune old database rows and save database backups.

Metrics (Prometheus)

The API exposes Prometheus metrics at http://127.0.0.1:8000/metrics. The deploy nginx templates do not expose /metrics on the public domain. To aggregate worker + API metrics, set PROMETHEUS_MULTIPROC_DIR for all judge services and ensure the directory exists and is writable by the judge user. If Prometheus runs on a separate VM, expose metrics on a private-network endpoint and scrape that private endpoint (instead of exposing metrics publicly).

Use a runtime path outside git, for example:

PROMETHEUS_MULTIPROC_DIR=/var/lib/judge/prometheus-multiproc

Deploy automation:

  • judge/deploy/apply.sh prepares the multiprocess directory.
  • judge/deploy/judge-metrics-init.service clears stale multiprocess shard files on startup.
  • judge/deploy/monitoring/apply-prometheus.sh applies repo-managed Prometheus config + alert rules.
  • judge/deploy/monitoring/install-monitoring.sh installs monitoring packages and enables services (Prometheus/Alertmanager/Grafana, Ubuntu-only).
  • judge/deploy/monitoring/apply-alertmanager.sh applies Alertmanager routing for Telegram/email notifications and enforces single-node runtime flags for prometheus-alertmanager.
  • judge/deploy/monitoring/apply-grafana.sh provisions Grafana datasource and dashboard.
  • judge/deploy/monitoring/send-test-alert.sh injects a synthetic alert to test Telegram/email delivery through Alertmanager.

Worker liveness and recovery:

  • Worker template services use systemd watchdog (Type=notify + WatchdogSec).
  • Workers send readiness/watchdog pings from the parent loop.
  • Worker-missing alerts use node-exporter systemd unit-state metrics (node_systemd_unit_state) instead of app heartbeat shard files.

Problem format

See judge/problems/README.md for source problem files and runtime hidden-test generation.

starter.py is canonical problem corpus data for editor seed code, but it is not part of judge runtime loading or execution. The runtime still executes only the canonical problem spec plus compiled/public hidden testcases.

Deploy

VM deploy templates and scripts live in judge/deploy/. See judge/deploy/README.md for the full setup steps.