A self-hosted technical-workflow evaluation dashboard for local OpenAI-compatible APIs, Open WebUI and OpenRouter. Synthetic tasks contain no personal conversation history. Provider keys stay server-side. No Docker socket, SSH credentials, GPU devices or host folders are mounted.
- Live run ledger with progress and cancellation.
- Local/Open WebUI/OpenRouter model discovery and separate quant labels.
- Paired multi-model campaigns, track filters, seeds, context bands and repeats.
- Verified pass/fail/error counts, critical violations and loop counts.
- Per-track bars, labeled context accuracy curves and latency/quality plots.
- Case-level prompts, responses, tool evidence and JSON/CSV exports.
| Suite | Cases per requested context band | Distinct scenario families |
|---|---|---|
| Smoke | 13 | 13 |
| Core | 52 | 13 |
| Extended | 78 | 13 |
Variants are generated instances, not additional independent task families. Families include exact recall, corrected state, multi-hop lookup, running configuration versus file intent, missing evidence, log diagnosis, exact identifiers, strict JSON, untrusted instructions, in-context memory, verified tool updates, transient-fault recovery and protected-production boundaries.
Requirements: Docker Engine with Compose, a private IPv4 address on the Docker host, outbound access to your chosen model APIs.
- Clone this repository, or copy this directory to your Docker host.
- Create
.envwithBIND_IP=your-private-LAN-IP. Do not use0.0.0.0or a public IP. - Run
docker compose config -q, thendocker compose up -d --build. - Visit
http://your-private-LAN-IP:8798. The login token is generated in the persistent volume. Retrieve it locally withdocker compose exec broodbench python -c "from pathlib import Path; print(Path('/data/admin.token').read_text())". Enter it only in the dashboard login form; do not send it to chat.
The container's internal wildcard bind is only inside its network namespace; the published host port is explicitly private. Do not add a public reverse proxy, router forward or Tailscale Funnel. Private HTTP is suitable only on trusted networks; use a trusted private HTTPS proxy or SSH tunnel where interception is a concern. Router forwards and existing tunnels require an independent network audit.
Add provider type Open WebUI and its origin, e.g. http://192.0.2.10:3080. Supply an Open WebUI API key in the dashboard if required; create/enable API access in your Open WebUI account. Use Test / Discover to retrieve model identifiers and register selected models. Never reuse a chat password as an API key.
Open WebUI can apply model presets, system prompts, filters or pipelines. Results measure that configured route, not necessarily an unmodified underlying model. For controlled quant comparisons, prefer the direct inference server's OpenAI-compatible endpoint and record any middleware.
Choose OpenAI compatible, enter http://host:port/v1, optionally supply its API key, discover models and register them. Network access is from the Docker container, so localhost means the container itself; use the server's private IP. This app does not load, unload or change model configurations.
Discovery preserves the serving engine's advertised context limit (for example vLLM's max_model_len) rather than substituting a universal 32K limit. The registry distinguishes the effective benchmark limit from the latest engine-reported value. Missing metadata is labeled unknown/default, not presented as a measured engine limit.
Use Models & providers → Edit context to save a manual override, refresh engine metadata, or switch back to the engine value. Refresh preserves a manual override. Registry edits affect future benchmark validation only: they do not reconfigure the engine and do not rewrite historical run configurations. A manual value above the engine's actual capacity cannot make that engine accept a larger request.
Choose OpenRouter, enter your key in the password field and discover its model catalog. The destination is fixed to https://openrouter.ai/api/v1. Keys are encrypted at rest using a key in the data volume; protect backups of that volume as secrets. The app never retrieves keys from another application. Paid runs require explicit consent and finite call/cost budgets. Cost labels must distinguish provider-reported spend from estimates. Unknown cost is not free. OpenRouter may route to changing underlying providers; record/pin routing where reproducibility requires it.
The normal launch workflow uses Category sweep: each test family receives a small, deterministic spread of short, intermediate and long attempts. It does not run every case at every length or pad every case to a global maximum. Smoke schedules 3 attempts per family, Core 4, and Extended 6; only one attempt per family targets the long end. The registered model context is a ceiling, with output/tool headroom reserved; comparisons use the smallest selected model limit so the workload is paired. An optional lower ceiling limits the sweep rather than assigning that length to all tests.
Preview the actual family/context distribution, total attempts and prompt budget before starting. Plans are snapshotted with the run. Fixed contexts (advanced) retains the old explicit-band stress-test behavior; historical runs keep their original configuration and scores.
For compatible local servers, category-sweep requests use the serving tokenizer before generation. Only generated archived distractor records may be reduced to fit; task facts, instructions and tool history must never be truncated. Actual submitted prompts and admission metadata are retained. Without a compatible tokenizer, the requested band remains approximate and conservative byte-based admission guards the model limit.
Long category-sweep cases get an automatic prefill allowance on top of the per-call timeout: timeout + (context − 8,192) ÷ 400 seconds, capped at 30 minutes. 400 tok/s is deliberately below typical local prefill so slow-but-working engines are not recorded as failures. The effective timeout is stored per call. Fixed-context mode keeps the exact operator timeout. Timeouts are reported as request timeouts, not model answers.
Two load benchmarks run outside the quality suites and grade nothing. Both accept private OpenAI-compatible/OpenWebUI endpoints only (OpenRouter is refused), and both put a unique leading nonce on every request so prefix caching cannot share KV across requests, waves or context bands.
- Concurrency test — one context band and a list of in-flight levels (1–64). For each level, a prefill probe wave of N simultaneous
max_tokens=1requests, thenrequests_per_levelgeneration requests with at most N in flight. - PR smoke test — a fixed matrix for comparing engine builds such as vLLM fork PRs: concurrency 1 and 8 up to 16K context (2K/4K/8K/16K) and concurrency 1 and 4 up to 128K (32K/64K/128K). That is 14 (context, concurrency) pairs and 275 requests per model at the default 16 requests per level. The matrix is fixed so every build produces rows that line up for comparison.
context_capclamps every band for models whose registered limit cannot hold the 128K row; clamped rows keep both the used and the requested band. The plan is snapshotted with the run.
Aggregate output tok/s = completion tokens ÷ wave wall time. Per-request decode is a client stream estimate; server prefill/decode rates come from vLLM /metrics counter deltas only when the endpoint is idle at both boundaries and the completed-request count plus token usage match exactly, otherwise they are unavailable with a stated reason. Long bands get the same prefill allowance as category sweeps (timeout + (context − 8,192) ÷ 400 seconds, capped at 1,800 s) and the effective timeout is stored per level. CSV export gives one row per (context, concurrency) level.
Register separate entries for each model/quant, even if the remote model ID is reused. Add family, quant label, context limit and provenance notes (artifact hash, engine version, cache precision, template, offload). Operator-provided labels are not hardware attestation.
Select multiple model entries, a suite, seed, repeat count and context bands. Serial execution avoids saturating your model lane by default. All selected models receive the same generated cases. If different quants share a single mutable endpoint, do not register both and assume this app swaps weights: serve distinct endpoints or run separate campaigns while manually verifying the loaded artifact.
Stop preserves completed evidence; unexecuted cases are not scored as model failures. Restarted in-flight work is marked interrupted rather than silently resumed and duplicated. Results persist in SQLite on the named volume. JSON/CSV exports contain prompts, responses, verdicts and metadata—not provider secrets.
The custom task engine covers synthetic operational questions and bounded tool workflows with deterministic graders. Agent tools mutate isolated in-memory service worlds, not your actual services. This is a deliberate safety boundary, not proof of real shell/coding ability.
- Long-context bands are approximate unless verified by the serving tokenizer. Provider input-token usage measures an actual request including wrapper overhead, not a guaranteed exact target context.
- Memory tests exercise in-context recall and state updates. This version is not a general cross-session memory-system benchmark.
- Generated case instances share scenario templates; report template-family counts as well as sample counts. Variants are not independent task families.
- No arbitrary model-generated code or shell commands execute. Real repository repair, browser operation and external benchmark adapters are future tracks, not existing capabilities.
- Task wall time is not engine throughput. New runs store individual streamed-call latency, time to first generated delta, actual provider input/output/cache token counts and finish state. True prefill/decode rates use explicit server timings when supplied; streaming decode estimates are separately labeled and can be affected by chunk batching. A single output chunk is insufficient for a decode estimate. Missing measurements and all historical unrecorded timings stay unknown; TTFT is never presented as pure prefill.
- The Suite guide documents each actual family, purpose, grader and limitations; evidence and comparison views break results down by family and requested context. Errors remain in the attempt denominator. These custom families are not HumanEval or the reference channel's agent tests.
- This release is an experimental evaluation instrument, not a certified public leaderboard. Inspect case evidence and validate graders before drawing quant-equivalence conclusions. A small score gap is not statistical equivalence.
Open a run's evidence, or select a test family in its comparison breakdown, to inspect individual attempts. Keep model, requested context, repeat and original verdict attached to every case. Expected output and actual output are shown separately; tool traces retain ordered inputs and outcomes. Full prompts and raw evidence remain available in expandable sections.
Failure explanations are derived from recorded evidence using fixed rules, not a model-generated diagnosis. A strict literal mismatch is not proof of a reasoning error or a semantic-equivalence judgment. A correct recorded final tool state does not erase missing readback, a protected-action attempt, truncation or a harness rejection. Historical generic exceptions may not identify a rejected action; those gaps must be labeled unknown. New records retain bounded error details and explicitly rejected tool attempts without marking them executed. Inspection does not regrade or rewrite stored results.
In New run, select one registered private/local model, describe the goal, and choose AI proposes settings. This makes one inference call (up to 1,024 output tokens; 60-second limit), not a benchmark campaign. The server checks strict JSON and fixed policy limits. Invalid answers are shown as failures rather than silently repaired. Paid/public endpoints are not supported for proposals.
Review the rationale, checks and raw answer. Apply proposal fills the form only; Start run remains a separate action. The model never changes the graders, provider secrets, spending authorization or serving configuration. This tests instruction-following/configuration generation, not whether a model can independently design a statistically sound benchmark.
State lives in named volume broodbench_broodbench-data (verify exact name via docker volume ls). It includes SQLite results, encrypted provider credentials and encryption/login keys. Never publish the volume or include it in a source archive.
For a consistent backup, stop only BroodBench, then archive its named volume using an approved local backup tool; keep the backup private. Copying a live SQLite file without its WAL can lose recent writes. Restore the entire volume including keys, not just the database. Source updates do not remove the data volume.
Off switch: docker compose down from this directory. Do not add -v unless you intend to permanently erase results and credentials. To roll back a deployment, stop this app and rebuild the previous saved source/image while preserving the volume; inspect schema compatibility first. No production serving service is controlled by this stack.
Python 3.12 recommended. Create a project venv, install requirements-dev.txt, then run python -m playwright install chromium and python -m pytest tests -q. The browser test uses installed Google Chrome when available and Playwright Chromium otherwise. Run node tests/ui_contract.test.cjs for UI contract checks and pip-audit -r requirements.txt for known dependency advisories.
Docker verification: docker build --target test -t broodbench:test . then docker run --rm --network none broodbench:test. This runs the non-browser tests under the image's actual Python version against local HTTP fixtures, not external models. CI also exercises browser tests. No API key is needed for tests.
Start one Uvicorn worker only; the scheduler is single-owner. Do not use multiple replicas against the same queue. See SECURITY.md before deployment. License: MIT.