evals

evals/

The small-model tool-calling evaluation harness. It points a real local model (GPT-OSS, Gemma 4, Qwen3 — served via vLLM or llama.cpp's OpenAI-compatible endpoint) at a server's tools and measures callability:

• Tool-selection accuracy — given a natural-language task, is the right tool chosen?
• Argument-filling success rate — are arguments populated correctly across N runs?
• Degradation signals — tools that score poorly (too many tools, oversized enums, nested args) are flagged as design defects to fix, reported as success rates rather than pass/fail.

Task sets live here as YAML. Run outputs are written to results/ (gitignored). A tool that passes contract tests but fails here has a design problem — simplify the schema, do not lower the bar.

Scorecard & multi-server eval

To score a single server against a single model, use evals/run.py directly (pass --server all to run the combined 28-tool registry — every server's tools registered at once — which is the ad-hoc composition test: does a model still pick the right tool when many platforms' tools are all on the table together):

uv run python -m evals.run --server all \
  --base-url http://localhost:8000/v1 --model <model-id> --runs 3

To sweep every model in models.yaml against every server (plus all) and produce a model x server matrix, use evals/scorecard.py:

uv run python -m evals.scorecard --base-url http://localhost:11434/v1 --runs 1
# narrow the sweep:
uv run python -m evals.scorecard --base-url ... --models gpt-oss:20b-c128k --servers defender,all
# re-run cells already recorded instead of skipping them:
uv run python -m evals.scorecard --base-url ... --force
# compute and print without persisting anything:
uv run python -m evals.scorecard --base-url ... --no-write

Each (model, server) cell is written incrementally to evals/results/<date>.json (gitignored, resumable — an interrupted sweep picks up where it left off and already-recorded cells are skipped unless --force), and the run regenerates SCORECARD.md, the checked-in markdown table of the latest sweep.

Ollama: cap loaded models for a multi-model sweep

Ollama keeps each model resident for ~5 minutes after use (keep_alive), so a sequential sweep can hold several large models in memory at once and OOM on the third or fourth load — cells then fail with "server disconnected" / "all connection attempts failed" (the sweep records these as error cells and continues; it is not a harness fault). Cap Ollama to one resident model so it unloads before loading the next:

# systemd (persists):
sudo systemctl edit ollama   # add:  [Service]\nEnvironment="OLLAMA_MAX_LOADED_MODELS=1"
sudo systemctl restart ollama
# or when launching manually:
OLLAMA_MAX_LOADED_MODELS=1 ollama serve

Then resume the sweep — the recorded ok cells are skipped and only the failed ones re-run (delete the error cells from the results JSON first, since any present cell is skipped without --force).

Multi-step (agentic) skill eval

evals/agentic.py + evals/agentic_scorecard.py measure whether a model can drive a whole SKILL.md procedure, not just pick one tool. Each scenario in evals/scenarios/*.yaml injects the skill's live ## Procedure, runs a multi-step tool-calling loop against deterministic mock tool results, and scores a dual metric — tool-coverage% (order-tolerant) and goal-reached% (keyword check on the final answer, where keywords are grounded in tool output, not the task). It is local-only (needs Ollama); the harness logic is covered offline by evals/tests/test_agentic.py.

Run the matrix (evict between models on a memory-constrained box, as with the scorecard):

for tag in $(python -c "import yaml;[print(m['tag']) for m in yaml.safe_load(open('evals/models.yaml'))]"); do
  uv run python -m evals.agentic_scorecard --models "$tag" --date 2026-07-12
  curl -s http://localhost:11434/api/chat -d "{\"model\":\"$tag\",\"messages\":[],\"keep_alive\":0}" >/dev/null
done

Results render to evals/AGENTIC.md (a skill × model matrix). Ministral 3 was removed from models.yaml — it emits no OpenAI tool_calls, so it scores 0 on both evals.

Watching a sweep (dashboard)

A full matrix at runs=3 takes 12-18 hours, and SCORECARD.md is only written at the end. To watch it live:

uv run python scripts/eval_dashboard.py     # http://127.0.0.1:8765

Read-only and safe to start, stop, or restart mid-sweep — it only reads evals/results/*.json, which run_matrix rewrites after every cell.

It shows progress with a column-weighted ETA (the all column is several times slower per cell, so a single pooled mean under-estimates badly), the matrix as a heatmap that fills in live, and each model's trend across past runs. Any error/unusable cell is raised as an alert — unusable means the model made no tool call at all on a whole task set, which is a serving problem (context too small for the schema), not a capability result.

Two distinctions the views keep deliberately visible, because collapsing them would misreport coverage as a result: a pending cell is never drawn as a low score, and in the trend a model dropped from the roster reads differently from one not yet reached by a running sweep.

Never serve this repo with python -m http.server - it would expose .env.defender and every other credential file over HTTP. The dashboard maps no URL to a filesystem path and binds 127.0.0.1 only.