Skip to content

README.md

Latest commit

 

History

History
187 lines (152 loc) · 7.13 KB

README.md

File metadata and controls

187 lines (152 loc) · 7.13 KB
Raw

@phatblat/evalkit

A framework for evaluating a CLI coding agent across a matrix of models × thinking levels × fixtures × repetitions. It runs the agent against a scratch git repo and grades what actually landed in git state — commits, diffs, git status — never the session transcript. A crashed or hung run is a failed run, not a missing data point: composite is forced to 0 for any cell that does not exit 0.

evalkit is subject-agnostic. It never imports a concrete subject under test — the consumer implements the EvalTask interface for whatever it wants graded (a CLI skill, a prompt, an agent harness feature) and injects it through a taskRegistry(). The reference consumer is phatblat/cmt-eval, which evaluates a /git:commit skill.

Requirements

  • Bun ≥ 1.3.14. The package depends on Bun.spawn/spawnSync, Bun.Glob, Bun.file/Bun.write, and Bun's ESM TOML loader (import() on a .toml path). Node cannot run this package.
  • omp on PATH. catalog() shells omp models --json to resolve model selectors, validate thinking-level support, and price runs.
  • @types/bun in the consumer's own devDependencies. evalkit ships raw TypeScript (no build step); a consumer's tsc needs Bun's ambient types to type-check the imported sources directly.

Install

bun add @phatblat/evalkit

Local development (unpublished changes)

Because evalkit ships raw TypeScript with no build step, a consumer can develop against an unpublished checkout via bun link rather than waiting on a publish:

# in this repo
just link          # registers "@phatblat/evalkit" globally

# in the consumer repo
bun link --save @phatblat/evalkit

bun link is path-independent (unlike file:~/..., which bun does not tilde-expand, or link:<path>, which only accepts a registered name) and survives the consumer living in a git worktree at an unpredictable path.

The EvalTask contract

export type Group = {
  type: string;
  subject: string;
  paths: string[];
};

export type Fixture = {
  id: string;
  dir: string;
  description: string;
  expectedCommits: number;
  groups: Group[];
  timeoutS: number;
};

// Metric values are numbers (continuous scores/counts), booleans (pass/fail
// checks), or strings (free-form diagnostic fields not used in composites).
export type Metrics = Record<string, number | boolean | string>;

export type PreparedRun = { repo: string; baseTip: string };

export interface EvalTask {
  readonly name: string; // e.g. "git-commit"
  fixtures(): Promise<Fixture[]>; // enumerates the subject's fixtures
  prepare(fixture: Fixture, workdir: string): Promise<PreparedRun>;
  prompt(fixture: Fixture): string;
  grade(fixture: Fixture, prepared: PreparedRun): Promise<Metrics>;
  composite(m: Metrics, weights: Record<string, number>): number;
}

The consumer supplies one or more EvalTask implementations to taskRegistry([...]), which rejects duplicate names at construction. evalkit resolves a suite's task = "<name>" string against that registry — there is no module-global registration and no static import of a subject anywhere in this package.

A minimal consumer entry point

#!/usr/bin/env bun
import { runCli, taskRegistry } from "@phatblat/evalkit";
import { gitCommitTask } from "./tasks/git-commit/task.ts";

try {
  await runCli(process.argv.slice(2), {
    name: "cmt-eval",
    tasks: taskRegistry([gitCommitTask]),
    scratchNamespace: "cmt-eval",
  });
} catch (err) {
  console.error(err instanceof Error ? err.message : String(err));
  process.exit(1);
}

runCli dispatches the built-in commands (scan, estimate, eval, report, compare) against the injected registry and directory layout. CliOptions.commands lets a consumer merge in subject-specific commands (e.g. fixture mining) that appear in the same usage banner and dispatch table.

Suite TOML schema

task = "git-commit"                              # key in the injected task registry
models = ["anthropic/claude-haiku-4-5"]          # omp selectors, validated against `omp models --json`
thinking = ["low"]                               # each level checked against each model's support
fixtures = ["mise-zsh-2", "harness-ci-docs-3"]   # ids returned by the task's fixtures()
reps = 1                                         # integer >= 1
perProviderJobs = 1                              # integer >= 1; > 1 sets contended=true
retries = 2                                      # integer >= 0

[weights]                                        # MUST sum to 1 ± 0.001
countMatch = 0.40
groupingScore = 0.30
convMsgs = 0.10
trailerOk = 0.10
treeClean = 0.05
typeMatch = 0.05

Validation collects every problem in one pass (SuiteError.problems[]) before throwing, so a suite author sees every mistake — an unknown model selector, an unsupported thinking level, a weight sum off by more than 0.001 — in a single run.

Scheduling model

Cells are expanded rep-major: every repetition sweeps the whole matrix, so a transient provider slowdown biases at most one rep, never a whole model. Cells are grouped by provider and run serially within a provider, in parallel across providers — most OAuth-backed provider pools throttle under concurrency, which would corrupt the wall-clock metric. Two shared cursors under Promise.all implement this: one across providers (capped at jobs), one within each provider (capped at perProviderJobs, default 1). contended is recorded per row whenever perProviderJobs > 1 — a contended row's timings are not comparable to a serialized one.

Invariants

  1. Scratch repos must live outside $HOME. scratchRoot(namespace, runId, cellId) throws if $TMPDIR resolves inside the home directory, because omp's AGENTS.md discovery walks ancestor directories — a scratch repo under ~/dev/... would silently leak the consumer's own AGENTS.md into every graded run's context. namespace keeps two consumers' scratch trees from colliding.
  2. composite is forced to 0 for any run that did not exit 0. A crashed run is a failed run, not a missing data point.
  3. Unpriced models legitimately report costUsd = 0. ModelInfo.priced: false marks a subscription/local model — never treat 0 as free-and-fast without checking priced.
  4. evalkit MUST NOT import a concrete subject. The registry arrives through taskRegistry() / CliOptions.tasks; a static import of a task here would recreate the coupling this package exists to avoid.

Commands

Run just --list for the full recipe list with descriptions:

Recipe Purpose
deps mise install + bun install
check tsc --noEmit
test (alias t) bun test — framework unit tests, no tokens, no network
link bun link — register this checkout as a linkable package
unlink bun unlink
publish-dry bun publish --dry-run — preview the tarball without publishing
publish bun publish --access public — real and irreversible
clean rm -rf node_modules

License

MIT — see LICENSE.md.