Skip to content

README.md

Latest commit

 

History

History
157 lines (109 loc) · 11.2 KB

README.md

File metadata and controls

157 lines (109 loc) · 11.2 KB
Raw

@tpsdev-ai/flair-bench

A standalone embedding recall benchmark for flair. Run real recall numbers — precision@3, MRR — against any GGUF embedding model, batch-compare several, get a host-aware recommendation, and (optionally) save a redacted, shareable result. No flair install required.

npx @tpsdev-ai/flair-bench run --model-file ./nomic-embed-text-v1.5.Q4_K_M.gguf

Why this exists

flair ships nomic-embed-text-v1.5 (Q4_K_M by default) for local semantic memory search. The question "would a different model/quant actually recall better for flair's use case, on my hardware?" used to require a full flair checkout, its recall-eval harness, and an ephemeral Harper instance. flair-bench packages the same corpus and the same scoring math into a small, dependency-light CLI/library you can point at any GGUF, anywhere.

Install / run

npx @tpsdev-ai/flair-bench run --model-file /path/to/model.gguf

or install it:

npm install -g @tpsdev-ai/flair-bench
flair-bench run --model-file /path/to/model.gguf

Requires Node.js >= 22. Downloads no models itself — point it at GGUF files you already have.

Commands

run — benchmark one or more models

flair-bench run --model-file a.gguf --model-file b.gguf --model-file c.gguf
flair-bench run --manifest models.txt          # one path per line, # comments allowed
flair-bench run --model-file a.gguf --json      # machine-readable output
flair-bench run --model-file a.gguf --label "local-m4-mini"   # freeform host/infra tag

Reports, per model: p@3/MRR (aggregate and per-kind — stress/trap/hard/clean, see "The corpus" below), ms/embed (serial, warm, N≥64), peak RSS delta, GGUF file size/BPW, embedding dimensions, and load time. A batch (more than one --model-file) ends with a ranked comparison table.

recommend — pick a model for this host

flair-bench recommend --model-file a.gguf --model-file b.gguf --model-file c.gguf

Fingerprints the host (platform, arch, RAM, CPU model, and the actual compute backend node-llama-cpp loaded — Metal/CUDA/Vulkan/CPU, plus GPU device name(s) when present — not inferred from the OS), runs the batch, and recommends the best measured recall the host can plausibly afford. See "Recommend heuristic, and its limits" below — this is deliberately simple and says so.

--share — save a redacted, shareable result

flair-bench run --model-file a.gguf --share

Writes a canonical JSON file per model (see "Share schema" below) and prints where the eventual hosted-submission endpoint would receive it — no network call is made; the endpoint is a config placeholder (SUBMISSION_ENDPOINT_PLACEHOLDER in src/share.ts) for a hosted site that doesn't exist yet.

--label: benchmarking infra, not just models

flair-bench's core use case extends past "which model" to "which model on which infra" — comparing a GGUF across, say, a free-tier Fabric host, a GPU-backed Fabric host, and a local Mac. --label <string> is a freeform, user-chosen string (e.g. "fabric-free-gcp", "fabric-gpu-a", "local-m4-mini") that becomes the grouping key in run/recommend output and in --share's hardware.label field. It is never auto-filled from the machine's real hostname — real hostnames never need to appear in a shared result. Combined with the measured hardware.backend/hardware.gpu fields (see below), a set of --share outputs across labeled hosts is a model × infra matrix.

Comparable to flair, with a caveat

flair-bench uses exactly the same corpus, the same p@3/MRR scoring math, and the same nomic search-prefix convention flair's own isolated recall-eval harness (test/bench/recall-harness/) uses — see "What's shared vs. copied" below. The one deliberate difference: Harper's live /SemanticSearch endpoint ranks candidates through an HNSW approximate index; flair-bench has no HNSW graph, so it ranks every query against every corpus record by exact cosine similarity instead. For a corpus this size (251 records), exact search is strictly more informative, not less — but it means a number from this tool and a number from the harness aren't guaranteed to be bit-identical, only very close. See this repo's PR for the validation that quantifies exactly how close (spoiler: they match).

What's shared vs. copied (and how it's kept honest)

  • Corpus (src/corpus-v2.ts): a build-time copy of test/bench/recall-harness/corpus-v2.ts. A standalone, npx-able package can't import from the monorepo's test/ directory at runtime (it isn't published), so the corpus is copied at commit time via scripts/sync-corpus.mjs and kept honest by test/corpus-sync.test.ts, which deep-equals this copy's exported CORPUS/QUERIES against the live harness source on every bun test run inside the monorepo checkout. Any drift fails CI loudly.
  • Scorer (src/scorer.ts): a faithful hand-replication of the harness's statsFor() (which isn't exported, so it can't be imported directly). test/scorer-sync.test.ts reads the harness source's raw text and asserts it still contains the exact formula fragments this package replicated — a tripwire against silent drift.
  • Prefix convention (src/prefixes.ts): re-implements the same search_document: /search_query: string-prepend resources/embeddings-provider.ts gets from harper-fabric-embeddings' engine, since flair-bench talks to node-llama-cpp directly and has no HFE wrapper in the loop. Keyed on model filename today (see the file's own header) — there's an upstream proposal to carry this convention in the GGUF/HFE registration surface itself instead of every consumer re-deriving it from a filename pattern: harper-fabric-embeddings#4.

The corpus

251 synthetic records across 30 topic clusters, 126 hand-written ground-truth queries in four kinds:

  • stress — durability/recency adversarial pairs (does a fresher-but-wrong record outrank an older-but-correct one?)
  • trap — cross-cluster lexical traps (the same ambiguous term used in two different domains, e.g. "transaction" in database internals vs. finance ops)
  • hard — near-duplicate-cluster disambiguation
  • clean — unambiguous single-best-answer sanity floor

See test/bench/recall-harness/corpus-v2.ts's header (in the main flair repo) for the full design rationale.

Recommend heuristic, and its limits

recommend picks the model with the best measured MRR among those whose peak RSS delta fits within a RAM headroom budget (default: 50% of currently-available RAM) and whose ms/embed is under a latency ceiling (default: 500ms — deliberately generous). Ties broken by faster ms/embed. If nothing fits the budget, it falls back to ranking the full set and says so explicitly rather than returning nothing.

What it doesn't know, stated plainly rather than hidden behind a confident-sounding number:

  • RSS is measured for a single model loaded and queried serially, one request at a time, in this one process. A real server holding the model resident under concurrent requests will use more memory than this measurement shows.
  • Latency is single-request serial ms/embed — not throughput under concurrency, not batched-request latency.
  • The RAM/latency thresholds are simple fixed fractions/ceilings (--ram-headroom, --latency-threshold), not a learned or host-class-aware model.
  • "Available RAM" comes from Node's os.freemem(), which on macOS in particular tends to report far less than what's actually usable — macOS treats most inactive/file-cache pages as reclaimable-but-not-"free", so os.freemem() can read a couple of GiB on a machine that's actually got plenty of headroom. On Linux, os.freemem() is closer to the truth (MemFree, not MemAvailable) but still not identical to it. Treat the RAM-budget gate as directionally useful, not authoritative — a "didn't fit the budget" fallback note is worth a second look on macOS specifically before concluding a model genuinely doesn't fit.

The recommendation always cites the actual measured numbers it's based on (e.g. "X because p@3 0.984 vs 0.976 (MRR 0.950 vs 0.946) at 22.1ms/embed and 612 MiB peak RSS on your 40.0GiB-available metal host") — never a bare model name with no evidence.

Share schema

--share writes one JSON file per model:

{
  "toolVersion": "0.1.0",
  "timestamp": "2026-07-12T00:00:00.000Z",
  "model": {
    "name": "nomic-embed-text-v1.5",
    "fileBasename": "nomic-embed-text-v1.5.Q4_K_M.gguf",
    "sha256": "…",
    "quant": "Q4_K_M",
    "paramsApprox": 136731648,
    "dims": 768
  },
  "hardware": {
    "label": "local-m4-mini",
    "platform": "darwin",
    "arch": "arm64",
    "cpuModel": "Apple M4 Pro",
    "ramGiB": 48,
    "gpu": "Apple M4 Pro",
    "backend": "metal"
  },
  "results": {
    "aggregate": { "n": 126, "p3": 0.976, "mrr": 0.946 },
    "perKind": { "stress": { … }, "trap": { … }, "hard": { … }, "clean": { … } },
    "msPerEmbedSerialWarm": 22.1,
    "peakRssMiB": 612.1
  }
}

Privacy: this document NEVER includes a hostname, a filesystem path, or a username. model.fileBasename is a basename only (never the directory it lives in); hardware.label is whatever freeform string you passed to --label — it defaults to nothing, never your machine's real hostname. test/share-schema.test.ts gates this contract.

Stubbed: the eventual hosted submission endpoint (a site to browse shared results — Nathan's "an option to share benchmarks") doesn't exist yet. --share writes the file locally and prints submission endpoint not yet configured — file saved at <path>; the endpoint URL in the code is a placeholder constant, and no network call is ever made.

Library use

The public API (src/index.ts) has no process.exit, no console.* calls anywhere in its call graph — runBenchmark()/recommend() return structured data (BatchResult/RecommendResult); progress is reported via an optional onProgress callback; rendering (pretty text / JSON) lives separately in src/format.ts. src/cli.ts is a thin argv-parsing layer on top, and is the only place in the package that prints or sets an exit code.

This shape is deliberate: a future flair bench subcommand on the main flair CLI is expected to import runBenchmark/recommend directly and drive its own UI, rather than shelling out to this package's bin.

import { runBenchmark, recommend } from "@tpsdev-ai/flair-bench";

const batch = await runBenchmark({ modelFiles: ["./model.gguf"], label: "my-host" });
console.log(batch.models[0].aggregate); // { n: 126, p3: 0.976, mrr: 0.946 }

const picked = await recommend({ modelFiles: ["./a.gguf", "./b.gguf"] });
console.log(picked.recommendation?.reason);

Development

bun test                 # unit tests (scorer, prefixes, cosine, recommend heuristic, share schema, sync checks)
bun run build             # tsc build to dist/
bun run sync:corpus       # regenerate src/corpus-v2.ts from the harness source (run after editing the harness corpus)

test/corpus-sync.test.ts and test/scorer-sync.test.ts only pass inside a full monorepo checkout (they read test/bench/recall-harness/ two levels up) — that's expected; they're the drift guard for maintainers, not something a consumer of the published package ever runs.