prompt-snapshot

Vitest-for-prompts. A file-based prompt eval runner with snapshots and assertions. No dashboard, no SaaS, no sign-up — just .prompt.ts files in your repo and a CLI that runs them in CI.

Why this exists

If you ship any LLM feature, you should be testing your prompts the way you test your code — in CI, on every change. The existing options (Langfuse, DeepEval, Phoenix, Promptfoo) are heavy observability platforms shaped for enterprise teams: SDK init, hosted dashboards, sign-ups, YAML configs, whole new abstractions to learn.

A solo dev or small team wants this instead:

• Colocated .prompt.ts files next to the code that uses the prompt.
• A TypeScript-native API with types, autocomplete, and Zod schemas.
• Snapshot testing — run a prompt, capture the output, fail the build if it drifts.
• Deterministic assertions (contains, matchesSchema, …) for the parts that matter.
• One CLI, JSON output, stable exit codes — CI-friendly on day one.

That's what prompt-snapshot is.

30-second demo

Create summarizer.prompt.ts:

import { defineSuite, contains, shorterThan } from "@swarmclawai/prompt-snapshot";

export default defineSuite({
  name: "summarizer",
  model: { provider: "anthropic", model: "claude-haiku-4-5", temperature: 0 },
  prompt: ({ doc }: { doc: string }) =>
    `Summarize the following in one sentence.\n\n${doc}`,
  cases: [
    {
      name: "short-doc",
      input: { doc: "The cat sat on the mat. The dog watched from across the room." },
      assertions: [contains("cat", { caseInsensitive: true }), shorterThan(300)],
      snapshot: true,
    },
  ],
});

Run it:

export ANTHROPIC_API_KEY=sk-...
npx @swarmclawai/prompt-snapshot@latest run

First run writes a snapshot to __snapshots__/summarizer.snap.json. Every run after that fails if the output drifts. prompt-snapshot update refreshes snapshots on purpose.

Install

pnpm add -D @swarmclawai/prompt-snapshot
# or
npm i -D @swarmclawai/prompt-snapshot

Commands

Command	Purpose
prompt-snapshot run [path]	Run every suite under a path
prompt-snapshot update [path]	Run + rewrite snapshots
prompt-snapshot list [path]	Discover suites without running
prompt-snapshot help-agents	Print the machine-readable catalog

Every command accepts --json and returns a one-line JSON envelope. Exit codes: 0 all passed, 1 some failed or suites wouldn't load, 2 internal error.

Assertions

Assertion	Checks
contains(s, { caseInsensitive? })	output contains the substring
notContains(s, { caseInsensitive? })	output does not contain the substring
matches(/regex/)	output matches a regex
matchesSchema(zodSchema)	output is JSON (possibly fenced) and matches the schema
startsWith(s)	trimmed output starts with the prefix
oneOf([...])	trimmed output equals one of the options
longerThan(n) / shorterThan(n)	char length bound

Combine them freely per case. Failures are reported with rule name + message.

Snapshots

Set snapshot: true on a case (or on the suite) and the first run writes a normalized copy of the output to <dir>/__snapshots__/<suite>.snap.json. Subsequent runs diff the new output against the saved one and fail on drift. Run with update to rewrite.

Normalization trims trailing whitespace and collapses trailing newlines — so whitespace noise doesn't cause false failures, but substantive drift does.

Providers

Provider	Env var	Notes
anthropic	ANTHROPIC_API_KEY	Uses @anthropic-ai/sdk. Defaults: temperature: 0, max_tokens: 1024
openai	OPENAI_API_KEY	Uses openai SDK
openai-compatible	varies	Point baseURL at Ollama, LM Studio, vLLM, or any OpenAI-compatible API
mock	—	Only via programmatic injection (see below)

Using from Vitest

prompt-snapshot is a library first, a CLI second. Drive it directly from Vitest if you prefer:

import { test, expect } from "vitest";
import { runSuite, createMockProvider } from "@swarmclawai/prompt-snapshot";
import suite from "./summarizer.prompt.js";

test("summarizer suite", async () => {
  const res = await runSuite(suite, {
    provider: () => createMockProvider({ respond: () => "A cat story." }),
  });
  expect(res.failed).toBe(0);
});

The provider factory makes it trivial to inject a mock for unit tests and a real client for integration tests.

Built for coding agents

Every swarmclawai CLI follows the same agent conventions so Claude Code, Cursor, Cline, Aider, Codex et al can drive them without guessing:

• --json everywhere, one-line envelope on stdout
• Stderr for logs, stdout for data
• Stable exit codes: 0 / 1 / 2
• Non-interactive by default; missing inputs error cleanly
• prompt-snapshot help-agents returns the entire command catalog as JSON

See AGENTS.md for the full machine-readable reference.

How it compares

	prompt-snapshot	Langfuse	DeepEval	Promptfoo	Phoenix
Zero setup, no dashboard	✅	❌	❌	✅	❌
TypeScript-native API	✅	partial	❌	partial	❌
Snapshot testing	✅	❌	❌	❌	❌
Zod-schema assertions	✅	❌	❌	❌	❌
Pluggable providers (incl. local)	✅	✅	partial	✅	✅
CI-first	✅	partial	✅	✅	partial

Roadmap

• similarTo(reference, { threshold }) — embedding-based similarity
• judge(criteria, { model }) — LLM-as-judge (opt-in, cost-aware)
• Parallel case execution with configurable concurrency
• A thin Vitest reporter plugin
• GitHub Action template for running on every PR

Contributing

See CONTRIBUTING.md.

License

MIT