Add evals support (offline + online) under logfire/evals

Co-authored-by: Petyo Ivanov <underlog@gmail.com>

Author: dmontagu

.changeset/evals-port.md

@@ -0,0 +1,12 @@
+---
+'logfire': minor
+'@pydantic/logfire-node': minor
+---
+
+Add evals support — offline + online evaluations.
+
+A new `logfire/evals` subpath exports `Dataset`, `Case`, `Evaluator`, built-in evaluators (`Equals`, `EqualsExpected`, `Contains`, `IsInstance`, `MaxDuration`, `HasMatchingSpan`, `LLMJudge`), report-level evaluators (`ConfusionMatrixEvaluator`, `PrecisionRecallEvaluator`, `ROCAUCEvaluator`, `KolmogorovSmirnovEvaluator`), and `withOnlineEvaluation` for runtime monitoring.
+
+Emitted OTel spans, log events, and report analyses are wire-compatible with the Python `pydantic-evals` package, so experiments, cases, report-level charts, and live evaluations show up automatically in the Logfire web UI without any additional configuration. Datasets serialize to / deserialize from the same YAML and JSON format Python uses (`Dataset.toFile` / `Dataset.fromFile`, `Dataset.jsonSchema()`), with filesystem helpers supported in Node, Bun, and Deno.
+
+`logfire.configure()` now auto-installs the evals span-tree processor; users on a custom `TracerProvider` can install it manually with `getEvalsSpanProcessor()` from `logfire/evals`.

.gitignore

@@ -1,5 +1,7 @@
 node_modules
 packages/*/*.tgz
+coverage
+packages/*/coverage
 .env
 scratch/
 **/.turbo

.node-version

@@ -0,0 +1 @@
+24.14.1

AGENTS.md

@@ -0,0 +1,137 @@
+# Agent Instructions
+
+This file is the canonical project guide for coding agents. `CLAUDE.md` is kept as a symlink for tools that still look for that name. Keep this file tool-neutral and update it when project commands, package layout, or non-obvious conventions change.
+
+## Project Overview
+
+This repository is the Pydantic Logfire JavaScript SDK monorepo. It provides OpenTelemetry-based tracing and logging packages for Node.js, browsers, Cloudflare Workers, and standalone manual tracing.
+
+## Repository Layout
+
+- `packages/logfire-api` publishes `logfire`, the core manual tracing API.
+- `packages/logfire-node` publishes `@pydantic/logfire-node`, which adds Node.js SDK setup and automatic instrumentation.
+- `packages/logfire-cf-workers` publishes `@pydantic/logfire-cf-workers`, which adapts Logfire to Cloudflare Workers.
+- `packages/logfire-browser` publishes `@pydantic/logfire-browser`, which adapts Logfire to browser tracing.
+- `packages/tooling-config` contains shared build and lint configuration.
+- `examples/` contains runnable examples for Express, Next.js, Deno, Cloudflare Workers, browser usage, and related integrations.
+
+## Environment
+
+- Use the Node.js version in `.node-version`. The root `package.json` also enforces Node.js 24 through `engines`.
+- Use pnpm 10.28.0. The package manager is pinned in `packageManager`.
+- Run workspace commands from the repository root unless a package-level command is explicitly needed.
+
+## Useful Commands
+
+Install dependencies:
+
+```bash
+pnpm install
+```
+
+Build all packages:
+
+```bash
+pnpm run build
+```
+
+Build packages in watch-style development mode:
+
+```bash
+pnpm run dev
+```
+
+Run all package tests:
+
+```bash
+pnpm run test
+```
+
+Run all checks before broad or release-oriented changes:
+
+```bash
+pnpm run check
+```
+
+Run focused checks for one package:
+
+```bash
+pnpm --filter logfire test
+pnpm --filter @pydantic/logfire-node test
+pnpm --filter @pydantic/logfire-browser typecheck
+```
+
+Run a single Vitest test by name from a package filter:
+
+```bash
+pnpm --filter logfire test -- -t "test name pattern"
+```
+
+Format or verify formatting:
+
+```bash
+pnpm run format
+pnpm run format-check
+```
+
+Add a changeset when a package-visible change needs a release note or version bump:
+
+```bash
+pnpm run changeset-add
+```
+
+## Architecture Notes
+
+- `logfire` wraps OpenTelemetry tracing APIs with convenience methods such as `span`, `startSpan`, `info`, `debug`, `warn`, and `error`.
+- Runtime packages depend on `logfire` and add environment-specific configuration and instrumentation.
+- Message templates are formatted by `logfireFormatWithExtras()` in `packages/logfire-api/src/formatter.ts`; template fields become structured attributes.
+- Sensitive data scrubbing is handled by `AttributeScrubber` in `packages/logfire-api/src/AttributeScrubber.ts`.
+- Trace IDs use ULIDs through `ULIDGenerator`.
+- Logfire spans use `logfire.span_type`: `log` for point-in-time events and `span` for duration-based work.
+
+## Development Conventions
+
+- Prefer existing package patterns, helpers, and OpenTelemetry abstractions over introducing new wrappers.
+- Keep changes scoped to the package or example relevant to the task.
+- Update examples or docs when public behavior, configuration, or package usage changes.
+- Add or update tests for behavior changes. If a package has minimal tests or `--passWithNoTests`, still run typecheck/build for that package when relevant.
+- Avoid adding production dependencies without a clear need; keep workspace dependency and lockfile changes together.
+- Do not put agent-specific or vendor-specific instructions here unless they are explicitly about repository compatibility. Use generic wording that applies to any coding agent.
+
+## Testing Guidance
+
+- Tests use Vitest and usually live alongside source files as `*.test.ts`.
+- Prefer exact assertions over fuzzy matching for stable output. Use `toBe` or `toEqual` with deterministic inputs instead of `toContain` or broad regex matching.
+- When testing formatted errors or stack output, mock stack strings so assertions stay deterministic.
+- For changes under `packages/logfire-api/src/evals`, consider the focused coverage script:
+
+```bash
+pnpm --filter logfire run coverage:evals
+```
+
+## Package-Specific Notes
+
+- `packages/logfire-node/src/logfireConfig.ts` owns Node SDK configuration and environment variable handling.
+- Relevant environment variables include `LOGFIRE_TOKEN`, `LOGFIRE_SERVICE_NAME`, `LOGFIRE_SERVICE_VERSION`, `LOGFIRE_ENVIRONMENT`, `LOGFIRE_CONSOLE`, `LOGFIRE_SEND_TO_LOGFIRE`, and `LOGFIRE_DISTRIBUTED_TRACING`.
+- `packages/logfire-api` is the base API package and should not depend on runtime-specific packages.
+- Cloudflare Workers code should stay compatible with Worker runtime constraints.
+- Browser code should avoid Node-only APIs.
+
+## Examples
+
+Use examples to validate integration behavior when package-level tests do not cover the runtime path. Check the target example's `package.json` before running it, because scripts vary by example.
+
+Typical flow:
+
+```bash
+cd examples/express
+pnpm install
+pnpm run dev
+```
+
+## Agent Guidance Maintenance
+
+- Keep this file concise and operational. Prefer links or file paths over copied explanations when the code is self-describing.
+- If a subdirectory needs different build, test, or safety rules, add a nested `AGENTS.md` in that directory instead of growing this root file.
+- Instructions closest to the edited file should be treated as more specific than this root file.
+- Preserve the `CLAUDE.md` symlink unless the repository drops compatibility with tools that read it.

CLAUDE.md

@@ -0,0 +1 @@
+AGENTS.md

CLAUDE.md

@@ -259,6 +259,42 @@ Optionally, you can use the Logfire API package for creating manual spans.
 Install the `logfire` NPM package and call the respective methods
 from your code.
 
+## Evaluations
+
+The `logfire/evals` subpath provides offline datasets and online evaluation
+wrappers that emit Logfire-compatible OpenTelemetry spans and log events. Add
+`logfire` as a direct dependency in projects that import this subpath.
+
+```ts
+import { Case, Dataset, Equals, EqualsExpected, withOnlineEvaluation } from 'logfire/evals'
+
+const dataset = new Dataset<{ text: string }, string>({
+  cases: [new Case({ expectedOutput: 'HELLO', inputs: { text: 'hello' }, name: 'hello' })],
+  evaluators: [new EqualsExpected()],
+  name: 'uppercase',
+})
+
+await dataset.evaluate(({ text }) => text.toUpperCase())
+
+const monitored = withOnlineEvaluation(async (text: string) => text.toUpperCase(), {
+  evaluators: [new Equals({ value: 'HELLO' })],
+  target: 'uppercase',
+})
+await monitored('hello')
+```
+
+Dataset file helpers are available in Node, Bun, and Deno. Browser and
+Cloudflare Worker usage is limited to in-memory datasets and online evaluation;
+browser offline runs should keep `maxConcurrency: 1` because there is no
+`AsyncLocalStorage` equivalent.
+
+Dataset YAML/JSON uses the Python `pydantic-evals` field names for portable
+files, for example `predicted_from`, `expected_from`, and snake_case
+`SpanQuery` keys. Online evaluator `context.inputs` is built from JavaScript
+function parameter names when they can be inspected; pass
+`extractArgs: ['name', ...]` when bundled or minified code needs stable input
+names, or `extractArgs: false` to keep positional input values.
+
 ### Configuring the instrumentation
 
 The `logfire.configure` function accepts a set of configuration options that