feat: add n8n_evaluations tool for reading evaluation test runs by czlonkowski · Pull Request #935 · czlonkowski/n8n-mcp

czlonkowski · 2026-07-15T21:59:40Z

What

New consolidated read-only tool n8n_evaluations exposing the evaluation test-run endpoints n8n shipped in its Public API in 2.30.0 (n8n-io/n8n#33455):

Action	n8n endpoint	Returns
`list_runs`	`GET /workflows/{id}/test-runs`	Paginated runs, `status` filter
`get_run`	`GET /workflows/{id}/test-runs/{runId}`	Run with aggregated metrics, final result, case count
`list_cases`	`GET /workflows/{id}/test-runs/{runId}/test-cases`	Per-case inputs/outputs/metrics + `executionId` (default limit 20 — cases can be large)

Why

Evaluation results are the observability signal for AI workflows: poll runs, export results to external dashboards, compare metric aggregates across prompt/model versions, catch regressions. Each case links to its execution via executionId, composing with n8n_executions for drill-down.

Error mapping

The feature has three confusable failure modes; the handlers disambiguate:

403 → the API key lacks testRun scopes. Keys created before n8n 2.30 silently don't have them and must be re-created (also covers unlicensed evaluations).
404 on a pre-2.30 instance → "requires n8n 2.30+; this instance runs X" (uses the cached instance version).
404 on 2.30+ → wrong workflow id, or a runId that belongs to a different workflow.
Empty run list → hint that runs exist only for workflows with an executed evaluation trigger.

Scope notes

Read-only by design. Trigger/cancel endpoints (feat(API): Add public API endpoints to trigger and cancel evaluation test runs n8n-io/n8n#33979) are merged upstream but unreleased; they'll be added as run/cancel actions in this same tool when n8n ships them — no schema break.
Registered in TOOL_OPERATION_PARAM, so DISABLED_TOOL_OPERATIONS=n8n_evaluations:... per-action gating works.
Skills-pack copy (data/skills/n8n-mcp-tools-expert/SKILL.md) updated in-repo; the upstream n8n-skills source repo needs the matching one-line edit before its next sync.

Testing

7 new API-client tests (URL construction, hostile-id rejection, legacy-array envelope, error transformation)
15 new handler tests (happy paths, zod validation, 403/404 disambiguation incl. unknown-version fallback, pagination defaults/notes)
Full unit suite green (5,271 tests), typecheck and build clean

Conceived by Romuald Członkowski - www.aiadvisors.pl/en

🤖 Generated with Claude Code

Three read methods for n8n's evaluation Public API (n8n >= 2.30): listTestRuns, getTestRun, listTestCases, with types transcribed from the sanitized public DTOs. Conceived by Romuald Członkowski - www.aiadvisors.pl/en Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>

handleListTestRuns / handleGetTestRun / handleListTestCases. 403 maps to API-key scope guidance (keys created before n8n 2.30 lack testRun scopes); 404 disambiguates pre-2.30 instances from wrong ids via the cached instance version. Conceived by Romuald Członkowski - www.aiadvisors.pl/en Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>

Consolidated action tool: list_runs / get_run / list_cases, wired into dispatch, per-operation gating (TOOL_OPERATION_PARAM), and tool docs. Conceived by Romuald Członkowski - www.aiadvisors.pl/en Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>

Conceived by Romuald Członkowski - www.aiadvisors.pl/en Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>

Copilot

Pull request overview

Adds a new consolidated, read-only MCP tool (n8n_evaluations) to read n8n Public API evaluation test-run data (runs, run details, and per-case results), including API client support, handler-level validation/error mapping, and documentation updates across tool registries and docs.

Changes:

Introduces N8nApiClient methods and TypeScript types for evaluation test-run endpoints (list_runs, get_run, list_cases).
Adds new MCP handlers with input validation and disambiguated 403/404 error guidance (scopes vs. pre-2.30 vs. wrong IDs).
Registers and documents the new tool across tool definitions, docs index, README, skills-pack copy, and changelog.

Reviewed changes

Copilot reviewed 15 out of 16 changed files in this pull request and generated 2 comments.

Show a summary per file

File	Description
tests/unit/services/n8n-api-client.test.ts	Adds API-client tests for evaluation endpoints (params, hostile IDs, legacy envelopes, error wrapping).
tests/unit/mcp/handlers-evaluations.test.ts	Adds handler tests for success paths, validation, pagination notes, and 403/404 mappings.
src/types/n8n-api.ts	Adds types for test run summaries, test cases, params, and list responses.
src/services/n8n-api-client.ts	Implements `listTestRuns`, `getTestRun`, and `listTestCases` in the API client.
src/mcp/tools-n8n-manager.ts	Registers the new `n8n_evaluations` tool and enables per-action gating via `TOOL_OPERATION_PARAM`.
src/mcp/tools-documentation.ts	Adds `n8n_evaluations` to the quick tool list.
src/mcp/tool-docs/workflow_management/n8n-evaluations.ts	New detailed tool documentation (actions, prerequisites, examples, pitfalls).
src/mcp/tool-docs/workflow_management/index.ts	Exports the new evaluations tool doc.
src/mcp/tool-docs/index.ts	Registers `n8n_evaluations` in the documentation map.
src/mcp/server.ts	Adds server-side validation/dispatch wiring for `n8n_evaluations`.
src/mcp/handlers-n8n-manager.ts	Adds Zod schemas, handler implementations, and error-mapping for evaluation test runs.
README.md	Documents `n8n_evaluations` in the tools list.
package.json	Bumps package version to 2.65.0.
package-lock.json	Updates lockfile for the release.
data/skills/n8n-mcp-tools-expert/SKILL.md	Updates skills-pack tool list to include `n8n_evaluations`.
CHANGELOG.md	Adds a 2.65.0 entry describing the new evaluations tool and behavior.

💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.

czlonkowski · 2026-07-16T08:23:13Z

+      case 'n8n_evaluations':
+        // Requires action parameter, runId validation done in dispatch based on action
+        validationResult = args.action
+          ? { valid: true, errors: [] }
+          : { valid: false, errors: [{ field: 'action', message: 'action is required' }] };
+        break;


Fixed in cd4e881 — the first-layer validation case now reports a missing workflowId alongside action.

czlonkowski · 2026-07-16T08:23:15Z

+    const response = await client.listTestRuns(input.workflowId, {
+      status: input.status as TestRunStatus | undefined,
+      limit: input.limit || 100,
+      cursor: input.cursor,
+    });
+
+    return {
+      success: true,
+      data: {
+        testRuns: response.data,
+        returned: response.data.length,
+        nextCursor: response.nextCursor,
+        hasMore: !!response.nextCursor,
+        ...(response.data.length === 0 ? {
+          _note: 'No test runs. Runs exist only for workflows with an evaluation trigger that have been executed at least once.'
+        } : {}),
+        ...(response.nextCursor ? {
+          _note: 'More test runs available. Use cursor to get next page.'
+        } : {})
+      }
+    };


Fixed in cd4e881 — replaced the two conditional spreads with a single computed note (empty-list hint takes precedence, and it's now filter-aware when a status filter matches nothing).

github-actions · 2026-07-15T22:11:06Z

Test Results Summary

📊 Artifacts

Generated at Thu, 16 Jul 2026 11:02:50 GMT
Commit: 6aee80f
Run: #1318

codecov · 2026-07-15T22:11:07Z

Codecov Report

❌ Patch coverage is 86.12100% with 39 lines in your changes missing coverage. Please review.

Files with missing lines	Patch %	Lines
src/mcp/server.ts	0.00%	29 Missing ⚠️
src/mcp/handlers-n8n-manager.ts	93.96%	7 Missing ⚠️
src/services/n8n-api-client.ts	91.66%	3 Missing ⚠️

📢 Thoughts on this report? Let us know!

npm install in the feature worktree added peer entries (playwright, browserbase, watsonx and friends) unrelated to this change. Restore the lockfile from main and re-apply only the version bump. Conceived by Romuald Członkowski - www.aiadvisors.pl/en Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>

- Consolidate the three duplicated catch blocks into handleTestRunError (follows the file's handleCrudError idiom); single computed _note - Fetch the instance version on 404 when the cache is cold, so a pre-2.30 instance gets version guidance on a session's first call - Filter-aware empty-list note (status filter no longer implies evaluations are unconfigured); 403 hint also names licensing and workflow access as causes - Validate workflowId at the first validation layer (Copilot) - Tests: empty-string coercion, cursor passthrough, filter-aware note, cold-cache version fetch incl. fetch failure, 403 on get_run Conceived by Romuald Członkowski - www.aiadvisors.pl/en Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>

Copilot

Pull request overview

Copilot reviewed 15 out of 16 changed files in this pull request and generated 1 comment.

+      return {
+        success: false,
+        error: 'Workflow or test run not found. A runId must belong to the given workflowId; check both ids.',
+        code: error.code,
+      };


The previous restore (899c467) used the local main ref, which predated the n8n 2.30.x dependency update - npm ci failed with missing @smithy/* entries. origin/main's lockfile is what CI validated on #933; re-apply only the 2.65.0 version fields on top of it. Conceived by Romuald Członkowski - www.aiadvisors.pl/en Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>

Copilot

Pull request overview

Copilot reviewed 15 out of 16 changed files in this pull request and generated no new comments.

Live testing against a pre-2.30 instance showed the test-runs path is swallowed by the 2.29-era workflow-versions route, whose spec rejects unknown query params - so the forced limit=100 produced 'Unknown query parameter' instead of the 404 our error mapping explains. n8n's server default is the same 100; send limit only when the caller sets one. Conceived by Romuald Członkowski - www.aiadvisors.pl/en Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>

Copilot

Pull request overview

Copilot reviewed 15 out of 16 changed files in this pull request and generated no new comments.

czlonkowski · 2026-07-16T10:27:00Z

Live end-to-end verification against n8n 2.30.6 (self-hosted, SQLite):

list_runs → completed run with aggregated metrics (custom exact_match: 0.667 = mean of 1/1/0, plus n8n's automatic token/executionTime metrics), finalResult: success, testCaseCount: 3
get_run → matching run detail
list_cases → 3 cases with per-case metrics and executionId linking to real executions
Error mappings verified live: 403 scope guidance fired on a pre-2.30 API key (before key re-creation); 404 ids-guidance fired for a bogus runId; empty-list hint fired on a workflow without runs
Live testing on a pre-2.30 instance also caught and fixed the forced-limit bug (0a07f57 → follow-up in the branch)

Observation for the record: on this SQLite instance the API returns executionId as a number (e.g. 5) although upstream's DTO declares string | null — we pass it through untouched, and n8n_executions accepts it stringified, so no action needed on our side.

n8n-mcp v2.65.0 (PR czlonkowski/n8n-mcp#935) adds a read-only tool for n8n 2.30's evaluation Public API. Skill updates: - tools-expert SKILL.md: added to the requires-API tool inventory - tools-expert WORKFLOW_GUIDE.md: full section (list_runs/get_run/ list_cases usage, key-scope 403 gotcha, three-way 404, pagination guidance, executionId drill-down into n8n_executions) + Additional Tools entry - using-n8n-mcp-skills router: Test & run bullet with the read-only and key re-creation caveats All guidance mirrors live-verified behavior against n8n 2.30.6. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>

Mirrors czlonkowski/n8n-skills@05eea48 so the shipped data/skills copy matches the source: WORKFLOW_GUIDE.md section for n8n_evaluations and the router's Test & run bullet. Conceived by Romuald Członkowski - www.aiadvisors.pl/en Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>

Copilot

Pull request overview

Copilot reviewed 17 out of 18 changed files in this pull request and generated no new comments.

czlonkowski and others added 4 commits July 15, 2026 23:50

docs: document n8n_evaluations tool; bump to 2.65.0

4faa8c1

Conceived by Romuald Członkowski - www.aiadvisors.pl/en Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>

Copilot AI review requested due to automatic review settings July 15, 2026 21:59

Copilot started reviewing on behalf of czlonkowski July 15, 2026 22:00 View session

Copilot AI reviewed Jul 15, 2026

View reviewed changes

czlonkowski and others added 2 commits July 16, 2026 10:22

Copilot AI review requested due to automatic review settings July 16, 2026 08:22

Copilot started reviewing on behalf of czlonkowski July 16, 2026 08:22 View session

czlonkowski mentioned this pull request Jul 16, 2026

n8n_evaluations: add run/cancel actions when n8n releases the trigger/cancel Public API #936

Open

Copilot AI reviewed Jul 16, 2026

View reviewed changes

Comment thread src/mcp/handlers-n8n-manager.ts

Comment on lines +1973 to +1977

return {

success: false,

error: 'Workflow or test run not found. A runId must belong to the given workflowId; check both ids.',

code: error.code,

};

Copilot AI review requested due to automatic review settings July 16, 2026 08:32

Copilot started reviewing on behalf of czlonkowski July 16, 2026 08:32 View session

Copilot AI reviewed Jul 16, 2026

View reviewed changes

Copilot AI review requested due to automatic review settings July 16, 2026 08:55

Copilot started reviewing on behalf of czlonkowski July 16, 2026 08:55 View session

Copilot AI reviewed Jul 16, 2026

View reviewed changes

czlonkowski mentioned this pull request Jul 16, 2026

Node DB missing nodes-base.evaluationTrigger (Evaluation Trigger) #937

Closed

Copilot AI review requested due to automatic review settings July 16, 2026 10:54

Copilot started reviewing on behalf of czlonkowski July 16, 2026 10:55 View session

Copilot AI reviewed Jul 16, 2026

View reviewed changes

czlonkowski merged commit 62f635b into main Jul 16, 2026
15 checks passed

czlonkowski deleted the feat/n8n-evaluations-tool branch July 16, 2026 11:24

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Uh oh!

feat: add n8n_evaluations tool for reading evaluation test runs#935

feat: add n8n_evaluations tool for reading evaluation test runs#935
czlonkowski merged 9 commits into
mainfrom
feat/n8n-evaluations-tool

czlonkowski commented Jul 15, 2026

Uh oh!

Copilot AI left a comment

Uh oh!

czlonkowski Jul 16, 2026

Uh oh!

czlonkowski Jul 16, 2026

Uh oh!

github-actions Bot commented Jul 15, 2026 •

edited

Loading

Uh oh!

codecov Bot commented Jul 15, 2026 •

edited

Loading

Uh oh!

Copilot AI left a comment

Uh oh!

Copilot AI left a comment

Uh oh!

Copilot AI left a comment

Uh oh!

czlonkowski commented Jul 16, 2026

Uh oh!

Copilot AI left a comment

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

2 participants

Uh oh!

Conversation

czlonkowski commented Jul 15, 2026

What

Why

Error mapping

Scope notes

Testing

Uh oh!

Copilot AI left a comment

Choose a reason for hiding this comment

Pull request overview

Reviewed changes

Uh oh!

czlonkowski Jul 16, 2026

Choose a reason for hiding this comment

Uh oh!

czlonkowski Jul 16, 2026

Choose a reason for hiding this comment

Uh oh!

github-actions Bot commented Jul 15, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Test Results Summary

📊 Artifacts

Uh oh!

codecov Bot commented Jul 15, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Codecov Report

Uh oh!

Copilot AI left a comment

Choose a reason for hiding this comment

Pull request overview

Uh oh!

Copilot AI left a comment

Choose a reason for hiding this comment

Pull request overview

Uh oh!

Copilot AI left a comment

Choose a reason for hiding this comment

Pull request overview

Uh oh!

czlonkowski commented Jul 16, 2026

Uh oh!

Copilot AI left a comment

Choose a reason for hiding this comment

Pull request overview

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

2 participants

github-actions Bot commented Jul 15, 2026 •

edited

Loading

codecov Bot commented Jul 15, 2026 •

edited

Loading