diff --git a/CHANGELOG.md b/CHANGELOG.md index 62231d3c7..90dfb4153 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -7,6 +7,12 @@ and this project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0 ## [Unreleased] +## [2.65.0] - 2026-07-15 + +### Added + +- **`n8n_evaluations` tool — read evaluation test runs.** n8n 2.30 shipped Public API read endpoints for evaluation test runs (n8n-io/n8n#33455); the new consolidated tool exposes them: `action='list_runs'` (paginated, status filter), `action='get_run'` (aggregated metrics and final result), and `action='list_cases'` (per-case inputs/outputs/metrics, default limit 20 — paginate, cases can be large). Requires n8n ≥ 2.30 **and an API key created on 2.30+**: older keys silently lack the `testRun` scopes and get a 403 — the tool's error messages spell this out, and a 404 on a pre-2.30 instance is disambiguated from a wrong workflow/run id via the cached instance version. Read-only; triggering or cancelling runs via API is not yet supported by n8n (upstream n8n-io/n8n#33979 is merged but unreleased) and will be added as new actions when it ships. + ## [2.64.1] - 2026-07-14 ### Changed diff --git a/README.md b/README.md index 68626789b..d914b6585 100644 --- a/README.md +++ b/README.md @@ -388,6 +388,7 @@ These tools require `N8N_API_URL` and `N8N_API_KEY` in your configuration. #### Execution Management - **`n8n_test_workflow`** - Test/trigger workflow execution (webhook, form, chat) - **`n8n_executions`** - Unified execution management (list, get, delete) +- **`n8n_evaluations`** - Read evaluation test runs (list runs, aggregated metrics, per-case results; n8n 2.30+) #### Data Table Management - **`n8n_manage_datatable`** - Manage n8n data tables and rows (list, get, create, update, delete) diff --git a/data/skills/n8n-mcp-tools-expert/SKILL.md b/data/skills/n8n-mcp-tools-expert/SKILL.md index 0e4a3c5a5..9a7ebd567 100644 --- a/data/skills/n8n-mcp-tools-expert/SKILL.md +++ b/data/skills/n8n-mcp-tools-expert/SKILL.md @@ -293,6 +293,7 @@ See [OPERATIONS_GUIDE.md](OPERATIONS_GUIDE.md) for examples. - n8n_list_workflows, n8n_get_workflow, n8n_delete_workflow - n8n_test_workflow - n8n_executions +- n8n_evaluations (n8n 2.30+; read evaluation test runs — API key must be created on 2.30+ for testRun scopes) - n8n_deploy_template - n8n_workflow_versions - n8n_autofix_workflow diff --git a/data/skills/n8n-mcp-tools-expert/WORKFLOW_GUIDE.md b/data/skills/n8n-mcp-tools-expert/WORKFLOW_GUIDE.md index 62f585932..1835bf840 100644 --- a/data/skills/n8n-mcp-tools-expert/WORKFLOW_GUIDE.md +++ b/data/skills/n8n-mcp-tools-expert/WORKFLOW_GUIDE.md @@ -843,6 +843,53 @@ n8n_executions({ --- +## n8n_evaluations (EVALUATION TEST RUNS) + +**Use when**: Reading evaluation test runs — polling a run started in the editor, comparing metrics across runs, pulling per-case results into a report or dashboard. + +Read-only. Requires n8n >= 2.30 **and an API key created on 2.30+** — keys created earlier silently lack the `testRun` scopes, so a 403 means "re-create the API key", not a bug. Runs exist only for workflows with an evaluation trigger that have been run from the n8n editor; triggering runs via the public API is not yet supported by n8n (planned upstream, will arrive as `run`/`cancel` actions). + +### List Test Runs +```javascript +n8n_evaluations({ + action: "list_runs", + workflowId: "workflow-id", + status: "completed" // new, running, completed, error, cancelled +}) +``` + +### Get a Run (aggregated metrics) +```javascript +n8n_evaluations({ + action: "get_run", + workflowId: "workflow-id", + runId: "run-id" +}) +// → status, finalResult (success/error/warning), metrics, testCaseCount +// metrics is a flat name → number/boolean map: your custom metrics plus +// n8n's automatic ones (promptTokens, completionTokens, totalTokens, executionTime) +``` + +### Per-Case Results +```javascript +n8n_evaluations({ + action: "list_cases", + workflowId: "workflow-id", + runId: "run-id" +}) +// Default limit 20 — per-case inputs/outputs can be large; paginate with +// cursor rather than raising the limit. +// Each case carries an executionId — drill into the underlying execution +// with n8n_executions({action: "get", id: executionId, mode: "error"}) +``` + +**Gotchas**: +- A 404 can mean three things: the instance predates 2.30, the workflowId is wrong, or the runId belongs to a different workflow (the tool's error message disambiguates using the instance version) +- Evaluations are license/quota-gated in n8n — an instance without the feature simply has no runs +- Compare `metrics` across runs of the same workflow to catch prompt/model regressions + +--- + ## Workflow Lifecycle **Standard pattern**: @@ -946,6 +993,7 @@ update → update → update → ... (56s avg between edits) - `n8n_workflow_versions` - Version control & rollback - `n8n_test_workflow` - Trigger execution - `n8n_executions` - Manage executions +- `n8n_evaluations` - Read evaluation test runs (n8n 2.30+, read-only) - `n8n_manage_datatable` - Data table and row management - `n8n_manage_credentials` - Credential CRUD + schema discovery - `n8n_audit_instance` - Security audit (built-in + custom scan) diff --git a/data/skills/using-n8n-mcp-skills/SKILL.md b/data/skills/using-n8n-mcp-skills/SKILL.md index 852a65414..a9e3ed6b2 100644 --- a/data/skills/using-n8n-mcp-skills/SKILL.md +++ b/data/skills/using-n8n-mcp-skills/SKILL.md @@ -1,6 +1,6 @@ --- name: using-n8n-mcp-skills -description: Use when building, editing, validating, testing, or debugging an n8n workflow through the n8n-mcp MCP server — designing a flow, configuring a node, writing an expression or Code node, wiring credentials, or fixing one that misbehaves. The entry-point skill for the n8n-mcp-skills pack: it routes you to the right specialist skill, gives working knowledge of every n8n-mcp tool from turn one, and states the rules that keep workflows from breaking in production. Always consult it first on any n8n, workflow, node, or automation task — even a quick one-off, and even when the user names no skill — because n8n's surface drifts between versions and the specialist skills prevent silent failures. +description: "Use when building, editing, validating, testing, or debugging an n8n workflow through the n8n-mcp MCP server — designing a flow, configuring a node, writing an expression or Code node, wiring credentials, or fixing one that misbehaves. The entry-point skill for the n8n-mcp-skills pack: it routes you to the right specialist skill, gives working knowledge of every n8n-mcp tool from turn one, and states the rules that keep workflows from breaking in production. Always consult it first on any n8n, workflow, node, or automation task — even a quick one-off, and even when the user names no skill — because n8n's surface drifts between versions and the specialist skills prevent silent failures." --- # Using the n8n-mcp Skills @@ -130,6 +130,7 @@ closes the gap where a tool's full description isn't loaded until first use. **Test & run** - `n8n_test_workflow` — runs real nodes (Code, HTTP, DB writes, sends all fire). Ask the user before running when side effects exist. - `n8n_executions` — list/inspect executions. **There is no `execute_workflow` tool.** +- `n8n_evaluations` — read evaluation test runs (n8n ≥ 2.30): list runs, aggregated metrics, per-case results. Read-only — runs are started from the n8n editor, not the API; a 403 usually means the API key predates 2.30 (re-create it for the testRun scopes). **Data, credentials, audit** - `n8n_manage_datatable` — Data Table CRUD, filtering, dry-run. diff --git a/package-lock.json b/package-lock.json index 5913d787e..a3a838a97 100644 --- a/package-lock.json +++ b/package-lock.json @@ -1,12 +1,12 @@ { "name": "n8n-mcp", - "version": "2.64.1", + "version": "2.65.0", "lockfileVersion": 3, "requires": true, "packages": { "": { "name": "n8n-mcp", - "version": "2.64.1", + "version": "2.65.0", "license": "MIT", "dependencies": { "@modelcontextprotocol/sdk": "1.28.0", diff --git a/package.json b/package.json index e1b55a40f..f538ecfad 100644 --- a/package.json +++ b/package.json @@ -1,6 +1,6 @@ { "name": "n8n-mcp", - "version": "2.64.1", + "version": "2.65.0", "description": "Integration between n8n workflow automation and Model Context Protocol (MCP)", "main": "dist/index.js", "types": "dist/index.d.ts", diff --git a/src/mcp/handlers-n8n-manager.ts b/src/mcp/handlers-n8n-manager.ts index df3ca9fc7..6225982b1 100644 --- a/src/mcp/handlers-n8n-manager.ts +++ b/src/mcp/handlers-n8n-manager.ts @@ -13,6 +13,7 @@ import { ExecutionFilterOptions, ExecutionMode, Credential, + TestRunStatus, } from '../types/n8n-api'; import type { TriggerType, TestWorkflowInput } from '../triggers/types'; import { @@ -525,6 +526,25 @@ const listExecutionsSchema = z.object({ includeData: z.boolean().optional(), }); +const listTestRunsSchema = z.object({ + workflowId: z.string(), + status: optionalEmptyAware(z.enum(['new', 'running', 'completed', 'error', 'cancelled'])), + limit: z.number().min(1).max(250).optional(), + cursor: optionalEmptyAware(z.string()), +}); + +const getTestRunSchema = z.object({ + workflowId: z.string(), + runId: z.string(), +}); + +const listTestCasesSchema = z.object({ + workflowId: z.string(), + runId: z.string(), + limit: z.number().min(1).max(250).optional(), + cursor: optionalEmptyAware(z.string()), +}); + const workflowVersionsSchema = z.object({ mode: z.enum(['list', 'get', 'rollback', 'delete', 'prune']), workflowId: z.string().optional(), @@ -1915,6 +1935,132 @@ export async function handleDeleteExecution(args: unknown, context?: InstanceCon } } +// Evaluation Test Run Handlers (n8n >= 2.30) + +const TEST_RUN_SCOPE_HINT = + 'n8n rejected the request (403). The API key lacks testRun scopes - keys created before n8n 2.30 do not have them; re-create the API key on n8n 2.30+. Other causes: evaluations not licensed on this plan, or the key\'s owner lacks access to this workflow.'; + +/** + * Builds the error response for the evaluation handlers. Mirrors handleCrudError + * but adds evaluation-specific guidance for the three failure modes that are easy + * to confuse from raw HTTP statuses alone: a pre-2.30 instance, an API key without + * testRun scopes, and a runId that does not belong to the given workflow. + */ +async function handleTestRunError(error: unknown, context?: InstanceContext): Promise { + if (error instanceof z.ZodError) { + return { success: false, error: 'Invalid input', details: { errors: error.errors } }; + } + if (error instanceof N8nApiError) { + if (error.statusCode === 403) { + return { success: false, error: TEST_RUN_SCOPE_HINT, code: error.code }; + } + if (error.statusCode === 404) { + // A 404 from a pre-2.30 instance means the endpoint doesn't exist, not + // that the ids are wrong. The version cache is usually cold on a fresh + // session, so fetch it (one extra request, failure path only). + const client = getN8nApiClient(context); + let version = client?.getCachedVersionInfo() ?? null; + if (!version && client) { + version = await client.getVersion().catch(() => null); + } + if (version && (version.major < 2 || (version.major === 2 && version.minor < 30))) { + return { + success: false, + error: `The evaluation API requires n8n 2.30 or later; this instance runs ${version.version}. Upgrade the instance to read test runs.`, + code: error.code, + }; + } + return { + success: false, + error: 'Workflow or test run not found. A runId must belong to the given workflowId; check both ids.', + code: error.code, + }; + } + return { success: false, error: getUserFriendlyErrorMessage(error), code: error.code }; + } + return { success: false, error: error instanceof Error ? error.message : 'Unknown error occurred' }; +} + +export async function handleListTestRuns(args: unknown, context?: InstanceContext): Promise { + try { + const client = ensureApiConfigured(context); + const input = listTestRunsSchema.parse(args || {}); + + // Send limit only when the caller sets one: n8n's server default is the + // same 100, and pre-2.30 instances (no test-runs routes) reject unknown + // query params before returning the 404 our error mapping explains. + const response = await client.listTestRuns(input.workflowId, { + status: input.status as TestRunStatus | undefined, + limit: input.limit, + cursor: input.cursor, + }); + + const note = response.data.length === 0 + ? (input.status + ? `No test runs with status '${input.status}' for this workflow.` + : 'No test runs. Runs exist only for workflows with an evaluation trigger that have been executed at least once.') + : response.nextCursor + ? 'More test runs available. Use cursor to get next page.' + : undefined; + + return { + success: true, + data: { + testRuns: response.data, + returned: response.data.length, + nextCursor: response.nextCursor, + hasMore: !!response.nextCursor, + ...(note ? { _note: note } : {}) + } + }; + } catch (error) { + return handleTestRunError(error, context); + } +} + +export async function handleGetTestRun(args: unknown, context?: InstanceContext): Promise { + try { + const client = ensureApiConfigured(context); + const input = getTestRunSchema.parse(args || {}); + + const response = await client.getTestRun(input.workflowId, input.runId); + + return { + success: true, + data: response + }; + } catch (error) { + return handleTestRunError(error, context); + } +} + +export async function handleListTestCases(args: unknown, context?: InstanceContext): Promise { + try { + const client = ensureApiConfigured(context); + const input = listTestCasesSchema.parse(args || {}); + + const response = await client.listTestCases(input.workflowId, input.runId, { + limit: input.limit || 20, + cursor: input.cursor, + }); + + return { + success: true, + data: { + testCases: response.data, + returned: response.data.length, + nextCursor: response.nextCursor, + hasMore: !!response.nextCursor, + ...(response.nextCursor ? { + _note: 'More test cases available. Paginate rather than raising limit - per-case inputs/outputs can be large.' + } : {}) + } + }; + } catch (error) { + return handleTestRunError(error, context); + } +} + // System Tools Handlers export async function handleHealthCheck(context?: InstanceContext): Promise { diff --git a/src/mcp/server.ts b/src/mcp/server.ts index 0076f421f..8e2055c21 100644 --- a/src/mcp/server.ts +++ b/src/mcp/server.ts @@ -1337,6 +1337,17 @@ export class N8NDocumentationMCPServer { ? { valid: true, errors: [] } : { valid: false, errors: [{ field: 'action', message: 'action is required' }] }; break; + case 'n8n_evaluations': { + // Every action of this tool requires action and workflowId; + // runId validation is done in dispatch based on action. + const evalErrors: Array<{ field: string; message: string }> = []; + if (!args.action) evalErrors.push({ field: 'action', message: 'action is required' }); + if (!args.workflowId) evalErrors.push({ field: 'workflowId', message: 'workflowId is required' }); + validationResult = evalErrors.length === 0 + ? { valid: true, errors: [] } + : { valid: false, errors: evalErrors }; + break; + } case 'n8n_manage_datatable': validationResult = args.action ? { valid: true, errors: [] } @@ -1839,6 +1850,26 @@ export class N8NDocumentationMCPServer { throw new Error(`Unknown action: ${execAction}. Valid actions: get, list, delete`); } } + case 'n8n_evaluations': { + this.validateToolParams(name, args, ['action', 'workflowId']); + const evalAction = args.action; + switch (evalAction) { + case 'list_runs': + return n8nHandlers.handleListTestRuns(args, this.instanceContext); + case 'get_run': + if (!args.runId) { + throw new Error('runId is required for action=get_run'); + } + return n8nHandlers.handleGetTestRun(args, this.instanceContext); + case 'list_cases': + if (!args.runId) { + throw new Error('runId is required for action=list_cases'); + } + return n8nHandlers.handleListTestCases(args, this.instanceContext); + default: + throw new Error(`Unknown action: ${evalAction}. Valid actions: list_runs, get_run, list_cases`); + } + } case 'n8n_health_check': // No required parameters - supports mode='status' (default) or mode='diagnostic' if (args.mode === 'diagnostic') { diff --git a/src/mcp/tool-docs/index.ts b/src/mcp/tool-docs/index.ts index 19bc1659c..7a6b3990e 100644 --- a/src/mcp/tool-docs/index.ts +++ b/src/mcp/tool-docs/index.ts @@ -22,6 +22,7 @@ import { n8nAutofixWorkflowDoc, n8nTestWorkflowDoc, n8nExecutionsDoc, + n8nEvaluationsDoc, n8nWorkflowVersionsDoc, n8nDeployTemplateDoc, n8nManageDatatableDoc, @@ -63,6 +64,7 @@ export const toolsDocumentation: Record = { n8n_autofix_workflow: n8nAutofixWorkflowDoc, n8n_test_workflow: n8nTestWorkflowDoc, n8n_executions: n8nExecutionsDoc, + n8n_evaluations: n8nEvaluationsDoc, n8n_workflow_versions: n8nWorkflowVersionsDoc, n8n_deploy_template: n8nDeployTemplateDoc, n8n_manage_datatable: n8nManageDatatableDoc, diff --git a/src/mcp/tool-docs/workflow_management/index.ts b/src/mcp/tool-docs/workflow_management/index.ts index 7f92998c4..603a7a8e3 100644 --- a/src/mcp/tool-docs/workflow_management/index.ts +++ b/src/mcp/tool-docs/workflow_management/index.ts @@ -8,6 +8,7 @@ export { n8nValidateWorkflowDoc } from './n8n-validate-workflow'; export { n8nAutofixWorkflowDoc } from './n8n-autofix-workflow'; export { n8nTestWorkflowDoc } from './n8n-test-workflow'; export { n8nExecutionsDoc } from './n8n-executions'; +export { n8nEvaluationsDoc } from './n8n-evaluations'; export { n8nWorkflowVersionsDoc } from './n8n-workflow-versions'; export { n8nDeployTemplateDoc } from './n8n-deploy-template'; export { n8nManageDatatableDoc } from './n8n-manage-datatable'; diff --git a/src/mcp/tool-docs/workflow_management/n8n-evaluations.ts b/src/mcp/tool-docs/workflow_management/n8n-evaluations.ts new file mode 100644 index 000000000..5710e19d3 --- /dev/null +++ b/src/mcp/tool-docs/workflow_management/n8n-evaluations.ts @@ -0,0 +1,71 @@ +import { ToolDocumentation } from '../types'; + +export const n8nEvaluationsDoc: ToolDocumentation = { + name: 'n8n_evaluations', + category: 'workflow_management', + essentials: { + description: 'Read evaluation test runs for a workflow: list runs, get a run with aggregated metrics, or fetch per-case results. Read-only; requires n8n >= 2.30.', + keyParameters: ['action', 'workflowId', 'runId', 'status'], + example: 'n8n_evaluations({action: "list_runs", workflowId: "abc123", status: "completed"})', + performance: 'Fast (50-200ms); list_cases payloads can be large - paginate', + tips: [ + 'action="list_runs": runs for a workflow, filterable by status, newest first', + 'action="get_run": one run with aggregated metrics and final result', + 'action="list_cases": per-case inputs/outputs/metrics - default limit 20, paginate rather than raising it', + 'Requires an API key created on n8n 2.30+ (older keys lack testRun scopes - re-create the key)', + 'Runs exist only for workflows with an evaluation trigger that have been executed' + ] + }, + full: { + description: `**Actions:** +- list_runs: List evaluation test runs for a workflow (paginated, status filter, newest first) +- get_run: Retrieve a single test run with aggregated metrics, final result, and case count +- list_cases: Retrieve per-case results of a run - inputs, outputs, metrics, and the executionId of each case + +**Prerequisites:** +- n8n 2.30.0 or later (the evaluation Public API shipped in 2.30) +- API key with testRun scopes - keys created before 2.30 do not have them and must be re-created +- Evaluation runs exist only for workflows with a configured evaluation trigger that have been run from the n8n editor (triggering runs via the public API is not yet supported by n8n) + +**Reading results:** +- Run metrics are a flat map of metric name to number/boolean (aggregates across cases) +- finalResult is success/error/warning once a run completes +- Each case links to its underlying execution via executionId - use n8n_executions with that id to inspect the full execution +- Compare metrics across runs of the same workflow to track prompt/model regressions`, + parameters: { + action: { type: 'string', required: true, description: 'Operation: "list_runs", "get_run", or "list_cases"' }, + workflowId: { type: 'string', required: true, description: 'Workflow ID the test runs belong to' }, + runId: { type: 'string', required: false, description: 'Test run ID (required for get_run and list_cases)' }, + status: { type: 'string', required: false, description: 'For list_runs: filter by "new", "running", "completed", "error", or "cancelled"' }, + limit: { type: 'number', required: false, description: 'Results per page, 1-250. Defaults: 100 (list_runs), 20 (list_cases)' }, + cursor: { type: 'string', required: false, description: 'Pagination cursor from a previous response' } + }, + returns: `list_runs: { testRuns, returned, nextCursor, hasMore }. get_run: run object { id, status, runAt, completedAt, metrics, errorCode, errorDetails, finalResult, testCaseCount, createdAt, updatedAt }. list_cases: { testCases, returned, nextCursor, hasMore } where each case has { id, status, runAt, completedAt, metrics, errorCode, errorDetails, inputs, outputs, executionId }.`, + examples: [ + 'n8n_evaluations({action: "list_runs", workflowId: "abc123"}) - all runs, newest first', + 'n8n_evaluations({action: "list_runs", workflowId: "abc123", status: "completed", limit: 10}) - recent completed runs', + 'n8n_evaluations({action: "get_run", workflowId: "abc123", runId: "run456"}) - aggregated metrics for one run', + 'n8n_evaluations({action: "list_cases", workflowId: "abc123", runId: "run456"}) - first 20 case results', + 'n8n_evaluations({action: "list_cases", workflowId: "abc123", runId: "run456", cursor: "..."}) - next page' + ], + useCases: [ + 'Poll a run started in the n8n editor and report when it completes', + 'Compare metric aggregates across runs to catch prompt or model regressions', + 'Pull per-case failures and inspect the underlying executions via n8n_executions', + 'Export evaluation results to an external dashboard' + ], + performance: 'Each call is a single n8n API request. list_cases responses carry raw per-case inputs/outputs - keep limit small and paginate.', + errorHandling: 'A 403 means the API key lacks testRun scopes (keys created before n8n 2.30 must be re-created), evaluations are not licensed on the plan, or the key\'s owner lacks access to the workflow. A 404 can mean the instance predates 2.30, the workflow id is wrong, or the runId belongs to a different workflow - the tool checks the instance version to disambiguate.', + bestPractices: [ + 'Filter list_runs by status="completed" when you only need finished results', + 'Keep list_cases limit at the default 20 and paginate; raise it only when cases are known to be small', + 'Store run ids, not case payloads, when tracking results over time' + ], + pitfalls: [ + 'API keys created before n8n 2.30 silently lack testRun scopes - a 403 means re-create the key, not a bug', + 'A 404 can mean the instance predates 2.30, the workflow id is wrong, or the runId belongs to a different workflow', + 'Evaluations are license/quota-gated in n8n - instances without the feature simply have no runs' + ], + relatedTools: ['n8n_executions', 'n8n_test_workflow', 'n8n_workflow_versions'] + } +}; diff --git a/src/mcp/tools-documentation.ts b/src/mcp/tools-documentation.ts index 9c8641147..f2f9d000b 100644 --- a/src/mcp/tools-documentation.ts +++ b/src/mcp/tools-documentation.ts @@ -157,6 +157,7 @@ When working with Code nodes, always start by calling the relevant guide: - n8n_autofix_workflow - Auto-fix common issues - n8n_test_workflow - Test/trigger workflows (webhook, form, chat, execute) - n8n_executions - Unified execution management (action='get'/'list'/'delete') +- n8n_evaluations - Read evaluation test runs (action='list_runs'/'get_run'/'list_cases', n8n 2.30+) - n8n_health_check - Check n8n API connectivity - n8n_workflow_versions - Version history and rollback - n8n_deploy_template - Deploy templates directly to n8n instance diff --git a/src/mcp/tools-n8n-manager.ts b/src/mcp/tools-n8n-manager.ts index f326ddcd1..5024d6a88 100644 --- a/src/mcp/tools-n8n-manager.ts +++ b/src/mcp/tools-n8n-manager.ts @@ -492,6 +492,48 @@ export const n8nManagementTools: ToolDefinition[] = [ openWorldHint: true, }, }, + { + name: 'n8n_evaluations', + description: `Read evaluation test runs for a workflow (read-only). Requires n8n >= 2.30 and an API key created on 2.30+ (testRun scopes). Actions: list_runs=list runs for a workflow, get_run=single run with aggregated metrics, list_cases=per-case results (paginate - cases can be large). Triggering runs via API is not yet supported by n8n.`, + inputSchema: { + type: 'object', + properties: { + action: { + type: 'string', + enum: ['list_runs', 'get_run', 'list_cases'], + description: 'Operation: list_runs=list test runs, get_run=run details with metrics, list_cases=per-case results' + }, + workflowId: { + type: 'string', + description: 'Workflow ID the test runs belong to (required)' + }, + runId: { + type: 'string', + description: 'Test run ID (required for action=get_run or action=list_cases)' + }, + status: { + type: 'string', + enum: ['new', 'running', 'completed', 'error', 'cancelled'], + description: 'For action=list_runs: filter by run status' + }, + limit: { + type: 'number', + description: 'Results per page (1-250). Defaults: n8n server default (100) for list_runs, 20 for list_cases (per-case inputs/outputs can be large)' + }, + cursor: { + type: 'string', + description: 'Pagination cursor from previous response' + } + }, + required: ['action', 'workflowId'] + }, + annotations: { + title: 'Read Evaluation Test Runs', + readOnlyHint: true, + destructiveHint: false, + openWorldHint: true, + }, + }, // System Tools { @@ -736,6 +778,7 @@ Old backups are also pruned automatically (10 most recent per workflow, plus an */ export const TOOL_OPERATION_PARAM: Record = { 'n8n_executions': 'action', + 'n8n_evaluations': 'action', 'n8n_workflow_versions': 'mode', }; diff --git a/src/services/n8n-api-client.ts b/src/services/n8n-api-client.ts index 5a16555db..521c7e33d 100644 --- a/src/services/n8n-api-client.ts +++ b/src/services/n8n-api-client.ts @@ -7,6 +7,12 @@ import { Execution, ExecutionListParams, ExecutionListResponse, + TestRunSummary, + TestCaseExecution, + TestRunListParams, + TestCaseListParams, + TestRunListResponse, + TestCaseListResponse, Credential, CredentialListParams, CredentialListResponse, @@ -495,6 +501,47 @@ export class N8nApiClient { } } + // Evaluation test runs (n8n >= 2.30) + + async listTestRuns(workflowId: string, params: TestRunListParams = {}): Promise { + try { + const response = await this.client.get( + `/workflows/${encodeApiPathSegment(workflowId, 'workflowId')}/test-runs`, + { params } + ); + return this.validateListResponse(response.data, 'test runs'); + } catch (error) { + throw handleN8nApiError(error); + } + } + + async getTestRun(workflowId: string, runId: string): Promise { + try { + const response = await this.client.get( + `/workflows/${encodeApiPathSegment(workflowId, 'workflowId')}/test-runs/${encodeApiPathSegment(runId, 'runId')}` + ); + return response.data; + } catch (error) { + throw handleN8nApiError(error); + } + } + + async listTestCases( + workflowId: string, + runId: string, + params: TestCaseListParams = {} + ): Promise { + try { + const response = await this.client.get( + `/workflows/${encodeApiPathSegment(workflowId, 'workflowId')}/test-runs/${encodeApiPathSegment(runId, 'runId')}/test-cases`, + { params } + ); + return this.validateListResponse(response.data, 'test cases'); + } catch (error) { + throw handleN8nApiError(error); + } + } + // Webhook Execution async triggerWebhook(request: WebhookRequest): Promise { try { diff --git a/src/types/n8n-api.ts b/src/types/n8n-api.ts index 27bcbc7c0..1ddec3265 100644 --- a/src/types/n8n-api.ts +++ b/src/types/n8n-api.ts @@ -297,6 +297,66 @@ export interface ExecutionListResponse { nextCursor?: string | null; } +// Evaluation test runs (n8n Public API >= 2.30) +export type TestRunStatus = 'new' | 'running' | 'completed' | 'error' | 'cancelled'; +export type TestRunFinalResult = 'success' | 'error' | 'warning'; +export type TestCaseExecutionStatus = + | 'new' + | 'running' + | 'evaluation_running' + | 'success' + | 'error' + | 'warning' + | 'cancelled'; + +export interface TestRunSummary { + id: string; + status: TestRunStatus; + runAt: string | null; + completedAt: string | null; + metrics: Record | null; + errorCode: string | null; + errorDetails: Record | null; + finalResult: TestRunFinalResult | null; + testCaseCount: number; + createdAt: string; + updatedAt: string; +} + +export interface TestCaseExecution { + id: string; + status: TestCaseExecutionStatus; + runAt: string | null; + completedAt: string | null; + metrics: Record | null; + errorCode: string | null; + errorDetails: Record | null; + inputs: Record | null; + outputs: Record | null; + executionId: string | null; +} + +export interface TestRunListParams { + status?: TestRunStatus; + limit?: number; + cursor?: string; +} + +export interface TestCaseListParams { + limit?: number; + cursor?: string; +} + +export interface TestRunListResponse { + data: TestRunSummary[]; + nextCursor?: string | null; +} + +export interface TestCaseListResponse { + data: TestCaseExecution[]; + nextCursor?: string | null; +} + export interface CredentialListParams { limit?: number; cursor?: string; diff --git a/tests/unit/mcp/handlers-evaluations.test.ts b/tests/unit/mcp/handlers-evaluations.test.ts new file mode 100644 index 000000000..1ebdea123 --- /dev/null +++ b/tests/unit/mcp/handlers-evaluations.test.ts @@ -0,0 +1,339 @@ +import { describe, it, expect, vi, beforeEach, afterEach } from 'vitest'; +import { N8nApiClient } from '@/services/n8n-api-client'; +import { N8nApiError } from '@/utils/n8n-errors'; + +// Mock dependencies +vi.mock('@/services/n8n-api-client'); +vi.mock('@/config/n8n-api', () => ({ + getN8nApiConfig: vi.fn(), +})); +vi.mock('@/services/n8n-validation', () => ({ + validateWorkflowStructure: vi.fn(), + hasWebhookTrigger: vi.fn(), + getWebhookUrl: vi.fn(), +})); +vi.mock('@/utils/logger', () => ({ + logger: { + info: vi.fn(), + error: vi.fn(), + debug: vi.fn(), + warn: vi.fn(), + }, + Logger: vi.fn().mockImplementation(() => ({ + info: vi.fn(), + error: vi.fn(), + debug: vi.fn(), + warn: vi.fn(), + })), + LogLevel: { + ERROR: 0, + WARN: 1, + INFO: 2, + DEBUG: 3, + }, +})); + +describe('Evaluation Handlers (n8n_evaluations)', () => { + let mockApiClient: any; + let handlers: any; + let getN8nApiConfig: any; + + const completedRun = { + id: 'run1', + status: 'completed', + runAt: '2026-07-15T10:00:00.000Z', + completedAt: '2026-07-15T10:05:00.000Z', + metrics: { accuracy: 0.9 }, + errorCode: null, + errorDetails: null, + finalResult: 'success', + testCaseCount: 3, + createdAt: '2026-07-15T10:00:00.000Z', + updatedAt: '2026-07-15T10:05:00.000Z', + }; + + beforeEach(async () => { + vi.clearAllMocks(); + + mockApiClient = { + listTestRuns: vi.fn(), + getTestRun: vi.fn(), + listTestCases: vi.fn(), + getCachedVersionInfo: vi.fn().mockReturnValue(null), + getVersion: vi.fn().mockResolvedValue(null), + }; + + getN8nApiConfig = (await import('@/config/n8n-api')).getN8nApiConfig; + + vi.mocked(getN8nApiConfig).mockReturnValue({ + baseUrl: 'https://n8n.test.com', + apiKey: 'test-key', + timeout: 30000, + maxRetries: 3, + }); + + vi.mocked(N8nApiClient).mockImplementation(() => mockApiClient); + + handlers = await import('@/mcp/handlers-n8n-manager'); + }); + + afterEach(() => { + if (handlers) { + const clientGetter = handlers.getN8nApiClient; + if (clientGetter) { + vi.mocked(getN8nApiConfig).mockReturnValue(null); + clientGetter(); + } + } + }); + + describe('handleListTestRuns', () => { + it('should return runs with pagination info', async () => { + mockApiClient.listTestRuns.mockResolvedValue({ data: [completedRun], nextCursor: null }); + + const result = await handlers.handleListTestRuns({ workflowId: 'wf1', status: 'completed' }); + + expect(result.success).toBe(true); + expect(result.data.testRuns).toHaveLength(1); + expect(result.data.returned).toBe(1); + expect(result.data.hasMore).toBe(false); + expect(mockApiClient.listTestRuns).toHaveBeenCalledWith('wf1', { + status: 'completed', + limit: undefined, + cursor: undefined, + }); + }); + + it('should include a pagination note when more pages exist', async () => { + mockApiClient.listTestRuns.mockResolvedValue({ data: [completedRun], nextCursor: 'next' }); + + const result = await handlers.handleListTestRuns({ workflowId: 'wf1' }); + + expect(result.success).toBe(true); + expect(result.data.hasMore).toBe(true); + expect(result.data._note).toContain('cursor'); + }); + + it('should include a hint when no runs exist', async () => { + mockApiClient.listTestRuns.mockResolvedValue({ data: [], nextCursor: null }); + + const result = await handlers.handleListTestRuns({ workflowId: 'wf1' }); + + expect(result.success).toBe(true); + expect(result.data.testRuns).toHaveLength(0); + expect(result.data._note).toContain('evaluation'); + }); + + it('should reject a limit above 250', async () => { + const result = await handlers.handleListTestRuns({ workflowId: 'wf1', limit: 500 }); + + expect(result.success).toBe(false); + expect(result.error).toBe('Invalid input'); + expect(mockApiClient.listTestRuns).not.toHaveBeenCalled(); + }); + + it('should reject a missing workflowId', async () => { + const result = await handlers.handleListTestRuns({}); + + expect(result.success).toBe(false); + expect(result.error).toBe('Invalid input'); + }); + + it('should map 403 to API key scope guidance', async () => { + mockApiClient.listTestRuns.mockRejectedValue(new N8nApiError('Forbidden', 403, 'FORBIDDEN')); + + const result = await handlers.handleListTestRuns({ workflowId: 'wf1' }); + + expect(result.success).toBe(false); + expect(result.error).toContain('testRun scopes'); + expect(result.error).toContain('re-create'); + }); + + it('should use a filter-aware note when a status filter matches nothing', async () => { + mockApiClient.listTestRuns.mockResolvedValue({ data: [], nextCursor: null }); + + const result = await handlers.handleListTestRuns({ workflowId: 'wf1', status: 'error' }); + + expect(result.success).toBe(true); + expect(result.data._note).toContain("status 'error'"); + expect(result.data._note).not.toContain('evaluation trigger'); + }); + + it('should coerce empty-string status and cursor to undefined', async () => { + mockApiClient.listTestRuns.mockResolvedValue({ data: [], nextCursor: null }); + + const result = await handlers.handleListTestRuns({ workflowId: 'wf1', status: '', cursor: '' }); + + expect(result.success).toBe(true); + expect(mockApiClient.listTestRuns).toHaveBeenCalledWith('wf1', { + status: undefined, + limit: undefined, + cursor: undefined, + }); + }); + + it('should pass the cursor through', async () => { + mockApiClient.listTestRuns.mockResolvedValue({ data: [completedRun], nextCursor: null }); + + await handlers.handleListTestRuns({ workflowId: 'wf1', cursor: 'page2' }); + + expect(mockApiClient.listTestRuns).toHaveBeenCalledWith('wf1', { + status: undefined, + limit: undefined, + cursor: 'page2', + }); + }); + }); + + describe('handleGetTestRun', () => { + it('should return the run', async () => { + mockApiClient.getTestRun.mockResolvedValue(completedRun); + + const result = await handlers.handleGetTestRun({ workflowId: 'wf1', runId: 'run1' }); + + expect(result.success).toBe(true); + expect(result.data).toEqual(completedRun); + expect(mockApiClient.getTestRun).toHaveBeenCalledWith('wf1', 'run1'); + }); + + it('should reject missing runId', async () => { + const result = await handlers.handleGetTestRun({ workflowId: 'wf1' }); + + expect(result.success).toBe(false); + expect(result.error).toBe('Invalid input'); + expect(mockApiClient.getTestRun).not.toHaveBeenCalled(); + }); + + it('should map 404 on a pre-2.30 instance to version guidance', async () => { + mockApiClient.getCachedVersionInfo.mockReturnValue({ + version: '2.29.1', + major: 2, + minor: 29, + patch: 1, + }); + mockApiClient.getTestRun.mockRejectedValue(new N8nApiError('Not found', 404, 'NOT_FOUND')); + + const result = await handlers.handleGetTestRun({ workflowId: 'wf1', runId: 'run1' }); + + expect(result.success).toBe(false); + expect(result.error).toContain('2.30'); + expect(result.error).toContain('2.29.1'); + }); + + it('should map 404 on a 2.30+ instance to not-found guidance', async () => { + mockApiClient.getCachedVersionInfo.mockReturnValue({ + version: '2.30.4', + major: 2, + minor: 30, + patch: 4, + }); + mockApiClient.getTestRun.mockRejectedValue(new N8nApiError('Not found', 404, 'NOT_FOUND')); + + const result = await handlers.handleGetTestRun({ workflowId: 'wf1', runId: 'run1' }); + + expect(result.success).toBe(false); + expect(result.error).toContain('belong'); + }); + + it('should map 404 with unknown instance version to not-found guidance', async () => { + mockApiClient.getCachedVersionInfo.mockReturnValue(null); + mockApiClient.getTestRun.mockRejectedValue(new N8nApiError('Not found', 404, 'NOT_FOUND')); + + const result = await handlers.handleGetTestRun({ workflowId: 'wf1', runId: 'run1' }); + + expect(result.success).toBe(false); + expect(result.error).toContain('belong'); + }); + + it('should fetch the version on 404 when the cache is cold', async () => { + mockApiClient.getCachedVersionInfo.mockReturnValue(null); + mockApiClient.getVersion.mockResolvedValue({ + version: '2.29.1', + major: 2, + minor: 29, + patch: 1, + }); + mockApiClient.getTestRun.mockRejectedValue(new N8nApiError('Not found', 404, 'NOT_FOUND')); + + const result = await handlers.handleGetTestRun({ workflowId: 'wf1', runId: 'run1' }); + + expect(mockApiClient.getVersion).toHaveBeenCalled(); + expect(result.success).toBe(false); + expect(result.error).toContain('2.30'); + expect(result.error).toContain('2.29.1'); + }); + + it('should fall back to not-found guidance when the version fetch fails', async () => { + mockApiClient.getCachedVersionInfo.mockReturnValue(null); + mockApiClient.getVersion.mockRejectedValue(new Error('network down')); + mockApiClient.getTestRun.mockRejectedValue(new N8nApiError('Not found', 404, 'NOT_FOUND')); + + const result = await handlers.handleGetTestRun({ workflowId: 'wf1', runId: 'run1' }); + + expect(result.success).toBe(false); + expect(result.error).toContain('belong'); + }); + + it('should map 403 to API key scope guidance', async () => { + mockApiClient.getTestRun.mockRejectedValue(new N8nApiError('Forbidden', 403, 'FORBIDDEN')); + + const result = await handlers.handleGetTestRun({ workflowId: 'wf1', runId: 'run1' }); + + expect(result.success).toBe(false); + expect(result.error).toContain('testRun scopes'); + }); + }); + + describe('handleListTestCases', () => { + it('should default limit to 20', async () => { + mockApiClient.listTestCases.mockResolvedValue({ data: [], nextCursor: null }); + + const result = await handlers.handleListTestCases({ workflowId: 'wf1', runId: 'run1' }); + + expect(result.success).toBe(true); + expect(mockApiClient.listTestCases).toHaveBeenCalledWith('wf1', 'run1', { + limit: 20, + cursor: undefined, + }); + }); + + it('should return cases with pagination info and size warning', async () => { + const testCase = { + id: 'case1', + status: 'success', + runAt: null, + completedAt: null, + metrics: { accuracy: 1 }, + errorCode: null, + errorDetails: null, + inputs: { question: 'hi' }, + outputs: { answer: 'hello' }, + executionId: 'exec1', + }; + mockApiClient.listTestCases.mockResolvedValue({ data: [testCase], nextCursor: 'next' }); + + const result = await handlers.handleListTestCases({ workflowId: 'wf1', runId: 'run1' }); + + expect(result.success).toBe(true); + expect(result.data.testCases).toEqual([testCase]); + expect(result.data.hasMore).toBe(true); + expect(result.data._note).toContain('Paginate'); + }); + + it('should reject missing runId', async () => { + const result = await handlers.handleListTestCases({ workflowId: 'wf1' }); + + expect(result.success).toBe(false); + expect(result.error).toBe('Invalid input'); + }); + + it('should map 403 to API key scope guidance', async () => { + mockApiClient.listTestCases.mockRejectedValue(new N8nApiError('Forbidden', 403, 'FORBIDDEN')); + + const result = await handlers.handleListTestCases({ workflowId: 'wf1', runId: 'run1' }); + + expect(result.success).toBe(false); + expect(result.error).toContain('testRun scopes'); + }); + }); +}); diff --git a/tests/unit/services/n8n-api-client.test.ts b/tests/unit/services/n8n-api-client.test.ts index ae1037521..b4c1ed5a7 100644 --- a/tests/unit/services/n8n-api-client.test.ts +++ b/tests/unit/services/n8n-api-client.test.ts @@ -1085,6 +1085,73 @@ describe('N8nApiClient', () => { }); }); + describe('evaluation test runs', () => { + beforeEach(() => { + client = new N8nApiClient(defaultConfig); + }); + + it('should list test runs with params', async () => { + const response = { data: [{ id: 'run1', status: 'completed' }], nextCursor: null }; + mockAxiosInstance.get.mockResolvedValue({ data: response }); + + const result = await client.listTestRuns('wf1', { status: 'completed', limit: 50, cursor: 'abc' }); + + expect(mockAxiosInstance.get).toHaveBeenCalledWith('/workflows/wf1/test-runs', { + params: { status: 'completed', limit: 50, cursor: 'abc' }, + }); + expect(result).toEqual(response); + }); + + it('should reject hostile workflow ids before making a request', async () => { + await expect(client.listTestRuns('a/../b')).rejects.toThrow(); + expect(mockAxiosInstance.get).not.toHaveBeenCalled(); + }); + + it('should get a single test run', async () => { + const run = { id: 'run1', status: 'completed', finalResult: 'success' }; + mockAxiosInstance.get.mockResolvedValue({ data: run }); + + const result = await client.getTestRun('wf1', 'run1'); + + expect(mockAxiosInstance.get).toHaveBeenCalledWith('/workflows/wf1/test-runs/run1'); + expect(result).toEqual(run); + }); + + it('should reject hostile run ids before making a request', async () => { + await expect(client.getTestRun('wf1', '../../executions')).rejects.toThrow(); + expect(mockAxiosInstance.get).not.toHaveBeenCalled(); + }); + + it('should list test cases with pagination params', async () => { + const response = { data: [{ id: 'case1', executionId: 'exec1' }], nextCursor: 'next' }; + mockAxiosInstance.get.mockResolvedValue({ data: response }); + + const result = await client.listTestCases('wf1', 'run1', { limit: 20 }); + + expect(mockAxiosInstance.get).toHaveBeenCalledWith('/workflows/wf1/test-runs/run1/test-cases', { + params: { limit: 20 }, + }); + expect(result.nextCursor).toBe('next'); + }); + + it('should wrap legacy array responses when listing test runs', async () => { + mockAxiosInstance.get.mockResolvedValue({ data: [{ id: 'run1' }] }); + + const result = await client.listTestRuns('wf1'); + + expect(result).toEqual({ data: [{ id: 'run1' }], nextCursor: null }); + }); + + it('should throw N8nApiError on HTTP failure', async () => { + await mockAxiosInstance.simulateError('get', { + message: 'Forbidden', + response: { status: 403, data: { message: 'forbidden' } }, + }); + + await expect(client.listTestRuns('wf1')).rejects.toThrow(N8nApiError); + }); + }); + describe('triggerWebhook', () => { beforeEach(() => { client = new N8nApiClient(defaultConfig);