feat: public validation benchmark — reproducible probe suite (announced in #900)

[czlonkowski]

Context

Announced in Discussion #900 as the first near-term deliverable: a public, reproducible validation benchmark so anyone can verify the validation claims in the July 2026 competitive analysis themselves, against both the official n8n MCP server and n8n-mcp.

The seed already exists: the five broken-workflow probes from the analysis (§6.1, §10) — unknown node type, impossible typeVersion, missing required field, duplicate node names, AI Agent without a model — all of which the official validator currently accepts as valid:true while n8n-mcp rejects all five.

Scope (v1 — validation honesty only)

• Probe suite: the five seed probes plus an expanded matrix of invalid configurations — community-node configs, expression syntax errors, connection/graph errors (invalid input index, orphaned nodes), per-node required-field cases across common nodes.
• A small harness that runs the suite as an MCP client against (a) an official n8n MCP endpoint and (b) an n8n-mcp endpoint, and emits a scored report (caught / warned / silently passed, per probe).
• Edit-cost measurement shapes from the analysis (insert-node-mid-flow; one-line large-field edit) as a secondary, optional module.
• Methodology doc: what each probe tests, why it matters for agent loops (the valid:true stop-signal problem), how to run it against your own instance.
• Published results for the current n8n release, with the intent to re-run per n8n release.

Design decisions to settle

• Location: standalone repo (neutral optics, own README) vs scripts/benchmark/ here. Leaning standalone.
• Result format: machine-readable (JSON) + rendered markdown table.

Out of scope (v2+)

End-to-end task-success benchmarking (build/edit/debug scenarios scored by outcome) — larger effort, separate issue when v1 ships.

[github-actions]

🎯 Agentic Issue Triage

Type: Feature request / Roadmap deliverable (v1)
Author: Repo owner (@czlonkowski), announced in Discussion #900
Existing labels: enhancement, roadmap — added documentation (methodology doc is in scope) and help wanted (community contributions could accelerate this)

This issue tracks the creation of a public, reproducible validation benchmark comparing n8n-mcp's workflow validator against the official n8n MCP server. Five seed probes already exist from the July 2026 competitive analysis. The v1 scope is focused on "validation honesty" — whether validators correctly reject invalid workflows.

---

📋 Sub-task Breakdown

Probe Suite

• Port the five existing probes from competitive analysis §6.1 / §10 into a dedicated probe corpus (JSON fixtures or TypeScript objects)
• Expand probe matrix:
  • Community-node configs (unknown type strings)
  • typeVersion impossibilities (e.g., version 999 on a core node)
  • Missing required fields (per-node, covering common nodes: HTTP Request, Set, Code, IF, etc.)
  • Duplicate node name / id collisions
  • AI Agent node missing model sub-node
  • Expression syntax errors ({{ $json.foo unclosed, wrong variable scope)
  • Graph/connection errors: invalid input index, orphaned (disconnected) nodes, cycles in non-loop contexts

Harness

• Implement an MCP client harness (likely using @modelcontextprotocol/sdk) that:
  • Connects to a configurable MCP endpoint (stdio or HTTP)
  • Calls n8n_validate_workflow (or equivalent) for each probe
  • Records valid: true/false, any error messages, and latency
• Score each result as: caught / warned / silently-passed
• Output: JSON report + rendered markdown table

Edit-Cost Module (optional / secondary)

• Implement insert-node-mid-flow and one-line large-field edit scenarios
• Token-count the requests/responses using a tokeniser (e.g., tiktoken)

Methodology Doc

• Explain what each probe tests
• Explain the valid:true stop-signal problem in agent loops
• How-to-run guide for self-hosting the harness against a custom instance

Published Results

• Run harness against current n8n release; commit results to repo
• Set up a CI job (or manual note) to re-run per n8n release

🏗️ Design Considerations

Repo location:

• The issue leans toward a standalone repo for neutral optics. That's the right call for credibility — a benchmark in the same repo as one of the tools being benchmarked raises methodology questions.
• If staying in this repo (e.g., scripts/benchmark/), a clear separation of concerns and a note in the README about independence is important.

Result format:

• JSON (machine-readable) + Markdown table is sensible. Consider also SARIF or a simple CSV for potential future tooling integration.
• Results should include the n8n package version and n8n-mcp version tested, so comparisons are reproducible.

Probe reproducibility:

• Probes should be pure JSON/TypeScript fixtures with no runtime dependencies on a live n8n instance. This ensures anyone can clone and run without a running n8n server.

False-positive concern:

• The harness should test that n8n-mcp does not reject valid workflows — a validator that rejects everything scores 100% on "catching invalid" but is useless. Include a suite of known-good workflows as a control group.

🔍 Relevant Codebase Pointers

The existing validation infrastructure in this repo that would inform probe design:

• src/services/workflow-validator.ts — main workflow validation entry point
• src/services/enhanced-config-validator.ts — operation-aware per-node validation
• src/services/expression-validator.ts — n8n expression syntax validation
• src/services/config-validator.ts — multi-profile validation (minimal → strict)
• src/services/type-structure-service.ts — complex type structure validation (filter, resourceMapper, etc.)
• src/scripts/test-workflow-validation.ts — existing test patterns to adapt as probe seeds
• src/scripts/test-n8n-validate-workflow.ts — covers the n8n_validate_workflow MCP tool specifically
• src/mcp/server.ts — where the MCP tool handler lives; useful for understanding what the harness needs to call

📚 Useful Resources

• MCP TypeScript SDK — for the client harness
• n8n Workflow JSON schema — ground truth for valid workflow structure
• Existing benchmark frameworks for MCP — reference pattern for MCP evaluation harnesses
• Discussion Where n8n-mcp goes from here — now that n8n ships its own MCP server #900 — project direction context
• July 2026 competitive analysis — §6.1 and §10 contain the five seed probes

💡 Implementation Nudges

1. Start with the control group (valid workflows) before invalid probes — it ensures the harness wiring is correct before adding edge cases.
2. Probe IDs matter — give each probe a stable, human-readable ID (e.g., probe-unknown-node-type, probe-missing-required-field-http-url) so results tables are navigable and regressions are traceable.
3. Harness should be idempotent — running it twice against the same endpoint should produce identical output (no side effects on the n8n instance).
4. Version pinning — the harness should record n8n-nodes-base version and n8n-mcp version in the output header so results snapshots are self-describing.
5. CI integration — even a simple GitHub Actions workflow that runs the probe suite on each PR (against a mocked or embedded validator) would catch regressions early.

Generated by Agentic Triage for issue #904 · ● 261.4K · ◷

To install this agentic workflow, run

gh aw add githubnext/agentics/workflows/issue-triage.md@13377ddf7e35c2b6e47aa58f45acb228fba902c8

[Sehlani042]

I can help with a small first slice here, but I want to avoid guessing the benchmark location before you settle the standalone-vs-in-repo question.

Would you prefer an initial contribution in a standalone benchmark repo, or under something like scripts/benchmark/ / benchmarks/validation/ in this repo?

The narrow slice I would start with is intentionally neutral and not a full runner:

• the five seed invalid-workflow probes as stable fixtures
• one valid control workflow
• a small result classification schema, e.g. caught | warned | silently-passed | error
• a methodology stub explaining why valid: true on invalid workflows is a stop-signal failure

I would leave official-endpoint assumptions and final comparative claims out of the first patch until the benchmark home is clear.

[czlonkowski]

@Sehlani042 Standalone repo — decision made, and it now exists: https://github.com/czlonkowski/n8n-validation-benchmark

Three reasons for standalone: a benchmark living inside one of the tools it measures invites methodology doubts no README disclaimer can cure; it needs its own release cadence (re-run per n8n release, independent of n8n-mcp versions); and both servers should be ordinary configurable endpoints to the harness — neither privileged.

The skeleton has the fixtures layout, the fixture envelope and result-snapshot schemas (schema/), a methodology draft (docs/METHODOLOGY.md), and pre-registered specs for the five seeds (fixtures/README.md) under Apache-2.0. Your slice is exactly the right first PR into it:

• The five seed probes as stable fixtures — the IDs and defect specs are pre-registered in fixtures/README.md (probe-unknown-node-type, probe-missing-required-field-http-url, …), so your fixtures can target them directly.
• Valid control workflows — yes, and more than a control group: false-alarm rate will be a first-class scored axis alongside miss rate, for both servers. A validator that cries wolf is as useless to an agent loop as one that rubber-stamps. We expect to be measured on this too.
• Classification — caught | warned | silently-passed | error for probes; passed | warned | wrongly-rejected | error for controls. This is now encoded in schema/result.schema.json; treat the schema as the source of truth, and if you'd shape it differently, PR against the schema.
• Methodology stub around the stop-signal problem — exactly right. There's a draft in the repo with open questions marked (wrong-reason rejections, mechanical defect matching); expanding or challenging it is in scope for your PR.

Two scope notes so the first slice stays defensible: v1 stays at the five seeds + controls — the expanded matrix (community nodes, expressions, graph errors) comes after v1 ships and re-runs cleanly per n8n release. And agreed on leaving official-endpoint adapters and comparative claims out of the first patch — fixtures and schema are endpoint-neutral by design. Result snapshots pin the benchmark commit, the n8n ground-truth release, and each endpoint's server version in the header, so every published table is self-describing.

Thanks for deliberately starting narrow — this is exactly how I hoped #900 would work.

[Liohtml]

The five pre-registered v1 probes plus the two suggested controls are now proposed as czlonkowski/n8n-validation-benchmark#1 — schema-validated, and spot-checked against n8n-mcp 2.63.2 (all five probes caught with a single defect-identifying error; both controls pass clean, no warnings). Authoring decisions that needed judgment (trigger in every probe, deterministic node ids, bare type string for the unknown-node probe) are flagged in the PR description for review.