refactor(ai-builder): Implement unified evaluations harness

[OlegIvaniv]

Summary

This PR rewrites the AI Workflow Builder evaluations into a single v2 harness with:

• One runner (local + langsmith backends) that owns dataset setup, concurrency, timeouts, scoring, and artifact writing.
• Multiple evaluators (LLM-as-judge, pairwise, programmatic, similarity) that are backend-agnostic and return a shared Feedback[] format.
• A single CLI entrypoint that only orchestrates (parse args → build config → run).
• A comprehensive Jest test suite covering the tricky behavior (scoring, CLI parsing, concurrency limiting, LangSmith trace behavior, CSV loading).

Why

The previous evals were hard to extend safely because responsibilities were duplicated across multiple runners:

• Each runner owned its own CLI parsing and argument transformation, often with subtly different semantics.
• Dataset setup was scattered and inconsistent (local vs LangSmith, different formats per runner).
• Concurrency was not a single “knob”; nested parallelism (judge panels, multi-generation) could multiply total LLM calls and hit provider/LangSmith rate limits.
• LangSmith integration was fragile: we were seeing missing metrics/traces and occasional 403 payload issues from large trace uploads.
• There was little/no automated test coverage, so refactors were risky and regressions were hard to catch.

What changed (high level)

1) Unified runner / harness

The core entrypoint is evaluations/harness/runner.ts (runEvaluation(config)), which now owns:

• Running a dataset in local mode (CLI prompt/CSV/test-case) and LangSmith mode (dataset name or preloaded examples).
• Global concurrency control via a single llmCallLimiter (p-limit), used for generation and evaluator LLM calls.
• Per-operation timeouts via withTimeout() (best-effort; does not cancel underlying requests).
• Stable scoring and pass/fail status derived from Feedback.kind.
• Optional artifact output (--output-dir) in local mode.

Evaluators are now “plugins” that don’t care about backends; they just evaluate a workflow + context and return feedback.

2) Shared types + feedback contract

All evaluators emit Feedback[] (evaluations/harness/harness-types.ts):

• evaluator: stable evaluator id (e.g. llm-judge, pairwise, programmatic)
• metric: metric key (supports dot-path sub-metrics)
• score: normalized 0..1
• kind: score | metric | detail (used by the scorer; detail should not affect overall scoring)
• comment: optional explanation / violations

This contract is what keeps the harness readable: the runner doesn’t need evaluator-specific coercions or ad-hoc post-processing.

3) LangSmith integration fixes (reliable traces + smaller payloads)

Key decisions in the LangSmith runner:

• Do not wrap the top-level evaluate() target with traceable(); the LangSmith SDK wraps it and attaches critical options (linking to examples, run lifecycle, client/project settings).
• Do wrap inner operations with traceable() for child traces and explicitly pass client: lsClient so child traces attach to the same run tree.
• Flush pending trace batches before returning.

To reduce payload size (and avoid 403 multipart errors on large runs), we keep “minimal tracing” enabled by default and filter heavy state fields via hideInputs/hideOutputs (evaluations/langsmith/trace-filters.ts). We explicitly keep messages untrimmed to avoid breaking downstream expectations.

4) Metric key compatibility for comparing old vs new runs

LangSmith metric key mapping is centralized (evaluations/harness/feedback.ts):

• LLM-judge metrics are unprefixed (e.g. overallScore, maintainability.workflowOrganization) to match historical dashboards.
• Programmatic metrics remain prefixed (e.g. programmatic.trigger) so they don’t collide with LLM-judge root keys.
• Pairwise preserves v1 metric keys (e.g. pairwise_primary, pairwise_total_violations, etc.) as raw counts/ratios as before; additional judge/gen details are namespaced (e.g. pairwise.judge1).

5) Directory structure (separation by responsibility)

The evals folder is reorganized so ownership is obvious:

• evaluations/cli/: CLI entry + arg parsing + CSV loader
• evaluations/harness/: runner, lifecycle logging, scoring, artifacts, helper utilities
• evaluations/evaluators/: evaluator factories (llm-judge, pairwise, programmatic, similarity)
• evaluations/judge/: LLM-judge internals (schema + category evaluators + workflow evaluator)
• evaluations/langsmith/: LangSmith helpers (types + trace filters)
• evaluations/support/: environment setup (LLM + node types + LangSmith client), node loading, reports, test-case generation
• evaluations/programmatic/python/: unchanged layout (Python similarity tooling)

Legacy/categorization evaluation artifacts were removed as part of cleanup.

Behavior changes / notes

• Concurrency is now intended to be controlled primarily via --concurrency (global limiter); nested LLM calls are routed through the same limiter to prevent multiplicative parallelism.
• Local mode can write artifacts via --output-dir (one folder per example + summary.json).
• LangSmith mode rejects local-only prompt sources (--prompt, --prompts-csv, --test-case) and requires --dataset.

Tests

This PR adds/expands Jest coverage for:

• CLI parsing edge cases and validation
• CSV prompt loading (including dos/donts)
• Local + LangSmith runner behavior (including dataset context extraction)
• Scoring invariance when new detail metrics are added
• Concurrency limiting of judge panels / multi-generation via p-limit
• Trace filter behavior (keeps messages untrimmed while filtering heavy fields)
• Metric key mapping contracts (LangSmith comparability)

Run from packages/@n8n/ai-workflow-builder.ee:

pnpm lint
pnpm typecheck
pnpm test:eval

How to verify (manual)

From packages/@n8n/ai-workflow-builder.ee:

# Local smoke test
pnpm eval --prompt "Create a workflow that..." --verbose --output-dir ./.data/out/llm-judge-local

# Pairwise local
pnpm eval:pairwise --prompt "Create a workflow that..." --dos "Must use Notion" --donts "No HTTP Request node" --verbose --output-dir ./.data/out/pairwise-local

# LangSmith (requires LANGSMITH_API_KEY)
pnpm eval:langsmith --dataset "workflow-builder-canvas-prompts" --name "v2-smoke" --max-examples 5 --concurrency 5 --verbose

Related Linear tickets, Github issues, and Community forum posts

Review / Merge checklist

• PR title and summary are descriptive. (conventions)
• Docs updated or follow-up ticket created.
• Tests included.
• PR Labeled with release/backport (if the PR is an urgent fix that needs to be backported)

[cubic-dev-ai]

13 issues found across 109 files

Note: This PR contains a large number of files. cubic only reviews up to 75 files per PR, so some files may not have been reviewed.

Prompt for AI agents (all issues)

Check if these issues are valid — if so, understand the root cause of each and fix them.


<file name="packages/@n8n/ai-workflow-builder.ee/evaluations/langsmith/trace-filters.ts">

<violation number="1" location="packages/@n8n/ai-workflow-builder.ee/evaluations/langsmith/trace-filters.ts:57">
P2: Rule violated: **Prefer Typeguards over Type casting**

Use a type guard instead of `as` for type narrowing. The pattern `typeof x === 'object' && x !== null` can be encapsulated in a reusable type guard like `isRecord(value): value is Record<string, unknown>`.</violation>

<violation number="2" location="packages/@n8n/ai-workflow-builder.ee/evaluations/langsmith/trace-filters.ts:75">
P2: Rule violated: **Prefer Typeguards over Type casting**

Use a type guard instead of `as` for type narrowing. This is the same pattern as `summarizeWorkflow` - consider creating a shared `isRecord()` type guard.</violation>

<violation number="3" location="packages/@n8n/ai-workflow-builder.ee/evaluations/langsmith/trace-filters.ts:141">
P2: Rule violated: **Prefer Typeguards over Type casting**

Use a type guard instead of `as` for type narrowing. Consider creating a type guard function that validates the structure (e.g., `hasNodes(value): value is { nodes?: unknown[] }`).</violation>

<violation number="4" location="packages/@n8n/ai-workflow-builder.ee/evaluations/langsmith/trace-filters.ts:202">
P2: Rule violated: **Prefer Typeguards over Type casting**

Use a type guard instead of `as` for type narrowing. The `typeof === 'object'` check validates runtime type but a type guard would properly narrow the TypeScript type.</violation>
</file>

<file name="packages/@n8n/ai-workflow-builder.ee/evaluations/harness/lifecycle.ts">

<violation number="1" location="packages/@n8n/ai-workflow-builder.ee/evaluations/harness/lifecycle.ts:306">
P2: Inverted verbose condition: `onEvaluatorError` only logs in non-verbose mode, which is the opposite of other hooks (`onExampleStart`, `onExampleComplete`). Evaluator errors will be silently suppressed in verbose mode.</violation>

<violation number="2" location="packages/@n8n/ai-workflow-builder.ee/evaluations/harness/lifecycle.ts:392">
P2: Rule violated: **Prefer Typeguards over Type casting**

Use a type predicate instead of `as` for type narrowing after filtering. The `filter(Boolean)` pattern doesn't automatically narrow types in TypeScript - use a type guard function instead.</violation>
</file>

<file name="packages/@n8n/ai-workflow-builder.ee/evaluations/evaluators/programmatic/index.ts">

<violation number="1" location="packages/@n8n/ai-workflow-builder.ee/evaluations/evaluators/programmatic/index.ts:72">
P2: Missing feedback entries for `nodes` and `credentials` evaluation results. The underlying `programmaticEvaluation` computes and returns both `result.nodes` and `result.credentials` with `.score` and `.violations` properties (same structure as other categories), but these are not included in the feedback array. This means these metric scores won't be visible on dashboards for debugging/analysis, even though they contribute to the overall score.</violation>
</file>

<file name="packages/@n8n/ai-workflow-builder.ee/evaluations/__tests__/multi-gen.test.ts">

<violation number="1" location="packages/@n8n/ai-workflow-builder.ee/evaluations/__tests__/multi-gen.test.ts:51">
P2: Test description doesn't match assertion: the test says "should return 2 for 2 judges" but the assertion expects 1, not 2.</violation>

<violation number="2" location="packages/@n8n/ai-workflow-builder.ee/evaluations/__tests__/multi-gen.test.ts:55">
P2: Test description doesn't match assertion: the test says "should return 3 for 4 judges" but the assertion expects 2, not 3.</violation>
</file>

<file name="packages/@n8n/ai-workflow-builder.ee/evaluations/harness/multi-gen.ts">

<violation number="1" location="packages/@n8n/ai-workflow-builder.ee/evaluations/harness/multi-gen.ts:56">
P2: For even numbers of judges, `Math.ceil(numJudges / 2)` returns the threshold for a tie (50%), not a majority (>50%). With 2 judges, this returns 1 (50%), and with 4 judges, it returns 2 (50%). A true majority threshold should use `Math.floor(numJudges / 2) + 1`.</violation>
</file>

<file name="packages/@n8n/ai-workflow-builder.ee/evaluations/__tests__/cli.test.ts">

<violation number="1" location="packages/@n8n/ai-workflow-builder.ee/evaluations/__tests__/cli.test.ts:401">
P3: This test is redundant with 'should exit with 0 when pass rate >= 70%' above - both use `passed: 7, totalExamples: 10` which is exactly 70%. Consider removing this duplicate test or changing the values to truly test the boundary (e.g., `passed: 70, totalExamples: 100` to make the "exactly 70%" intent clearer).</violation>
</file>

<file name="packages/@n8n/ai-workflow-builder.ee/evaluations/__tests__/evaluators/pairwise.test.ts">

<violation number="1" location="packages/@n8n/ai-workflow-builder.ee/evaluations/__tests__/evaluators/pairwise.test.ts:497">
P2: Test assertion doesn't verify parallelism as the test name claims. `expect(callTimes).toHaveLength(3)` only confirms all calls were made, not that they ran in parallel. Consider checking that all timestamps fall within a small window (e.g., all within 10ms of the first call) to verify true parallel execution.</violation>
</file>

<file name="packages/@n8n/ai-workflow-builder.ee/evaluations/__tests__/trace-filters.test.ts">

<violation number="1" location="packages/@n8n/ai-workflow-builder.ee/evaluations/__tests__/trace-filters.test.ts:27">
P2: Test doesn't verify filtering behavior as claimed. The `cachedTemplates` input only contains `templateId` and `name` - the exact properties that `summarizeCachedTemplates` preserves. Consider adding extra properties (e.g., `workflow: { nodes: [] }`) to the template object to verify they are actually filtered out.</violation>
</file>

_{Reply with feedback, questions, or to request a fix. Tag @cubic-dev-ai to re-run a review.}

[cubic-dev-ai]

P2: Rule violated: Prefer Typeguards over Type casting

Use a type guard instead of as for type narrowing. Consider creating a type guard function that validates the structure (e.g., hasNodes(value): value is { nodes?: unknown[] }).

Prompt for AI agents

Check if this issue is valid — if so, understand the root cause and fix it. At packages/@n8n/ai-workflow-builder.ee/evaluations/langsmith/trace-filters.ts, line 141:

<comment>Use a type guard instead of `as` for type narrowing. Consider creating a type guard function that validates the structure (e.g., `hasNodes(value): value is { nodes?: unknown[] }`).</comment>

<file context>
@@ -0,0 +1,250 @@
+	if (!workflow || typeof workflow !== 'object') {
+		return workflow;
+	}
+	const wf = workflow as { nodes?: unknown[] };
+	if (wf.nodes && wf.nodes.length > WORKFLOW_SUMMARY_THRESHOLD) {
+		return summarizeWorkflow(workflow);
</file context>

✅ Addressed in 3ed9be4

[cubic-dev-ai]

P2: Rule violated: Prefer Typeguards over Type casting

Use a type guard instead of as for type narrowing. This is the same pattern as summarizeWorkflow - consider creating a shared isRecord() type guard.

Prompt for AI agents

Check if this issue is valid — if so, understand the root cause and fix it. At packages/@n8n/ai-workflow-builder.ee/evaluations/langsmith/trace-filters.ts, line 75:

<comment>Use a type guard instead of `as` for type narrowing. This is the same pattern as `summarizeWorkflow` - consider creating a shared `isRecord()` type guard.</comment>

<file context>
@@ -0,0 +1,250 @@
+function summarizeCachedTemplates(templates: unknown[]): Array<Record<string, unknown>> {
+	return templates.map((t) => {
+		if (!t || typeof t !== 'object') return { unknown: true };
+		const template = t as Record<string, unknown>;
+		return {
+			templateId: template.templateId,
</file context>

✅ Addressed in 3ed9be4

[cubic-dev-ai]

P2: Rule violated: Prefer Typeguards over Type casting

Use a type guard instead of as for type narrowing. The pattern typeof x === 'object' && x !== null can be encapsulated in a reusable type guard like isRecord(value): value is Record<string, unknown>.

Prompt for AI agents

Check if this issue is valid — if so, understand the root cause and fix it. At packages/@n8n/ai-workflow-builder.ee/evaluations/langsmith/trace-filters.ts, line 57:

<comment>Use a type guard instead of `as` for type narrowing. The pattern `typeof x === 'object' && x !== null` can be encapsulated in a reusable type guard like `isRecord(value): value is Record<string, unknown>`.</comment>

<file context>
@@ -0,0 +1,250 @@
+		return { unknown: true };
+	}
+
+	const wf = workflow as Record<string, unknown>;
+	const nodes = wf.nodes as Array<{ name?: string }> | undefined;
+	const connections = wf.connections as Record<string, unknown> | undefined;
</file context>

✅ Addressed in 3ed9be4

[cubic-dev-ai]

P2: Rule violated: Prefer Typeguards over Type casting

Use a type guard instead of as for type narrowing. The typeof === 'object' check validates runtime type but a type guard would properly narrow the TypeScript type.

Prompt for AI agents

Check if this issue is valid — if so, understand the root cause and fix it. At packages/@n8n/ai-workflow-builder.ee/evaluations/langsmith/trace-filters.ts, line 202:

<comment>Use a type guard instead of `as` for type narrowing. The `typeof === 'object'` check validates runtime type but a type guard would properly narrow the TypeScript type.</comment>

<file context>
@@ -0,0 +1,250 @@
+		// Handle workflowContext if present
+		if (filtered.workflowContext && typeof filtered.workflowContext === 'object') {
+			filtered.workflowContext = filterWorkflowContext(
+				filtered.workflowContext as Record<string, unknown>,
+			);
+		}
</file context>

✅ Addressed in 3ed9be4

[cubic-dev-ai]

P2: Rule violated: Prefer Typeguards over Type casting

Use a type predicate instead of as for type narrowing after filtering. The filter(Boolean) pattern doesn't automatically narrow types in TypeScript - use a type guard function instead.

Prompt for AI agents

Check if this issue is valid — if so, understand the root cause and fix it. At packages/@n8n/ai-workflow-builder.ee/evaluations/harness/lifecycle.ts, line 392:

<comment>Use a type predicate instead of `as` for type narrowing after filtering. The `filter(Boolean)` pattern doesn't automatically narrow types in TypeScript - use a type guard function instead.</comment>

<file context>
@@ -0,0 +1,437 @@
+export function mergeLifecycles(
+	...lifecycles: Array<Partial<EvaluationLifecycle> | undefined>
+): EvaluationLifecycle {
+	const validLifecycles = lifecycles.filter(Boolean) as Array<Partial<EvaluationLifecycle>>;
+
+	return {
</file context>

Suggested change

	const validLifecycles = lifecycles.filter(Boolean) as Array<Partial<EvaluationLifecycle>>;
	const validLifecycles = lifecycles.filter(
	(lc): lc is Partial<EvaluationLifecycle> => Boolean(lc),
	);

✅ Addressed in 3ed9be4

[cubic-dev-ai]

P2: Test description doesn't match assertion: the test says "should return 2 for 2 judges" but the assertion expects 1, not 2.

Prompt for AI agents

Check if this issue is valid — if so, understand the root cause and fix it. At packages/@n8n/ai-workflow-builder.ee/evaluations/__tests__/multi-gen.test.ts, line 51:

<comment>Test description doesn't match assertion: the test says "should return 2 for 2 judges" but the assertion expects 1, not 2.</comment>

<file context>
@@ -0,0 +1,159 @@
+			expect(getMajorityThreshold(5)).toBe(3);
+		});
+
+		it('should return 2 for 2 judges', () => {
+			expect(getMajorityThreshold(2)).toBe(1);
+		});
</file context>

Suggested change

	it('should return 2 for 2 judges', () => {
	it('should return 1 for 2 judges', () => {

✅ Addressed in 3ed9be4

[cubic-dev-ai]

P2: For even numbers of judges, Math.ceil(numJudges / 2) returns the threshold for a tie (50%), not a majority (>50%). With 2 judges, this returns 1 (50%), and with 4 judges, it returns 2 (50%). A true majority threshold should use Math.floor(numJudges / 2) + 1.

Prompt for AI agents

Check if this issue is valid — if so, understand the root cause and fix it. At packages/@n8n/ai-workflow-builder.ee/evaluations/harness/multi-gen.ts, line 56:

<comment>For even numbers of judges, `Math.ceil(numJudges / 2)` returns the threshold for a tie (50%), not a majority (>50%). With 2 judges, this returns 1 (50%), and with 4 judges, it returns 2 (50%). A true majority threshold should use `Math.floor(numJudges / 2) + 1`.</comment>

<file context>
@@ -0,0 +1,97 @@
+	if (!Number.isFinite(numJudges) || numJudges < 1) {
+		throw new Error(`numJudges must be >= 1 (received ${String(numJudges)})`);
+	}
+	return Math.ceil(numJudges / 2);
+}
+
</file context>

Suggested change

	return Math.ceil(numJudges / 2);
	return Math.floor(numJudges / 2) + 1;

[cubic-dev-ai]

P2: Test assertion doesn't verify parallelism as the test name claims. expect(callTimes).toHaveLength(3) only confirms all calls were made, not that they ran in parallel. Consider checking that all timestamps fall within a small window (e.g., all within 10ms of the first call) to verify true parallel execution.

Prompt for AI agents

Check if this issue is valid — if so, understand the root cause and fix it. At packages/@n8n/ai-workflow-builder.ee/evaluations/__tests__/evaluators/pairwise.test.ts, line 497:

<comment>Test assertion doesn't verify parallelism as the test name claims. `expect(callTimes).toHaveLength(3)` only confirms all calls were made, not that they ran in parallel. Consider checking that all timestamps fall within a small window (e.g., all within 10ms of the first call) to verify true parallel execution.</comment>

<file context>
@@ -0,0 +1,500 @@
+				generateWorkflow: mockGenerateWorkflow,
+			});
+
+			expect(callTimes).toHaveLength(3);
+		});
+	});
</file context>

[cubic-dev-ai]

P2: Test doesn't verify filtering behavior as claimed. The cachedTemplates input only contains templateId and name - the exact properties that summarizeCachedTemplates preserves. Consider adding extra properties (e.g., workflow: { nodes: [] }) to the template object to verify they are actually filtered out.

Prompt for AI agents

Check if this issue is valid — if so, understand the root cause and fix it. At packages/@n8n/ai-workflow-builder.ee/evaluations/__tests__/trace-filters.test.ts, line 27:

<comment>Test doesn't verify filtering behavior as claimed. The `cachedTemplates` input only contains `templateId` and `name` - the exact properties that `summarizeCachedTemplates` preserves. Consider adding extra properties (e.g., `workflow: { nodes: [] }`) to the template object to verify they are actually filtered out.</comment>

<file context>
@@ -0,0 +1,38 @@
+
+		const msg = { type: 'ai', content: 'hello' };
+		const input: KVMap = {
+			cachedTemplates: [{ templateId: 't1', name: 'Template' }],
+			messages: [msg],
+		};
</file context>

Suggested change

	cachedTemplates: [{ templateId: 't1', name: 'Template' }],
	cachedTemplates: [{ templateId: 't1', name: 'Template', workflow: { nodes: [] }, fullDefinition: 'large data' }],

✅ Addressed in 3ed9be4

[cubic-dev-ai]

P3: This test is redundant with 'should exit with 0 when pass rate >= 70%' above - both use passed: 7, totalExamples: 10 which is exactly 70%. Consider removing this duplicate test or changing the values to truly test the boundary (e.g., passed: 70, totalExamples: 100 to make the "exactly 70%" intent clearer).

Prompt for AI agents

Check if this issue is valid — if so, understand the root cause and fix it. At packages/@n8n/ai-workflow-builder.ee/evaluations/__tests__/cli.test.ts, line 401:

<comment>This test is redundant with 'should exit with 0 when pass rate >= 70%' above - both use `passed: 7, totalExamples: 10` which is exactly 70%. Consider removing this duplicate test or changing the values to truly test the boundary (e.g., `passed: 70, totalExamples: 100` to make the "exactly 70%" intent clearer).</comment>

<file context>
@@ -0,0 +1,459 @@
+				await expect(runV2Evaluation()).rejects.toThrow('process.exit(1)');
+			});
+
+			it('should exit with 0 when pass rate is exactly 70%', async () => {
+				mockRunEvaluation.mockResolvedValue(createMockSummary({ totalExamples: 10, passed: 7 }));
+
</file context>

[burivuhster]

Nice refactor! And kudos for the test coverage.
Couple of suggestions/comments

[burivuhster]

Is there a specific reason we need to pass generateWorkflow twice in the config?

[burivuhster]

Would it make sense to move this check to environment init logic?

[burivuhster]

Can we unify this and use --repetitions parameter everywhere instead?

[burivuhster]

This function is a bit hard to read because of many undefineds, ternary and spread operators and different logic depending on whether we have a header.
Here's the Claude's take on how it can be siplified:

export function loadTestCasesFromCsv(csvPath: string): TestCase[] {
  const resolvedPath = path.isAbsolute(csvPath) ? csvPath : path.resolve(process.cwd(), csvPath);

  if (!existsSync(resolvedPath)) {
    throw new Error(`CSV file not found at ${resolvedPath}`);
  }

  const rows = parseCsv(readFileSync(resolvedPath, 'utf8'));

  if (rows.length === 0) {
    throw new Error('The provided CSV file is empty');
  }

  const header = isHeaderRow(rows[0]) ? rows[0] : undefined;
  const dataRows = header ? rows.slice(1) : rows;

  if (dataRows.length === 0) {
    throw new Error('No prompt rows found in the provided CSV file');
  }

  const col = (name: string, ...aliases: string[]) => {
    if (!header) return undefined;
    for (const n of [name, ...aliases]) {
      const idx = detectColumnIndex(header, n);
      if (idx !== undefined) return idx;
    }
    return undefined;
  };

  const promptIdx = col('prompt') ?? 0;
  const idIdx = col('id');
  const dosIdx = col('dos', 'do');
  const dontsIdx = col('donts', 'dont');

  const testCases: TestCase[] = [];

  for (let i = 0; i < dataRows.length; i++) {
    const row = dataRows[i];
    const prompt = sanitizeValue(row[promptIdx]);
    if (!prompt) continue;

    const testCase: TestCase = {
      id: sanitizeValue(row[idIdx!]) || `csv-case-${i + 1}`,
      prompt,
    };

    const dos = sanitizeValue(row[dosIdx!]);
    const donts = sanitizeValue(row[dontsIdx!]);

    if (dos || donts) {
      testCase.context = {};
      if (dos) testCase.context.dos = dos;
      if (donts) testCase.context.donts = donts;
    }

    testCases.push(testCase);
  }

  if (testCases.length === 0) {
    throw new Error('No valid prompts found in the provided CSV file');
  }

  return testCases;
}

[burivuhster]

Should this whole judge/evaluators directory be moved under evaluators/llm-judge?

[burivuhster]

Could also be a single array referenceWorkflows, consumers may pass array of a single item instead of referenceWorkflow

[burivuhster]

Not sure how much work is to move workflow generation upper in the call chain, but it would be nice to make evaluator agnostic of the multi-generation and just always process a single generation. It would require some higher-level code doing the metrics aggregation though.

[burivuhster]

🤯

[burivuhster]

Suggested change

	);
	const evalDurationMs = Date.now() - evalStart;
	const totalDurationMs = Date.now() - startTime;
	const score = calculateExampleScore(feedback);
	const status = hasErrorFeedback(feedback)
	? 'error'
	: determineStatus({ score, passThreshold });
	stats.total++;
	stats.scoreSum += score;
	stats.durationSumMs += totalDurationMs;
	if (status === 'pass') stats.passed++;
	else if (status === 'fail') stats.failed++;
	else stats.errors++;
	const result: ExampleResult = {
	);
	const evalDurationMs = Date.now() - evalStart;
	const totalDurationMs = Date.now() - startTime;
	const score = calculateExampleScore(feedback);
	const status = hasErrorFeedback(feedback)
	? 'error'
	: determineStatus({ score, passThreshold });
	stats.total++;
	stats.scoreSum += score;
	stats.durationSumMs += totalDurationMs;
	if (status === 'pass') stats.passed++;
	else if (status === 'fail') stats.failed++;
	else stats.errors++;
	const result: ExampleResult = {

(Just line breaks to improve readability)

[burivuhster]

ASCII-art schemes have rough edges 😅

[mike12345567]

Really great cleanup, really tidies this up which was starting to get a little out of control!

Left a few comments I think we should consider addressing, one other thing as well, I think it would be nice to maintain some of the old functionality we had, or at least have a package.json command to achieve it. Being able to run the matrix suite with our old prompts could be useful, because they've been a benchmark so far as we have an idea of how they score, I think it would also be nice to have a command for the old default output directory and markdown report generation - the JSON is really useful, but its also handy to have a quick report we can look through.

Really like the generation splitting up the workflow JSON and feedback JSON, thats a massive pain reduction, always found myself trying to scrape the correct workflow out of the massive results JSON!

[mike12345567]

Should this also emit the score for the bestPractices?

[mike12345567]

Does the similarity evals need a weighting as well?

[mike12345567]

Is this test suite useful, I don't think we usually do type testing, I'm wondering if its just an over-reach by Claude!

[mike12345567]

Do we not have an option now to run with our standard set of eval prompts as we did before?

[mike12345567]

Would it be possible to add a --help command to list some of this information? Tried it locally, think it could be helpful.

[cubic-dev-ai]

1 issue found across 41 files (changes from recent commits).

Prompt for AI agents (all issues)

Check if these issues are valid — if so, understand the root cause of each and fix them.


<file name="packages/@n8n/ai-workflow-builder.ee/evaluations/README.md">

<violation number="1" location="packages/@n8n/ai-workflow-builder.ee/evaluations/README.md:81">
P3: The diagram is missing `'detail'` from the `kind` type. The actual `Feedback` interface (and the Feedback section later in this file) defines `kind: 'score' | 'metric' | 'detail'`.</violation>
</file>

_{Reply with feedback, questions, or to request a fix. Tag @cubic-dev-ai to re-run a review.}

[cubic-dev-ai]

P3: The diagram is missing 'detail' from the kind type. The actual Feedback interface (and the Feedback section later in this file) defines kind: 'score' | 'metric' | 'detail'.

Prompt for AI agents

Check if this issue is valid — if so, understand the root cause and fix it. At packages/@n8n/ai-workflow-builder.ee/evaluations/README.md, line 81:

<comment>The diagram is missing `'detail'` from the `kind` type. The actual `Feedback` interface (and the Feedback section later in this file) defines `kind: 'score' | 'metric' | 'detail'`.</comment>

<file context>
@@ -38,47 +44,43 @@ popd
+        F1["evaluator: string"]
+        F2["metric: string"]
+        F3["score: 0-1"]
+        F4["kind: 'score' | 'metric'"]
+        F5["comment?: string"]
+    end
</file context>

Suggested change

	F4["kind: 'score' | 'metric'"]
	F4["kind: 'score' | 'metric' | 'detail'"]