feat(ai-builder): Improve eval verifier and mock handler reliability (no-changelog)

[JoseBra]

Summary

• Use structured output on the verifier to eliminate "No verification result" failures — response is now schema-validated instead of parsed from free-form text
• Add retry logic to the mock handler with configurable maxRetries (default 1) to recover from transient LLM failures
• Improve failure categorization — verifier now traces data flow before attributing failures, correctly distinguishing builder logic errors from mock data issues
• Remove hardcoded maxTokens overrides, let the AI SDK use model defaults
• Make test case scenarios less brittle (remove exact count expectations, use minimal response hints)

Related Linear ticket

https://linear.app/n8n/issue/TRUST-34

Test plan

• Mock handler tests: 32/32 pass
• Typecheck clean (instance-ai)
• Full suite run: zero "No verification result", 1 mock_issue out of 27 scenarios

🤖 Generated with Claude Code

[codecov]

Codecov Report

❌ Patch coverage is 92.30769% with 1 line in your changes missing coverage. Please review.

Files with missing lines	Patch %	Lines
...s/cli/src/modules/instance-ai/eval/mock-handler.ts	92.30%	1 Missing ⚠️

📢 Thoughts on this report? Let us know!

[cubic-dev-ai]

2 issues found across 5 files

Prompt for AI agents (unresolved issues)

Check if these issues are valid — if so, understand the root cause of each and fix them. If appropriate, use sub-agents to investigate and fix each issue separately.


<file name="packages/@n8n/instance-ai/evaluations/data/workflows/rest-api-data-pipeline.json">

<violation number="1" location="packages/@n8n/instance-ai/evaluations/data/workflows/rest-api-data-pipeline.json:11">
P2: The happy-path success criteria dropped verification of the required post count, so this test can pass even if the workflow misses part of the requested Slack summary.</violation>
</file>

<file name="packages/@n8n/instance-ai/evaluations/checklist/verifier.ts">

<violation number="1" location="packages/@n8n/instance-ai/evaluations/checklist/verifier.ts:12">
P1: Structured output schema does not match the verifier prompt contract (array + nullable fields), so valid model responses can be rejected and still fall back to missing verification results.</violation>
</file>

Architecture diagram

sequenceDiagram
    participant Runner as Eval Runner
    participant Mock as Mock Handler
    participant LLM as AI Provider (SDK)
    participant Verifier as Eval Verifier

    Note over Runner,Verifier: Phase 1: Workflow Execution (Mocking)

    Runner->>Mock: interceptRequest(node, context)
    
    loop NEW: Retry Logic (maxRetries)
        Mock->>LLM: generate(userPrompt)
        Note right of LLM: CHANGED: Using model default maxTokens
        alt Success
            LLM-->>Mock: JSON specification
            Mock->>Mock: materializeSpec()
        else LLM Error / Timeout
            LLM-->>Mock: Transient Failure
            Note over Mock: Log warning & decrement retries
        end
    end

    alt All Retries Failed
        Mock-->>Runner: Return _evalMockError object
    else Success
        Mock-->>Runner: Return mocked response body/status
    end

    Note over Runner,Verifier: Phase 2: Result Verification

    Runner->>Verifier: verify(artifact, checklist)
    
    Verifier->>LLM: NEW: structuredOutput(schema)
    Note right of LLM: Uses checklistResultSchema (Zod)
    
    LLM-->>Verifier: Validated JSON Object
    
    alt Schema Validation Success
        Verifier->>Verifier: CHANGED: Trace data flow
        Note right of Verifier: Categorize: builder_issue vs mock_issue
        Verifier-->>Runner: List of ChecklistResult
    else NEW: Schema Validation Failure (null)
        Verifier->>Verifier: Log warning (No results)
        Verifier-->>Runner: Empty results array
    end

Loading

_{Reply with feedback, questions, or to request a fix. Tag @cubic-dev-ai to re-run a review, or fix all with cubic.}