Implement per-case MCP overrides and related documentation

[roark47]

This pull request introduces support for per-case mocked MCP server overrides, allowing each test case to specify its own mocked MCP fixture while keeping server and tool names consistent. This enables more flexible and isolated testing scenarios, especially when running cases in parallel. The implementation includes configuration merging logic, schema and validation updates, documentation, and comprehensive tests.

Per-case mocked MCP server overrides

• Added support for case-level mcp.servers overrides in test case YAML files. Case-level servers merge with eval-level servers by name: a matching name replaces the eval-level entry, and new names are appended. Each case can now specify its own mocked fixture, with strict validation to ensure isolation and correctness. [1] [2] [3]

• Documented the new per-case MCP override feature in both the English and Chinese guides, including configuration examples, merging rules, and constraints. [1] [2]

Testing and validation

• Added end-to-end and unit tests to verify that per-case MCP overrides work as intended, including config merging, error handling, and that the correct fixture is used for each case. [1] [2] [3]

Example and fixture files

• Provided example skills, evals, case files, and fixture YAMLs to demonstrate and test per-case MCP override behavior. These include cases for "project-open" and "project-closed" that each use a different mocked fixture. [1] [2] [3] [4] [5] [6] [7]

Changelog

• Added a changelog entry describing the new per-case mocked MCP override feature and its merging semantics.

[Copilot]

Pull request overview

This PR adds per-case overrides for mocked MCP servers so each case can swap mocked fixtures (via config_ref) while keeping server/tool names stable, enabling isolated behavior across cases (including parallel execution).

Changes:

• Introduces merge semantics for case-level mcp.servers on top of eval-level MCP config (replace-by-name, append new).
• Updates evaluator runtime preparation to provision MCP per case when overrides are present, while preserving the cached fast-path when not.
• Adds validator coverage, unit/e2e tests, fixtures, and documentation/changelog entries describing the feature and constraints.

Reviewed changes

Copilot reviewed 21 out of 21 changed files in this pull request and generated 2 comments.

Show a summary per file

File	Description
skills/skill-upper/references/eval-yaml.md	Documents case-level mocked MCP override behavior in the Skill reference.
internal/mcp/provisioner_test.go	Adds unit test proving same server name can embed different mocked fixtures via different config_ref.
internal/evaluator/evaluator.go	Switches to per-case MCP provisioning when case-level overrides are present.
internal/evaluator/evaluator_test.go	Adds tests validating override application and rejecting case-level mode: real.
internal/config/validator.go	Adds case-level MCP validation logic for MVP constraints.
internal/config/validator_test.go	Extends validator test matrix for valid/invalid case-level MCP overrides.
internal/config/schema.go	Adds mcp block to CaseConfig schema with documented merge semantics.
internal/config/mcp_merge.go	Adds MergeCaseMCP implementation for eval/case MCP server merging.
internal/config/mcp_merge_test.go	Adds unit tests for merge semantics, immutability, and error cases.
internal/config/loader_test.go	Adds loader tests ensuring case-level mcp is parsed and defaults behave.
e2e/testdata/mcp-case-overrides/SKILL.md	Adds a smoke skill used by e2e to exercise case overrides.
e2e/testdata/mcp-case-overrides/evals/fixtures/mcp/project-open.yaml	Adds OPEN fixture for per-case override.
e2e/testdata/mcp-case-overrides/evals/fixtures/mcp/project-default.yaml	Adds default fixture (UNKNOWN) for eval-level baseline.
e2e/testdata/mcp-case-overrides/evals/fixtures/mcp/project-closed.yaml	Adds CLOSED fixture for per-case override.
e2e/testdata/mcp-case-overrides/evals/eval.yaml	Adds eval config wiring default server + two case files.
e2e/testdata/mcp-case-overrides/evals/cases/project-open.yaml	Adds case file overriding fixture to OPEN.
e2e/testdata/mcp-case-overrides/evals/cases/project-closed.yaml	Adds case file overriding fixture to CLOSED.
e2e/mcp_test.go	Adds end-to-end test that runs two cases with per-case mocked overrides.
docs/zh/guide/writing-evals.md	Documents per-case mocked MCP overrides (Chinese).
docs/guide/writing-evals.md	Documents per-case mocked MCP overrides (English).
CHANGELOG.md	Adds changelog entry describing SUP-0003 behavior and constraints.

---

💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.