[evals] Investigate skipped test cases in skill-eval pipeline run 20260526.1

[helen229]

Background

In the meeting on 2026-05-27, Praveen flagged that test cases are being skipped in the skill-eval pipeline run 20260526.1 and asked for an investigation.

Goal

Understand why some Vally skill-eval test cases are being skipped in the pipeline run linked above, and either fix the skips or document the expected reason.

Tasks

• Reproduce / inspect the linked pipeline run and enumerate which test cases were skipped.
• Determine the cause for each skip (e.g., conditional matrix exclusion, missing env var, Vally host bug, stimulus error mis-categorised as skip, etc.).
• Add a focused test case (or matrix entry) that surfaces the skip behavior so it doesn't regress silently.
• File any host-level bugs discovered against @microsoft/vally-cli upstream.
• Update this issue with findings and link any follow-up PRs.

Related

• Parent epic: Adoption of Vally #15063 (Adoption of Vally)
• Migration tracking: Migrate Benchmarks + Tool invocation from evaluate #15124
• Recent flaky-test triage: [evals] Flaky-test report: 5/15 evals failing  #15696