Skip to content

[evals] Investigate skipped test cases in skill-eval pipeline run 20260526.1 #15765

@helen229

Description

@helen229

Background

In the meeting on 2026-05-27, Praveen flagged that test cases are being skipped in the skill-eval pipeline run 20260526.1 and asked for an investigation.

Goal

Understand why some Vally skill-eval test cases are being skipped in the pipeline run linked above, and either fix the skips or document the expected reason.

Tasks

  • Reproduce / inspect the linked pipeline run and enumerate which test cases were skipped.
  • Determine the cause for each skip (e.g., conditional matrix exclusion, missing env var, Vally host bug, stimulus error mis-categorised as skip, etc.).
  • Add a focused test case (or matrix entry) that surfaces the skip behavior so it doesn't regress silently.
  • File any host-level bugs discovered against @microsoft/vally-cli upstream.
  • Update this issue with findings and link any follow-up PRs.

Related

Metadata

Metadata

Assignees

Labels

AzSDK Tools AgentIssue related to the AzSDK Tools Agent.

Type

No type
No fields configured for issues without a type.

Projects

No projects

Milestone

No milestone

Relationships

None yet

Development

No branches or pull requests

Issue actions