Since #2504 bumped tests/.coder-eval-version to 0.9.5, the activation gate fails for every PR that touches SKILL.md frontmatter.
Symptom (deterministic, reproduced twice on #2496):
Skipping task file .../gate.yaml — ValueError: Invalid task definition: 1 validation error for TaskDefinition
success_criteria.0.skill_triggered.stop_when
Extra inputs are not permitted [type=extra_forbidden, input_value='auto', input_type=str]
ERROR: .../suite.json missing
Process completed with exit code 2
Failing runs: https://github.com/UiPath/skills/actions/runs/31366050504 (job 93384597028 and the re-run 93386751703, identical error).
Cause: tests/scripts/activation_gate.py still emits stop_when: auto inside the skill_triggered success criterion (the template comment notes it pairs with the experiment's armed stop_early, required since coder_eval >= 0.9.1). coder-eval 0.9.5 no longer accepts stop_when at that position, so gate.yaml is skipped, suite.json never materializes, and the job exits 2 before any eval runs.
Impact: any PR changing SKILL.md frontmatter (when_to_use etc.) fails CI regardless of content. I worked around it in #2496 by dropping the frontmatter portion of the change.
Likely fix: move/rename stop_when per the 0.9.5 schema in the gate template (or pin back until the template is updated).
🤖 Generated with Claude Code
Since #2504 bumped tests/.coder-eval-version to 0.9.5, the activation gate fails for every PR that touches SKILL.md frontmatter.
Symptom (deterministic, reproduced twice on #2496):
Failing runs: https://github.com/UiPath/skills/actions/runs/31366050504 (job 93384597028 and the re-run 93386751703, identical error).
Cause:
tests/scripts/activation_gate.pystill emitsstop_when: autoinside theskill_triggeredsuccess criterion (the template comment notes it pairs with the experiment's armedstop_early, required since coder_eval >= 0.9.1). coder-eval 0.9.5 no longer acceptsstop_whenat that position, so gate.yaml is skipped, suite.json never materializes, and the job exits 2 before any eval runs.Impact: any PR changing SKILL.md frontmatter (when_to_use etc.) fails CI regardless of content. I worked around it in #2496 by dropping the frontmatter portion of the change.
Likely fix: move/rename
stop_whenper the 0.9.5 schema in the gate template (or pin back until the template is updated).🤖 Generated with Claude Code