Skill install failure is only a warning, so the job succeeds without the skill

[JaganGopalkrish]

Summary

When a skill declared in skills: cannot be installed, the workflow logs a warning and continues to success. The agent then runs without that skill, and nothing in the run status indicates anything is missing.

Why this is easy to miss

install_frontmatter_skills.cjs emits ::warning::Failed to install skill '<ref>', collect_skill_install_failures.cjs records the count, and main() finishes with:

if (failureCount > 0) {
  core.warning(`${failureCount} skill(s) failed to install — see agent failure issue/comment for details`);
}

There is no core.setFailed, so the job is green. The remaining skills install normally, the agent starts, produces plausible output, and the only evidence is a warning inside a collapsed step of a passing run.

I hit this with a skill that carried operational cost and safety limits for a tool the agent drives. The agent ran without those limits and the run looked entirely healthy. I found it only by reading install logs of a run that had already passed.

Reproduction

Reference a skill in a private or internal repository that your account can read but the workflow's GITHUB_TOKEN cannot:

skills:
  - some-org/private-repo/skills/example@<sha>

Observed (gh-aw v0.86.2):

Installing skill reference: some-org/private-repo/skills/example@<sha>
could not resolve version: ref "<sha>" not found as branch, tag, or commit in some-org/private-repo
##[warning]Failed to install skill 'some-org/private-repo/skills/example@<sha>': The process '/usr/bin/gh' failed with exit code 1
Installed 6 skill file(s)
##[warning]1 skill(s) failed to install — details will be reported in the agent failure issue/comment

Job conclusion: success.

Two things compound it:

1. A permission failure is reported as ref ... not found as branch, tag, or commit, which reads like a typo'd ref rather than an auth problem. The ref existed and was correct.
2. Because the outcome is identical either way, there is no signal to prompt a look.

(The correct fix for my case was per-skill auth via github-token: on the skills: entry, which works well — this report is only about the failure being silent.)

Suggestion

Fail the job by default when a declared skill does not install. A workflow that asked for a skill and didn't get it is not in the state its author specified, and skills increasingly carry safety and policy content where silent absence is the worst outcome.

If opt-out is wanted:

skills:
  on-install-failure: fail   # default
  entries:
    - owner/repo/skills/example@<sha>

The information is already collected and surfaced as an output — it just doesn't affect the result. The minimal change is core.setFailed in place of the final core.warning in collect_skill_install_failures.cjs.

It would also help to distinguish 404-from-permissions from a genuinely bad ref in the install error message, since with a repo-scoped GITHUB_TOKEN the two are indistinguishable.

I'm happy to raise a PR for either or both if you'd like — just let me know which behaviour you'd prefer as the default.

[pelikhan]

@copilot the skill installation should be surfaced in the agent failure issue/comment with a structured message specialized. The agent should be skipped if the required skills are not available.