You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
Onboarding PRs add components.d/<slug>.yml entries pointing at skill directories in a source repo. Nothing in this repo's CI verifies that those source skills carry a skill.oms.sig matching their current content.
The result is a failure that only surfaces after merge: the sync reverts the skill (for an existing one) or drops it entirely (for a new one, since there's no previously-signed version to fall back on). The onboarding PR itself shows all checks green, because content-integrity validates this repository's tree — not the source repository's.
Evidence
Both of these happened on 2026-08-31:
paidf-augmentation — onboarded via Add PAIDF Augmentation and Auto-Labeling skills #501 with all checks green. The next sync dropped it: 1 signed file(s) do not match skill.oms.sig — new skill, no prior version. Verifying by hand, 11 of 12 signed files matched; BENCHMARK.md did not.
The common cause is sequencing: BENCHMARK.md is regenerated after the signing run, so the signature covers an earlier report. It is an easy mistake to make and currently invisible until after merge.
Proposal
Add a check that runs on PRs touching components.d/**. For each declared path:
Fetch the skill directory from the source repo at the component's ref (default main).
Parse skill.oms.sig and extract the signed resource digests.
Recompute SHA-256 for each signed file and compare.
Fail with the specific file names that do not match.
The same pass can cheaply assert the other onboarding hard requirements, which are also only enforced by reviewer discipline today:
SKILL.md, skill.oms.sig, and a skill card are present
evals/evals.json is present
BENCHMARK.md exists and reports Overall verdict: PASS with real Tier 3 results
Acceptance criteria
An onboarding PR whose source skill has content not covered by its signature fails CI, naming the offending files.
A PR where every declared skill is correctly signed passes.
The check reads the source repo read-only and needs no signing credentials.
Failure output tells the author what to do: re-run signing at source so the signature covers current content.
Why it is worth doing
Post-merge drops are expensive to diagnose. The catalog looks correct, the PR looks correct, and the skill silently never appears. Moving this left turns a confusing absence into an actionable CI failure at the point where someone can fix it.
Problem
Onboarding PRs add
components.d/<slug>.ymlentries pointing at skill directories in a source repo. Nothing in this repo's CI verifies that those source skills carry askill.oms.sigmatching their current content.The result is a failure that only surfaces after merge: the sync reverts the skill (for an existing one) or drops it entirely (for a new one, since there's no previously-signed version to fall back on). The onboarding PR itself shows all checks green, because
content-integrityvalidates this repository's tree — not the source repository's.Evidence
Both of these happened on 2026-08-31:
paidf-augmentation— onboarded via Add PAIDF Augmentation and Auto-Labeling skills #501 with all checks green. The next sync dropped it:1 signed file(s) do not match skill.oms.sig — new skill, no prior version. Verifying by hand, 11 of 12 signed files matched;BENCHMARK.mddid not.paidf-orchestration(feat: onboard PAIDF Orchestration component #507) — the same defect in all four declared skills, caught only because the signature was checked manually during review:write-dagBENCHMARK.mdorchestration-setupBENCHMARK.mdevent-video-generation-workflowBENCHMARK.mdimage-attribute-augmentation-workflowBENCHMARK.mdThe common cause is sequencing:
BENCHMARK.mdis regenerated after the signing run, so the signature covers an earlier report. It is an easy mistake to make and currently invisible until after merge.Proposal
Add a check that runs on PRs touching
components.d/**. For each declaredpath:ref(defaultmain).skill.oms.sigand extract the signed resource digests.The same pass can cheaply assert the other onboarding hard requirements, which are also only enforced by reviewer discipline today:
SKILL.md,skill.oms.sig, and a skill card are presentevals/evals.jsonis presentBENCHMARK.mdexists and reportsOverall verdict: PASSwith real Tier 3 resultsAcceptance criteria
Why it is worth doing
Post-merge drops are expensive to diagnose. The catalog looks correct, the PR looks correct, and the skill silently never appears. Moving this left turns a confusing absence into an actionable CI failure at the point where someone can fix it.