Skip to content

Verify source-repo signatures on onboarding PRs before merge #508

Description

@mosheabr

Problem

Onboarding PRs add components.d/<slug>.yml entries pointing at skill directories in a source repo. Nothing in this repo's CI verifies that those source skills carry a skill.oms.sig matching their current content.

The result is a failure that only surfaces after merge: the sync reverts the skill (for an existing one) or drops it entirely (for a new one, since there's no previously-signed version to fall back on). The onboarding PR itself shows all checks green, because content-integrity validates this repository's tree — not the source repository's.

Evidence

Both of these happened on 2026-08-31:

  • paidf-augmentation — onboarded via Add PAIDF Augmentation and Auto-Labeling skills #501 with all checks green. The next sync dropped it: 1 signed file(s) do not match skill.oms.sig — new skill, no prior version. Verifying by hand, 11 of 12 signed files matched; BENCHMARK.md did not.

  • paidf-orchestration (feat: onboard PAIDF Orchestration component #507) — the same defect in all four declared skills, caught only because the signature was checked manually during review:

    Skill Files matching Mismatch
    write-dag 4 / 5 BENCHMARK.md
    orchestration-setup 8 / 9 BENCHMARK.md
    event-video-generation-workflow 13 / 14 BENCHMARK.md
    image-attribute-augmentation-workflow 14 / 15 BENCHMARK.md

The common cause is sequencing: BENCHMARK.md is regenerated after the signing run, so the signature covers an earlier report. It is an easy mistake to make and currently invisible until after merge.

Proposal

Add a check that runs on PRs touching components.d/**. For each declared path:

  1. Fetch the skill directory from the source repo at the component's ref (default main).
  2. Parse skill.oms.sig and extract the signed resource digests.
  3. Recompute SHA-256 for each signed file and compare.
  4. Fail with the specific file names that do not match.

The same pass can cheaply assert the other onboarding hard requirements, which are also only enforced by reviewer discipline today:

  • SKILL.md, skill.oms.sig, and a skill card are present
  • evals/evals.json is present
  • BENCHMARK.md exists and reports Overall verdict: PASS with real Tier 3 results

Acceptance criteria

  • An onboarding PR whose source skill has content not covered by its signature fails CI, naming the offending files.
  • A PR where every declared skill is correctly signed passes.
  • The check reads the source repo read-only and needs no signing credentials.
  • Failure output tells the author what to do: re-run signing at source so the signature covers current content.

Why it is worth doing

Post-merge drops are expensive to diagnose. The catalog looks correct, the PR looks correct, and the skill silently never appears. Moving this left turns a confusing absence into an actionable CI failure at the point where someone can fix it.

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Type

    No type

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions