Skip to content

auditor: match repo-root SKILL.md in the discover probe; seed video-shotcraft - #790

Open
xiaolai wants to merge 1 commit into
mainfrom
auditor/seed-video-shotcraft
Open

auditor: match repo-root SKILL.md in the discover probe; seed video-shotcraft#790
xiaolai wants to merge 1 commit into
mainfrom
auditor/seed-video-shotcraft

Conversation

@xiaolai

@xiaolai xiaolai commented Aug 7, 2026

Copy link
Copy Markdown
Owner

Two changes, one cause: Vincentwei1021/video-shotcraft is a 3.7k-star skill repo the pipeline could never have found.

1. Discover probe missed repo-root SKILL.md

The artifact-count expression in auditor-discover.yml had no pattern anchored at a repo-root SKILL.md — the canonical single-skill layout in the open Agent Skills spec, where the repo itself is the skill directory:

  • ^[^/]+/SKILL\.md$ requires one leading directory
  • skills/.*/SKILL\.md$ and the .claude/skills/... variant require that literal segment

So a bare root SKILL.md matched nothing. bin/nlpm-check handles this layout fine (it rglobs for SKILL.md), which is why the gap went unnoticed — it is probe-specific.

Blast radius, measured: video-shotcraft has 660 files and ~116 NL artifacts. The probe scored it 1 (.claude-plugin/plugin.json alone) against MIN_ARTIFACTS=5. Invisible to discovery, and absent from all 280 registry entries.

Verified against the real tree: 1 → 2 after the fix.

2. Seeded the repo by hand

2 is still under the floor of 5, so the fix alone does not surface it. Seeded manually with discovery_strategy: "manual-seed" and audit issue #789; the registry write went through atomic-registry-write.sh and the metadata counters were bumped by the same jq discover uses.

Open question, not addressed here

The probe counts layout markers, not the reference layer. video-shotcraft's actual NL surface is 104 shot cards under references/shots/ plus 7 docs under references/, and no pattern reaches any of them — hence 2 rather than ~116. nlpm's own skills use references/ the same way, so any repo shaped "one root skill + a large reference tree" is undercounted by roughly two orders of magnitude.

Whether the probe should count a skill's references/**/*.md is a real change to discovery breadth, so I left it alone rather than deciding it inside a bug fix.

Verification

Check Result
python3 -m unittest discover -s tests 90 passed
Registry JSON parse valid
Workflow YAML parse valid
bin/nlpm-check . clean
jq expression against real 660-blob tree 1 → 2

Context

Manual audit of the repo at 0022ec4: nlpm-check clean, security PASS, NL score 98 (SKILL.md 95, plugin.json 100, agents/openai.yaml 100). One finding, contributed upstream as Vincentwei1021/video-shotcraft#28. Expected to clear the exemplar gate once the pipeline audits it.

…hotcraft

The artifact probe had no pattern anchored at a repo-root SKILL.md — the
canonical single-skill layout in the open Agent Skills spec, where the repo
itself is the skill directory. `^[^/]+/SKILL\.md$` needs one leading
directory and the `skills/`-prefixed patterns need that literal segment, so
a root SKILL.md matched nothing.

Found auditing Vincentwei1021/video-shotcraft (3.7k stars, 660 files,
~116 NL artifacts): it probed at 1 artifact — `.claude-plugin/plugin.json`
alone — against a MIN_ARTIFACTS=5 floor, so discovery could never surface
it. bin/nlpm-check finds root SKILL.md fine (rglob); the gap was
probe-specific.

The fix takes that repo 1 -> 2, still under the floor. Its real NL surface
is a 104-card references/shots/ tree that no probe pattern reaches, so the
repo is seeded by hand here (audit issue #789) rather than waiting on a
broader change to how the probe counts reference layers.
@xiaolai
xiaolai force-pushed the auditor/seed-video-shotcraft branch from ca52b4b to eb5a023 Compare August 7, 2026 04:34
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant