Evaluate skill-validator vs asm eval and decide integration strategy

[luongnv89]

Type

feature (high confidence) — research/spike, outcome may be code or a "won't do" decision record

Description

Investigate agent-ecosystem/skill-validator (Go, MIT, ~116★ — "Validate Skill content against the Agent Skill specification, with additional content density and quality checks") and decide how, or whether, to incorporate it into ASM's eval pipeline.

Before committing to integration, weigh effort vs. result. The goal of this issue is the decision, not the integration itself. Two realistic paths:

1. Cherry-pick — identify the specific checks skill-validator does that asm eval (src/evaluator.ts) does not, and port just those rules into our existing evaluator. Pros: no new runtime dep, no Go toolchain for users, keeps evalSummary shape stable. Cons: one-time port work, we take ownership of the logic.
2. Add as a new eval provider — invoke the Go binary as a subprocess (or via a small wrapper) and merge its results into evalSummary as a new category / provider. Pros: automatic updates when upstream improves, richer spec-conformance signal. Cons: adds a Go binary dependency (packaging + CI implications), duplicates work with our existing categories, extra runtime cost per skill on preindex.

The deliverable is a written recommendation backed by data. Only proceed to implementation if the cost/benefit clearly justifies it.

Investigation plan:

1. Run skill-validator against >=10 skills already in the curated index (mix of high/low eval scores).
2. Diff findings against asm eval output for the same skills. Catalogue: (a) checks unique to skill-validator, (b) checks unique to asm eval, (c) overlap.
3. Estimate porting effort per unique check (LOC, complexity, test surface).
4. Estimate integration effort for the provider path (packaging, distribution, CI, fallback when Go is unavailable).
5. Produce a recommendation: cherry-pick specific checks, add as provider, or drop. Include what to port / what to skip and why.

Related components:

• src/evaluator.ts
• data/skill-index/*.json (evalSummary shape)
• scripts/preindex.ts
• Packaging story if provider path: package.json, postinstall, bun vs node runtime (see Make bun an optional runtime dependency (CLI runs on node, bun only for TUI) #221)

Reporter Context

consider integrating into eval toolset: https://github.com/agent-ecosystem/skill-validator

update: need to evaluate carefully the effort and the result to see if it worth to add it as a new eval provider or we can just cherry pick the different into our existing asm eval

Acceptance Criteria

• Comparative run on >=10 indexed skills with diff table (unique-to-validator, unique-to-asm-eval, overlap) attached to the issue or PR (high confidence)
• Effort estimate for each path: cherry-pick (LOC + test surface per check) and new-provider (packaging, CI, fallback) (high confidence)
• Written recommendation: cherry-pick / add provider / drop — with reasoning rooted in the diff and effort data (high confidence)
• If cherry-pick wins: list of specific checks to port, with pointers to upstream rules (medium confidence)
• If provider wins: packaging plan that does not require Go toolchain for default ASM users (medium confidence)

Metadata

• Priority: low/medium (quality improvement, not blocking users)
• Estimated effort: S for the spike, M-L if implementation follows
• Suggested labels: feature, eval, tooling, research, spike