Skip to content

Commit 9f80ef5

Browse files
committed
Expand CONTRIBUTING.md with skill development template
Adds directory structure, SKILL.md frontmatter format, eval requirements, check_output.py template, registration steps, and pattern registry format.
1 parent c3ea4c0 commit 9f80ef5

1 file changed

Lines changed: 118 additions & 0 deletions

File tree

CONTRIBUTING.md

Lines changed: 118 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -56,3 +56,121 @@ Before opening a PR:
5656
2. Update docs when commands, outputs, contracts, or release behavior changes.
5757
3. Update `CHANGELOG.md` for release-facing changes.
5858
4. Explain whether skill output behavior changed.
59+
60+
## Creating a New Skill
61+
62+
Use this template to add a new skill to the collection.
63+
64+
### 1. Directory Structure
65+
66+
```
67+
skills/omv-<name>/
68+
SKILL.md — skill definition (frontmatter name must match directory)
69+
references/ — detailed guidance loaded on demand
70+
patterns/<ecosystem>.md — ecosystem-specific patterns (if applicable)
71+
scripts/
72+
check_output.py — heuristic eval checker
73+
evals/
74+
evals.json — eval scenarios (minimum 3)
75+
golden/ — stable golden outputs
76+
contracts/ — copies of consumed contracts
77+
```
78+
79+
### 2. SKILL.md Frontmatter
80+
81+
```yaml
82+
---
83+
name: omv-<name>
84+
description: |
85+
One paragraph describing when to invoke this skill.
86+
---
87+
```
88+
89+
The `name` field MUST match the directory basename exactly.
90+
91+
### 3. Eval Requirements
92+
93+
Every skill needs at least 3 eval scenarios covering:
94+
95+
- **Happy path** — normal successful invocation
96+
- **Edge case** — boundary condition or unusual input
97+
- **Error/blocked path** — graceful handling of invalid or impossible requests
98+
99+
Each eval in `evals.json` must have:
100+
- `id`: unique integer
101+
- `prompt`: the invocation string
102+
- `expected_output`: human-readable description
103+
- `files`: list of golden output paths
104+
- `assertions`: list of `{type, text}` pairs checked by `check_output.py`
105+
106+
### 4. check_output.py Template
107+
108+
```python
109+
#!/usr/bin/env python3
110+
"""Heuristic checker for omv-<name> eval outputs."""
111+
112+
from __future__ import annotations
113+
import argparse, json, re, sys
114+
from pathlib import Path
115+
from typing import Any
116+
117+
SKILL_DIR = Path(__file__).resolve().parents[1]
118+
119+
def load_eval(evals_path: Path, eval_id: int) -> dict[str, Any]:
120+
data = json.loads(evals_path.read_text(encoding="utf-8"))
121+
item = next((e for e in data["evals"] if e["id"] == eval_id), None)
122+
if item is None:
123+
raise SystemExit(f"unknown eval id: {eval_id}")
124+
return item
125+
126+
def check(assertion_type: str, text: str) -> bool:
127+
# Add assertion checks here
128+
raise SystemExit(f"unknown assertion type: {assertion_type}")
129+
130+
def main() -> None:
131+
parser = argparse.ArgumentParser()
132+
parser.add_argument("--eval-id", type=int, required=True)
133+
parser.add_argument("--output", type=Path, required=True)
134+
parser.add_argument("--evals", type=Path, default=SKILL_DIR / "evals" / "evals.json")
135+
args = parser.parse_args()
136+
eval_item = load_eval(args.evals, args.eval_id)
137+
output = args.output.read_text(encoding="utf-8")
138+
failures = [
139+
a["type"] for a in eval_item.get("assertions", [])
140+
if not check(str(a["type"]), output)
141+
]
142+
if failures:
143+
print("FAIL: " + ", ".join(failures), file=sys.stderr)
144+
raise SystemExit(1)
145+
print(f"OK: eval {args.eval_id} heuristic assertions passed")
146+
147+
if __name__ == "__main__":
148+
main()
149+
```
150+
151+
### 5. Registration
152+
153+
After creating the skill:
154+
155+
1. Run `python3 scripts/validate_skill.py skills/omv-<name>` to verify structure.
156+
2. Add the skill to `registry.yaml` with version, produces/consumes bindings.
157+
3. Run `python3 scripts/sync_skill_assets.py` to sync shared references.
158+
4. Run `npm test` to verify no regressions.
159+
160+
### 6. Pattern Registry (if applicable)
161+
162+
If your skill uses ecosystem-specific vulnerability patterns, add them to `shared/references/patterns/<ecosystem>.md` with this structure:
163+
164+
```markdown
165+
## <Vuln Class>: <short description>
166+
167+
- Source pattern: ...
168+
- Sink signature: ...
169+
- Common misuse: ...
170+
- Expected guard: ...
171+
- Evidence criteria: ...
172+
- False-positive checks: ...
173+
- CWE: CWE-XXX
174+
```
175+
176+
Currently supported ecosystems: npm, python, go, rust, java, ruby, php, csharp, swift, dart, elixir, perl.

0 commit comments

Comments
 (0)