Beta Testing: pai-pkg Multi-Source Registry & Skill Lifecycle

[mellanon]

Beta Testing Request

Project: pai-pkg
Version: 0.1.0
Seeking testers: @jcfischer

pai-pkg now has a working multi-source registry, upgrade lifecycle, and publishing pipeline. Ready for external testing before community launch.

What to Test

1. Install & Setup

git clone https://github.com/mellanon/pai-pkg.git
cd pai-pkg && bun install && bun link
pai-pkg source list                    # Should show pai-collab source
pai-pkg source update                  # Should fetch 4 community packages

2. Search & Discovery (apt-cache model)

pai-pkg search doc                     # Finds _DOC from community registry
pai-pkg search hello                   # Finds hello-pai tool
pai-pkg search nonexistent             # Should show no results gracefully

3. Install from Registry

pai-pkg install _DOC                   # Install from community registry
pai-pkg install hello-pai              # Install a tool (gets CLI shim)
pai-pkg list                           # Should show installed packages
pai-pkg info _DOC                      # Should show capabilities + release notes

4. Publishing Your Own Skill

# Scaffold a new skill
pai-pkg init my-test-skill --type skill

# Or a tool
pai-pkg init my-test-tool --type tool --author jcfischer

# Check generated files: pai-manifest.yaml, package.json, README.md, skill/SKILL.md

Then follow publishing SOP to submit a PR adding an entry to skills/REGISTRY.yaml.

5. Install from Direct Git URL (custom tier)

pai-pkg install https://github.com/jcfischer/specflow-bundle
# Should show capabilities with risk warning (custom tier)
# Should prompt for confirmation

6. Upgrade Lifecycle (apt-get upgrade model)

pai-pkg upgrade --check                # Compare installed vs registry versions
pai-pkg upgrade                        # Upgrade all
pai-pkg upgrade _DOC                   # Upgrade specific package

7. Audit & Security

pai-pkg audit                          # Summary: capability surface + cross-tier warnings
pai-pkg audit --verbose                # Full pairwise capability combination list
pai-pkg verify _DOC                    # Verify manifest integrity

8. Lifecycle Management

pai-pkg disable _DOC                   # Disable (preserves repo)
pai-pkg enable _DOC                    # Re-enable
pai-pkg remove hello-pai               # Full uninstall

9. Multi-Source Registry

# Add your own registry source
pai-pkg source add my-registry https://raw.githubusercontent.com/you/registry/main/REGISTRY.yaml --tier community
pai-pkg source update                  # Should fetch from all sources
pai-pkg search <keyword>               # Should search across all sources
pai-pkg source remove my-registry

10. Version Management

Each skill repo should have:

• pai-manifest.yaml with version field
• Git tag matching the version (v1.0.0)
• GitHub Release with release notes
• pai-pkg info <name> displays release notes via gh CLI

Key Areas of Interest

• Usability: Is the CLI intuitive? Are error messages helpful?
• Trust model: Does the tier display (official/community/custom) make sense during install?
• Capability display: Are the risk levels and capability listings clear?
• Publishing flow: Can you successfully scaffold + publish a skill?
• Edge cases: What breaks? Missing manifests, bad URLs, network failures?

Environment Requirements

• Bun v1.0+
• Git
• Optional: GitHub CLI (gh) for release notes in pai-pkg info

How to Report

Comment on this issue with findings, or open separate issues on pai-pkg for bugs.

Related

• Closes testing gap from Review Request: pai-pkg Security Architecture + Skill Lifecycle Design #106 (security architecture review — now closed)
• Publishing SOP
• pai-pkg README

[mellanon]

cc @danielmiessler — pai-pkg is a package manager for PAI that enables skill discovery, installation, and sharing. Would love your feedback on the approach.

The pai-pkg README has the full overview and the testing steps above cover the main flows.

[jcfischer]

Beta Test Report: pai-pkg v0.2.0

Tester: @jcfischer (agent: Ivy)
Date: 2026-03-25
Environment: macOS 14, Bun 1.3.10, Git, gh CLI
Test plan: All 10 areas from issue description

---

Results Summary

#	Test Area	Result	Notes
1	Install & Setup	✅ PASS	bun link works, source list shows pai-collab
2	Search & Discovery	✅ PASS	search doc → _DOC, search hello → hello-pai, search nonexistent → graceful "no matches"
3	Install from Registry	✅ PASS	_DOC v1.0.1 and hello-pai v1.0.0 installed cleanly, capabilities displayed, release notes shown in info
4	Publishing / Scaffold	✅ PASS	init test-skill --type skill --author jcfischer generates correct structure with pai-manifest.yaml, skill/SKILL.md, package.json
5	Direct Git URL Install	✅ PASS	install https://github.com/jcfischer/specflow-bundle — installed SpecFlow v0.1.0 with full capabilities
6	Upgrade Lifecycle	✅ PASS	upgrade --check → "All packages are up to date"
7	Audit & Security	✅ PASS	audit shows correct capability surface (3 read, 4 write, 2 bash). verify works, caught uncommitted changes
8	Lifecycle Management	✅ PASS	disable → list shows ⏸️ → enable → verify → remove — full cycle works
9	Multi-Source Registry	✅ PASS	source update fetched 4 packages from pai-collab registry
10	Version Management	✅ PASS	info _DOC shows release notes from GitHub Release

Observations

What Works Well

• The apt mental model is effective. source update → search → install → upgrade --check feels natural for anyone who's used apt/brew.
• Release notes in info via gh CLI — nice touch. Pulls directly from GitHub Releases.
• Scaffold generates clean structure — manifest, SKILL.md, workflows, gitignore. Ready to fill in.
• Capability display is clear. Green/amber/red emoji for read/write/bash with risk level.
• Verify catches real issues. Detected uncommitted changes in installed repo.

Bugs Found (Already Filed)

• signal: Contrib prep complete — clean branch available for review #1 — EEXIST when target skill directory is a real directory (not symlink). PR signal: Independent review (automated + community) #2 submitted with fix.
• signal: Package and submit PR to danielmiessler/PAI #3 — TypeError when depends_on.packages entry has no repo URL.

Minor Issues (Not Filed)

1. verify reports "Uncommitted changes detected" for freshly-installed packages. This is because bun install creates bun.lock and node_modules/ which aren't committed in the cloned repo. Consider adding bun.lock and node_modules to the "expected untracked" list, or running git checkout -- . after install.

2. audit --verbose output is identical to audit when there are no warnings. The verbose flag doesn't seem to add extra detail for the happy path. Consider showing the full per-skill capability breakdown even when no warnings.

3. Scaffold creates src/ directory but no file in it — empty dir. Minor, but could confuse new users.

Two Repos Now pai-pkg Ready

We've added pai-manifest.yaml + skill/ to:

• pii-pseudonymizer — pai-pkg install https://github.com/jcfischer/pii-pseudonymizer
• specflow-bundle — pai-pkg install https://github.com/jcfischer/specflow-bundle (monorepo with skill/ symlink to packages/specflow/)

Both install cleanly and show correct capabilities in info and audit.

Usability Feedback

• CLI is intuitive. Command naming follows apt/brew conventions closely.
• Error messages are helpful for common cases (not found, already installed).
• Tier display during install — didn't see this triggered for community packages (auto-approved with -y). Would be good to test without -y to see the confirmation prompt.
• source update is fast — 4 packages fetched in under a second.

---

Overall: Ready for community launch. The core lifecycle (search → install → use → upgrade → remove) works end-to-end. The two filed bugs (#1, #3) are install-time edge cases, not blockers for the happy path.

[mellanon]

Thanks @jcfischer — excellent test report. All findings addressed:

Bugs:

• EEXIST on real directories + TypeError on missing repo URL — both were fixed in `fd3c629` (before this test)

Fuzz findings (#4):

• F1 (init path traversal) — fixed in `15715eb`, names with `/`, `\`, or `..` now rejected
• F2 (source URL validation) — deferred, error surfaces at `source update` time
• Issue closed

Minor feedback:

• `verify` false positive on `bun.lock`/`node_modules` — fixed in `15715eb`, these are now excluded from the dirty check
• `audit --verbose` happy path — noted, will add per-skill breakdown in a future pass
• Empty `src/` in scaffold — the `src/lib/` dir is created for skills; will review

Two repos pai-pkg ready — great to see specflow-bundle and pii-pseudonymizer installable. When you're ready to publish to the community registry, open a PR adding entries to `skills/REGISTRY.yaml` per the publishing SOP.

Release notes updated at https://github.com/mellanon/pai-pkg/releases/tag/v0.2.0

[mellanon]

All reported issues now resolved in v0.2.0:

From beta test report:

• ✅ verify false positive on bun.lock/node_modules — fixed
• ✅ audit --verbose now shows per-skill capability breakdown
• ✅ Scaffold no longer creates empty src/ directory

From fuzz testing (#4):

• ✅ F1: init path traversal — names with path separators rejected
• ✅ F2: source add URL validation — rejects invalid URLs at add time
• ✅ F3: empty scaffold dirs — src/ removed from skill scaffold

@jcfischer run `pai-pkg self-update` to pull all fixes. Release notes updated at https://github.com/mellanon/pai-pkg/releases/tag/v0.2.0