feat: Add support for skills

[patricka3125]

Overview

Adds a skills system to CAO that enables reusable, composable blocks of instructional content that can be shared across agent profiles. Skills are lazily loaded via MCP tool calls rather than injected into the system prompt upfront, preserving context window budget.

Addresses issue #143. See docs/skills-design.md for the full design document.

Motivation

Agent profiles today are monolithic — when multiple agents need the same domain knowledge (testing conventions, communication protocols, coding standards), that knowledge must be duplicated across each profile. Skills solve this by separating agent identity (who the agent is) from agent capability (what the agent knows how to do). Skills are defined once, referenced by name in profile frontmatter, and loaded on-demand by the agent at runtime.

Key Changes

Skill file format and store

• Skills are folders containing a SKILL.md with YAML frontmatter (name, description) and a Markdown body
• All skills live in ~/.aws/cli-agent-orchestrator/skills/ — a single user-writable store with no built-in/custom split
• Two builtin skills ship with the package: cao-supervisor-protocols and cao-worker-protocols

CLI commands (cao skills)

• cao skills list — lists all installed skills with name and description
• cao skills add <folder-path> — installs a skill from a local folder (with --force flag for overwrites)
• cao skills remove <name> — removes an installed skill

Skill seeding via cao init

• cao init now seeds builtin skills into the user store, skipping any that already exist to preserve user edits

Server-side skill retrieval

• New GET /skills/{name} HTTP endpoint on cao-server returns skill body content
• New load_skill MCP tool on cao-mcp-server wraps the HTTP call for agent access
• Path traversal validation on all skill name inputs

Provider skill injection

Skills are delivered to agents via three different mechanisms depending on what each provider supports natively:

Runtime prompt injection (Claude Code, Codex, Gemini CLI, Kimi CLI)

A skill catalog (names + descriptions + load_skill usage instruction) is built at terminal creation time and appended to the system prompt via each provider's CLI flag (e.g. --append-system-prompt, -c developer_instructions). The agent then calls the load_skill MCP tool to load full skill content on demand.

Native skill:// resources (Kiro CLI)

cao install writes a skill:// glob URI (skill://<SKILLS_DIR>/*/SKILL.md) into the Kiro agent JSON's resources field. Kiro natively supports progressive loading — only skill metadata (name, description) is loaded at startup, with full content loaded on demand when the agent determines it's relevant. No prompt baking or refresh on cao skills add/remove is needed.

Baked prompt at install time (Q CLI)

Note that there is no official documentation listing Q CLI supports skill:// as a resource, I've elected to keep the solution similar to the primitive approach done for Copilot CLI.

cao install bakes the skill catalog text into the agent JSON's prompt field via compose_agent_prompt(). On cao skills add/remove, installed Q agent JSONs are refreshed to reflect the updated catalog.

Baked prompt at install time (Copilot CLI)

cao install bakes the skill catalog into the .agent.md body (appended after the profile's system prompt). On cao skills add/remove, installed Copilot agent .agent.md files are refreshed to reflect the updated catalog. Non-CAO-managed Copilot agents (no matching context file) are left untouched.

Provider	Mechanism	Refresh on skill change	Skill discovery
Claude Code	Runtime prompt injection	Automatic (built at terminal creation)	load_skill MCP tool
Codex	Runtime prompt injection	Automatic	load_skill MCP tool
Gemini CLI	Runtime prompt injection	Automatic	load_skill MCP tool
Kimi CLI	Runtime prompt injection	Automatic	load_skill MCP tool
Kiro CLI	Native skill:// resources	Automatic (glob)	Kiro native progressive loading
Q CLI	Baked prompt at install	cao skills add/remove triggers refresh	load_skill MCP tool
Copilot CLI	Baked prompt at install	cao skills add/remove triggers refresh	load_skill MCP tool

Test plan

• Unit tests: uv run pytest test/ --ignore=test/e2e -v — covers skill model, utils, CLI commands, API endpoint, MCP tool, terminal service integration, and provider changes
• E2E tests: uv run pytest -m e2e test/e2e/test_skills.py -v — covers skill catalog injection and skill API endpoint with a running server
• Manual: cao init seeds builtin skills, cao skills list/add/remove work as expected

[patricka3125]

E2E Test Report: cao skills CLI

Branch: feat/skills (2c2a081)
Run via: uv run cao from project source

Help Text

Command	Status	Notes
cao --help	✅	Shows skills with description "Manage installed skills."
cao skills --help	✅	Shows add, list, remove subcommands with correct descriptions
cao skills add --help	✅	Shows FOLDER_PATH argument, --force flag
cao skills remove --help	✅	Shows NAME argument
cao skills list --help	✅	Clean, no extra options

Happy Path

Test	Status	Output
skills list (empty)	✅	"No skills found"
skills add (built-in skill)	✅	"Skill 'cao-worker-protocols' installed successfully"
skills add (second skill)	✅	"Skill 'cao-supervisor-protocols' installed successfully"
skills list (2 skills)	✅	Sorted table with Name + Description columns, 100-char separator
skills add --force (overwrite)	✅	Overwrites without error
skills remove	✅	"Skill 'cao-worker-protocols' removed successfully"
skills list (after remove)	✅	Shows only remaining skill
skills add (custom skill)	✅	Custom SKILL.md installed to ~/.aws/cli-agent-orchestrator/skills/
Filesystem verification	✅	SKILL.md copied to correct directory

Error Handling

Test	Status	Output
Duplicate add (no --force)	✅	"Error: Skill '...' already exists. Use --force to overwrite it." (exit 1)
Remove nonexistent skill	✅	"Error: Skill '...' does not exist." (exit 1)
Add from nonexistent path	✅	Click validates path: "Path '...' does not exist." (exit 2)
Add folder without SKILL.md	✅	"Error: Missing SKILL.md in skill folder: ..." (exit 1)
Add with invalid frontmatter	✅	Pydantic validation error with field details (exit 1)
Add with name/folder mismatch	✅	"Error: Skill folder name '...' does not match skill name '...'" (exit 1)
Path traversal on remove	✅	"Error: Invalid skill name '...': must not contain '/', '', or '..'" (exit 1)

Summary

18/18 tests passed. All subcommands work correctly. Help descriptions are clear and consistent. Error messages are specific and actionable. Path traversal protection works. The skill store at ~/.aws/cli-agent-orchestrator/skills/ is created on demand and cleaned up properly.

[patricka3125]

E2E Test Report: Provider Skill Injection (d1a04ce)

Prerequisites

• CAO server running (cao-server on port 9889)
• cao init has been run (seeds builtin skills)
• Kiro CLI, Q CLI, and Copilot CLI config directories exist

Reproduction Steps

# 1. Clean previous installs
rm -f ~/.kiro/agents/developer.json ~/.aws/amazonq/cli-agents/developer.json ~/.copilot/agents/developer.agent.md

# 2. Install developer agent for all three providers
cao install developer --provider kiro_cli
cao install developer --provider q_cli
cao install developer --provider copilot_cli

# 3. Inspect agent configs
cat ~/.kiro/agents/developer.json | python3 -m json.tool    # Expect: skill:// in resources, no prompt field
cat ~/.aws/amazonq/cli-agents/developer.json | python3 -m json.tool  # Expect: catalog in prompt field
cat ~/.copilot/agents/developer.agent.md  # Expect: profile body + catalog in markdown body

# 4. Add two test skills and verify propagation
mkdir -p /tmp/alpha-skill && cat > /tmp/alpha-skill/SKILL.md << 'SKILL'
---
name: alpha-skill
description: First test skill
---
# Alpha Skill
SKILL
mkdir -p /tmp/beta-skill && cat > /tmp/beta-skill/SKILL.md << 'SKILL'
---
name: beta-skill
description: Second test skill
---
# Beta Skill
SKILL
cao skills add /tmp/alpha-skill
cao skills add /tmp/beta-skill

# 5. Verify Kiro glob captures both new skills
ls ~/.aws/cli-agent-orchestrator/skills/*/SKILL.md  # Should list 4 files
# Verify Kiro JSON unchanged (no prompt, same glob URI)
# Verify Q prompt contains alpha-skill and beta-skill
# Verify Copilot body contains alpha-skill and beta-skill

# 6. Remove test skills
cao skills remove alpha-skill
cao skills remove beta-skill

# 7. Verify cleanup: test skills gone, builtins remain

Test Flow Walkthrough

Setup: Two built-in skills (cao-worker-protocols, cao-supervisor-protocols) installed. Previous agent configs cleaned.

Step 1 — Install developer agent for all three baked providers via cao install developer --provider <provider>.

Step 2–4 — Assert agent config shapes for each provider after install.

Step 5 — Add a test skill (e2e-test-skill) and verify it propagates to Q and Copilot (baked prompt refresh), while Kiro is unaffected (uses glob).

Step 6 — Remove the test skill and verify it disappears from Q and Copilot, while builtin skills and profile content are preserved.

Step 7 — Non-CAO agent protection. Create an external Copilot agent with no matching context file, run a skill add/remove cycle, verify it's untouched.

Step 8 — Refresh count. Verify "Refreshed 2 installed agent(s)" (Q + Copilot only, Kiro excluded).

Step 9 — Kiro glob captures multiple skills. Add two test skills (alpha-skill, beta-skill), verify the skill:// glob resolves to 4 SKILL.md files, verify all have valid frontmatter.

Kiro CLI — Native skill:// Resources

Assertion	Status
JSON keys: allowedTools, description, mcpServers, name, resources, tools, useLegacyMcpJson (no prompt)	✅
resources contains 1 file:// URI pointing to agent-context/developer.md	✅
resources contains 1 skill:// URI ending in /**/SKILL.md	✅
prompt field correctly omitted (profile.prompt is None)	✅
After cao skills add: no prompt field, glob unchanged	✅
After cao skills remove: no prompt field, unchanged	✅
Not included in refresh count	✅

Kiro CLI — Glob Captures Multiple Skills

Assertion	Status
Glob skill://<SKILLS_DIR>/**/SKILL.md resolves to 4 matches after adding 2 test skills	✅
alpha-skill/SKILL.md captured by glob	✅
beta-skill/SKILL.md captured by glob	✅
cao-worker-protocols/SKILL.md captured by glob	✅
cao-supervisor-protocols/SKILL.md captured by glob	✅
All matched SKILL.md files have valid frontmatter (name, description) and body content	✅

Q CLI — Baked Prompt

Assertion	Status
JSON keys include prompt	✅
resources contains file:// only, no skill://	✅
prompt starts with ## Available Skills (catalog-only)	✅
prompt contains cao-worker-protocols and cao-supervisor-protocols	✅
prompt contains get_skill MCP tool instruction	✅
After cao skills add: e2e-test-skill appears in prompt	✅
After cao skills remove: e2e-test-skill gone, cao-worker-protocols remains	✅

Copilot CLI — Baked .agent.md Body

Assertion	Status
Frontmatter keys: description, name	✅
Body starts with # DEVELOPER AGENT (profile system_prompt)	✅
Body contains ## Available Skills after profile content	✅
Body contains cao-worker-protocols and cao-supervisor-protocols	✅
Body contains get_skill MCP tool instruction	✅
Profile body appears before skill catalog (correct ordering)	✅
After cao skills add: e2e-test-skill appears, DEVELOPER AGENT preserved	✅
After cao skills remove: e2e-test-skill gone, profile + builtins preserved	✅

Non-CAO Agent Protection

Assertion	Status
External Copilot agent body unchanged after skill add/remove cycle	✅

Refresh Count

Assertion	Status
cao skills add refreshes 2 agents (Q + Copilot, not Kiro)	✅
cao skills remove refreshes 2 agents (Q + Copilot, not Kiro)	✅

Summary

34/34 assertions passed. Kiro uses native skill:// glob resources — verified that the glob correctly captures multiple dynamically-added skills without any agent config refresh. Q and Copilot receive baked prompt catalogs that refresh on skill changes. Non-CAO-managed agents are protected from modification.

[patricka3125]

hi @fanhongy , I have made numerous changes to ensure all CLI providers that do not accept prompts as a CLI flag (namely Kiro, Q, Copilot CLI) will still have access to CAO skills. Please take a look at the updated PR description for details. I have also included a user guide docs/skills.md in the PR. Thanks for reviewing!

[fanhongy]

LGTM