skill-scanner

Problem

AI skills and instruction bundles are a new software supply-chain and identity attack surface. Developers and agents often treat these files as trusted setup or execution guidance, even though malicious or prompt-injected skills can steal secrets, exfiltrate data, pull remote payloads, and deliver malware through shell commands or other risky behavior.

What it does

It discovers AI skills and instruction files, analyzes them for any hidden and malicious behavior, and produces human-readable and machine-readable results for local review or CI gating.

Demo

Analysis

skill-scanner combines:

• LLM analysis
• VirusTotal analysis

Use cases

Use it to:

• review that trust surface before execution
• surface malicious instructions before a developer runs them
• validate local workflows or CI with machine-readable security results

Why it matters

• Malicious skills can steal secrets, exfiltrate data, pull remote payloads, and deliver malware.
• From magic to malware: How OpenClaw's agent skills become an attack surface

Architecture flow

Requirements

• Python 3.11+
• uv
• LiteLLM-compatible model configuration and/or VirusTotal API key (at least one analyzer)

Install (from PyPI)

uv pip install skill-scanner

Base install includes LLM analysis and VirusTotal support.

Install (from source)

uv sync --group dev

Run with:

uv run skill-scanner --help

Alias:

uv run skillscan --help

What gets scanned

By default, discover and scan detect common AI skill, instruction, prompt, agent, rule, and command artifacts such as SKILL.md, AGENTS.md, CLAUDE.md, GEMINI.md, copilot-instructions.md, *.instructions.md, *.prompt.md, *.agent.md, .claude/commands/*.md, .mdc, and .cursorrules. It also inspects installed VS Code extension package.json manifests to find contributed chat skills and then discovers the referenced SKILL.md files.

Default discovered locations follow the documented conventions for:

• Codex: AGENTS.md, .agents/skills/, and .codex/skills/
• Claude Code: CLAUDE.md, .claude/skills/, .claude/agents/, .claude/commands/, and Claude marketplace skill directories
• GitHub Copilot: .github/copilot-instructions.md, .github/instructions/, .github/skills/, and repo-local agent files
• VS Code AI customization: AGENTS.md, .github/prompts/, .github/agents/, and prompt/instruction files used by editor-integrated chat flows
• VS Code extensions: installed extension manifests are inspected for contributed chat skills, and referenced SKILL.md files are discovered as extension-scope targets
• Cursor: .cursor/rules/ (.mdc), .cursorrules, and .cursor/skills/
• Windsurf: .windsurf/skills/ and ~/.codeium/windsurf/skills/
• Gemini CLI: GEMINI.md, .gemini/skills/, .gemini/agents/, and .gemini/extensions/
• Cline: .cline/skills/ and .clinerules/
• OpenCode: .opencode/skills/ and .opencode/agents/

The scanner also supports repo, user, system, and extension variants of these locations where they are implemented in the tool or documented by the vendor.

Use --path to target a specific file or folder. --path discovery is deterministic and only emits files that match known discovery roots/patterns (plus a direct SKILL.md at the provided path root). It does not treat arbitrary *.md files as targets.

Default discover behavior:

• discover attempts all scopes (repo, user, system, extension).
• repo scope is only active when your current directory is inside a git repository.
• Filesystem traversal errors are non-fatal; discovery returns partial results. Use --verbose to inspect warnings.

Quick start

# See targets
uv run skill-scanner discover --format json

# Discover only user scope
uv run skill-scanner discover --scope user

# Show detailed discovery warnings
uv run skill-scanner discover --verbose

# Verify key/model configuration
uv run skill-scanner doctor

# Run live API checks (fails non-zero if checks fail)
uv run skill-scanner doctor --check

# Run a scan (with whichever analyzers are configured)
uv run skill-scanner scan --format summary

Configuration

scan requires at least one analyzer enabled.

• There is no built-in model default. Set SKILLSCAN_MODEL explicitly or pass --model.
• If SKILLSCAN_MODEL plus either SKILLSCAN_API_KEY or SKILLSCAN_BASE_URL is available, LLM analysis runs.
• If only VT_API_KEY is available, VT runs and LLM analysis is disabled.
• If both LLM config and VT_API_KEY are available, both analyzers run and VT context is passed into the LLM analysis.
• You can disable either analyzer with --no-ai or --no-vt.
• LLM analysis requires an explicit SKILLSCAN_MODEL or --model value.
• When --fail-on is set, analyzer failures surfaced in notes also cause a non-zero exit to avoid false-clean scans.

Use doctor --check to verify model/API/base URL connectivity. Use models.litellm.ai to choose a supported LiteLLM model string.

Supported configuration paths:

1. Environment variables:

export SKILLSCAN_MODEL=openai/gpt-5.4
export SKILLSCAN_API_KEY=your-llm-key
export VT_API_KEY=your-vt-key
uv run skill-scanner scan --format summary

2. Local or user config file:

• project-local: ./skill-scanner.toml
• user-level: ~/.config/skill-scanner/config.toml

model = "openai/gpt-5.4"
api_key = "your-llm-key"
vt_api_key = "your-vt-key"

Local model config example:

model = "ollama/llama3.1"
base_url = "http://localhost:11434"

3. CLI overrides for a single run:

uv run skill-scanner scan \
  --model openai/gpt-5.4 \
  --api-key "$SKILLSCAN_API_KEY" \
  --format summary

Hosted model env example:

SKILLSCAN_MODEL=openai/gpt-5.4
SKILLSCAN_API_KEY=op://Developer/LLM/api_key
VT_API_KEY=op://Developer/VirusTotal/api_key

Local model example:

SKILLSCAN_MODEL=ollama/llama3.1
SKILLSCAN_BASE_URL=http://localhost:11434

1Password setup

Recommended approach: keep secrets out of skill-scanner.toml and store them as 1Password secret references in .env.

Example .env:

SKILLSCAN_MODEL=openai/gpt-5.4
SKILLSCAN_API_KEY=op://Developer/LLM/api_key
VT_API_KEY=op://Developer/VirusTotal/api_key

Run the scanner through 1Password CLI so those references are resolved at runtime:

op run --env-file=.env -- uv run skill-scanner scan --format summary

To verify configuration before scanning:

op run --env-file=.env -- uv run skill-scanner doctor --check

If you prefer shell exports instead of an env file:

export SKILLSCAN_MODEL=openai/gpt-5.4
export SKILLSCAN_API_KEY="$(op read 'op://Developer/LLM/api_key')"
export VT_API_KEY="$(op read 'op://Developer/VirusTotal/api_key')"
uv run skill-scanner scan --format summary

Important:

• skill-scanner does not auto-load .env files on its own.
• op://... references are resolved when you use op run or op read.
• If you want to keep secrets in a config template, render it first with op inject; do not commit the rendered file.

Security best practice:

• Prefer a 1Password Service Account scoped to only the vault/items required for scanning (least privilege).

References:

• https://developer.1password.com/docs/cli/secret-references/
• https://developer.1password.com/docs/cli/reference/commands/run/
• https://developer.1password.com/docs/service-accounts/

Output formats

scan --format supports:

• table (default)
• summary
• json
• sarif

You can write output to a file with --output <path>. For json, sarif, and summary, the written file matches the selected format. For table, the terminal still shows the Rich table and --output writes JSON. Interactive table and summary scans show a live Rich progress bar while targets are being analyzed.

Risk output is severity-classified, not numerically scored. The final risk_level is the strongest surviving signal from llm and vt. Structured output exposes llm_findings, vt_findings, llm_risk_level, vt_risk_level, risk_level, vt_report, and notes.

Useful commands

# List providers
uv run skill-scanner providers

# Scan one path only
uv run skill-scanner scan --path ./some/skill/folder --format summary

# Increase scan concurrency (default: 8)
uv run skill-scanner scan --jobs 16 --format summary

# Enable verbose logs for troubleshooting
uv run skill-scanner scan --verbose --format summary

# List discovered targets without running analyzers
uv run skill-scanner scan --list-targets

# Discover targets from user scope only
uv run skill-scanner discover --scope user --format table

# Discover with detailed traversal diagnostics
uv run skill-scanner discover --verbose --format table

# Scan only selected discovered targets (repeat --target)
uv run skill-scanner scan --target /absolute/path/to/SKILL.md --target /absolute/path/to/AGENTS.md --format summary

# Filter to medium+
uv run skill-scanner scan --min-severity medium --format summary

# Non-zero exit if high+ findings exist
uv run skill-scanner scan --fail-on high --format summary

# Verbose doctor checks
uv run skill-scanner doctor --check --verbose

--list-targets can be used without API keys because it only runs discovery and exits.

Discovery troubleshooting (macOS/Windows)

# macOS/Windows: default discover should complete without crashing
uv run skill-scanner discover --format table

# Scoped check: verify user skill paths only
uv run skill-scanner discover --scope user --format table

Windows known-path sanity check:

• create %USERPROFILE%\\.clinerules\\skills\\demo\\SKILL.md
• run uv run skill-scanner discover --platform cline --scope user --format table
• confirm the demo skill appears in output

Exit behavior

• 0: scan completed and fail threshold not hit
• 1: --fail-on threshold matched, or an analyzer failure was recorded while --fail-on was set
• 2: no analyzers enabled (for example missing keys combined with flags), or --target did not match any discovered target

doctor --check exit behavior:

• 0: all executed checks passed
• 1: one or more checks failed

Notes and truncation visibility

scan now surfaces per-target notes in table and summary output, including:

• analyzer failures (LLM or VirusTotal)
• VirusTotal timeout fallback when the latest available file verdict is used
• payload truncation when files are skipped due to the 400k-character LLM payload limit
• unreadable files excluded from payload construction

Contributing

See CONTRIBUTING.md for setup, testing, and PR guidelines.

Roadmap (in progress)

• Improve system prompt hardening to uncover more threat patterns
• Support multiple providers via LiteLLM model strings
• Baseline / suppression / false-positive management
• Ollama / local LLM provider support via LiteLLM base_url
• Configurable risk classification policy
• GitHub Actions template for automated scans
• Improve fixtures to include realistic malicious skills

Version bump workflow

Use uv version so version updates stay command-driven and lock state remains consistent.

# Patch bump (e.g., 0.1.2 -> 0.1.3)
uv version --bump patch

# Or set an explicit version
uv version 0.3.0

Notes:

• pyproject.toml is the canonical version source.
• uv.lock is generated by uv and should not be edited manually.