fetch skills from cloudflare/skills at build time

[elithrar]

Dynamically fetches skills from cloudflare/skills repo at build time instead of storing them statically.

Replaces #27707's static approach with a prebuild script that:

• Fetches all skills from cloudflare/skills via GitHub API
• Generates index.json from SKILL.md frontmatter
• Caches for 1 hour to reduce API calls
• Falls back to stale cache on fetch failures
• Uses GITHUB_TOKEN when available (CI) for higher rate limits

Security:

• Only allows cloudflare/* repos
• Validates download URLs are raw.githubusercontent.com
• Prevents path traversal in file writes

Config via skills.config.json:

{
  "skills_repo": "cloudflare/skills",
  "skills_path": "skills",
  "branch": "main",
  "cache_control": 3600,
  "output_dir": "public/.well-known/skills"
}

[github-actions]

This pull request requires reviews from CODEOWNERS as it changes files that match the following patterns:

Pattern	Owners
*	@cloudflare/pcx-technical-writing
package.json	@cloudflare/content-engineering
*.ts	@cloudflare/content-engineering, @kodster28

[elithrar]

Tested locally:

~/repos/cloudflare-docs (skills-at-build-time ✓)
ins ➜ curl -s http://localhost:1111/.well-known/skills/index.json

{"skills":[{"name":"agents-sdk","description":"Build AI agents on Cloudflare Workers using the Agents SDK. Load when creating stateful agents, durable workflows, real-time WebSocket apps, scheduled tasks, MCP servers, or chat applications. Covers Agent class, state management, callable RPC, Workflows integration, and React hooks.","files":["SKILL.md","references/callable.md","references/codemode.md","references/email.md","references/mcp.md","references/state-scheduling.md","references/streaming-chat.md","references/workflows.md"]},{"name":"building-ai-agent-on-cloudflare","description":"Builds AI agents on Cloudflare using the Agents SDK with state management, real-time WebSockets, scheduled tasks, tool integration, and chat capabilities. Generates production-ready agent code deployed to Workers. Use when: user wants to \"build an agent\", \"AI agent\", \"chat agent\", \"stateful agent\", mentions \"Agents SDK\", needs \"real-
time AI\", \"WebSocket AI\", or asks about agent \"state management\", \"scheduled tasks\", or \"tool calling\".","files":["SKILL.md","references/agent-patterns.md","references/examples.md","references/state-patterns.md","references/troubleshooting.md"]},{"name":"building-mcp-server-on-cloudflare","description":"Builds remote MCP (Model Context Protocol) servers on Cloudflare Workers with tools, OAuth authentication, and production deployment. Generates server code, configures auth providers, and deploys to Workers. Use when: user wants to \"build MCP server\", \"create MCP tools\", \"remote MCP\", \"deploy MCP\", add \"OAuth to MCP\", or mentions Model Context Protocol on Cloudflare. Also triggers on \"MCP authentication\" or \"MCP deployment\".","files":["SKILL.md","references/examples.md","references/oauth-setup.md","references/troubleshooting.md"]}]}%
~/repos/cloudflare-docs (skills-at-build-time ✓)
ins ➜ curl -s http://localhost:1111/.well-known/skills/index.json
{"skills":[{"name":"agents-sdk","description":"Build AI agents on Cloudflare Workers using the Agents SDK. Load when creating stateful agents, durable workflows, real-time WebSocket apps, scheduled tasks, MCP servers, or chat applications. Covers Agent class, state management, callable RPC, Workflows integration, and React hooks.","files":["SKILL.md","references/callable.md","references/codemode.md","references/email.md","references/mcp.md","references/state-scheduling.md","references/streaming-chat.md","references/workflows.md"]},{"name":"building-ai-agent-on-cloudflare","description":"Builds AI agents on Cloudflare using the Agents SDK with state management, real-time WebSockets, scheduled tasks, tool integration, and chat capabilities. Generates production-ready agent code deployed to Workers. Use when: user wants to \"build an agent\", \"AI agent\", \"chat agent\", \"stateful agent\", mentions \"Agents SDK\", needs \"real-time AI\", \"WebSocket AI\", or asks about agent \"state management\", \"scheduled tasks\", or \"tool calling\".","files":["SKILL.md","references/agent-patterns.md","references/examples.md","references/state-patterns.md","references/troubleshooting.md"]},{"name":"building-mcp-server-on-cloudflare","description":"Builds remote MCP (Model Context Protocol) servers on Cloudflare Workers with tools, OAuth authentication, and production deployment. Generates server code, configures auth providers, and deploys to Workers. Use when: user wants to \"build MCP server\", \"create MCP tools\", \"remote MCP\", \"deploy MCP\", add \"OAuth to MCP\", or mentions Model Context Protocol on Cloudflare. Also triggers on \"MCP authentication\" or \"MCP deployment\".","files":["SKILL.md","references/examples.md","references/oauth-setup.md","references/troubleshooting.md"]}]}%

[github-actions]

Preview URL: https://04d4f005.preview.developers.cloudflare.com
Preview Branch URL: https://skills-at-build-time.preview.developers.cloudflare.com

[kodster28]

This needs to be re-factored to avoid any dependency on the GitHub API.

We don't want to tie our build process into anything GitHub rate limits.

a) we had a version of this previously with the Wrangler release notes and it caused the build to fail sporadically. I don't want to re-introduce these kinds of dependencies.
b) impairs contributor experience. If I naively come to this repo the first time and try and build the repo, I'll get an error.

Couldn't you load local files and get the same experience here? Or -- at a minimum -- point this at the same cache built into GitHubCode.

[elithrar]

Maybe? We're still needing to fetch this though - how does https://gh-code.developers.cloudflare.com/${repo}/${commit}/${file} work?

[elithrar]

See updates :-) @kodster28

[elithrar]

Switched to gh-code proxy for file fetches:

• 2 GitHub API calls total (commit SHA + tree listing) instead of 300+
• All file contents fetched via gh-code.developers.cloudflare.com - no rate limits
• Removed GITHUB_TOKEN support (not needed with only 2 API calls)
• Fails open locally (skips skills if fetch fails, warns)
• Fails closed in CI (no cache = build failure)

[kodster28]

We'll be addressing this in a different way, more deets internally