Add support for externally-hosted DocC websites#29
Conversation
mattt
force-pushed
the
mattt/external
branch
from
08219a7 to
e8714da
Compare
There was a problem hiding this comment.
Pull request overview
This pull request adds support for externally-hosted Swift-DocC documentation sites, resolving issues #28 and #12. It enables users to proxy non-Apple DocC sites (e.g., third-party Swift package documentation) through sosumi.ai using a new /external/ route pattern.
Changes:
- Adds external DocC support with URL pattern
/external/https://<host>/documentation/<path> - Implements comprehensive access controls including robots.txt checking, host allowlist/blocklist, and SSRF protection
- Adds link rewriting in rendered markdown to properly handle external origins
- Includes new MCP tool
fetchExternalDocumentationfor AI clients
Reviewed changes
Copilot reviewed 11 out of 11 changed files in this pull request and generated 4 comments.
Show a summary per file
| File | Description |
|---|---|
| src/lib/external/policy.ts | New module implementing access control policies, robots.txt parsing, and SSRF protection for external hosts |
| src/lib/external/fetch.ts | Handles fetching external DocC JSON data and building proper JSON URLs from documentation paths |
| src/lib/external/index.ts | Exports external module functionality |
| src/lib/reference/render.ts | Adds externalOrigin option and rewrites documentation links to use /external/ prefix |
| src/lib/mcp.ts | Adds fetchExternalDocumentation tool to MCP server |
| src/index.ts | Implements /external/* HTTP route and error handling for ExternalAccessError |
| tests/external.test.ts | Comprehensive tests for external DocC support including policy enforcement and rendering |
| tests/integration.test.ts | Integration test for external DocC fetching |
| tests/url.test.ts | Tests for external URL to JSON URL conversion |
| public/index.html | Adds bot policy section and external documentation examples |
| README.md | Documents external DocC support and access controls |
💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.
There was a problem hiding this comment.
Pull request overview
Copilot reviewed 12 out of 12 changed files in this pull request and generated 5 comments.
💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.
There was a problem hiding this comment.
Pull request overview
Copilot reviewed 13 out of 13 changed files in this pull request and generated 5 comments.
💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.
| export async function fetchExternalDocCJSON(sourceUrl: URL): Promise<AppleDocJSON> { | ||
| const jsonUrl = buildExternalDocCJsonUrl(sourceUrl) | ||
| const response = await fetch(jsonUrl.toString(), { | ||
| headers: { | ||
| "User-Agent": EXTERNAL_DOC_USER_AGENT, |
There was a problem hiding this comment.
fetchExternalDocCJSON is exported but does not enforce validateExternalDocumentationUrl/assertExternalDocumentationAccess itself. This makes it easy for future call sites to fetch arbitrary external URLs without applying the host/robots/X-Robots-Tag policies. Consider either (a) making this function internal (not exported) or (b) moving the policy enforcement into this function so it’s safe-by-default.
Resolves #28
Resolves #12