Skip to content

Commit 18efd9b

Browse files
ceilf6claude
andauthored
feat(mcp-web-fetch): add @frontagent/mcp-web-fetch adapter package (unwired) (#358)
* feat(mcp-web-fetch): add @frontagent/mcp-web-fetch adapter package (unwired) Adds a new workspace package providing an agent web_fetch MCP tool: fetch a URL and return cleaned, readable text (HTML→text) for looking up library/ framework docs, API references, and error resolutions during planning and execution. Mirrors the @frontagent/mcp-filesense adapter shape (zero runtime deps, ported engine, *Schema + handle*Tool dispatch, tools.security.test.ts). Ports the SSRF-hardened fetch logic from the standalone ceilf6/websense repo: url-safety guards (reject private/loopback/link-local/cloud-metadata hosts, DNS-resolution check against rebinding, credential rejection), per-redirect-hop re-validation, AbortController timeout, byte-cap, and a pure HTML→text extractor. This package is NOT yet wired into the agent's MCP client registries; that is a follow-up PR. Part of #357 (incremental delivery, step 2 of 3). Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> * fix(mcp-web-fetch): bind SSRF check to connection + enforce hard limits Addresses repo-guard's two HIGH findings on the adapter: 1. DNS-rebinding (TOCTOU): the previous pre-resolve check did not bind to the actual fetch connection, so a domain could resolve to a public IP at check time and a private/metadata IP at connect time. Route fetch through an undici Agent whose connect.lookup rejects private/loopback/ link-local resolved addresses at connection time, so the validated address IS the connected address. Adds undici dependency (security- justified). Synchronous parseAndValidateUrl pre-check is retained for fast rejection of bad protocol/credentials/literal-private hosts. 2. Caller-amplifiable limits: maxBytes/timeoutMs/maxRedirects were soft defaults a caller could overshoot. Add clampLimit() enforcing positive integers capped at hard ceilings (bytes 5MB, timeout 60s, redirects 10) in the engine, plus minimum/maximum constraints in the tool schema. New tests for clampLimit and isConnectionAddressBlocked. Part of #357. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> * fix(mcp-web-fetch): block IPv4-mapped IPv6 SSRF bypass (hex & expanded) repo-guard HIGH finding: isPrivateIpv6 only caught the dotted form ::ffff:127.0.0.1 and checked just the first hextet, so equivalents like ::ffff:7f00:1 (127.0.0.1), ::ffff:0a00:1 (10.0.0.1), and expanded 0:0:0:0:0:ffff:127.0.0.1 mapped to loopback/private IPv4 but slipped through — an SSRF bypass shared by the pre-check and the connect-time dispatcher. Rewrite isPrivateIpv6 to fully normalize the address (expandIpv6 via node:net isIP, embedded-IPv4-tail handling, :: expansion to 8 hextets), then classify: ::/::1, IPv4-mapped and IPv4-compatible forms delegate to isPrivateIpv4; fc00::/7 and fe80::/10 by mask. Adds regression tests for hex/expanded mapped loopback/private forms and bracketed-host URLs. Part of #357. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> * test(mcp-web-fetch): add runtime SSRF/limit regression tests + dns all:false Addresses repo-guard's remaining non-blocking notes: - Force all:false in the safeDispatcher dns.lookup so the resolved address is always a single string (not an array when undici passes all:true). - Add an internal test-only fetchImpl injection seam to fetchUrl so the runtime security paths can be verified deterministically without network. - New engine.test.ts: redirect hop to 127.0.0.1 is rejected before the second fetch; oversized body is truncated to maxBytes; maxRedirects cap throws; happy-path HTML→text/title/status. 40/40 tests pass. Part of #357. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> * test(mcp-web-fetch): export and directly test the dispatcher safeLookup repo-guard asked to verify the ACTUAL connection-time DNS guard (the function undici's dispatcher calls), not just the pure helper + mocked fetch. Extract the dispatcher's inline lookup into an exported safeLookup (behavior unchanged; matches Node's net.LookupFunction signature) and add engine.lookup.test.ts asserting it rejects localhost/127.0.0.1/::1 (private, resolved locally — no network) and allows 8.8.8.8. This proves the exact lookup wired into the safe undici Agent blocks private resolved addresses. 44/44 tests. Part of #357. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> --------- Co-authored-by: Claude Opus 4.8 <noreply@anthropic.com>
1 parent 6bda08e commit 18efd9b

17 files changed

Lines changed: 1150 additions & 0 deletions

packages/mcp-web-fetch/.gitignore

Lines changed: 2 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,2 @@
1+
node_modules/
2+
dist/
Lines changed: 29 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,29 @@
1+
{
2+
"name": "@frontagent/mcp-web-fetch",
3+
"version": "2.1.1",
4+
"description": "MCP Web-Fetch Adapter - Agent-friendly URL fetching (HTML→text) with SSRF guards for FrontAgent",
5+
"type": "module",
6+
"main": "./dist/index.js",
7+
"types": "./dist/index.d.ts",
8+
"exports": {
9+
".": {
10+
"import": "./dist/index.js",
11+
"types": "./dist/index.d.ts"
12+
}
13+
},
14+
"scripts": {
15+
"build": "tsc",
16+
"dev": "tsc --watch",
17+
"test": "vitest run",
18+
"typecheck": "tsc --noEmit",
19+
"clean": "rm -rf dist"
20+
},
21+
"dependencies": {
22+
"undici": "^6.21.0"
23+
},
24+
"devDependencies": {
25+
"@types/node": "^25.9.1",
26+
"typescript": "^6.0.3",
27+
"vitest": "^4.1.7"
28+
}
29+
}
Lines changed: 49 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,49 @@
1+
import type { LookupAddress } from 'node:dns';
2+
import { describe, expect, it } from 'vitest';
3+
import { safeLookup } from './engine.js';
4+
5+
/**
6+
* These tests exercise `safeLookup` directly — the function undici's
7+
* dispatcher actually calls at connection time. All cases use local-only
8+
* resolution (loopback hostnames and IP literals), so no external network
9+
* access is required:
10+
* - Literal IPs are returned by `dns.lookup` without a network query.
11+
* - 'localhost' is resolved via the hosts file (local).
12+
*/
13+
function runLookup(hostname: string): Promise<{
14+
err: NodeJS.ErrnoException | null;
15+
address: string | LookupAddress[];
16+
family?: number;
17+
}> {
18+
return new Promise((resolve) => {
19+
safeLookup(hostname, {}, (err, address, family) => {
20+
resolve({ err, address, family });
21+
});
22+
});
23+
}
24+
25+
describe('safeLookup', () => {
26+
it('blocks localhost (resolves to a loopback address)', async () => {
27+
const { err } = await runLookup('localhost');
28+
expect(err).not.toBeNull();
29+
expect(err?.message).toMatch(/private|blocked/i);
30+
});
31+
32+
it('blocks the literal IPv4 loopback address 127.0.0.1', async () => {
33+
const { err } = await runLookup('127.0.0.1');
34+
expect(err).not.toBeNull();
35+
expect(err?.message).toMatch(/private|blocked/i);
36+
});
37+
38+
it('blocks the literal IPv6 loopback address ::1', async () => {
39+
const { err } = await runLookup('::1');
40+
expect(err).not.toBeNull();
41+
expect(err?.message).toMatch(/private|blocked/i);
42+
});
43+
44+
it('allows a literal public IPv4 address', async () => {
45+
const { err, address } = await runLookup('8.8.8.8');
46+
expect(err).toBeNull();
47+
expect(address).toBe('8.8.8.8');
48+
});
49+
});
Lines changed: 74 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,74 @@
1+
import { describe, expect, it, vi } from 'vitest';
2+
import { fetchUrl } from './engine.js';
3+
4+
describe('fetchUrl', () => {
5+
it('rejects a redirect to a private address without following it', async () => {
6+
const mock = vi.fn(
7+
async () =>
8+
new Response(null, {
9+
status: 302,
10+
headers: { location: 'http://127.0.0.1/secret' },
11+
}),
12+
);
13+
14+
await expect(
15+
fetchUrl('https://example.com/', { fetchImpl: mock as unknown as typeof fetch }),
16+
).rejects.toThrow();
17+
18+
expect(mock).toHaveBeenCalledTimes(1);
19+
});
20+
21+
it('truncates the body when it exceeds maxBytes', async () => {
22+
const largeBody = 'x'.repeat(1000);
23+
const mock = vi.fn(
24+
async () =>
25+
new Response(largeBody, {
26+
status: 200,
27+
headers: { 'content-type': 'text/plain' },
28+
}),
29+
);
30+
31+
const result = await fetchUrl('https://example.com/', {
32+
fetchImpl: mock as unknown as typeof fetch,
33+
maxBytes: 100,
34+
});
35+
36+
expect(result.truncated).toBe(true);
37+
expect(result.bytes).toBeLessThanOrEqual(100);
38+
});
39+
40+
it('throws once the redirect count exceeds maxRedirects', async () => {
41+
const mock = vi.fn(
42+
async () =>
43+
new Response(null, {
44+
status: 302,
45+
headers: { location: 'http://example.org/next' },
46+
}),
47+
);
48+
49+
await expect(
50+
fetchUrl('https://example.com/', {
51+
fetchImpl: mock as unknown as typeof fetch,
52+
maxRedirects: 2,
53+
}),
54+
).rejects.toThrow();
55+
});
56+
57+
it('returns text and title for a successful HTML response', async () => {
58+
const mock = vi.fn(
59+
async () =>
60+
new Response('<title>Hi</title><p>Hello</p>', {
61+
status: 200,
62+
headers: { 'content-type': 'text/html' },
63+
}),
64+
);
65+
66+
const result = await fetchUrl('https://example.com/', {
67+
fetchImpl: mock as unknown as typeof fetch,
68+
});
69+
70+
expect(result.status).toBe(200);
71+
expect(result.title).toBe('Hi');
72+
expect(result.text).toContain('Hello');
73+
});
74+
});

0 commit comments

Comments
 (0)