Skip to content

feat(scrape): declare headers in scrapeParamsSchema so custom headers reach the API#273

Open
kamalrana22 wants to merge 1 commit into
firecrawl:mainfrom
kamalrana22:feat/scrape-headers-passthrough-67
Open

feat(scrape): declare headers in scrapeParamsSchema so custom headers reach the API#273
kamalrana22 wants to merge 1 commit into
firecrawl:mainfrom
kamalrana22:feat/scrape-headers-passthrough-67

Conversation

@kamalrana22

Copy link
Copy Markdown

Summary

Fixes #67.

Custom HTTP headers (Cookie, Authorization, User-Agent, ...) passed to firecrawl_scrape never reach the Firecrawl API, even though the API fully supports a headers option on /v2/scrape.

Root cause

The scrape handler already spreads all options through transformScrapeParams() into client.scrape() — but headers was never declared in scrapeParamsSchema, and zod object validation silently strips unknown keys. So the parameter is dropped during validation, before the handler ever sees it. (An earlier implementation passed unvalidated args through, which is why the workaround described in #67 used to work and no longer does.)

Changes

  • Declare headers as an optional Record<string, string> in scrapeParamsSchema, with a description so MCP clients know when to use it.
  • No handler changes needed: once the key survives validation, the existing spread already delivers it to the API. Because the same schema backs scrapeOptions in firecrawl_crawl and firecrawl_agent, the passthrough works there too.

Verification

Built and ran the server over stdio against a self-hosted Firecrawl instance, calling firecrawl_scrape with a Cookie header against https://httpbin.org/cookies (echoes received cookies):

  • Before: {cookies: {}} — header silently dropped
  • After: {cookies: {mcpfix: issue67}}, and tools/list now advertises headers in the firecrawl_scrape input schema

Use cases this unlocks: scraping pages behind cookie-based auth, regional consent walls (e.g. Google's EU consent page via its consent cookie), and sites that reject the default headless User-Agent.

… reach the API

The scrape handler already spreads all validated options through
transformScrapeParams into client.scrape(), but headers was never
declared in scrapeParamsSchema, and zod object validation strips
unknown keys. Any Cookie/Authorization/User-Agent header passed by an
MCP client was therefore silently dropped before the API call.

Declaring headers as an optional string record restores the
passthrough for firecrawl_scrape and for every tool that reuses the
schema via scrapeOptions (crawl, agent).

Verified against a self-hosted instance by scraping
https://httpbin.org/cookies with headers {Cookie: mcpfix=issue67}
over stdio: previously echoed {cookies: {}}, now echoes the cookie.

Fixes firecrawl#67
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Custom header support for authentication scenarios

1 participant