feat(scrape): declare headers in scrapeParamsSchema so custom headers reach the API#273
Open
kamalrana22 wants to merge 1 commit into
Open
Conversation
… reach the API The scrape handler already spreads all validated options through transformScrapeParams into client.scrape(), but headers was never declared in scrapeParamsSchema, and zod object validation strips unknown keys. Any Cookie/Authorization/User-Agent header passed by an MCP client was therefore silently dropped before the API call. Declaring headers as an optional string record restores the passthrough for firecrawl_scrape and for every tool that reuses the schema via scrapeOptions (crawl, agent). Verified against a self-hosted instance by scraping https://httpbin.org/cookies with headers {Cookie: mcpfix=issue67} over stdio: previously echoed {cookies: {}}, now echoes the cookie. Fixes firecrawl#67
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
Fixes #67.
Custom HTTP headers (
Cookie,Authorization,User-Agent, ...) passed tofirecrawl_scrapenever reach the Firecrawl API, even though the API fully supports aheadersoption on/v2/scrape.Root cause
The scrape handler already spreads all options through
transformScrapeParams()intoclient.scrape()— butheaderswas never declared inscrapeParamsSchema, and zod object validation silently strips unknown keys. So the parameter is dropped during validation, before the handler ever sees it. (An earlier implementation passed unvalidated args through, which is why the workaround described in #67 used to work and no longer does.)Changes
headersas an optionalRecord<string, string>inscrapeParamsSchema, with a description so MCP clients know when to use it.scrapeOptionsinfirecrawl_crawlandfirecrawl_agent, the passthrough works there too.Verification
Built and ran the server over stdio against a self-hosted Firecrawl instance, calling
firecrawl_scrapewith a Cookie header against https://httpbin.org/cookies (echoes received cookies):{cookies: {}}— header silently dropped{cookies: {mcpfix: issue67}}, andtools/listnow advertisesheadersin thefirecrawl_scrapeinput schemaUse cases this unlocks: scraping pages behind cookie-based auth, regional consent walls (e.g. Google's EU consent page via its consent cookie), and sites that reject the default headless User-Agent.