I am attempting to use the firecrawl MCP server to scrape a website that requires authentication.
On the firecrawl API, one can specify custom headers like so:
curl -X POST https://api.firecrawl.dev/v1/scrape \
-H "Authorization: Bearer YOUR_FIRECRAWL_KEY" \
-H "Content-Type: application/json" \
-d '{
"url": "https://protected.example.com",
"headers": {
"X-Custom-Auth": "token123"
}
}'
However, headers does not seem to be available as a valid parameter on the input schema for the firecrawl_scrape tool, or any other tool, in the MCP server.
I'm almost surprised that this is the case. Is there a work-around or is this on the roadmap to add support for custom headers to scrape requests?
Thank you!
Update: I've found that passing the headers object actually works as expected for the scrape command because the implementation spreads all parameters to the client. So, the header just passes through and things work.
const firecrawlTransport = new StdioClientTransport({
command: "npx",
args: ["-y", "firecrawl-mcp"],
env: {
FIRECRAWL_API_KEY: process.env.FIRECRAWL_API_KEY!,
PATH: process.env.PATH || "/usr/local/bin:/usr/bin:/bin",
},
});
const firecrawlClient = new Client(
{
name: "firecrawl-client",
version: "1.0.0",
},
{
capabilities: {},
}
);
await firecrawlClient.connect(firecrawlTransport);
const result = await firecrawlClient.request(
{
method: "tools/call",
params: {
name: "firecrawl_scrape",
arguments: {
url: `https://example.com`,
formats: ["html"],
onlyMainContent: true,
// Pass the cookie in the headers
headers: {
Authorization: "Bearer <token>",
},
},
},
},
CallToolResultSchema
);However, this does not work consistently, like for example the extract tool explicitly re-maps the available options and so the headers option is lost.
I am attempting to use the firecrawl MCP server to scrape a website that requires authentication.
On the firecrawl API, one can specify custom headers like so:
However,
headersdoes not seem to be available as a valid parameter on the input schema for thefirecrawl_scrapetool, or any other tool, in the MCP server.I'm almost surprised that this is the case. Is there a work-around or is this on the roadmap to add support for custom headers to scrape requests?
Thank you!
Update: I've found that passing the headers object actually works as expected for the scrape command because the implementation spreads all parameters to the client. So, the header just passes through and things work.
However, this does not work consistently, like for example the extract tool explicitly re-maps the available options and so the headers option is lost.