-
Notifications
You must be signed in to change notification settings - Fork 6.1k
Open
Description
Problem
Firecrawl's self-hosted deployments need proxy support for reliable scraping, but:
- No built-in proxy provider configuration for self-hosted users
PROXY_SERVERandPROXY_USERNAME/PROXY_PASSWORDenv vars exist but lack provider examples- Mobile carrier proxies (real 4G/5G IPs) provide the highest success rate but are undocumented
- Self-hosted users must figure out proxy integration on their own
Proposed Solution
1. Simple env configuration (works today):
# .env for self-hosted Firecrawl
PROXY_SERVER=http://gw.proxies.sx:7000
PROXY_USERNAME=psx_USER-mbl-us
PROXY_PASSWORD=yourpassword2. Country-targeted proxies:
# US mobile proxy
PROXY_USERNAME=psx_USER-mbl-us
# Germany mobile proxy
PROXY_USERNAME=psx_USER-mbl-de
# UK mobile proxy
PROXY_USERNAME=psx_USER-mbl-gb3. Proxy rotation via username tokens:
# Session sticky (same IP per session)
PROXY_USERNAME=psx_USER-mbl-us-sid-mysession
# Auto-rotate every request
PROXY_USERNAME=psx_USER-mbl-us-rot-1Why Mobile Proxies for Firecrawl
- Real 4G/5G carrier tower IPs — highest trust score of any proxy type
- Sites that block residential proxies cannot block mobile carrier IPs
- 155+ real modems across US, DE, GB, FR, ES, PL
- $4/GB — comparable to premium residential
- Standard HTTP proxy auth — works with existing
PROXY_SERVERenv vars
Additional Context
- This is primarily a documentation contribution for self-hosted users
- Hosted Firecrawl manages its own proxy infrastructure (this is for self-hosted only)
- Proxies.sx — mobile proxy infrastructure for AI agents
- Gateway:
gw.proxies.sx:7000(HTTP) /gw.proxies.sx:7001(SOCKS5) - x402 payment support: Pay-per-request with USDC micropayments
Reactions are currently unavailable
Metadata
Metadata
Assignees
Labels
No labels