TRAWL

Welcome to TRAWL! 👋

Self-hosted web scraping engine with best-effort JS challenge and CAPTCHA solving.
Dedicated flows for Cloudflare, Akamai Bot Manager, and Imperva/Incapsula (best effort), plus Turnstile, reCAPTCHA, hCaptcha, and GeeTest.
Much faster and more reliable FlareSolverr & Byparr alternative and drop-in replacement for your *arr stack.

Features

2-6x faster - compared to FlareSolverr or Byparr it returns much faster with higher success rate
4-tier execution - plain HTTP fetch → cached browser session → fresh challenge solve → residential proxy
Challenge-aware HTTP/HTTPS proxy - direct forwarding for normal traffic, automatic tier escalation for detected walls, plus WebSockets, binary bodies, and Range/206 support
Multi-WAF handling - dedicated Cloudflare, Akamai Bot Manager, and Imperva/Incapsula detection and browser flows
Native captcha solving - CF Turnstile/Interstitial, reCAPTCHA v2 (free STT), hCaptcha, GeeTest v4 Slide
Camoufox Firefox - fingerprint-patched at the C++/Juggler level to reduce automation signals
Session cache - solved cookies and browser identity stored in Redis; accepted sessions can avoid a fresh solve
FlareSolverr compatible - works with Prowlarr, Jackett, Sonarr, and the full *arr ecosystem out of the box
No paid solver API required - reCAPTCHA audio can use Google's free STT endpoint or an optional local Whisper service

Quick start

# Clone and configure
git clone https://github.com/germondai/trawl
cd trawl
cp .env.example .env

# Start scraper + Redis
docker compose up -d

# Verify
curl http://localhost:8191/health

First boot takes 15–30s while the browser pool warms up. Subsequent starts are fast.

NAS app catalogs

Prefer a one-click installation? TRAWL is available from the community app catalogs for both TrueNAS and Unraid:

TrueNAS Community Apps — open Apps → Discover Apps and search for TRAWL.
Unraid Community Apps — open the Apps tab and search for Trawl.

Thanks to the TrueNAS and Unraid community contributors who packaged and published these integrations.

API

FlareSolverr-compatible (`/v1`)

curl -X POST http://localhost:8191/v1 \
  -H 'Content-Type: application/json' \
  -d '{"cmd":"request.get","url":"https://nowsecure.nl","maxTimeout":60000}'

Native API (`/scrape`)

Returns richer metadata: tier, timings, sessionCached, full cookie list.

curl -X POST http://localhost:8191/scrape \
  -H 'Content-Type: application/json' \
  -d '{"url":"https://nowsecure.nl","maxTimeout":60000}'

Connect Prowlarr / Jackett

Set the FlareSolverr URL to:

http://localhost:8191        # running on the same host
http://trawl:8191            # running via Docker Compose on the same network

Challenge-bypassing HTTP/HTTPS proxy

Some sites bind their Cloudflare clearance to the solving browser's full connection fingerprint. The /v1 flow can't help there: Prowlarr keeps only the cookie + user-agent and re-fetches the page with its own HTTP client, which Cloudflare re-challenges — the cookie isn't portable. For those indexers, enable TRAWL's forward proxy and add it to Prowlarr as an HTTP proxy:

MITM_PROXY_ENABLED=true
MITM_PROXY_PORT=8192
MITM_PROXY_CA_DIR=/data/proxy-ca   # persist the CA (mount a volume)
MITM_PROXY_MAX_TIER=4              # cap escalation (e.g. 3 to stay off residential)

By default the listener binds 0.0.0.0 so clients on a Docker bridge network can reach it; set MITM_PROXY_HOST=127.0.0.1 to restrict it to loopback on a bare-metal host.

Install the proxy's CA into the client's trust store so it accepts the per-host certs: curl http://<trawl-host>:8191/proxy-ca.crt → add to the Prowlarr container's CA store (e.g. a linuxserver /custom-cont-init.d script that copies it to /usr/local/share/ca-certificates/ and runs update-ca-certificates).
Prowlarr → Settings → Indexer Proxies → HTTP, host <trawl-host>, port 8192. Give it a tag if only selected indexers should use it.

Ordinary requests use a direct HTTP/TLS path. Small HTML, JSON, and text responses are buffered for challenge detection; detected challenges escalate through the same tier pipeline as POST /scrape. Videos and large binary responses stream directly. Range requests are forwarded end to end and can escalate when their response is a detected challenge; WebSocket upgrades use a direct relay without browser escalation.

See the complete proxy documentation for routing details, supported traffic, limitations, CA installation, and client examples.

⚠️ A MITM proxy can impersonate any host to a client that trusts its CA. Only expose it on a private interface (localhost / a private Docker network), never publicly.

Installing the proxy CA certificate

The proxy self-generates a root CA on first run. Its certificate and private key are persisted under MITM_PROXY_CA_DIR (default /data/proxy-ca). Per-host certificates are minted and cached in memory while TRAWL runs; they do not need separate installation because they are signed by the persistent root. Every client that uses the proxy must trust that root. Without it, HTTPS fails with ERR_CERT_AUTHORITY_INVALID (browsers) or PKIX path building failed (Java).

Download the CA once per client:

curl http://<trawl-host>:8191/proxy-ca.crt -o trawl-ca.crt
# or in a Docker setup where the API isn't reachable from outside:
docker cp trawl:/data/proxy-ca/ca.crt ./trawl-ca.crt

macOS (system keychain — affects most apps including Safari, curl, wget)

sudo security add-trusted-cert -d -r trustRoot \
  -k /Library/Keychains/System.keychain ./trawl-ca.crt
# Verify
security find-certificate -c "TRAWL MITM Proxy CA"
# Remove later
sudo security delete-certificate -c "TRAWL MITM Proxy CA" \
  /Library/Keychains/System.keychain

Linux (Debian/Ubuntu — system-wide for curl, wget, apt, etc.)

sudo cp trawl-ca.crt /usr/local/share/ca-certificates/trawl-ca.crt
sudo update-ca-certificates
# Verify
awk '/BEGIN/{c++} c==2' /etc/ssl/certs/ca-certificates.crt | grep -c "TRAWL MITM"

Linux (RHEL/Fedora/Amazon)

sudo cp trawl-ca.crt /etc/pki/ca-trust/source/anchors/trawl-ca.crt
sudo update-ca-trust

Firefox and NSS trust stores

Firefox installations that do not use operating-system roots need a per-profile NSS import:

# Firefox 115+ uses a file-backed NSS DB; older versions use the legacy libnssdb format.
# The certutil command is the same either way.
certutil -A -n "TRAWL MITM" -t "CT,C,C" -i trawl-ca.crt \
  -d sql:$HOME/.mozilla/firefox/<profile-dir>
# Or via Firefox UI: Settings → Privacy & Security → Certificates → View Certificates →
# Authorities → Import… → check "Trust this CA to identify websites".
# Profile dir location: about:profiles in Firefox.

Chrome / Chromium (Linux: separate from system trust)

Chrome uses the system trust store on macOS and Windows but has its own on Linux:

# Option A: launch Chrome with --user-data-dir + NSS DB update (same as Firefox).
# Option B: use Chrome's --ignore-certificate-errors-spki-list=<hash> (per-session, less safe).
# Option C: add the cert to the system store (above) — Chrome picks it up automatically on
# most Linux distros via the nss-tool lookup.

Java (including JDownloader)

# Find the JRE cacerts file for your client.
#   JDownloader:    <install>/jre/lib/security/cacerts
keytool -importcert -alias trawl -file trawl-ca.crt \
  -keystore "<path-to-cacerts>" -storepass changeit
# If `keytool` reports "Certificate already exists in keystore", use -delete first:
#   keytool -delete -alias trawl -keystore "<path-to-cacerts>" -storepass changeit

Prowlarr, Sonarr, and Radarr are .NET applications, not Java applications. For their Docker-based installations, add the CA to the container's Linux system trust store. A common LinuxServer pattern is a /custom-cont-init.d script:

# In the client's Compose service:
volumes:
  - ./trawl-ca.crt:/config/trawl-ca.crt:ro
  - ./install-trawl-ca.sh:/custom-cont-init.d/50-install-trawl-ca:ro

#!/usr/bin/with-contenv bash
cp /config/trawl-ca.crt /usr/local/share/ca-certificates/trawl-ca.crt
update-ca-certificates

LinuxServer runs scripts in /custom-cont-init.d/ when the container starts. Java clients such as JDownloader require the separate keytool import described above.

JDownloader 2 (Windows / macOS / Linux — manual install)

JDownloader bundles its own JRE; the CA must be imported into it.

Find the JRE: Settings → Advanced → Java Path (in JDownloader) or look in the install dir:
- Windows: C:\Program Files\JDownloader 2\jre\lib\security\cacerts
- macOS: /Applications/JDownloader 2.app/Contents/app/jre/lib/security/cacerts
- Linux: <install>/jre/lib/security/cacerts
Run the keytool -importcert command above against that file.
Restart JDownloader.

Windows (system trust store)

# Run PowerShell as Administrator.
Import-Certificate -FilePath .\trawl-ca.crt `
  -CertStoreLocation Cert:\LocalMachine\Root
# Remove later
Get-ChildItem Cert:\LocalMachine\Root | Where-Object { $_.Subject -like "*TRAWL MITM*" } | Remove-Item

Removing the CA (cleanup)

Every installation method has a symmetric removal path. Search your trust store for TRAWL MITM Proxy CA (the CA's CN) and delete that entry. The CA certificate and key also live at <MITM_PROXY_CA_DIR>/ca.crt and ca.key on the TRAWL host. Deleting either causes TRAWL to generate a new root on its next start, so existing clients must install the new certificate.

Tiers

Request
  │
  ▼
Tier 1: Plain HTTP fetch ────── success ──→ return (< 100ms)
  │ blocked
  ▼
Tier 2: Cached session ─────── success ──→ return (~500ms)
  │ cache miss / expired
  ▼
Tier 3: Fresh challenge solve ─ success ──→ cache + return
  │ IP flagged
  ▼
Tier 4: Residential proxy ──── success ──→ cache + return (15–45s)
  │ failed
  ▼
  error

Docker Compose files

File	Description
`docker-compose.yml`	Scraper + Redis (default)
`docker-compose.minimal.yml`	Scraper only, no Redis
`docker-compose.prod.yml`	Production: `restart: always`, memory limit, healthcheck

Docker images (one GHCR package, two tags)

Image tag	Built from	Runtime	Use case
`ghcr.io/germondai/trawl:latest`	`apps/api/Dockerfile`	Bun 1.3.14 (modern, AVX2)	Default — modern Linux amd64/arm64
`ghcr.io/germondai/trawl:baseline`	`apps/api/Dockerfile.baseline`	Bun 1.3.14 baseline (no AVX2)	Older CPUs / older kernels (Synology NAS, J4125, Atom-era)

Both tags live on the same ghcr.io/germondai/trawl package — they share the registry but use different Dockerfile sources. Pick whichever tag fits your hardware:

# Modern hardware (most users)
image: ghcr.io/germondai/trawl:latest

# Older CPUs without AVX2 / Synology / older kernels
image: ghcr.io/germondai/trawl:baseline

Synology note: many Synology NAS units (DSM 7.x on J4125 / older hardware) ship kernel 4.4.x, which Bun's modern runtime can't fully handle. Standard Bun requires kernel 5.1+ (5.6+ recommended); the baseline build degrades gracefully down to kernel 3.10. The :baseline tag is published for that case — confirmed working on a Synology DS920+ (Celeron J4125, DSM 7.3.2, kernel 4.4.302): the container starts cleanly, /health reports healthy, and it solves live Cloudflare challenges via /v1 (see #1). Published by independent GitHub Actions workflows: pushing v1.4.0 creates :1.4.0, :latest, :1.4.0-baseline, and :baseline; pushing main creates :nightly and :nightly-<sha>.

Releases & versioning

TRAWL follows Semantic Versioning. Pushing a v-prefixed git tag (e.g. v1.0.0) triggers publish.yml, which builds and pushes the matching un-prefixed Docker tag (ghcr.io/germondai/trawl:1.0.0) alongside a major-only tag (:1). :latest always tracks the tip of main; :sha-<shortsha> images are pushed on every main commit regardless of tags. See the Releases page for the full version history and CHANGELOG.md for what changed in each one.

To publish a specific past commit that had a major fix without waiting for the next tip-of-main release, tag that exact SHA and push it — publish.yml builds off the tag ref, not off main's current tip:

git tag -a v1.0.1 <sha> -m "..."
git push origin v1.0.1

Configuration

TRAWL supports HTTP proxies, authenticated HTTP proxies, and SOCKS5 proxies. The standard Compose files read proxy settings from the local .env file:

# Optional Tier 3 datacenter proxy
PROXY_URL=http://user:pass@datacenter.example.com:8080

# Optional Tier 4 residential proxy
RESIDENTIAL_PROXY_URL=socks5://user:pass@residential.example.com:1080

docker compose up -d

Leave either value empty to disable that proxy tier. Multiple endpoints can be separated with commas; larger pools can use the corresponding *_LIST_FILE variable. See Configuration → Proxies for pool and mounted-file examples.

Variable	Default	Description
`BROWSER_POOL_SIZE`	`3`	Warm Camoufox Firefox instances
`BROWSER_ACQUIRE_TIMEOUT_MS`	`15000`	How long `acquire()` polls for a free browser before HTTP 429 is returned
`BROWSER_RECYCLE_AFTER_CONTEXTS`	`8`	Rolling-replace after this many Tier 3/4 contexts; set `0` to disable
`BROWSER_CONTENT_PROCESSES`	`2`	Cap Firefox content processes per browser (`dom.ipc.processCount`); lowers RAM/CPU
`SESSION_TTL_SECONDS`	`3600`	Redis session cache TTL (seconds)
`REDIS_URL`	`redis://localhost:6379`	Redis connection string
`PROXY_URL`	—	Optional Tier 3 HTTP or SOCKS5 proxy, or comma-separated pool
`PROXY_LIST_FILE`	—	File containing one Tier 3 proxy URL per line
`RESIDENTIAL_PROXY_URL`	—	Enables Tier 4 proxy escalation
`RESIDENTIAL_PROXY_LIST_FILE`	—	File containing one Tier 4 proxy URL per line
`STT_URL`	—	Local Whisper endpoint for reCAPTCHA (optional)
`PORT`	`8191`	API listen port
`MITM_PROXY_ENABLED`	`false`	Enable the challenge-bypassing HTTP/HTTPS proxy
`MITM_PROXY_PORT`	`8192`	Forward-proxy listen port
`MITM_PROXY_HOST`	`0.0.0.0`	Bind address; `127.0.0.1` for loopback-only
`MITM_PROXY_CA_DIR`	`/data/proxy-ca`	Persistent root CA certificate and private-key directory
`MITM_PROXY_MAX_TIER`	`4`	Cap escalation used by the proxy (e.g. `3` to stay off residential)
`MITM_PROXY_DEBUG`	`false`	Log one line per proxied request (errors are always logged)

Stack

Built on a modern, fast-by-default stack: Bun + Elysia for the API, Redis for caching, Camoufox (hardened Firefox) for browser automation, and Nuxt for the web UI — no legacy Node/Express baggage.

Layer	Technology
Runtime	Bun
API	Elysia
Browser	Camoufox Firefox (via camoufox-js)
Session cache	Redis 8.8
Landing page	Nuxt 4
Documentation	VitePress

License

AGPL-3.0

Made with ❤️ by @germondai

Navigation Menu

TRAWL

Welcome to TRAWL! 👋

Features

Sponsors

Quick start

NAS app catalogs

API

FlareSolverr-compatible (`/v1`)

Native API (`/scrape`)

Connect Prowlarr / Jackett

Challenge-bypassing HTTP/HTTPS proxy

Installing the proxy CA certificate

macOS (system keychain — affects most apps including Safari, curl, wget)

Linux (Debian/Ubuntu — system-wide for curl, wget, apt, etc.)

Linux (RHEL/Fedora/Amazon)

Firefox and NSS trust stores

Chrome / Chromium (Linux: separate from system trust)

Java (including JDownloader)

JDownloader 2 (Windows / macOS / Linux — manual install)

Windows (system trust store)

Removing the CA (cleanup)

Tiers

Docker Compose files

Docker images (one GHCR package, two tags)

Releases & versioning

Configuration

Stack

License

FilesExpand file tree

README.md

Latest commit

History

README.md

File metadata and controls

TRAWL

Welcome to TRAWL! 👋

Features

Sponsors

Quick start

NAS app catalogs

API

FlareSolverr-compatible (/v1)

Native API (/scrape)

Connect Prowlarr / Jackett

Challenge-bypassing HTTP/HTTPS proxy

Installing the proxy CA certificate

macOS (system keychain — affects most apps including Safari, curl, wget)

Linux (Debian/Ubuntu — system-wide for curl, wget, apt, etc.)

Linux (RHEL/Fedora/Amazon)

Firefox and NSS trust stores

Chrome / Chromium (Linux: separate from system trust)

Java (including JDownloader)

JDownloader 2 (Windows / macOS / Linux — manual install)

Windows (system trust store)

Removing the CA (cleanup)

Tiers

Docker Compose files

Docker images (one GHCR package, two tags)

Releases & versioning

Configuration

Stack

License

FlareSolverr-compatible (`/v1`)

Native API (`/scrape`)