feat(engine): opt-in prewarm Chromium spare with smoke matrix and hardening

.github/workflows/ci.yml

@@ -4,7 +4,11 @@ on:
   pull_request:
   push:
     branches:
-      - "**"
+      - main
+
+concurrency:
+  group: ci-${{ github.workflow }}-${{ github.event.pull_request.number || github.ref }}
+  cancel-in-progress: true
 
 permissions:
   contents: read
@@ -104,13 +108,68 @@ jobs:
       - name: Verify OpenAPI snapshot
         run: make openapi-verify
 
+  docker-smoke-image:
+    name: Docker smoke image (linux/amd64, cache)
+    runs-on: ubuntu-latest
+    permissions:
+      contents: read
+      actions: write
+    steps:
+      - name: Check out repository
+        uses: actions/checkout@v7
+
+      - name: Set up Docker Buildx
+        uses: docker/setup-buildx-action@v4
+
+      - name: Build and cache smoke image
+        uses: docker/build-push-action@v7
+        with:
+          context: .
+          push: false
+          load: true
+          platforms: linux/amd64
+          tags: botasaurus-api-smoke:ci
+          cache-from: type=gha,scope=smoke-linux-amd64
+          cache-to: type=gha,mode=max,scope=smoke-linux-amd64
+
   smoke-test:
-    name: Run docker smoke test
+    name: ${{ matrix.title }}
+    needs: docker-smoke-image
     runs-on: ubuntu-latest
-    timeout-minutes: 30
+    timeout-minutes: ${{ matrix.timeout_minutes }}
+    permissions:
+      contents: read
+      actions: write
+    strategy:
+      fail-fast: false
+      matrix:
+        include:
+          - title: "Smoke (prewarm=false, cold)"
+            profile: contract-prewarm-off
+            timeout_minutes: 30
+          - title: "Smoke (prewarm=true, warm-handoff)"
+            profile: prewarm-on-warm-handoff
+            timeout_minutes: 20
     steps:
       - name: Check out repository
         uses: actions/checkout@v7
 
-      - name: Execute smoke test
+      - name: Set up Docker Buildx
+        uses: docker/setup-buildx-action@v4
+
+      - name: Load smoke image from cache
+        uses: docker/build-push-action@v7
+        with:
+          context: .
+          push: false
+          load: true
+          platforms: linux/amd64
+          tags: botasaurus-api-smoke:ci
+          cache-from: type=gha,scope=smoke-linux-amd64
+
+      - name: Execute smoke profile
+        env:
+          SMOKE_SKIP_BUILD: "1"
+          SMOKE_IMAGE_NAME: botasaurus-api-smoke:ci
+          SMOKE_PROFILE: ${{ matrix.profile }}
         run: make smoke

AGENTS.md

@@ -28,6 +28,7 @@ app/
     budget.py                # wall-clock budget math shared across tiers (elapsed_ms, step budgets)
     request_tier.py          # HTTP/curl_cffi path
     browser_tier.py          # Chromium path
+    warm_pool.py             # opt-in WarmDriverPool + DriverFingerprint
     strategies.py            # NavigationMode resolution, driver helpers
     driver_capabilities.py   # DriverProtocol + call_if_available adapter
     envelope.py              # success/error builders, UTF-8 HTML normalization
@@ -90,10 +91,11 @@ Conventions:
 | --- | --- | --- |
 | `ScraperEngine` | process (app.state) | shared |
 | `ThreadPoolExecutor` | process (app.state) | shared, sized by `SCRAPE_MAX_WORKERS` |
+| `WarmDriverPool` (opt-in) | process (engine.warm_pool) | single spare slot; refill on dedicated daemon thread — never the scrape executor |
 | `_active_request_ids` | in-process memory | shared; collision guard |
 | runtime dir `/tmp/scrape/<request_id>` | per request | isolated; deleted in `finally` |
-| browser profile | per request | isolated; no reuse |
-| Botasaurus Driver | per request | isolated; closed in `finally` |
+| browser profile | per request (or adopted spare-*) | isolated; no reuse across requests; warm spare dies with the adopting request |
+| Botasaurus Driver | may start before assignment | usage stays ≤1 request; closed in session `__exit__`; never returned to the pool |
 
 Multi-worker uvicorn breaks in-process collision detection unless request ids are sticky to a worker. Default to single-worker for isolation semantics.
 
@@ -122,13 +124,14 @@ Multi-worker uvicorn breaks in-process collision detection unless request ids ar
 - `POST /scrape` is async API over sync browser work (threadpool).
 - Each scrape request must use isolated runtime state:
   - request-scoped runtime dir `/tmp/scrape/<request_id>`
-  - request-scoped browser profile
-  - no cache/profile/driver reuse across requests
+  - request-scoped browser profile (cold path) or one-shot adopted `spare-*` profile (warm path)
+  - no cache/profile/driver reuse across requests (warm spare is closed after the adopting request)
+- Opt-in prewarm (`SCRAPE_PREWARM=true`, default off): single-slot `WarmDriverPool` builds a spare after a browser scrape finishes when the fingerprint matches. Refill runs on a dedicated daemon thread. Idle TTL (`SCRAPE_PREWARM_IDLE_TTL_SECONDS`, default `600`, `0`=never) and min refill interval (`SCRAPE_PREWARM_MIN_REFILL_SECONDS`, default `30`) bound idle RAM. Worst case during refill overlap: 2 Chromiums briefly. Cgroup-v2 best-effort skip above 70% memory. Docker Xvfb spike confirmed concurrent headed drivers are safe (`PREWARM_HEADLESS_ONLY=False`).
 - Cleanup is mandatory in `finally`:
   - close browser driver
-  - delete request runtime dir
+  - delete request runtime dir (and adopted spare dir when present)
   - remove in-memory active request id
-- Before each scrape, prune orphaned runtime dirs not tied to an active request id; ENOSPC on profile creation retries after another prune pass. Optional `SCRAPE_RUNTIME_MIN_FREE_BYTES` (default 256MiB) logs when the runtime filesystem is low.
+- Before each scrape, prune orphaned runtime dirs not tied to an active request id; warm-pool `live_spare_dirs()` protects ready, in-build, and adopted-in-use `spare-*` dirs (release via `release_adopted` on session exit). Orphan `spare-*` cleaned at pool init. ENOSPC on profile creation retries after another prune pass. Optional `SCRAPE_RUNTIME_MIN_FREE_BYTES` (default 256MiB) logs when the runtime filesystem is low.
 - Keep request-id collision/invariant guard (`_active_request_ids`) intact; raises `RequestIdCollisionError`.
 - `driver.requests.get` metadata is best-effort; metadata failure must not fail HTML success.
 - Keep strategy engine behavior:
@@ -142,8 +145,9 @@ Multi-worker uvicorn breaks in-process collision detection unless request ids ar
 ## Performance
 
 - Baseline bench: `PYTHONPATH=. .venv/bin/python3 scripts/bench_scrape.py --runs 10` (TestClient, `execution_mode=request`).
+- Browser cold vs warm: `SCRAPE_PREWARM=true` with `--execution-mode browser` (local Docker with `--shm-size=1gb --init` recommended); compare `boot_ms` from `scrape_boot` logs / p50 wall time. Record p50 `boot_ms` delta in the PR body when opening a prewarm PR.
 - Record p50 wall time when changing hot paths; avoid regressions vs prior baseline.
-- On low-RAM hosts (Docker `--memory=768m`, 1–2 vCPU), tune `SCRAPE_MAX_WORKERS` to `1` or `2`; higher values increase queue wait and swap without improving wall time.
+- On low-RAM hosts (Docker `--memory=768m`, 1–2 vCPU), tune `SCRAPE_MAX_WORKERS` to `1` or `2`; higher values increase queue wait and swap without improving wall time. Keep prewarm default-off until canary shows no RSS ceiling breach.
 - Browser tier skips XHR harvest before Cloudflare bypass; one consolidated `collect_page_state` pass runs after bypass.
 
 ## Types
@@ -164,6 +168,9 @@ Multi-worker uvicorn breaks in-process collision detection unless request ids ar
 - Run `make check` before finish (lint, test, typecheck, openapi-verify).
 - When Pydantic models or route response metadata change, run `make openapi` and commit the snapshot with the code change.
 - When API contract, Docker behavior, or error semantics change, also run `make smoke`.
-- `make smoke` must cover build, boot, `/health`, `/scrape` happy path, strategy override, retry path, isolation check, localhost guardrail.
+- `make smoke` (default `SMOKE_PROFILE=all`) covers prewarm **off** and **on**:
+  - `contract-prewarm-off` — build/boot, `/health`, `/scrape` happy path, strategy override, retry path, isolation (httpbingo cookies), localhost SSRF, request-tier, headers, `organic_get`, scroll, Sentry sidecar init.
+  - `prewarm-on-warm-handoff` — `SCRAPE_PREWARM=true`, two browser `example.com` scrapes, assert `scrape_boot warm_hit=False` then refill then `warm_hit=True` (no isolation pair; that stays on the off profile).
+  - CI Checks titles: `Smoke (prewarm=false, cold)` and `Smoke (prewarm=true, warm-handoff)`. One `docker-smoke-image` job warms Buildx GHA cache (`scope=smoke-linux-amd64`); matrix jobs `cache-from` + `SMOKE_SKIP_BUILD=1`.
 - If API contract, Docker behavior, or error semantics changed, update README in same change.
 - Keep commits scoped (infra vs API vs docs).

README.md

@@ -337,6 +337,9 @@ Use a **separate Sentry project** from html2rss-web (`BOTASAURUS_SENTRY_DSN` →
 | `SCRAPE_TIMEOUT_SECONDS` | `45` | Handler wall-clock budget in seconds (queue, browser boot, and work). |
 | `SCRAPE_WORK_TIMEOUT_SECONDS` | `30` | Post-boot navigate, selector wait, and scroll budget in seconds. |
 | `SCRAPE_RUNTIME_MIN_FREE_BYTES` | `268435456` (256 MiB) | Prune orphan runtime dirs when free space drops below this threshold. |
+| `SCRAPE_PREWARM` | `false` | Opt-in single-slot Chromium prewarm. When `true`, fingerprint-matching browser scrapes reuse a spare driver built after the previous scrape. Requires **single uvicorn worker** when enabled; multi-worker deployments do not share spare state and worker restarts delete foreign `spare-*` dirs. Keep off until canary on `--shm-size=1gb` + `init: true` shows stable RSS. |
+| `SCRAPE_PREWARM_IDLE_TTL_SECONDS` | `600` | Close an unused warm spare after this many seconds. `0` disables idle reap — one Chromium remains in RAM until process exit. |
+| `SCRAPE_PREWARM_MIN_REFILL_SECONDS` | `30` | Minimum interval between spare rebuilds. |
 
 ## Example Calls
 

app/config.py

@@ -65,10 +65,28 @@ class Settings(BaseSettings):
         default=256 * 1024 * 1024,
         validation_alias="SCRAPE_RUNTIME_MIN_FREE_BYTES",
     )
+    scrape_prewarm: bool = Field(default=False, validation_alias="SCRAPE_PREWARM")
+    scrape_prewarm_idle_ttl_seconds: int = Field(
+        default=600,
+        ge=0,
+        validation_alias="SCRAPE_PREWARM_IDLE_TTL_SECONDS",
+    )
+    scrape_prewarm_min_refill_seconds: int = Field(
+        default=30,
+        ge=0,
+        validation_alias="SCRAPE_PREWARM_MIN_REFILL_SECONDS",
+    )
     runtime_root: Path = Field(default=Path("/tmp/scrape"))
     environment: str = Field(default="production", validation_alias="ENVIRONMENT")
     sentry: SentrySettings = Field(default_factory=SentrySettings)
 
+    @field_validator("scrape_prewarm", mode="before")
+    @classmethod
+    def parse_scrape_prewarm(cls, value: object) -> bool:
+        if isinstance(value, bool):
+            return value
+        return str(value or "").strip().lower() in {"true", "1", "yes", "on"}
+
     @model_validator(mode="after")
     def validate_timeout_relationship(self) -> Settings:
         if self.scrape_work_timeout_seconds > self.scrape_timeout_seconds:

app/domain/scrape_service.py

@@ -192,12 +192,20 @@ async def _run(
                 phase.value,
                 timeout_result.diagnostics.attempts,
             )
-            emit_terminal_telemetry(timeout_result, http_status=504)
+            emit_terminal_telemetry(
+                timeout_result,
+                http_status=504,
+                warm_hit=progress.snapshot().warm_hit,
+            )
             return ScrapeOutcome(body=timeout_result, status_code=504)
 
         status_code = 200 if isinstance(result, ScrapeSuccess) else 502
         if isinstance(result, ScrapeError):
-            emit_terminal_telemetry(result, http_status=status_code)
+            emit_terminal_telemetry(
+                result,
+                http_status=status_code,
+                warm_hit=progress.snapshot().warm_hit,
+            )
         logger.info(
             "scrape_complete request_id=%s host=%s mode=%s tier=%s attempts=%s status=%d error_category=%s",
             result.diagnostics.request_id,

app/engine/browser_tier.py

@@ -24,6 +24,7 @@
     resolve_strategies,
     wait_for_readiness,
 )
+from app.engine.warm_pool import DriverFingerprint
 from app.infra.detector import ChallengeAssessment, ChallengeDetector
 from app.infra.metadata import MetadataExtractor, MetadataResult
 from app.infra.scrape_progress import ScrapeProgress
@@ -65,6 +66,28 @@ def settle_page_state(
     return html, meta, assessment, xhr_responses
 
 
+def _boot_storage_error(
+    target_url: str,
+    request_id: str,
+    started_monotonic: float,
+    exc: OSError,
+) -> ScrapeError:
+    if exc.errno == errno.ENOSPC:
+        detail = "Scrape runtime storage full"
+    else:
+        detail = "Scrape runtime storage unavailable"
+    return build_error(
+        target_url,
+        f"{detail}: {exc}",
+        request_id=request_id,
+        attempts=0,
+        render_ms=elapsed_ms(started_monotonic),
+        error_category=ErrorCategory.NAVIGATION_ERROR,
+        execution_tier=ExecutionTier.BROWSER_DRIVER,
+        timeout_phase=TimeoutPhase.BOOT,
+    )
+
+
 def run_browser_tier(
     payload: ScrapeRequest,
     session: ScrapeSession,
@@ -81,67 +104,89 @@ def run_browser_tier(
         TimeoutPhase.BOOT,
         execution_tier=ExecutionTier.BROWSER_DRIVER,
     )
-    try:
-        session.prepare_profile_dirs()
-    except OSError as exc:
-        if exc.errno == errno.ENOSPC:
-            detail = "Scrape runtime storage full"
-        else:
-            detail = "Scrape runtime storage unavailable"
-        return build_error(
-            target_url,
-            f"{detail}: {exc}",
-            request_id=request_id,
-            attempts=0,
-            render_ms=elapsed_ms(started_monotonic),
-            error_category=ErrorCategory.NAVIGATION_ERROR,
-            execution_tier=ExecutionTier.BROWSER_DRIVER,
-            timeout_phase=TimeoutPhase.BOOT,
+    fingerprint = DriverFingerprint.from_request(payload)
+    session.warm_fingerprint = fingerprint
+    warm_hit = False
+    boot_started = time.monotonic()
+
+    pool = session.engine.warm_pool
+    taken = pool.take(fingerprint) if pool is not None else None
+    if taken is not None:
+        driver, spare_dir = taken
+        # Assign immediately so session.__exit__ closes the spare on any raise.
+        session.driver = driver
+        session.profile_dir = spare_dir
+        session.adopted_profile_dir = spare_dir
+        try:
+            session.prepare_runtime_dir()
+        except OSError as exc:
+            return _boot_storage_error(target_url, request_id, started_monotonic, exc)
+        warm_hit = True
+    else:
+        try:
+            session.prepare_profile_dirs()
+        except OSError as exc:
+            return _boot_storage_error(target_url, request_id, started_monotonic, exc)
+
+        driver_window_size = (
+            [payload.window_size.width, payload.window_size.height]
+            if payload.window_size
+            else None
         )
 
+        try:
+            driver = cast(
+                DriverProtocol,
+                Driver(
+                    headless=payload.headless,
+                    enable_xvfb_virtual_display=not payload.headless,
+                    proxy=payload.proxy,
+                    profile=str(session.profile_dir),
+                    tiny_profile=True,
+                    block_images=payload.block_images,
+                    block_images_and_css=payload.block_images_and_css,
+                    wait_for_complete_page_load=payload.wait_for_complete_page_load,
+                    user_agent=payload.effective_user_agent,
+                    window_size=driver_window_size,
+                    lang=payload.lang,
+                    remove_default_browser_check_argument=True,
+                ),
+            )
+        except Exception as exc:
+            is_timeout = is_timeout_exception(exc)
+            return build_error(
+                target_url,
+                str(exc),
+                request_id=request_id,
+                attempts=0,
+                render_ms=elapsed_ms(started_monotonic),
+                error_category=(
+                    ErrorCategory.TIMEOUT
+                    if is_timeout
+                    else ErrorCategory.NAVIGATION_ERROR
+                ),
+                execution_tier=ExecutionTier.BROWSER_DRIVER,
+                timeout_phase=TimeoutPhase.BOOT,
+            )
+
+        session.driver = driver
+
+    session.warm_hit = warm_hit
+    progress.set_warm_hit(warm_hit)
+    boot_ms = int((time.monotonic() - boot_started) * 1000)
+    logger.info(
+        "scrape_boot request_id=%s warm_hit=%s boot_ms=%d",
+        request_id,
+        warm_hit,
+        boot_ms,
+    )
+
     strategies = resolve_strategies(payload.navigation_mode, payload.max_retries)
     attempts = 0
     collector = XhrCollector(target_url)
-    driver_window_size = (
-        [payload.window_size.width, payload.window_size.height]
-        if payload.window_size
-        else None
-    )
-
-    try:
-        driver = cast(
-            DriverProtocol,
-            Driver(
-                headless=payload.headless,
-                enable_xvfb_virtual_display=not payload.headless,
-                proxy=payload.proxy,
-                profile=str(session.profile_dir),
-                tiny_profile=True,
-                block_images=payload.block_images,
-                block_images_and_css=payload.block_images_and_css,
-                wait_for_complete_page_load=payload.wait_for_complete_page_load,
-                user_agent=payload.effective_user_agent,
-                window_size=driver_window_size,
-                lang=payload.lang,
-                remove_default_browser_check_argument=True,
-            ),
-        )
-    except Exception as exc:
-        is_timeout = is_timeout_exception(exc)
-        return build_error(
-            target_url,
-            str(exc),
-            request_id=request_id,
-            attempts=0,
-            render_ms=elapsed_ms(started_monotonic),
-            error_category=(
-                ErrorCategory.TIMEOUT if is_timeout else ErrorCategory.NAVIGATION_ERROR
-            ),
-            execution_tier=ExecutionTier.BROWSER_DRIVER,
-            timeout_phase=TimeoutPhase.BOOT,
-        )
+    driver = session.driver
+    assert driver is not None
 
-    session.driver = driver
     configure_driver(driver, payload, target_url, collector=collector)
     browser_ready_monotonic = time.monotonic()
     progress.mark(