feat(engine): opt-in prewarm Chromium spare with smoke matrix and hardening

[gildesmarais]

What changed

• Opt-in SCRAPE_PREWARM single-slot WarmDriverPool: fingerprint-gated Chromium spare, dedicated refill daemon (never the scrape executor), one-shot adoption per request (app/engine/warm_pool.py, app/engine/browser_tier.py, app/main.py).
• Browser tier adopts warm spare on fingerprint match; cold boot unchanged on pool miss or prewarm off. Spare dies with the adopting request (never returned to the pool).
• Production hardening: atomic prune snapshot of active request ids + live_spare_dirs() (app/engine/orchestrator.py); unified _probe_health / _abort_adopted_take so probe failures close the driver and drop adopted-dir protection; stale spare reap when _desired fingerprint changes (fixes refill stall at SCRAPE_PREWARM_IDLE_TTL_SECONDS=0).
• Lifespan coverage: tests/api/test_prewarm_lifespan.py asserts SCRAPE_PREWARM=false leaves engine.warm_pool is None, true attaches WarmDriverPool, and real shutdown stops refill (post-exit notify_scrape_finished does not rebuild); also parse_scrape_prewarm env string matrix.
• Smoke profiles (scripts/smoke.sh): SMOKE_PROFILE=contract-prewarm-off (full contract suite) and prewarm-on-warm-handoff (two browser scrapes; assert scrape_boot warm_hit=False then refill then warm_hit=True). Default all runs both. SMOKE_SKIP_BUILD + empty-env-safe container start.
• CI: docker-smoke-image warms Buildx GHA cache (scope=smoke-linux-amd64); matrix legs titled Smoke (prewarm=false, cold) and Smoke (prewarm=true, warm-handoff) with fail-fast: false and SMOKE_SKIP_BUILD=1. Workflow push limited to main (PR checks via pull_request only) plus concurrency cancel-in-progress.
• Test support: consolidated fakes in tests/support/fakes.py (TrackingDriver, InstrumentedDriver, HealthCheckStub); typed factory functions replace dataclass lambdas; concurrent take/notify race fixed (nudge notify after barrier + clean adopted spare).
• Ops docs: single uvicorn worker when prewarm on; idle_ttl=0 holds RAM until process exit; smoke/CI coverage recorded (README.md, AGENTS.md).

Commits vs main (newest first):

• 5ca676f ci: run PR checks once via pull_request only
• 2524045 fix(smoke): tolerate empty env when starting scrape container
• 7b79c63 fix(tests): assert real warm-pool shutdown in lifespan test
• be94cb3 docs: record prewarm smoke coverage and CI image cache
• 636c486 ci(smoke): cached image job and named prewarm matrix
• 15a6ef0 test(smoke): add SMOKE_PROFILE off and warm-handoff
• de10e1a test(engine): cover create_app prewarm on and off via lifespan
• 56f81e8 fix(tests): stabilize concurrent warm-pool take/notify test
• c890065 refactor(tests): replace dataclass lambdas with typed factory functions
• 355d098 fix(engine): reap stale warm spare when desired fingerprint changes
• a5bfca3 docs(ops): state prewarm single-worker and idle TTL requirements
• 85a0992 test(engine): lock warm-path configure and idle-spare invariants
• 3e1ce6f fix(engine): atomic prune protection and unified adopted-take abort
• da0aeb4 refactor(tests): consolidate warm pool driver fakes
• 7a6f9ec fix(engine): protect adopted spare dirs from prune during handoff
• e0ec5a2 docs(ops): document prewarm flag, memory posture, and invariant change
• 1f1d121 feat(engine): serve browser scrapes from prewarmed driver when fingerprint matches
• d8ba007 feat(engine): add warm driver pool with fingerprint gating

Why

Cold Chromium boot dominates browser-tier latency. A single prewarmed spare cuts boot time when fingerprints match, without changing default memory posture (SCRAPE_PREWARM=false). Hardening closes prune races, health-probe leaks, and stale-spare refill stalls. Smoke profiles + cached CI matrix prove cold contract and warm handoff without double-building the image; push limited to main avoids duplicate PR check runs.

Risk

• Memory: warm spare holds an extra Chromium (briefly two during refill); canary under --memory before production enable.
• Deployment: prewarm is in-process — single uvicorn worker (or sticky routing) required.
• Fingerprint: cookies/headers/block_trackers applied per request in configure_driver(), not in fingerprint hash — idle-spare + warm-path tests lock this.
• Default off: no runtime change unless SCRAPE_PREWARM=true.
• CI time: two smoke matrix legs; shared image cache mitigates rebuild cost.

Review map

Large PR (~2557 lines, 27 files). Tests first:

1. tests/engine/test_warm_pool.py — pool lifecycle, TTL, stale fingerprint reap, health-probe abort, concurrent take/notify, idle refill
2. tests/engine/test_warm_wiring.py — end-to-end adoption, prune barrier, cookies/headers on warm hit, isolation
3. tests/api/test_prewarm_lifespan.py — create_app attach/shutdown for prewarm on/off; env parse
4. tests/support/fakes.py — shared instrumented drivers and health-check stubs
5. app/engine/warm_pool.py — spare build/refill/TTL/take/abort; module docstring invariants
6. app/engine/browser_tier.py — warm hit vs cold boot wiring
7. app/engine/orchestrator.py — atomic prune snapshot with live_spare_dirs()
8. scripts/smoke.sh — SMOKE_PROFILE off / warm-handoff / all; skip-build
9. .github/workflows/ci.yml — docker-smoke-image cache job; named matrix; push→main only
10. README.md / AGENTS.md — ops constraints and isolation invariants

Validation

• make check (lint, tests, pyright strict, openapi-verify, spectral, hadolint)
• CI smoke matrix: Smoke (prewarm=false, cold) + Smoke (prewarm=true, warm-handoff) via cached image
• Local canary: make smoke / warm handoff under memory limit; record p50 boot_ms delta cold vs warm