fix: clamp wait_timeout, UTF-8 HTML, and gate CI on Ruff plus Hadolint (#35)

* fix: clamp wait_timeout and return scrape envelope on schema 422

Gem clients send wait=28 against a 20s service cap; clamp inbound wait so scrape still runs, and keep remaining schema 422s in the scrape envelope the gem already parses.

* fix: utf-8 html headers and request-tier 403 escalate

Botasaurus HTML must be labeled UTF-8 so clients do not treat octet-stream as abort; 401/403/429 on the request tier already escalate to browser, and tests now lock that El País/NOS path.

* test: lock utf-8 identity and clamp wait on POST /scrape

Already-correct Caffè/CJK HTML must survive normalize; README timeout and content-type now match the 20s UTF-8 runtime.

* chore(lint): add Ruff with Makefile and CI

Host-side lint/format as the fast gate so Docker smoke is not the only quality check.

* style: apply Ruff autofix so check and format pass

Bring the existing modules in line with the new linter so CI lint is green on this branch.

Author: gildesmarais

.github/workflows/ci.yml

@@ -10,6 +10,26 @@ permissions:
   contents: read
 
 jobs:
+  ruff:
+    name: Ruff
+    runs-on: ubuntu-latest
+    steps:
+      - name: Check out repository
+        uses: actions/checkout@v6
+
+      - name: Run Ruff
+        # ruff-action has v4.1.0 / v4.0.0 tags, not a floating v4 major.
+        uses: astral-sh/ruff-action@v4.1.0
+        with:
+          version: "0.16.3"
+          args: check --output-format=github
+
+      - name: Check Ruff format
+        uses: astral-sh/ruff-action@v4.1.0
+        with:
+          version: "0.16.3"
+          args: format --check
+
   hadolint:
     name: Hadolint
     runs-on: ubuntu-latest
@@ -20,6 +40,17 @@ jobs:
       - name: Lint Dockerfile
         run: docker run --rm -i hadolint/hadolint < Dockerfile
 
+  lint:
+    name: Lint
+    if: ${{ !cancelled() }}
+    needs: [ruff, hadolint]
+    runs-on: ubuntu-latest
+    steps:
+      - name: Require Ruff and Hadolint
+        run: |
+          test "${{ needs.ruff.result }}" = "success"
+          test "${{ needs.hadolint.result }}" = "success"
+
   unit-tests:
     name: Run unit tests
     runs-on: ubuntu-latest

.gitignore

@@ -1,3 +1,4 @@
 .ruff_cache/
+.venv/
 __pycache__/
 *.pyc

AGENTS.md

@@ -9,8 +9,10 @@
 
 - Endpoints: `GET /health`, `POST /scrape`.
 - Stable legacy `/scrape` fields: `url`, `final_url`, `status_code`, `headers`, `html`, `error`, `metadata_error`.
+- When `html` is present, document `headers` `content-type` is `text/html; charset=utf-8` and `html` is UTF-8-normalized.
 - Additive diagnostics fields (current contract): `request_id`, `attempts`, `strategy_used`, `render_ms`, `blocked_detected`, `challenge_detected`, `error_category`, `execution_tier`, `detected_challenge`, `xhr_responses`.
-- Request options (current contract): `execution_mode`, `navigation_mode`, `max_retries`, `wait_for_selector`, `wait_timeout_seconds`, `scroll`, `scroll_to_bottom`, `block_images`, `block_trackers`, `headers`, `cookies`.
+- Request options (current contract): `execution_mode`, `navigation_mode`, `max_retries`, `wait_for_selector`, `wait_timeout_seconds`, `scroll`, `scroll_to_bottom`, `block_images`, `block_images_and_css`, `block_trackers`, `wait_for_complete_page_load`, `headers`, `cookies`, `user_agent`, `window_size`, `lang`, `headless`, `proxy`.
+- `wait_timeout_seconds` outside `[1, SCRAPE_TIMEOUT_SECONDS]` (default 20) is clamped into that range so `/scrape` still runs; remaining schema 422 bodies use the scrape envelope (`url`, `error`, `error_category`, `request_id`), not FastAPI `detail`.
 - Error codes:
   - `400` validation/resolution failure
   - `403` SSRF guardrail block

Makefile

@@ -1,4 +1,4 @@
-.PHONY: test build serve health scrape-example smoke lint check ready
+.PHONY: test build serve health scrape-example smoke lint lintfix check ready
 
 .DEFAULT_GOAL := check
 
@@ -7,9 +7,15 @@ PORT ?= 4010
 BASE_URL ?= http://localhost:$(PORT)
 
 lint:
+	ruff check .
+	ruff format --check .
 	docker run --rm -i hadolint/hadolint < Dockerfile
 
-check: lint
+lintfix:
+	ruff check --fix .
+	ruff format .
+
+check: lint test
 
 ready: check
 

README.md

@@ -8,7 +8,7 @@ Docker-only FastAPI service that uses [Botasaurus](https://github.com/omkarcloud
   - `GET /health`
   - `POST /scrape`
 - Intended usage: run and test through Docker only.
-- Runtime boundary: async FastAPI handler delegates sync browser work to a bounded threadpool (`SCRAPE_MAX_WORKERS`, default `4`), with a per-request timeout (`SCRAPE_TIMEOUT_SECONDS`, default `25`).
+- Runtime boundary: async FastAPI handler delegates sync browser work to a bounded threadpool (`SCRAPE_MAX_WORKERS`, default `4`), with a per-request timeout (`SCRAPE_TIMEOUT_SECONDS`, default `20`).
 - On-demand isolation-first runtime: every scrape request runs with an ephemeral browser profile and request-scoped runtime dir, then gets fully cleaned up.
 
 ## Prerequisites
@@ -139,22 +139,19 @@ Request options (contract):
   - `organic_get`: only `organic_get`
 - `max_retries`: `0..3`, default `2` (attempts = `1 + max_retries`, with `auto` capped by 3 strategy steps).
 - `wait_for_selector`: if set, response waits for selector before capture (routes to browser tier).
-- `wait_timeout_seconds`: selector wait timeout (default `15`, capped by service timeout).
+- `wait_timeout_seconds`: selector wait timeout (default `15`). Values outside `[1, SCRAPE_TIMEOUT_SECONDS]` (default `20`) are clamped into that range so scrape still runs.
 - `scroll` / `scroll_to_bottom`: if true, scrolls the page to trigger lazy-loaded feeds (routes to browser tier).
 - `block_images`: pass image blocking to driver. Default `true`.
+- `block_images_and_css`: pass image+css blocking to driver. Default `false`.
 - `block_trackers`: block tracking/ad networks and web fonts to speed up rendering. Default `true`.
+- `wait_for_complete_page_load`: pass page-load wait behavior to driver. Default `true`.
+- `user_agent`: explicit user agent string passed to driver.
 - `headers`: custom HTTP request headers forwarded to request client or browser session.
 - `cookies`: key-value cookies map forwarded to request client or browser session.
-
-Currently accepted passthrough options (implemented, not part of stable request-options contract):
-
-- `block_images_and_css`: pass image+css blocking to driver.
-- `wait_for_complete_page_load`: pass page-load wait behavior to driver.
-- `user_agent`: explicit user agent string passed to driver.
 - `window_size`: two-item integer list `[width, height]` passed to driver.
 - `lang`: browser language passed to driver (for example `en-US`).
 - `headless`: pass headless browser mode to driver. Default `false`.
-- `proxy`: proxy URL passed to driver.
+- `proxy`: proxy URL passed to driver. Invalid or blocked proxy URLs are rejected by SSRF guardrails.
 
 Success response shape (legacy fields preserved, additive diagnostics included):
 
@@ -164,7 +161,7 @@ Success response shape (legacy fields preserved, additive diagnostics included):
   "final_url": "https://example.com/",
   "status_code": 200,
   "headers": {
-    "content-type": "text/html"
+    "content-type": "text/html; charset=utf-8"
   },
   "html": "<!doctype html>...",
   "error": null,
@@ -184,8 +181,8 @@ Success response shape (legacy fields preserved, additive diagnostics included):
 
 Field behavior:
 
-- `html`: rendered page HTML.
-- `headers`, `status_code`, `final_url`: best-effort metadata and may be `null`.
+- `html`: rendered page HTML, UTF-8-normalized.
+- `headers`, `status_code`, `final_url`: best-effort metadata and may be `null`. When `html` is present, document `headers` `content-type` is `text/html; charset=utf-8`.
 - `error`: populated when scrape fails or challenge is detected on final attempt.
 - `metadata_error`: populated when metadata extraction fails but HTML scrape succeeds.
 - `request_id`: unique per request for tracing.
@@ -207,7 +204,7 @@ Status codes:
 - `200`: scrape completed without `error`.
 - `400`: URL rejected by validation (for example unresolved host).
 - `403`: URL blocked by SSRF guardrails.
-- `422`: request schema validation failed.
+- `422`: request schema validation failed. Body is the scrape error envelope (`url`, `error`, `error_category`, `request_id`), not FastAPI `{"detail":[...]}`.
 - `502`: scrape execution failure/challenge block.
 - `504`: scrape timed out.
 

app/detector.py

@@ -75,7 +75,9 @@ def detect(
             status_code in {401, 403, 429} if status_code is not None else False
         )
 
-        error_category = "challenge_block" if (challenge_detected or blocked_detected) else None
+        error_category = (
+            "challenge_block" if (challenge_detected or blocked_detected) else None
+        )
 
         return ChallengeAssessment(
             blocked_detected=blocked_detected,

app/engine.py

@@ -13,7 +13,7 @@
 
 from botasaurus.browser import Driver
 from botasaurus.request import Request
-from pydantic import BaseModel, Field, HttpUrl, field_validator
+from pydantic import BaseModel, Field, HttpUrl, ValidationInfo, field_validator
 
 from app.detector import ChallengeDetector
 from app.metadata import MetadataExtractor
@@ -26,7 +26,9 @@
 _RUNTIME_ROOT = Path("/tmp/scrape")
 
 ExecutionMode = Literal["auto", "request", "browser"]
-NavigationMode = Literal["auto", "get", "google_get", "google_get_bypass", "organic_get"]
+NavigationMode = Literal[
+    "auto", "get", "google_get", "google_get_bypass", "organic_get"
+]
 ErrorCategory = Literal[
     "timeout", "challenge_block", "navigation_error", "metadata_error"
 ]
@@ -73,6 +75,28 @@ class ScrapeRequest(BaseModel):
     headless: bool = False
     proxy: str | None = None
 
+    @field_validator("wait_timeout_seconds", mode="before")
+    @classmethod
+    def clamp_wait_timeout_seconds(cls, value: Any, info: ValidationInfo) -> Any:
+        if value is None:
+            return value
+        try:
+            numeric = int(value)
+        except (TypeError, ValueError):  # fmt: skip
+            return value
+
+        clamped = max(1, min(numeric, DEFAULT_SCRAPE_TIMEOUT_SECONDS))
+        if clamped != numeric:
+            url = info.data.get("url")
+            logger.info(
+                "request_field_clamped host=%s field=%s from=%s to=%s",
+                urlparse(str(url)).hostname if url else None,
+                "wait_timeout_seconds",
+                numeric,
+                clamped,
+            )
+        return clamped
+
     @field_validator("window_size")
     @classmethod
     def validate_window_size(cls, value: list[int] | None) -> list[int] | None:
@@ -93,6 +117,36 @@ def effective_user_agent(self) -> str | None:
         return None
 
 
+HTML_DOCUMENT_CONTENT_TYPE = "text/html; charset=utf-8"
+
+
+def utf8_normalize_html(html: str) -> str:
+    if not html:
+        return html
+    if not isinstance(html, str):
+        html = str(html)
+    try:
+        html = html.encode("latin-1").decode("utf-8")
+    except (UnicodeEncodeError, UnicodeDecodeError):  # fmt: skip
+        pass
+    return html.encode("utf-8", errors="replace").decode("utf-8")
+
+
+def html_document_headers(
+    html: str, headers: dict[str, str] | None
+) -> tuple[str, dict[str, str] | None]:
+    if not html:
+        return html, headers
+    normalized = utf8_normalize_html(html)
+    out: dict[str, str] = {}
+    for key, value in (headers or {}).items():
+        if str(key).lower() == "content-type":
+            continue
+        out[str(key)] = str(value)
+    out["content-type"] = HTML_DOCUMENT_CONTENT_TYPE
+    return normalized, out
+
+
 class ScrapeResponse(BaseModel):
     url: str
     final_url: str | None = None
@@ -134,6 +188,7 @@ def create_success(
         detected_challenge: str | None = None,
         xhr_responses: list[dict[str, Any]] | None = None,
     ) -> dict[str, Any]:
+        html, headers = html_document_headers(html, headers)
         return cls(
             url=url,
             final_url=final_url or url,
@@ -176,6 +231,7 @@ def create_error(
         metadata_error: str | None = None,
         xhr_responses: list[dict[str, Any]] | None = None,
     ) -> dict[str, Any]:
+        html, headers = html_document_headers(html, headers)
         return cls(
             url=url,
             final_url=final_url,
@@ -284,9 +340,7 @@ def unregister_request_id(self, request_id: str) -> None:
             self._active_request_ids.discard(request_id)
 
     @classmethod
-    def resolve_strategies(
-        cls, mode: NavigationMode, max_retries: int
-    ) -> list[str]:
+    def resolve_strategies(cls, mode: NavigationMode, max_retries: int) -> list[str]:
         max_attempts = 1 + max_retries
         if mode == "auto":
             ordered = ["google_get", "google_get_bypass", "get"]
@@ -298,11 +352,13 @@ def navigate(
         cls, driver: Driver, target_url: str, strategy: str, timeout_seconds: int
     ) -> None:
         if strategy == "organic_get":
-            method = getattr(driver, "organic_get", getattr(driver, "google_get", getattr(driver, "get")))
+            method = getattr(
+                driver, "organic_get", getattr(driver, "google_get", driver.get)
+            )
         elif strategy.startswith("google_get"):
-            method = getattr(driver, "google_get", getattr(driver, "get"))
+            method = getattr(driver, "google_get", driver.get)
         else:
-            method = getattr(driver, "get")
+            method = driver.get
 
         kwargs: dict[str, Any] = {}
         if strategy == "google_get_bypass":
@@ -441,16 +497,13 @@ def run_request_tier(
         remaining_budget = max(
             1,
             int(
-                DEFAULT_SCRAPE_TIMEOUT_SECONDS
-                - (time.monotonic() - started_monotonic)
+                DEFAULT_SCRAPE_TIMEOUT_SECONDS - (time.monotonic() - started_monotonic)
             ),
         )
 
         req_headers = dict(payload.headers) if payload.headers else {}
         proxies = (
-            {"http": payload.proxy, "https": payload.proxy}
-            if payload.proxy
-            else None
+            {"http": payload.proxy, "https": payload.proxy} if payload.proxy else None
         )
 
         req = Request()
@@ -497,8 +550,12 @@ def run_request_tier(
                 )
                 return None
 
-            error_msg = "Challenge block detected" if assessment.blocked_detected else None
-            error_cat: ErrorCategory | None = "challenge_block" if assessment.blocked_detected else None
+            error_msg = (
+                "Challenge block detected" if assessment.blocked_detected else None
+            )
+            error_cat: ErrorCategory | None = (
+                "challenge_block" if assessment.blocked_detected else None
+            )
 
             return ScrapeResponse.create_success(
                 target_url,
@@ -554,9 +611,7 @@ def run_browser_tier(
             lang=payload.lang,
             remove_default_browser_check_argument=True,
         )
-        self._configure_driver(
-            session.driver, payload, target_url, collector=collector
-        )
+        self._configure_driver(session.driver, payload, target_url, collector=collector)
 
         for attempt_index, strategy in enumerate(strategies, start=1):
             attempts = attempt_index
@@ -572,9 +627,7 @@ def run_browser_tier(
                 self.wait_for_readiness(
                     session.driver,
                     selector=payload.wait_for_selector,
-                    timeout_seconds=min(
-                        payload.wait_timeout_seconds, remaining_budget
-                    ),
+                    timeout_seconds=min(payload.wait_timeout_seconds, remaining_budget),
                 )
 
                 if payload.should_scroll:
@@ -598,11 +651,11 @@ def run_browser_tier(
                             assessment = ChallengeDetector.detect(
                                 html, meta.status_code, driver=session.driver
                             )
-                            xhr_responses = self._harvest_xhr(
-                                collector, session.driver
-                            )
+                            xhr_responses = self._harvest_xhr(collector, session.driver)
                         except Exception as exc:
-                            logger.debug("bypass_cloudflare_attempt_failed error=%s", str(exc))
+                            logger.debug(
+                                "bypass_cloudflare_attempt_failed error=%s", str(exc)
+                            )
 
                 if assessment.challenge_detected or assessment.blocked_detected:
                     logger.warning(
@@ -715,14 +768,11 @@ def execute(
             )
 
         with ScrapeSession(self, request_id) as session:
-            should_try_request_tier = (
-                payload.execution_mode == "request"
-                or (
-                    payload.execution_mode == "auto"
-                    and payload.navigation_mode == "auto"
-                    and not payload.wait_for_selector
-                    and not payload.should_scroll
-                )
+            should_try_request_tier = payload.execution_mode == "request" or (
+                payload.execution_mode == "auto"
+                and payload.navigation_mode == "auto"
+                and not payload.wait_for_selector
+                and not payload.should_scroll
             )
 
             if should_try_request_tier: