fix(docker): harden runtime against portal 403s

Install a less-detectable browser path in Docker (Chrome on x86_64; Chromium fallback on ARM64), align browser-compat fingerprint to Linux, and add update+rebuild wrapper modes for scheduled runs.

Made-with: Cursor

Author: UnexpectedFisting

Dockerfile

@@ -4,11 +4,21 @@ WORKDIR /app
 
 COPY requirements.txt /app/requirements.txt
 RUN pip install --no-cache-dir -r /app/requirements.txt
+# Playwright "chrome" channel isn't supported on Linux ARM64.
+# - x86_64: install real Chrome channel (best for avoiding 403 fingerprinting)
+# - arm64: fall back to Playwright Chromium
+RUN set -e; arch="$(uname -m)"; \
+    if [ "$arch" = "x86_64" ]; then \
+      python -m playwright install chrome; \
+    else \
+      python -m playwright install chromium; \
+    fi
 
 COPY src /app/src
 
 ENV PYTHONUNBUFFERED=1
 ENV PYTHONPATH=/app/src
+ENV STUDENTAID_MONARCH_RUNTIME=docker
 
 ENTRYPOINT ["python", "-m", "studentaid_monarch_sync"]
 

README.md

@@ -85,6 +85,7 @@ Pick **one** of the following and run it end-to-end. For most people (especially
 
 #### Runtime A: Docker (recommended) 🐳
 This repo includes a `docker-compose.yml` service that runs the sync as a **run-once** container.
+The Docker image now installs a real **Google Chrome** channel and uses a Linux-specific browser-compat profile, which reduces `HeadlessChrome`-style fingerprinting that can trigger portal `403 Access Denied` responses.
 
 ##### Docker Desktop (Windows/macOS)
 1. Install Docker Desktop and make sure `docker compose` works in a terminal.
@@ -122,7 +123,7 @@ To keep the scheduled command simple (and easy to update later), you can schedul
   - **Add arguments** (example):
 
 ```text
--NoProfile -File .\scripts\docker_sync.ps1 run --payments-since 2025-01-01
+-NoProfile -File .\scripts\docker_sync.ps1 update-run --payments-since 2025-01-01
 ```
 
   - **Start in**: `C:\path\to\repo` (the folder that contains `docker-compose.yml`)
@@ -131,14 +132,14 @@ To keep the scheduled command simple (and easy to update later), you can schedul
   - Create a LaunchAgent that runs:
 
 ```bash
-cd /path/to/repo && bash ./scripts/docker_sync.sh run --payments-since 2025-01-01
+cd /path/to/repo && bash ./scripts/docker_sync.sh update-run --payments-since 2025-01-01
 ```
 
 - **Linux (cron/systemd)**:
   - Use cron or a systemd timer to run the same command on your desired timeframe:
 
 ```bash
-cd /path/to/repo && bash ./scripts/docker_sync.sh run --payments-since 2025-01-01
+cd /path/to/repo && bash ./scripts/docker_sync.sh update-run --payments-since 2025-01-01
 ```
 
 ##### Unraid (NAS)
@@ -160,7 +161,7 @@ bash ./scripts/docker_sync.sh dry-run --payments-since 2025-01-01
 4. Schedule it (e.g., Unraid **User Scripts** plugin) with a daily command like:
 
 ```bash
-cd /path/to/repo && bash ./scripts/docker_sync.sh run --payments-since 2025-01-01
+cd /path/to/repo && bash ./scripts/docker_sync.sh update-run --payments-since 2025-01-01
 ```
 
 Keep `./data` persistent so sessions and the SQLite idempotency DB survive restarts.
@@ -340,6 +341,7 @@ See **Quick start → Runtime A: Docker (recommended)** for the Unraid schedulin
 <a id="403-access-denied"></a>
 - **HTTP 403 Access Denied / portal blocks the headless browser**
   - Some servicers (notably Nelnet) occasionally return a bare `HTTP 403 Access Denied` page to headless browsers that look like automation. The tool detects this and retries once with a fresh session automatically.
+  - Docker builds now install a real Chrome channel and use a Linux-aligned browser fingerprint. If you updated from an older image, rebuild it first with `docker compose build --no-cache`.
   - If it keeps failing, try the following in order:
     1. Run `sync --headful --manual-mfa` once to establish a fresh, trusted browser session stored under `data/servicer_storage_state_*.json`. Subsequent headless runs reuse that session.
     2. Add `--fresh-session` to force discarding any stale stored session before retrying.

scripts/docker_sync.ps1

@@ -1,6 +1,6 @@
 Param(
   [Parameter(Position = 0)]
-  [ValidateSet("setup-accounts", "preflight", "dry-run", "run")]
+  [ValidateSet("setup-accounts", "preflight", "dry-run", "run", "update", "update-run", "update-dry-run")]
   [string]$Mode = "run",
 
   [Parameter(ValueFromRemainingArguments = $true)]
@@ -24,6 +24,24 @@ $Service = "studentaid-monarch-sync"
 
 New-Item -ItemType Directory -Force -Path (Join-Path $RepoDir "data") | Out-Null
 
+function Invoke-GitPull {
+  if (Get-Command git -ErrorAction SilentlyContinue) {
+    if (Test-Path (Join-Path $RepoDir ".git")) {
+      # Best-effort update; keep it safe/non-destructive.
+      git pull --ff-only
+    }
+  }
+}
+
+function Invoke-ComposeBuild {
+  $buildArgs = @("build", "--pull")
+  if ($env:NO_CACHE -eq "1") {
+    $buildArgs += "--no-cache"
+  }
+  $buildArgs += $Service
+  docker compose @buildArgs
+}
+
 switch ($Mode) {
   "setup-accounts" {
     docker compose run --rm --build $Service setup-monarch-accounts --apply @Args
@@ -37,6 +55,20 @@ switch ($Mode) {
   "run" {
     docker compose run --rm $Service sync @Args
   }
+  "update" {
+    Invoke-GitPull
+    Invoke-ComposeBuild
+  }
+  "update-run" {
+    Invoke-GitPull
+    Invoke-ComposeBuild
+    docker compose run --rm $Service sync @Args
+  }
+  "update-dry-run" {
+    Invoke-GitPull
+    Invoke-ComposeBuild
+    docker compose run --rm $Service sync --dry-run @Args
+  }
 }
 
 

scripts/docker_sync.sh

@@ -8,6 +8,9 @@ set -euo pipefail
 #   ./scripts/docker_sync.sh preflight
 #   ./scripts/docker_sync.sh dry-run --payments-since 2025-01-01
 #   ./scripts/docker_sync.sh run --payments-since 2025-01-01
+#   ./scripts/docker_sync.sh update
+#   ./scripts/docker_sync.sh update-run --payments-since 2025-01-01
+#   ./scripts/docker_sync.sh update-dry-run --payments-since 2025-01-01
 
 REPO_DIR="$(cd "$(dirname "${BASH_SOURCE[0]}")/.." && pwd)"
 cd "${REPO_DIR}"
@@ -19,6 +22,24 @@ SERVICE="studentaid-monarch-sync"
 
 mkdir -p data
 
+_git_pull_ff_only() {
+  if command -v git >/dev/null 2>&1 && [ -d ".git" ]; then
+    # Best-effort update; keep it safe/non-destructive.
+    git pull --ff-only
+  fi
+}
+
+_compose_build() {
+  # Pull newer base image layers when available.
+  # Set NO_CACHE=1 to force a full rebuild.
+  local args=(build --pull)
+  if [ "${NO_CACHE:-}" = "1" ]; then
+    args+=(--no-cache)
+  fi
+  args+=("${SERVICE}")
+  docker compose "${args[@]}"
+}
+
 case "${MODE}" in
   setup-accounts)
     docker compose run --rm --build "${SERVICE}" setup-monarch-accounts --apply "$@"
@@ -32,9 +53,23 @@ case "${MODE}" in
   run)
     docker compose run --rm "${SERVICE}" sync "$@"
     ;;
+  update)
+    _git_pull_ff_only
+    _compose_build
+    ;;
+  update-run)
+    _git_pull_ff_only
+    _compose_build
+    docker compose run --rm "${SERVICE}" sync "$@"
+    ;;
+  update-dry-run)
+    _git_pull_ff_only
+    _compose_build
+    docker compose run --rm "${SERVICE}" sync --dry-run "$@"
+    ;;
   *)
     echo "Unknown mode: ${MODE}"
-    echo "Expected: setup-accounts | preflight | dry-run | run"
+    echo "Expected: setup-accounts | preflight | dry-run | run | update | update-run | update-dry-run"
     exit 2
     ;;
 esac

src/studentaid_monarch_sync/portal/client.py

@@ -2,6 +2,7 @@
 
 import logging
 import json
+import os
 import random
 import re
 import shutil
@@ -79,7 +80,19 @@ def __init__(
         "--disable-blink-features=AutomationControlled",
     ]
 
-    _BROWSER_COMPAT_INIT_SCRIPT = r"""
+    _WINDOWS_CHROME_USER_AGENT = (
+        "Mozilla/5.0 (Windows NT 10.0; Win64; x64) "
+        "AppleWebKit/537.36 (KHTML, like Gecko) "
+        "Chrome/131.0.0.0 Safari/537.36"
+    )
+
+    _LINUX_CHROME_USER_AGENT = (
+        "Mozilla/5.0 (X11; Linux x86_64) "
+        "AppleWebKit/537.36 (KHTML, like Gecko) "
+        "Chrome/131.0.0.0 Safari/537.36"
+    )
+
+    _BROWSER_COMPAT_INIT_SCRIPT_TEMPLATE = r"""
     (() => {
       // Mask navigator.webdriver
       try { Object.defineProperty(navigator, 'webdriver', { get: () => undefined }); } catch (_) {}
@@ -91,6 +104,13 @@ def __init__(
         });
       } catch (_) {}
 
+      // Keep platform aligned with the chosen runtime/user-agent.
+      try {
+        Object.defineProperty(navigator, 'platform', {
+          get: () => __NAVIGATOR_PLATFORM__,
+        });
+      } catch (_) {}
+
       // Chrome runtime presence (headless Chrome often lacks this)
       try {
         if (!window.chrome) { window.chrome = {}; }
@@ -191,6 +211,28 @@ def __init__(
     })();
     """
 
+    def _browser_compat_runtime(self) -> str:
+        runtime = (os.getenv("STUDENTAID_MONARCH_RUNTIME") or "").strip().lower()
+        if runtime in {"docker", "native"}:
+            return runtime
+        return "docker" if Path("/.dockerenv").exists() else "native"
+
+    def _browser_compat_user_agent(self) -> str:
+        if self._browser_compat_runtime() == "docker":
+            return self._LINUX_CHROME_USER_AGENT
+        return self._WINDOWS_CHROME_USER_AGENT
+
+    def _browser_compat_platform(self) -> str:
+        if self._browser_compat_runtime() == "docker":
+            return "Linux x86_64"
+        return "Win32"
+
+    def _browser_compat_init_script(self) -> str:
+        return self._BROWSER_COMPAT_INIT_SCRIPT_TEMPLATE.replace(
+            "__NAVIGATOR_PLATFORM__",
+            json.dumps(self._browser_compat_platform()),
+        )
+
     def _launch_browser(self, p, *, headless: bool, slow_mo: int):
         """
         Launch Chromium with anti-detection defaults.
@@ -242,11 +284,7 @@ def _create_browser_context(self, browser, *, storage_state: Optional[str] = Non
         Create a browser context with realistic fingerprint settings.
         """
         ctx_kwargs: dict = {
-            "user_agent": (
-                "Mozilla/5.0 (Windows NT 10.0; Win64; x64) "
-                "AppleWebKit/537.36 (KHTML, like Gecko) "
-                "Chrome/131.0.0.0 Safari/537.36"
-            ),
+            "user_agent": self._browser_compat_user_agent(),
             "color_scheme": "light",
             "viewport": {"width": 1920, "height": 1080},
             "locale": "en-US",
@@ -276,7 +314,7 @@ def _rewrite(route, request) -> None:
         except Exception:
             logger.debug("Failed to install dark-host rewrite route.", exc_info=True)
 
-        ctx.add_init_script(self._BROWSER_COMPAT_INIT_SCRIPT)
+        ctx.add_init_script(self._browser_compat_init_script())
         ctx.add_init_script(self._CONSENT_DISMISS_SCRIPT)
 
     def _human_delay(self, page: Page, min_ms: int = 80, max_ms: int = 250) -> None:

tests/test_parsing.py

@@ -292,3 +292,17 @@ def test_looks_like_access_denied_ignores_normal_page_content() -> None:
     c = _client()
     page = _BodyOnlyPage("Welcome back. Payment Activity is ready.")
     assert c._looks_like_access_denied(page) is False
+
+
+def test_browser_compat_uses_linux_profile_for_docker_runtime(monkeypatch: pytest.MonkeyPatch) -> None:
+    monkeypatch.setenv("STUDENTAID_MONARCH_RUNTIME", "docker")
+    c = _client()
+    assert "Linux x86_64" in c._browser_compat_user_agent()
+    assert c._browser_compat_platform() == "Linux x86_64"
+
+
+def test_browser_compat_uses_native_profile_when_runtime_forced_native(monkeypatch: pytest.MonkeyPatch) -> None:
+    monkeypatch.setenv("STUDENTAID_MONARCH_RUNTIME", "native")
+    c = _client()
+    assert "Windows NT 10.0" in c._browser_compat_user_agent()
+    assert c._browser_compat_platform() == "Win32"