Merge pull request #2077 from unclecode/release/v0.9.2

release: Crawl4AI v0.9.2

Author: unclecode

CONTRIBUTORS.md

@@ -2,7 +2,7 @@
 
 We would like to thank the following people for their contributions to Crawl4AI:
 
-## Core Team
+## Maintainers
 
 - [Unclecode](https://github.com/unclecode) - Project Creator and Main Developer
 - [Nasrin](https://github.com/ntohidi) - Project Manager and Developer
@@ -58,9 +58,11 @@ We would like to thank the following people for their contributions to Crawl4AI:
 - [microHoffman](https://github.com/microHoffman) - feat: add --json-ensure-ascii CLI flag for Unicode handling [#1668](https://github.com/unclecode/crawl4ai/pull/1668)
 - [fstark96](https://github.com/fstark96) - fix: skip channel='chromium' on Windows to prevent Chrome exit code 0 [#2051](https://github.com/unclecode/crawl4ai/pull/2051)
 - [TobiasWallura-xitaso](https://github.com/TobiasWallura-xitaso) - fix: use writable directories for supervisord pid and redis data in Docker [#2047](https://github.com/unclecode/crawl4ai/pull/2047)
-- [harshmathurx](https://github.com/harshmathurx) - fix: authenticate rate limit Redis storage in Docker [#2043](https://github.com/unclecode/crawl4ai/pull/2043)
+- [harshmathurx](https://github.com/harshmathurx) - fix: authenticate rate limit Redis storage in Docker [#2043](https://github.com/unclecode/crawl4ai/pull/2043), fix: add non-free apt source for nvidia-cuda-toolkit GPU builds [#2057](https://github.com/unclecode/crawl4ai/pull/2057)
 - [RajanChavada](https://github.com/RajanChavada) - chore: remove dead normalize_url definitions and accidental adaptive_crawler copy [#2042](https://github.com/unclecode/crawl4ai/pull/2042)
 - [bibi-creator](https://github.com/bibi-creator) - docs: split sponsors into Strategic Partners and Enterprise Sponsors [#2056](https://github.com/unclecode/crawl4ai/pull/2056), fix: update sponsor logos [#2052](https://github.com/unclecode/crawl4ai/pull/2052)
+- [reallav0](https://github.com/reallav0) - fix: clean up MemoryAdaptiveDispatcher tasks on stream close [#2072](https://github.com/unclecode/crawl4ai/pull/2072), fix: include Playwright headless shell in Docker image [#2067](https://github.com/unclecode/crawl4ai/pull/2067)
+- [Pitchfork-and-Torch](https://github.com/Pitchfork-and-Torch) - fix: Docker playground config type and monitor WebSocket auth [#2063](https://github.com/unclecode/crawl4ai/pull/2063)
 
 #### Feb-Alpha-1
 - [sufianuddin](https://github.com/sufianuddin) - fix: [Documentation for JsonCssExtractionStrategy](https://github.com/unclecode/crawl4ai/issues/651)

Dockerfile

@@ -1,7 +1,7 @@
 FROM python:3.12-slim-bookworm AS build
 
 # C4ai version
-ARG C4AI_VER=0.9.1
+ARG C4AI_VER=0.9.2
 ENV C4AI_VERSION=$C4AI_VER
 LABEL c4ai.version=$C4AI_VER
 
@@ -90,9 +90,10 @@ RUN apt-get update && apt-get dist-upgrade -y \
     && rm -rf /var/lib/apt/lists/*
 
 RUN if [ "$ENABLE_GPU" = "true" ] && [ "$TARGETARCH" = "amd64" ] ; then \
-    apt-get update && apt-get install -y --no-install-recommends \
+    echo "deb http://deb.debian.org/debian bookworm contrib non-free" >> /etc/apt/sources.list \
+    && apt-get update && apt-get install -y --no-install-recommends \
     nvidia-cuda-toolkit \
-    && apt-get clean \ 
+    && apt-get clean \
     && rm -rf /var/lib/apt/lists/* ; \
 else \
     echo "Skipping NVIDIA CUDA Toolkit installation (unsupported platform or GPU disabled)"; \
@@ -177,7 +178,9 @@ RUN crawl4ai-setup
 RUN playwright install --with-deps
 
 RUN mkdir -p /home/appuser/.cache/ms-playwright \
-    && cp -r /root/.cache/ms-playwright/chromium-* /home/appuser/.cache/ms-playwright/ \
+    && cp -r /root/.cache/ms-playwright/chromium-* \
+        /root/.cache/ms-playwright/chromium_headless_shell-* \
+        /home/appuser/.cache/ms-playwright/ \
     && chown -R appuser:appuser /home/appuser/.cache/ms-playwright
 
 RUN crawl4ai-doctor
@@ -226,4 +229,4 @@ ENV PYTHON_ENV=production
 # Start via entrypoint.sh, which resolves the socket-level auth/egress posture
 # (loopback unless a credential is present) and the redis password, then execs
 # supervisord.
-CMD ["bash", "entrypoint.sh"]
+CMD ["bash", "entrypoint.sh"]

README.md

@@ -37,9 +37,9 @@ Limited slots._
 
 Crawl4AI turns the web into clean, LLM ready Markdown for RAG, agents, and data pipelines. Fast, controllable, battle tested by a 50k+ star community.
 
-[✨ Check out latest update v0.9.1](#-recent-updates)
+[✨ Check out latest update v0.9.2](#-recent-updates)
 
-✨ **New in v0.9.1**: Patch release with 12 bug fixes across Docker, browser, and core. Adds `preserve_classes`/`preserve_tags` whitelist for PruningContentFilter, fixes Windows browser crash, Docker auth gate UI, HTTP timeout unit mismatch, and more. [Release notes →](https://github.com/unclecode/crawl4ai/blob/main/docs/blog/release-v0.9.1.md)
+✨ **New in v0.9.2**: Maintenance patch release. Fixes a `MemoryAdaptiveDispatcher` task/page leak when a streaming crawl is closed, Docker Playground "Advanced Config" and Monitor WebSocket auth, Playwright headless-shell packaging, and GPU (`ENABLE_GPU=true`) Docker builds. [Release notes →](https://github.com/unclecode/crawl4ai/blob/main/docs/blog/release-v0.9.2.md)
 
 ✨ Recent v0.9.0: Major secure-by-default release of the Docker API server. Auth is on by default, the server binds loopback unless given a token, and the request body is now an untrusted trust boundary. [Release notes →](https://github.com/unclecode/crawl4ai/blob/main/docs/blog/release-v0.9.0.md)
 
@@ -565,6 +565,21 @@ async def test_news_crawl():
 ## ✨ Recent Updates
 
 <details open>
+<summary><strong>Version 0.9.2 Release Highlights - Maintenance Bug Fixes</strong></summary>
+
+A maintenance patch release with bug fixes across the dispatcher, Docker, and GPU builds. `MemoryAdaptiveDispatcher` no longer leaks crawl tasks and browser pages when a streaming crawl is closed. Docker fixes cover the Playground "Advanced Config" 400, the Monitor WebSocket 500 under JWT auth, and Playwright headless-shell packaging. `ENABLE_GPU=true` Docker builds no longer fail on the CUDA toolkit.
+
+No new features, no breaking changes.
+
+```bash
+pip install -U crawl4ai
+```
+
+[Full v0.9.2 Release Notes →](https://github.com/unclecode/crawl4ai/blob/main/docs/blog/release-v0.9.2.md)
+
+</details>
+
+<details>
 <summary><strong>Version 0.9.1 Release Highlights - Bug Fixes & PruningContentFilter Whitelist</strong></summary>
 
 A patch release with 12 bug fixes and one new feature. The new `preserve_classes` / `preserve_tags` parameters for `PruningContentFilter` let you whitelist CSS classes or HTML tags that should never be pruned — useful for protecting short metadata elements like author names and timestamps.

crawl4ai/__version__.py

@@ -1,7 +1,7 @@
 # crawl4ai/__version__.py
 
 # This is the version that will be used for stable releases
-__version__ = "0.9.1"
+__version__ = "0.9.2"
 
 # For nightly builds, this gets set during build process
 __nightly_version__ = None

crawl4ai/async_dispatcher.py

@@ -534,6 +534,7 @@ async def run_urls_stream(
         config: Union[CrawlerRunConfig, List[CrawlerRunConfig]],
     ) -> AsyncGenerator[CrawlerTaskResult, None]:
         self.crawler = crawler
+        active_tasks = []
 
         # Start the memory monitor task
         memory_monitor = asyncio.create_task(self._memory_monitor_task())
@@ -550,7 +551,6 @@ async def run_urls_stream(
                 # Add to queue with initial priority 0, retry count 0, and current time
                 await self.task_queue.put((0, (url, task_id, 0, time.time())))
 
-            active_tasks = []
             completed_count = 0
             total_urls = len(urls)
 
@@ -614,8 +614,24 @@ async def run_urls_stream(
                 await self._update_queue_priorities()
 
         finally:
-            # Clean up
+            # Cancel and await every task owned by this stream before returning
+            # control to the caller. Otherwise a closed stream can leave crawls
+            # using browser pages and contexts in the background.
+            for task in active_tasks:
+                if not task.done():
+                    task.cancel()
+            if active_tasks:
+                await asyncio.gather(*active_tasks, return_exceptions=True)
+
+            # Discard URLs that were queued by this stream but never started.
+            while True:
+                try:
+                    self.task_queue.get_nowait()
+                except asyncio.QueueEmpty:
+                    break
+
             memory_monitor.cancel()
+            await asyncio.gather(memory_monitor, return_exceptions=True)
             if self.monitor:
                 self.monitor.stop()
 
@@ -769,4 +785,4 @@ async def run_urls(
             return await asyncio.gather(*tasks, return_exceptions=True)
         finally:
             if self.monitor:
-                self.monitor.stop()
+                self.monitor.stop()

deploy/docker/README.md

@@ -59,10 +59,10 @@ Pull and run images directly from Docker Hub without building locally.
 
 #### 1. Pull the Image
 
-Our latest stable release is `0.8.6`. Images are built with multi-arch manifests, so Docker automatically pulls the correct version for your system.
+Our latest stable release is `0.9.2`. Images are built with multi-arch manifests, so Docker automatically pulls the correct version for your system.
 
 ```bash
-# Pull the latest stable version (0.8.6)
+# Pull the latest stable version (0.9.2)
 docker pull unclecode/crawl4ai:0.8.6
 
 # Or use the latest tag

deploy/docker/server.py

@@ -482,8 +482,12 @@ def _config_from_json(data: dict) -> dict:
 app.include_router(init_job_router(redis, config, token_dep))
 
 # ── monitor router ──────────────────────────────────────────
+# Do not attach token_dep at router level: it is HTTP Request-only and breaks
+# the WebSocket upgrade on /monitor/ws (TypeError: _principal() missing 'request').
+# AuthGateMiddleware already authenticates HTTP + WS; destructive monitor
+# actions keep their own Depends(require_admin).
 from monitor_routes import router as monitor_router
-app.include_router(monitor_router, dependencies=[Depends(token_dep)])
+app.include_router(monitor_router)
 
 logger = logging.getLogger(__name__)
 

deploy/docker/static/playground/index.html

@@ -563,10 +563,13 @@ <h2 class="font-medium text-accent">🔥 Stress Test</h2>
             const code = cm.getValue().trim();
             if (!code) return {};
 
+            // Server requires `type` alongside `code` (CrawlerRunConfig | BrowserConfig).
+            // The UI already tracks this in #cfg-type; omitting it always 400s Advanced Config.
+            const cfgType = document.getElementById('cfg-type').value;
             const res = await authFetch('/config/dump', {
                 method: 'POST',
                 headers: { 'Content-Type': 'application/json' },
-                body: JSON.stringify({ code }),
+                body: JSON.stringify({ type: cfgType, code }),
             });
 
             const statusEl = document.getElementById('cfg-status');
@@ -652,9 +655,12 @@ <h2 class="font-medium text-accent">🔥 Stress Test</h2>
         // Detect if stream is requested inside payload
         function shouldUseStream(payload) {
             const toBool = (v) => v === true || (typeof v === 'string' && v.toLowerCase() === 'true');
-            const fromCrawler = payload && payload.crawler_config && payload.crawler_config.params && payload.crawler_config.params.stream;
+            // Successful /config/dump returns { type, params: { stream, ... } }
+            const fromCrawlerParams = payload && payload.crawler_config && payload.crawler_config.params && payload.crawler_config.params.stream;
+            // Fallback / minimal shapes may set stream at the crawler_config top level
+            const fromCrawlerTop = payload && payload.crawler_config && payload.crawler_config.stream;
             const direct = payload && payload.stream;
-            return toBool(fromCrawler) || toBool(direct);
+            return toBool(fromCrawlerParams) || toBool(fromCrawlerTop) || toBool(direct);
         }
 
         // Main run function
@@ -676,8 +682,8 @@ <h2 class="font-medium text-accent">🔥 Stress Test</h2>
                 const streamFlag = /stream\s*=\s*True/i.test(codeText);
                 const isCrawlEndpoint = document.getElementById('endpoint').value === 'crawl';
                 if (isCrawlEndpoint && streamFlag) {
-                    // Fallback: proceed with minimal config only for stream
-                    advConfig = { crawler_config: { stream: true } };
+                    // Fallback: minimal dump-shaped config so shouldUseStream + server load agree
+                    advConfig = { crawler_config: { type: 'CrawlerRunConfig', params: { stream: true } } };
                 } else {
                     updateStatus('error');
                     document.querySelector('#response-content code').textContent =

deploy/docker/tests/test_security_2026_04.py

@@ -138,14 +138,17 @@ def test_traversal_validator_removed(self):
 
 class TestMonitorAuthStructural(unittest.TestCase):
 
-    def test_monitor_router_has_auth(self):
+    def test_monitor_router_has_no_http_only_dep(self):
+        """Router-level token_dep was removed because it is HTTP-only and broke
+        the WebSocket upgrade on /monitor/ws.  Auth is enforced by
+        AuthGateMiddleware; destructive routes keep Depends(require_admin)."""
         with open(os.path.join(DEPLOY_DIR, "server.py")) as f:
             source = f.read()
-        # Find the line with monitor_router
         for line in source.splitlines():
             if "monitor_router" in line and "include_router" in line:
-                self.assertIn("dependencies=", line,
-                    "Monitor router must have dependencies=[Depends(token_dep)]")
+                self.assertNotIn("dependencies=", line,
+                    "Monitor router must NOT have router-level token_dep "
+                    "(breaks WS); auth is enforced by AuthGateMiddleware")
                 return
         self.fail("Could not find monitor_router include_router line")
 

deploy/docker/tests/test_security_container_posture.py

@@ -57,6 +57,19 @@ def test_artifact_dir_created_0700(self, dockerfile):
         assert "/var/lib/crawl4ai/outputs" in dockerfile
         assert "chmod 700 /var/lib/crawl4ai/outputs" in dockerfile
 
+    def test_playwright_headless_shell_is_copied_to_runtime_cache(self, dockerfile):
+        # Playwright launches chromium_headless_shell for headless crawls. The
+        # image runs as appuser, so both Chromium artifacts must be copied out
+        # of root's install cache.
+        cache_copy = re.search(
+            r"cp -r(?P<artifacts>(?:(?!&&).)*)/home/appuser/\.cache/ms-playwright/",
+            dockerfile,
+            re.DOTALL,
+        )
+        assert cache_copy, "Dockerfile must copy Playwright artifacts into appuser's cache"
+        assert "chromium-*" in cache_copy.group("artifacts")
+        assert "chromium_headless_shell-*" in cache_copy.group("artifacts")
+
     def test_runs_as_non_root(self, dockerfile):
         assert re.search(r"^USER\s+appuser", dockerfile, re.MULTILINE)