[Bug]: Unable to execute arun_many with managed browsers and cdp

[medmahmoudi26]

crawl4ai version

v0.7.6

Expected Behavior

Hello ! I am trying to use crawl4ai for concurrent authenticated crawling. However, I'm running into errors when combining cdp with arun_many.

In terminal, I launched a browser using cli:

• I created a profile using the cli and turned headless to off.
• I run a browser using cdp and my profile directory

crwl cdp -d /home/user/.crawl4ai/profiles/

A new instance was launched, I confirmed I could interact with it using native playwright.

Crawl4ai script:

import asyncio
from crawl4ai import AsyncWebCrawler, BrowserConfig, CrawlerRunConfig, CacheMode

# Define URLs to crawl
URLS = [
    "https://example.com",
    "https://httpbin.org/html",
    "https://www.python.org",
]

async def main():
    # Configure CDP browser connection
    browser_cfg = BrowserConfig(
        browser_type="cdp",
        cdp_url="http://localhost:9222",
        verbose=True,
    )
    
    # Configure crawler settings
    crawler_cfg = CrawlerRunConfig(
        cache_mode=CacheMode.BYPASS,
        page_timeout=60000,
        wait_until="domcontentloaded",
    )
    
    # Crawl all URLs using arun_many
    async with AsyncWebCrawler(config=browser_cfg) as crawler:
        results = await crawler.arun_many(urls=URLS, config=crawler_cfg)
        
        for result in results:
            print(f"\nURL: {result.url}")
            if result.success:
                print(f"✓ Success | Content length: {len(result.markdown)}")
            else:
                print(f"✗ Failed: {result.error_message}")

if __name__ == "__main__":
    asyncio.run(main())

Expectaed behaviour: multiple tabs should open in chromium and the crawls are executed in parallel.
Current behaviour: only one tab is opened, only one crawl is finished, the rest runs into errors.

Current Behavior

Only one tab is opened, only one crawl is finished, the rest runs into errors.

I think the bug might be coming from a race condition in the browser manager, all sessions are competing over the same page.

crawl4ai/crawl4ai/browser_manager.py

Line 1055 in 40173ee

page = pages[0]

Is this reproducible?

Yes

Inputs Causing the Bug

Steps to Reproduce

Code snippets

OS

Linux

Python version

3.11

Browser

No response

Browser version

No response

Error logs & Screenshots (if applicable)

No response

[Ahmed-Tawfik94]

Root Cause Analysis

The Issue

The root cause was page contention in concurrent crawling scenarios when using managed CDP (Chrome DevTools Protocol) browsers.

Detailed Explanation

In the BrowserManager.get_page() method, when multiple concurrent tasks called arun_many with a managed browser, all tasks were reusing the same browser page instead of creating new ones:

# PROBLEMATIC CODE:
if pages:
    page = pages[0]  # All concurrent tasks reused the first page

This caused several critical issues:

1. Race Conditions: Multiple tasks simultaneously navigating the same page
2. State Conflicts: One task's actions interfering with another's crawling operations
3. Target Closed Errors: Pages being closed by one task while others were still using them
4. Unpredictable Behavior: Tasks getting mixed up results or failing entirely

The Fix

The solution was to always create new pages for managed browsers instead of reusing existing ones:

# FIXED CODE:
if pages:
    # Always create a new page for managed browsers to support concurrent crawling
    async with self._page_lock:
        page = await context.new_page()

This ensures each concurrent task gets its own dedicated page, eliminating contention and allowing reliable concurrent crawling with managed CDP browsers.