PlaywrightCrawler __init__ method browser_new_context_options argument does not function

[ForumeFoundation]

According to the documentation, the browser_new_context_options are the keyword arguments that are accepted by the Playwright Browser new_context method, here:

Keyword arguments to pass to the browser new context method. These options are provided directly to Playwright's browser.new_context method. For more details, refer to the [Playwright documentation](https://playwright.dev/python/docs/api/class-browser#browser-new-context). This option should not be used if browser_pool is provided.

https://crawlee.dev/python/api/class/PlaywrightCrawler#__init__

The appropriate keywords are documented on this page:
https://playwright.dev/python/docs/api/class-browser#browser-new-context

One of those keywords is ** storage_state**.

But...

When I pass the browser_new_context_options to the PlaywrightCrawler.init method, it calls some other function called...

[crawlee.crawlers._playwright._playwright_crawler] WARN Retrying request to ___ due to: BrowserType.launch_persistent_context() got an unexpected keyword argument 'storage_state'. File "./crawlee/browsers/_playwright_browser.py", line 70, in new_context, self._context = await self._browser_type.launch_persistent_context(, ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~^, user_data_dir=user_data_dir, **launch_options, ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^, )

Please update the code to match the documentation.

[sushant-mutnale]

Hi there! I've investigated the traceback and confirmed the issue. It looks like browser_new_context_options (intended for Playwright's new_context) are being incorrectly passed to launch_persistent_context within crawlee/browsers/_playwright_browser. py. This leads to the TypeError since launch_persistent_context doesn't support parameters like storage_state.

I'm starting work on a fix to ensure these options are applied correctly to the appropriate Playwright methods. I'll follow up with a PR soon!

[Pijukatel]

Hello,

Playwrights methods new_context and launch_persistent_context have overlapping sets of arguments, but they are not the same. In our code, this is oversimplified as browser_new_context_options can be passed to both, which works for the overlapping set of the arguments.

Until any code change I can offer you two workarounds suggestions:

1. Use use_incognito_pages=True argument, which is compatible with storage_state

crawler = PlaywrightCrawler(
    ...
    browser_new_context_options={"storage_state":{...}},
    use_incognito_pages=True
)

2. Set up storage state manually in user_data_dir if using persistent context (use_incognito_pages=False)

[Pijukatel]

Btw. using storage_dir for persistent context is requested, but not implemented feature in Playwright microsoft/playwright#14949