[Feature Request]: Remove GDPR Banners

[nikolakretschmer]

What needs to be done?

Create a CrawlerRunConfig option/feature to reliably remove GDPR / Cookie-Consent popups and banners from crawled sites.

What problem does this solve?

With these banners/popups present the resulting markdown will often contain content of the cookie popup and not the actual site content.

Target users/beneficiaries

EU users

Current alternatives/workarounds

I've tried every trick under the sun to remove these obnoxious popups. Injecting JS-Code, loading a chrome/firefox extension, remove_overlay_elements ... nothing has worked so far. Until this fix is implemented, I might have to switch to playwright and use some other parser for the markdown.

Proposed approach

I don't know. You guys are the wizards.

[hansvangent]

Yes this please... those damn cookie banners are really preventing 99% of websites to be crawled and it is really unusable like this.