Allow Proxy Middleware Configuration for Specific Spiders

[SergeyKapshuchenko]

Good day !

I would like to know if it's possible to configure the proxy middleware in a way that it applies only to specific spiders instead of all spiders in a Scrapy project.

For example, I want to have a toggle or setting for each spider that decides whether proxies should be used or not. Is there a recommended way to achieve this using the Scrapy framework?

Example Use Case:
Spider A should use proxies.
Spider B should not use proxies.
Would it be possible to configure the middleware dynamically per spider or implement logic to check this toggle and enable/disable the middleware accordingly? If yes, could you provide guidance or examples?

Thank you!

[devfox-se]

Yes you can simply keep this middleware off for any spider that you dont need it by overriding settings on sniper creation. You can tailor any setting for a specific spider using update_settings classmethod

Best practice for middlewares is that it should have an ENABLED flag/value which users could use to enable/disable the middleware but this one does not have such parameter, instead you can clear the list of proxies and the middleware will deactivate itself automatically based on its initialization logic

@classmethod
def update_settings(cls, settings):
    super().update_settings(settings)
    settings["ROTATING_PROXY_LIST"].clear()

[SergeyKapshuchenko]

Thanks !

[superclarkk]

Another way to do it in the spider:

    @classmethod
    def update_settings(cls, settings):
        super().update_settings(settings)
        settings["DOWNLOADER_MIDDLEWARES"]['rotating_proxies.middlewares.RotatingProxyMiddleware'] = None
        settings["DOWNLOADER_MIDDLEWARES"]['rotating_proxies.middlewares.BanDetectionMiddleware'] = None

[github-actions]

This issue has been marked stale and will be closed in 7 days unless updated.