Scraping competitions from additional pages

[ScottishWolverine]

Currently the competitions scraper only chooses countries from the first page of a confederation (25 of them) and then scrapes the competitions from those countries. I've tried to extend the functionality to move through the different competitions pages so these can be scraped but unfortunately, I am not familiar enough with the framework to do it myself. Would you be able to provide some assistance for how I might achieve this?

[dcaribou]

Hi @ScottishWolverine,

Can you explain what you have tried and the problem you've found?

[ScottishWolverine]

Hi @dcaribou , thanks for the quick reply. I had initially thought about code to adapt the URL and access the additional pages but the URL does not appear to change when skipping cycling. I think some form of coding that can click the "next" button is needed but that's beyond my Python capabilities with the scrapy framework. Would you have any advice for how to implement this?

[dcaribou]

I think what you want is to "follow" the link. I'd suggest that you checkout the docs for following links in scrapy website.