iFrames?

[thomasrice]

Is there a way to easily scrape content from iFrames?

E.g. This page loads documents in an iFrame once you scroll down, which I can see if I launch it in non-headless mode and have a page_action to scroll down, but the returned page doesn't seem to have the contents of the iFrame.

I guess one way to do this it to check for page.css("iframe") then re-fetch the iframe URLs directly if they seem relevant.

[D4Vinci]

Yes, that's your best approach here. IFrames are almost always problematic in Web Scraping since they have their own policies and so on that the browser respects automatically