New request: Plants for a Future

[Samandamand]

• Website URL: https://PFAF.org
• License: Creative Commons
• Desired ZIM Title: PFAF
• Desired ZIM Description: free-to-use information database for those interested in edible and useful plants.
• Desired ZIM Icon –png (URL or attach one): https://www.plantetp.eu/wp-content/uploads/2022/01/plants-for-the-future-e1679909954788.jpg
• Language (ISO 639-3): Eng
• Is this a MediaWiki?: No?

[RavanJAltaie]

Recipe created
https://farm.openzim.org/recipes/pfaf.org_en_all
Will update the library link once ready

[RavanJAltaie]

The recipe has a problem, we are troubleshooting here
#1009

[Samandamand]

I don't know if this is any help but this person seems to have scraped the data from PFAF.org https://github.com/saulshanabrook/pfaf-data

[RavanJAltaie]

@benoit74 Does this make any progress to our case?
We had a problem that the website is using cloudflare

[benoit74]

Yes, this is a great news because this means we could definitely create a custom scraper. But as you know, this is a significant effort.

I suggest to wait for zimit2 and check what happens with this new scraper version, maybe it will work way better regarding Cloudflare (it proved to in some cases).

If it doesn't work better, then we will tag this issue "scraper needed" and wait for funding / an external contributor / ...

[benoit74]

I tried to run this again with Zimit2 and Crawler 1.x

Task is at https://farm.openzim.org/pipeline/7a856c5a-6dd4-459e-98b1-7f3fae593ab1

ZIM is at https://dev.library.kiwix.org/viewer#pfaf.org_en_all_2024-05 or https://dev.library.kiwix.org/#lang=eng&q=pfaf or https://mirror.download.kiwix.org/zim/.hidden/dev/pfaf.org_en_all_2024-05.zim

This time the crawling worked, but it makes me realize that website is highly dynamic and mainly centered around a search engine. We cannot hence scrape all plants successfully and the UI is mostly broken since the search relies on an online server which won't work inside the ZIM (EDIT: with zimit scraper).

Since it looks like we might benefit from another source of information to retrieve the DB, I propose to tag this "scraper_needed" and to wait (could take a long time, current lead time is ... years) for someone to create a scraper for this website, either by using existing github repo or by crawling the website with specific code.