Data source: web scraping 🚫

[raphodn]

One source of data could be web scraping.

Allowed ? NO

⚠️ Currently we don't allow this as a source ⚠️

• because of legal reasons
• we focus on prices in physical shops

Additional info

In the about page & in the OFF support

For legal and technical reasons, we don't consider scraping prices from retailers' websites as a valid way to contribute to Open Prices. We want to make sure that the prices we collect are accurate and up-to-date, and receiving scraped prices from contributors doesn't allow us to do that.
Price scraping is a considered option in a future version of Open Prices, but it would be done by Open Prices itself so that we can have a proof of the price based on the HTML page.

Related issues

• Automatically import prices from external grocery store APIs / Websites #512

[raphodn]

But if this were to change, here is a list of shops where data (product EAN, price) seems to be available online :

Shop	Flavor(s)	Link	EAN ? (URL, web page, code)	Comments
Intermarché (France)	OxF	3700214618240	- in the URL - in the code
Chronodrive (France)	OxF	3502110004239	- in the web page
Biocoop (France)	OxF	3760076003417	- in the code (image URL)
Naturalia (France)	OxF	3760076004698	- in the code (image URL)
Satoriz (France)	OxF	5411788026253	- in the web page
Sherpa (France)		7613031423958	- in the URL - in the code	- prestashop
Mon Epicerie Paris (France)	OFF	4020943139359	- in the URL - in the code
Picard (France)	OFF	3270160168125	- in the code (gtin13)
La Fourche (France)	OxF	3760324840139	- in the code
O'zando (France)	OFF	8720812311584	- in the web page
Castorama	OPF	3086960203702	- in the URL - in the web page
Leroy Merlin	OPF		- in the code ?
Botanic	OxF	3086960225285	- in the code
LaSante.net	OBF, OPF	3551101034366	- in the web page

• https://www.orientalpanda.com/fr/
• https://www.joueclub.fr/
• https://www.king-jouet.com/

[raphodn]

https://github.com/badlogic/heissepreise

• https://shop.rewe.de/api/products/?search=4002846034603
• https://www.aldi-nord.de/ (seems to not provide GTINs)

[CloCkWeRX]

alltheplaces.xyz does a lot of scraping where business have published schema.org data of their location.

The equivalent would be

• https://schema.org/Offer (has price)
• https://schema.org/Product

Google encourages folks to do this:
https://developers.google.com/search/docs/specialty/ecommerce/include-structured-data-relevant-to-ecommerce

I would guess a significant chunk of supermarkets publish some price data like this.

[CloCkWeRX]

Example of one of your URLs:

https://validator.schema.org/#url=https%3A%2F%2Fwww.botanic.com%2Fproduit%2F231168%2Fpot-blanc-eda-toscane-o-80-x-h-66-cm.html

[CloCkWeRX]

As a proof of concept, I created https://github.com/CloCkWeRX/alltheprices based on alltheplaces.xyz.

Example output:

2025-04-06 03:21:20 [scrapy.core.engine] DEBUG: Crawled (200) <GET https://www.botanic.com/produit/448109/hibiscus.html> (referer: https://www.botanic.com/Assets/Rbs/Seo/100541/fr_FR/Rbs_Catalog_Product.1.xml)
2025-04-06 03:21:20 [scrapy.core.scraper] DEBUG: Scraped from <200 https://www.botanic.com/produit/448109/hibiscus.html>
{'extras': {'@source_uri': 'https://www.botanic.com/produit/448109/hibiscus.html'},
 'image': 'https://botanic-botanic-storage.omn.proximis.com/Imagestorage/imagesSynchro/0/0/ea984d27164cea9bf70e210c6316a365b823d761_1536742.png',
 'name': 'HIBISCUS',
 'offers': {'@type': 'Offer',
            'availability': 'https://schema.org/InStock',
            'price': 23.99,
            'priceCurrency': 'EUR',
            'url': 'https://www.botanic.com/produit/448109/hibiscus.html'},
 'ref': '448109',
 'website': 'https://www.botanic.com/produit/448109/hibiscus.html'}

I will think about the best identifiers (gtin) to require, write a few more, think about filtering or mapping to openfoodfacts equivalent categories

[CloCkWeRX]

Did the first crawl via https://github.com/CloCkWeRX/alltheprices/releases/tag/2025-04-13

Unfortunately, GTINs are few and far between. Need to do a bit of work for normalisation of naming, extracting other attributes, etc. Need to think about offers structure(s)

[raphodn]

Hi, thanks for the investigation!
I didn't know about schema.org 👀
In any case, we won't be able to upload them anytime soon... :(

[CloCkWeRX]

So, I figure I'm going to make a complementary dataset slowly over time. What would be interesting is seeing where schemas can just be aligned or common services used; so that UIs could display/query both without having combined them in the ODBL sense.

[leonarf]

Hello, I have written a script that gather EAN and price per kilogram for 3 type of shop (carrefour, U, and intermarche) through their website. It use https://github.com/ultrafunkamsterdam/nodriver to navigate throught their website, selecting a given shop, and making request in the search bar. If you're interested, I can share it.