feat: Generate stores taxonomy from name-suggestion-index by CloCkWeRX · Pull Request #9607 · openfoodfacts/openfoodfacts-server

CloCkWeRX · 2023-12-31T07:28:42Z

Fixes #7632 ?

What

This adds a snapshot of the BSD 3 clause licenced name-suggestion-index records for shop/supermarket, transformed into a stores.txt taxinomy format.

There may be some mapping inconsistencies with countries.txt, as I used ISO3601 2 letter codes and added 1 or 2 minor adjustments.

While I added this as a snapshot in git, it would be trivial to fetch the live data from upstream.

Running/generating:

clockwerx@LAPTOP-2K4VT916:~/openfoodfacts-server/taxonomies$ node external/name-suggestion-index/shop/supermarket.js >> unused/stores.txt

What should be done to make this better

If there is a central list of languages from an i18n package or similar, that has mappings from countries, that should be used
Use of countries.txt or similar
Evaluate how important it is to keep stores.txt as is, vs just say, adopting the name-suggestion-index format.
Is the wikidata syntax right?
eslint?

…stion-index/main/data/brands/shop/supermarket.json @ 32150a9

codecov-commenter · 2023-12-31T08:48:59Z

Codecov Report

✅ All modified and coverable lines are covered by tests.
✅ Project coverage is 55.49%. Comparing base (0c7be85) to head (ea3c536).
⚠️ Report is 3 commits behind head on main.
✅ All tests successful. No failed tests found.

Additional details and impacted files

@@            Coverage Diff             @@
##             main    #9607      +/-   ##
==========================================
+ Coverage   50.30%   55.49%   +5.18%     
==========================================
  Files         103      104       +1     
  Lines       26398    26456      +58     
  Branches     6315     6324       +9     
==========================================
+ Hits        13280    14681    +1401     
+ Misses      11357     9850    -1507     
- Partials     1761     1925     +164

Flag	Coverage Δ
integration-test-group-1	`13.09% <ø> (?)`
integration-test-group-2	`9.21% <ø> (?)`
integration-test-group-3	`9.38% <ø> (?)`
integration-test-group-4	`33.42% <ø> (?)`
integration-test-group-5	`28.77% <ø> (?)`
integration-test-group-6	`9.14% <ø> (?)`
integration-test-group-7	`9.63% <ø> (?)`
integration-test-group-8	`28.31% <ø> (?)`
integration-test-group-9	`26.64% <ø> (?)`
unit-test-group-1	`25.02% <ø> (ø)`
unit-test-group-2	`31.28% <ø> (ø)`
unit-test-group-3	`33.59% <ø> (ø)`
unit-test-group-4	`22.29% <ø> (-0.01%)`	⬇️
unit-test-group-5	`28.30% <ø> (ø)`
unit-test-group-6	`21.52% <ø> (ø)`

Flags with carried forward coverage won't be shown. Click here to find out more.

☔ View full report in Codecov by Harness.
📢 Have feedback on the report? Share it here.

🚀 New features to boost your workflow:

📦 JS Bundle Analysis: Save yourself from yourself by tracking and limiting bundle sizes in JS merges.

sonarqubecloud · 2024-01-02T07:45:56Z

Quality Gate passed

Kudos, no new issues were introduced!

0 New issues
0 Security Hotspots
No data about Coverage
0.0% Duplication on New Code

See analysis details on SonarCloud

alexgarel · 2024-02-14T13:09:23Z

@teolemon can you review this PR ? I personally lack context to understand.

teolemon · 2024-02-14T18:01:23Z

cc @raphodn

teolemon

It looks really nice to me, I just required a review from the 2 Raphael from Open Prices, since this will potentially be really helpful for them.
What was your personal intent behind this PR ?

raphael0202 · 2025-07-01T12:02:31Z

I suspect what you have done with open prices supercedes this a bunch, as you model the specific OSM way or node ID, from the looks of it?

It's not really a superseed of it, as we currently have no way of knowing whether a specific store belongs to a bigger chain.

Basically, on Open Prices, I would like to have a new table that could be named something like store_brands, where a specific store in the stores table could be linked to this item.
This would open up the new features on Open Prices described above (and also help you on your use cases).

I will open a new discussion on Open Prices repo to discuss the matter.

edit: Done here: openfoodfacts/open-prices#863

raphodn · 2025-12-14T01:35:07Z

sorry, seeing this PR only now, this is really cool.
one thing we mentioned recently with @teolemon & @aleene is to transform the taxonomy to xx:
doable ?

raphodn · 2025-12-14T01:39:31Z

I wouldn't restrict to just shop/supermarket.json

OFF: shop/convenience, shop/alcohol, shop/wine, shop/bakery...

OBF: shop/beauty

OPF: shop/doityourself, shop/toys, shop/sports, shop/books...

OPFF: shop/pet

==> take all the shop/*

So, can do that, would the preference be 1x script per topic; or all in one?

all in one ? 😁
with a nsi_kv: shop/convenience ?

Definitely all in one. It would lead to a lot of duplication (or complication) if store-brands had to be repeated for each OxF type they were applicable for (e.g., most supermarkets/grocery or general stores/etc. would fit in all of them).

raphodn · 2025-12-14T01:43:11Z

I'm a big fan of the idea of leveraging NSI 👍
fyi I'm regularly contributing to the project, see openfoodfacts/open-prices#467 (comment)

Maybe store a nsi_id: carrefour-c622af to know that it comes from NSI ?

to easily find it back

and differentiate in the taxonomy shops coming from NSI from shops coming from OFF contributors ?

thus knowing which shops could be added back to NSI

Happy to split these out into an stores-nsi.txt or something of that nature to more clearly split it; I feel like at the time I did this manually.

Script likely needs to be re-run since it's been ~2 years.

I can have a look later this week and try to re-run the script myself.
if we switch to xx: then the country mapping could be removed ?

if we switch to xx: then the country mapping could be removed ?

I think country mapping could still be useful, esp. for cases where store-brands may have identical or near-identical names in different regions (see e.g. the 50 or so “COOP”s). (Mappings like country:en: en:United States of America,en:United States of America,en:United States of America,en:United States of America,en:United States of America are probably a bit excessive though…). Maybe this is too much “preemptive optimisation(/complication)” though, and could be easily solved with a human-added comment:en: key or similar where it’s actually needed.

Definitely +1 to using xx: similar to our brands.txt taxonomy though, with possible locale specific overrides for locales with scripts other than the first/primary xx: alias where relevant.

sonarqubecloud · 2025-12-14T05:01:14Z

Quality Gate passed

Issues
0 New issues
0 Accepted issues

Measures
0 Security Hotspots
0.0% Coverage on New Code
0.0% Duplication on New Code

See analysis details on SonarQube Cloud

aleene · 2025-12-14T15:29:24Z

Let us go ahead with this and readjust later.

teolemon · 2026-05-07T13:49:27Z

@raphodn @CloCkWeRX @aleene @Freso @raphael0202 Should I do as suggested by @aleene , approving and merging ? Otherwise, can someone else do so ?

sonarqubecloud · 2026-05-07T13:51:11Z

Quality Gate passed

Issues
0 New issues
0 Accepted issues

Measures
0 Security Hotspots
0.0% Coverage on New Code
0.0% Duplication on New Code

See analysis details on SonarQube Cloud

Freso · 2026-05-08T08:40:18Z

I’m fine with whatever!

teolemon

Approving this one since it's peripheral and useful

teolemon · 2026-07-06T08:23:27Z

+wikidata:en:Q483551
+country:en: en:China
+
+117.1140042,36.6507007,50:济南华联超市


@CloCkWeRX issue here

teolemon

some issues to fix

Copilot

Pull request overview

This PR adds a large snapshot of Name Suggestion Index (NSI) “shop/supermarket” brands into the stores.txt taxonomy format (plus a small amount of Wikidata enrichment for existing entries) and introduces a Node.js helper script intended to (re)generate that snapshot from NSI JSON.

Changes:

Extend taxonomies/unused/stores.txt with ~3.5k NSI-derived store entries and add a few missing wikidata:* properties for existing stores.
Add taxonomies/external/name-suggestion-index/shop/supermarket.js to transform NSI JSON into stores.txt-style blocks.
Update .github/labeler.yml so NSI taxonomy generator files are labeled as “Stores” changes.

Reviewed changes

Copilot reviewed 2 out of 4 changed files in this pull request and generated 9 comments.

File	Description
taxonomies/unused/stores.txt	Adds NSI-derived store taxonomy snapshot and some Wikidata properties; currently includes malformed entry headers that will be ignored by the taxonomy parser.
taxonomies/external/name-suggestion-index/shop/supermarket.js	Adds a generator script; currently mishandles non-ISO `locationSet.include` values, producing invalid `stores.txt` entry headers.
.github/labeler.yml	Expands Stores labeler globs to include the new NSI generator directory.

💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.

+// Do not use results without careful review
+const data = require('./supermarket.json');
+
+// A quick ISO 3601 to human labels (english) mapping.


+data["items"].forEach(function (record) {
+  var countryCodes = record.locationSet["include"];
+  var countryNames = [];
+  countryCodes.forEach(function (code) {
+	if (code == "001") { return; }
+
+	if (!iso3601[code.toString().split("-")[0]]) {
+		// console.debug("Unmapped ISO3601 code: " + code);
+		return;
+	} else {
+		countryNames.push("en:" + iso3601[code.toString().split("-")[0]]);
+	}
+  });
+
+  var primaryCountryCode = countryCodes[0].toString().split("-")[0];
+  var primaryLanguageCode = primaryCountryCode;
+  if (iso3601_to_language_mappings[primaryCountryCode]) {
+    primaryLanguageCode = iso3601_to_language_mappings[primaryCountryCode]
+  }
+


+001:Amazon Fresh
+wikidata:en:Q4740834
+


+foodland_eastern_us.geojson:FoodLand (USA)
+wikidata:en:Q5465271
+


+:masymas (Hijos de Luis Rodríguez)
+wikidata:en:Q6135987
+


+:masymas (Juan Fornés Fornés)
+wikidata:en:Q6135987
+


+:masymas (Luis Piña)
+wikidata:en:Q6135987
+


+117.1140042,36.6507007,50:东方冷库
+wikidata:en:Q123025086
+


+117.1140042,36.6507007,50:济南华联超市
+wikidata:en:Q123025079
+


…ba3d3546fb10

…permarket data processing

…egeneration Merge origin/main into feature/7632-stores-taxinomy. The only conflicted path was taxonomies/unused/stores.txt; everything else merged cleanly and is staged as-is. Reasoning per side: - origin/main touched taxonomies/unused/stores.txt with whitespace-only changes: it collapsed duplicated consecutive blank lines inside the hand-written section of the taxonomy. A diff between the merge base (a4879ce) and main shows exclusively deletions of empty lines, no additions or modifications of content lines, so there was no semantic change from main to reconcile. - This PR branch (HEAD) made two kinds of changes to the same file, which must both be preserved because the file was partly regenerated from the name-suggestion-index (NSI) dataset in this branch: 1. Added missing "wikidata:en:" identifiers to existing hand-written entries (Bio C' Bon Q54312551, Hyper U / U Express / Super U Q2529029, La vie claire Q3213589, Delhaize Q1184173, Colruyt Q2363991). 2. Appended the large "# Stores via name-suggestion-index" block at the end of the file, generated from data/brands/shop/supermarket.json at NSI commit a86dd15d303126f8cfc7831aa395ba3d3546fb10 (~4k store entries with Wikidata IDs and country coverage). Resolution applied: - Kept the entire NSI-generated block from HEAD verbatim. Since it was regenerated in this PR from the pinned NSI snapshot, re-editing it by hand would desynchronize it from its source; main contained nothing in that region anyway. - Kept main's blank-line normalization in the hand-written top section, so the file stays tidy where git auto-merged both sides already. - Removed only the three conflict marker lines ("<<<<<<< HEAD", "=======", ">>>>>>> main"); no entry, property or country line was altered beyond that. Resulting file = PR branch content + main's whitespace cleanup, i.e. the union of both sides without losing anything from either. Merge conflict resolved by Ox Alpha Free (Unlimited), running via OpenCode Zen. Co-authored-by: hangy <hangy@hangy.de>

sonarqubecloud · 2026-08-22T13:27:42Z

Quality Gate passed

Issues
0 New issues
0 Accepted issues

Measures
0 Security Hotspots
0.0% Coverage on New Code
0.0% Duplication on New Code

See analysis details on SonarQube Cloud

Freso · 2026-08-22T14:03:10Z

+}
+
+// Overrides for the language codes/mappings inferred by country code.
+// TODO: This list is far, far from accurate or complete and should be replaced with any internal/authoritative mappings


Would it be possible to use the language property from the countries taxonomy?

Freso · 2026-08-22T14:03:43Z

+
+// A quick ISO 3601 to human labels (english) mapping.
+// TODO: Ensure this matches exactly with countries.txt
+// TODO: Replace with a better lookup


Could this utilise the countries taxonomy?

CloCkWeRX added 4 commits December 31, 2023 17:13

Add a snapshot of https://raw.githubusercontent.com/osmlab/name-sugge…

2754512

…stion-index/main/data/brands/shop/supermarket.json @ 32150a9

Add a generator

1f028eb

Comment out debugging for the moment

e4d3bb6

Add initially generated stores from OSM NSI

19380cc

github-actions Bot assigned CloCkWeRX Dec 31, 2023

github-actions Bot added 🧬 Taxonomies https://wiki.openfoodfacts.org/Global_taxonomies JavaScript labels Dec 31, 2023

CloCkWeRX added 2 commits December 31, 2023 18:14

Unify naming

c9893a0

Append generated

d0d0ee2

github-actions Bot added the Stores label Dec 31, 2023

CloCkWeRX added 3 commits December 31, 2023 18:32

Deal with english speakers vs country

b4eae74

Fix script

793992e

Append generated stores

6863091

CloCkWeRX requested a review from a team as a code owner December 31, 2023 08:03

github-actions Bot added the GitHub Actions Pull requests that update Github_actions code label Dec 31, 2023

CloCkWeRX added 6 commits December 31, 2023 18:36

Only render country if available

9cf7066

Only render wikidata if available

af49be9

English in the UK

94f0f09

Regenerated list

c157240

Merge lists

de13ba8

Merge lists

2db44a8

Merge branch 'main' into feature/7632-stores-taxinomy

030f451

CloCkWeRX mentioned this pull request Jan 6, 2024

Taxonomize stores - Add autosuggest for stores #7632

Open

alexgarel changed the title ~~feat: Generate stores taxinomy from name-suggestion-index~~ feat: Generate stores taxonomy from name-suggestion-index Feb 6, 2024

teolemon requested a review from raphael0202 February 14, 2024 17:56

teolemon reviewed Feb 14, 2024

View reviewed changes

teolemon moved this to In Progress in JavaScript issues across Open Food Facts Jul 8, 2025

raphodn reviewed Dec 14, 2025

View reviewed changes

Merge branch 'main' into feature/7632-stores-taxinomy

9fcb62a

github-actions Bot removed the 💥 Merge Conflicts 💥 Merge Conflicts label Apr 20, 2026

Merge branch 'main' into feature/7632-stores-taxinomy

cada148

teolemon approved these changes Jul 6, 2026

View reviewed changes

github-project-automation Bot moved this from PRs to Done in 🍊 Open Food Facts Server issues Jul 6, 2026

teolemon reviewed Jul 6, 2026

View reviewed changes

teolemon requested a review from Copilot July 6, 2026 08:23

Copilot started reviewing on behalf of teolemon July 6, 2026 08:24 View session

Copilot AI reviewed Jul 6, 2026

View reviewed changes

github-actions Bot added the 💥 Merge Conflicts 💥 Merge Conflicts label Aug 20, 2026

hangy and others added 5 commits August 22, 2026 14:39

chore: Update NSI supermarkets to commit a86dd15d303126f8cfc7831aa395…

e3decb0

…ba3d3546fb10

typo

836c86f

fix: enhance country code validation and language code handling in su…

164d837

…permarket data processing

fix regexes and rerun script

ebe5279

github-actions Bot removed the 💥 Merge Conflicts 💥 Merge Conflicts label Aug 22, 2026

Freso reviewed Aug 22, 2026

View reviewed changes

		foodland_eastern_us.geojson:FoodLand (USA)
		wikidata:en:Q5465271

		117.1140042,36.6507007,50:东方冷库
		wikidata:en:Q123025086

		117.1140042,36.6507007,50:济南华联超市
		wikidata:en:Q123025079

		001:Amazon Fresh
		wikidata:en:Q4740834

		:masymas (Luis Piña)
		wikidata:en:Q6135987

Uh oh!

Uh oh!

Conversation

CloCkWeRX commented Dec 31, 2023 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

What

What should be done to make this better

Uh oh!

codecov-commenter commented Dec 31, 2023 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Codecov Report

Uh oh!

sonarqubecloud Bot commented Jan 2, 2024

Quality Gate passed

Uh oh!

alexgarel commented Feb 14, 2024

Uh oh!

teolemon commented Feb 14, 2024

Uh oh!

teolemon left a comment

Choose a reason for hiding this comment

Uh oh!

raphael0202 commented Jul 1, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

raphodn Dec 14, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

raphodn Dec 15, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

sonarqubecloud Bot commented Dec 14, 2025

Quality Gate passed

Uh oh!

aleene commented Dec 14, 2025

Uh oh!

teolemon commented May 7, 2026

Uh oh!

sonarqubecloud Bot commented May 7, 2026

Quality Gate passed

Uh oh!

Freso commented May 8, 2026

Uh oh!

teolemon left a comment

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

teolemon left a comment

Choose a reason for hiding this comment

Uh oh!

Copilot AI left a comment

Choose a reason for hiding this comment

Pull request overview

Reviewed changes

Uh oh!

sonarqubecloud Bot commented Aug 22, 2026

Quality Gate passed

Uh oh!

Choose a reason for hiding this comment

CloCkWeRX commented Dec 31, 2023 •

edited

Loading

codecov-commenter commented Dec 31, 2023 •

edited

Loading

raphael0202 commented Jul 1, 2025 •

edited

Loading

raphodn Dec 14, 2025 •

edited

Loading

raphodn Dec 15, 2025 •

edited

Loading