Train a language identifier model that works well on ingredient lists

[raphael0202]

Problem

We're currently using fasttext for language identification.
This is useful especially to detect the language of an ingredient list extracted automatically using a ML model, or added by a contributor.

However, fasttext was trained on data that is quite different from ingredient lists (Wikipedia, Tatoeba and SETimes).

Sometimes the model fails for obvious cases, such as this one (french ingredient list):

text: fraise (12%), framboise (10%)

predictions:
en, confidence=0.4291181
it, confidence=0.13040087
fr, confidence=0.0435654
ro, confidence=0.026255628
no, confidence=0.019594753
de, confidence=0.017750196
es, confidence=0.01671417
tr, confidence=0.015862297
sco, confidence=0.01577331
ms, confidence=0.015433003

This behaviour is mostly present for short ingredient lists.

We should explore training a new model for language identification using Open Food Facts data (especially ingredient lists).

Requirements

Using fasttext is not a requirement. We can either train a new fasttext model, or train it with pytorch/tensorflow and export it to ONNX format.

[korablique]

What I'm going to try (following the discussion on Slack):

• select products from OFF with >80% of recognized ingredients
• measure existing fasttext model quality on this data
• fine tune the fasttext model
• compare results

@raphael0202 If that's OK with you, could you please assign the issue to me?

[raphael0202]

Yes that's a good plan to start 👍
I've assigned you on this issue

[korablique]

Here is the number of texts for each language

en    422020
fr    299681
de     89880
es     46255
it     31801
nl     19983
pl      8401
pt      8119
sv      6128
bg      4453
ro      3771
fi      3726
ru      3610
nb      3591
cs      3500
th      3157
da      2021
hr      2015
hu      1962
ar      1104
el       943
ja       912
ca       824
sr       735
sl       727
sk       606
tr       506
lt       453
zh       436
et       370
lv       333
xx       318
no       315
uk       274
id       262
he       209
vi       121
is       113
la        89
in        72
ko        71
sq        70
iw        59
ka        54
ms        52
bs        37
fa        35
bn        33
gl        32
kk        25
mk        23
nn        18
hi        18
aa        17
uz        17
so        15
af        12
eu        11
az         8
be         7
cy         7
hy         7
tt         6
ku         5
km         4
te         4
ky         4
ur         4
mg         3
ty         3
ta         3
tg         3
my         3
tl         3
mo         2
sc         2
ir         2
ne         2
tk         2
am         2
mn         2
co         2
se         2
si         2
fj         1
ch         1
ug         1
yi         1
to         1
fo         1
mt         1
ht         1
ak         1
jp         1
oc         1
lb         1
mi         1
as         1
yo         1
ga         1
gd         1
ba         1
zu         1
mr         1

[baslia]

Hey, is that a requirements that the models needs to run locally (not from a server) ?
It seems that LLMs are good with detecting languages.

[korablique]

Tried to use clustering to fix mislabeled data.

I took the languages for which there are at least 100 texts (37 languages). Then took 100 texts for each language and used them as a training dataset (wanted then to get predictions for the entire dataset).

The texts were converted to embeddings using fasttext (get_sentence_vector method), the dimension was reduced from 256 to 66 to preserve 95% variance using PCA.
Tried 2 methods: gaussian mixture and HDBSCAN.
Gaussian mixture divides the data into only 3 clusters. And HDBSCAN classifies all new data as noise. The picture below shows the result of HDBSCAN clustering of training data. The clusters are difficult to separate.

Either clustering is not suitable for this task, or I am doing something wrong.

Now I will try to use another text classification model: lingua https://github.com/pemistahl/lingua-py to compare the predictions and confidence of two models. Then I'll take data in which the predictions of the models coincide and they are both confident and fine-tune one of them on this data.

[jeremyarancio]

Here's a really nice article summarizing different approaches for language detection, from statistical to deep learning
https://medium.com/besedo-engineering/language-identification-for-very-short-texts-a-review-c9f2756773ad

Would be great to have a validation dataset to estimate the performance of any solution.
This dataset can be manually annotated using https://languagetool.org/

[korablique]

How I got the distribution of texts languages:

1. selected ingredients_text_{LANG} field names from mongo DB:

docker exec -i mongodb-container mongo mydatabase --quiet --eval '
var cursor = db.mycollection.aggregate([
  { "$project": {
      "fields": { "$objectToArray": "$$ROOT" }
  }},
  { "$unwind": "$fields" },
  { "$match": { "fields.k": /^ingredients_text_/ }},
  { "$group": {
      "_id": null,
      "all_fields": { "$addToSet": "$fields.k" }
  }},
  { "$limit": 20 }
]);
if (cursor.hasNext()) {
  printjson(cursor.next().all_fields);
}' > field_names.json

2. then selected field values:

FIELDS=$(jq -r '.[]' field_names.json | paste -sd "," -)
docker exec -i mongodb-container mongo mydatabase --quiet --eval '
var fields = "'$FIELDS'".split(",");
var projection = {};
fields.forEach(function(field) { projection[field] = 1; });

var cursor = db.mycollection.find({}, projection).forEach(function(doc) {
    var cleanedDoc = {};
    fields.forEach(function(field) {
        if (doc[field] && doc[field] !== "") {
            cleanedDoc[field] = doc[field];
        }
    });
    if (Object.keys(cleanedDoc).length > 0) {
        printjson(cleanedDoc);
    }
});' > filtered_extracted_values.json

(but after that there are still some extra fields left, e.x. ingredients_text_with_allergens)

3. then I made a dictionary in which text is the key, language is the value:

ingredients_text_lang_dct = dict()

with open(os.path.join(data_dir, 'filtered_extracted_values.json'), 'r') as data_file:
    for dct in ijson.items(data_file, 'item'):
        for k, v in dct.items():
            if k == 'ingredients_text_with_allergens':
                continue
            lang = k[k.rfind('_') + 1:]

            # if the field is `ingredients_text_{LANG}_imported`
            if lang == 'imported':
                start = k[:k.rfind('_')].rfind('_') + 1
                end = k.rfind('_')
                lang = k[start:end]
            ingredients_text_lang_dct.update({v: lang})

@raphael0202

[korablique]

Would be great to have a validation dataset to estimate the performance of any solution. This dataset can be manually annotated using https://languagetool.org/

How many samples should it contain? Should I select an equal number of samples for each language or just random?
@jeremyarancio

[jeremyarancio]

Roughly 30 labels per language to start with I would say.
It's just to have an idea about the performances

Would be great to have a validation dataset to estimate the performance of any solution. This dataset can be manually annotated using https://languagetool.org/

How many samples should it contain? Should I select an equal number of samples for each language or just random?
@jeremyarancio

[baslia]

Here is the number of texts for each language

en    422020
fr    299681
de     89880
es     46255
it     31801
nl     19983
pl      8401
pt      8119
sv      6128
bg      4453
ro      3771
fi      3726
ru      3610
nb      3591
cs      3500
th      3157
da      2021
hr      2015
hu      1962
ar      1104
el       943
ja       912
ca       824
sr       735
sl       727
sk       606
tr       506
lt       453
zh       436
et       370
lv       333
xx       318
no       315
uk       274
id       262
he       209
vi       121
is       113
la        89
in        72
ko        71
sq        70
iw        59
ka        54
ms        52
bs        37
fa        35
bn        33
gl        32
kk        25
mk        23
nn        18
hi        18
aa        17
uz        17
so        15
af        12
eu        11
az         8
be         7
cy         7
hy         7
tt         6
ku         5
km         4
te         4
ky         4
ur         4
mg         3
ty         3
ta         3
tg         3
my         3
tl         3
mo         2
sc         2
ir         2
ne         2
tk         2
am         2
mn         2
co         2
se         2
si         2
fj         1
ch         1
ug         1
yi         1
to         1
fo         1
mt         1
ht         1
ak         1
jp         1
oc         1
lb         1
mi         1
as         1
yo         1
ga         1
gd         1
ba         1
zu         1
mr         1

Would it be possible to share the link to this original data set ? I am curious to have a look to it as well.
Thanks!