Migration from webL10n to i18next #4459

ac-mmi · 2025-02-24T12:03:43Z

@walterbender I have successfully migrated the internationalization system in Music Blocks from webL10n to i18next. I have tested the implementation, and it is working as expected. The only remaining task is to update the test suites, which are currently tailored for webL10n, to support the new i18next standards.

I made the following changes:

In loader.js: Added i18next initialization with i18nextHttpBackend to load translations from JSON files stored in the locales folder. This change allows the application to switch languages dynamically.
In Multiple JavaScript Files: Replaced instances of webL10n functionality with i18next's function for translations. This change was made across several files(activity.js,block.js,blocks.js,ActionBlocks.js,BooleanBlocks.js and more...)
Created locales Folder: Added a new folder named locales containing converted PO files into JSON format, supporting i18next's structure for managing translations.

github-actions · 2025-02-24T12:04:28Z

❌ Some Jest tests failed. Please check the logs and fix the issues before merging.

Failed Tests:

synthutils.test.js
SaveInterface.test.js
VolumeActions.test.js
musicutils.test.js
ToneActions.test.js
PitchActions.test.js
MeterActions.test.js
rubrics.test.js
turtle-singer.test.js
DrumActions.test.js
languagebox.test.js
DictActions.test.js
themebox.test.js
turtledefs.test.js
macros.test.js

walterbender · 2025-02-24T13:13:59Z

I am not sure why you changed the _() function to t(). It is quite an invasive and seemingly unnecessary change.
Is there a standard way to convert the PO files to their JSON representation?
It seems (at least) the es file has some encoding issues.

ac-mmi · 2025-02-24T13:20:46Z

@walterbender I switched from _() to t() because it's the standard function used by i18next. If preferred, I can alias _() to t() to minimize changes, make a wrapper function.
For the PO to JSON conversion, I used a custom Python script, ensuring comments were ignored and keys matched the original msgids. If there’s a preferred method, I’m open to adjustments.
I’ll check the encoding issues in the es file.

ac-mmi · 2025-02-24T13:30:55Z

@walterbender If you were referring to encoding issues in es.po (if I assumed that correctly), I ran a thorough check on the PO files, including es.po, and didn't find any encoding issues using a custom script that detects unescaped quotes and encoding errors. The only issues found were in other files, like ayc.po, quz.po, and te.po, where unescaped double quotes were present.

walterbender · 2025-02-24T13:32:16Z

I think keeping the _() wrapper is a good idea.
I am fine with a simple Python script for conversion of the PO files, but it would be good to test some of the more language-specific PO file features which we deliberately avoided with WebL10n, such as handling plurals and word order.
Please do look into the encoding issue. I only noticed it with ES but it could be elsewhere as well.

walterbender · 2025-02-24T14:00:33Z

look t your es.json file and you'll see that it is no longer utf-8. It is some weird encoding.

"Are you sure you want to clear the workspace?": "Â¿EstÃ¡s segura de que quieres borrar el espacio de trabajo?",

Should be:

"Are you sure you want to clear the workspace?": "¿Estás segura de que quieres borrar el espacio de trabajo?"",

github-actions · 2025-02-24T14:17:00Z

❌ Some Jest tests failed. Please check the logs and fix the issues before merging.

Failed Tests:

synthutils.test.js
VolumeActions.test.js
SaveInterface.test.js
musicutils.test.js
ToneActions.test.js
PitchActions.test.js
MeterActions.test.js
rubrics.test.js
turtle-singer.test.js
DrumActions.test.js
languagebox.test.js
DictActions.test.js
themebox.test.js
turtledefs.test.js
macros.test.js

walterbender · 2025-02-24T14:20:49Z

Not sure what you are doing to fix the encodings, but it is not correct. There should not be any strings with "Â" in them. It is not a glyph used in Spanish.

github-actions · 2025-02-24T15:05:20Z

❌ Some Jest tests failed. Please check the logs and fix the issues before merging.

Failed Tests:

synthutils.test.js
VolumeActions.test.js
SaveInterface.test.js
musicutils.test.js
ToneActions.test.js
PitchActions.test.js
MeterActions.test.js
rubrics.test.js
DrumActions.test.js
turtle-singer.test.js
languagebox.test.js
DictActions.test.js
themebox.test.js
turtledefs.test.js
macros.test.js

ac-mmi · 2025-02-24T15:08:12Z

@walterbender Could you please let me know the encoding format used in the PO files? This will help me improve the Python script for conversion and avoid similar issues in the future.

walterbender · 2025-02-24T15:26:00Z

Everything should be utf-8

omsuneri · 2025-02-24T16:21:31Z

@ac-mmi I saw many tests failures please try to resolve those along with the PR only.

omsuneri · 2025-02-24T16:30:19Z

@walterbender do we really need this change currently as I don't feel so
as this is part of one of the projects we have in GSOC let's wait till the best ideas we get as it is very early to decide the changes at this moment also @ac-mmi I feel we need a discussion before moving to i18n right now..

omsuneri · 2025-02-24T16:31:51Z

@walterbender do we really need this change currently as I don't feel so currently
as this is part of one of the projects we have in GSOC let's wait till the best ideas we get as it is very early to decide the changes at this moment also @ac-mmi I feel we need a discussion before moving to i18n right now..

walterbender · 2025-02-24T17:09:12Z

It needs work.

…locks into internalization-js

github-actions · 2025-02-24T17:20:25Z

✅ All Jest tests passed! This PR is ready to merge.

ac-mmi · 2025-02-24T17:22:32Z

@walterbender i have made changes in the _() functionality catering to i18next functions and also passing the test suits of the repo

walterbender · 2025-02-24T17:25:41Z

ES is still broken. Now all of the non-ASCII characters are just missing.
We need to discuss a number of other details, including how to handle the Kana/Kanji situation. And language detection in general.

github-actions · 2025-02-24T17:50:03Z

✅ All Jest tests passed! This PR is ready to merge.

….js so that renderLanguageSelectIcon() works correctly

walterbender · 2025-04-04T12:02:15Z

did you look at the quality report? it looks pretty nonsensical.

ac-mmi · 2025-04-04T12:21:25Z

@walterbender In the current report, I included:

The original English sentence
The back-translated English (from Catalan)
A lexical similarity score and confidence value
A flag to highlight low-confidence matches

walterbender · 2025-04-04T12:51:17Z

The fields are correct. The data in the fields is not.

ac-mmi · 2025-04-04T13:08:33Z

@walterbender I'm flagging translations based on a combination of back-translation, lexical similarity with the original English, and spelling errors in the back-translation.

For example:

"Refresh your browser to change your language preference.": {
  "Catalan translation":"Actualitzeu el navegador per canviar la vostra preferència d’idioma."
  "back_translated": "Update your browser to change your language preference.",
  "lexical_score": 0.81,
  "confidence_score": 81.0,
  "flag_for_review": false
}

Here, it’s a semantic match, but the lexical similarity is a bit low due to the word "Refresh" → "Update".

ac-mmi · 2025-04-04T17:57:29Z

@walterbender this is the new report
translation_quality_report_ca.json

walterbender · 2025-04-04T18:23:40Z

This report is starting to make sense.

Note that we may want to always flag anything that is ambiguous (and maybe it will help us improve the translator notes as well. For example, I noticed that "duck" was not flagged for review, but do we know if it is duck the noun or duck the verb?

ac-mmi · 2025-04-04T18:45:19Z

@walterbender for this "duck" situation i am going for POS tagging what do you say ?

walterbender · 2025-04-04T20:45:57Z

I don't think it is reasonable to tag every string in the code with POS. It is reasonable to expect decent translator hints.

ac-mmi · 2025-04-05T14:44:15Z

@walterbender I created a separate file, en_translation_hints.json based on en.json, that pairs each English string with a short context hint. It includes things like:

Whether the string is a button label, an error message, or a tooltip
If it’s a full sentence, a short label, a placeholder, etc.
Clarifying terms like "Project undefined".

Context file: en_translation_hints.json

walterbender · 2025-04-05T20:19:45Z

Is the separate file for the AI? The hints should be in the PO files.

ac-mmi · 2025-04-06T08:42:52Z

@walterbender but for i18next i am using .json files

walterbender · 2025-04-06T11:55:10Z

I thought that we were going to use the PO framework for translators and i18next for the implementation. I assumed that the AI translation would show up in the PO files so that the translators can review them.

ac-mmi · 2025-04-06T12:25:54Z

@walterbender I initially converted all existing PO files into JSON(present in locales folder), and the AI-generated translations now work directly with JSON files.

ac-mmi · 2025-04-06T14:19:34Z

@walterbender in my ai translation code both .po and .json files are created

ac-mmi · 2025-04-06T15:16:18Z

@walterbender I have sent you the po file for catalan in the chat with the context included

walterbender · 2025-04-06T15:30:47Z

it is fine that the AI uses JSON and that you put the hints into JSON as long as the source of truth remains the POs -- since there is extensive PO file support in the translator community.
And we can ditch altogether the localization.ini all together.

ac-mmi · 2025-04-06T15:53:40Z

@walterbender Also have you seen my proposal ,is there anything i am lacking or should research on ?

walterbender · 2025-04-06T16:29:38Z

I've gone through it multiple times. Time to submit it.

…

On Sun, Apr 6, 2025 at 11:54 AM Aman Chadha(IVIXMMI) < ***@***.***> wrote: @walterbender <https://github.com/walterbender> Also have you seen my proposal ,is there anything i am lacking or should research on ? — Reply to this email directly, view it on GitHub <#4459 (comment)>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/AA6PXYNJ736UDYYCBZCR67L2YFEZVAVCNFSM6AAAAABXX3ECWOVHI2DSMVQWIX3LMV43OSLTON2WKQ3PNVWWK3TUHMZDOOBRGQ4DOMRUGU> . You are receiving this because you were mentioned.Message ID: ***@***.***> [image: ac-mmi]*ac-mmi* left a comment (sugarlabs/musicblocks#4459) <#4459 (comment)> @walterbender <https://github.com/walterbender> Also have you seen my proposal ,is there anything i am lacking or should research on ? — Reply to this email directly, view it on GitHub <#4459 (comment)>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/AA6PXYNJ736UDYYCBZCR67L2YFEZVAVCNFSM6AAAAABXX3ECWOVHI2DSMVQWIX3LMV43OSLTON2WKQ3PNVWWK3TUHMZDOOBRGQ4DOMRUGU> . You are receiving this because you were mentioned.Message ID: ***@***.***>

-- Walter Bender Sugar Labs https://www.sugarlabs.org <http://www.sugarlabs.org>

ac-mmi · 2025-04-06T16:39:34Z

@walterbender Thank you, I’ve submitted the proposal.

ac-mmi · 2025-05-05T10:11:10Z

@walterbender this is the updated context file Also the word 'in' in the translation refers to the Japanese scale ?
en_context_updated.json

walterbender · 2025-05-05T12:55:36Z

Not sure how you are generating your context document. Ideally it would come directly from translator hints as they same context is needed by human translators.

Yes, "in" is the name of a Japanese mode.

Lots of strings are simply tagged "General UI String". Not sure the utility of that.

ac-mmi · 2025-05-05T13:00:49Z

@walterbender I just provided you with the basic outline of the context file. Yes, it does need human intervention. I’ve been generating the context using the GPT API, which is focused on giving basic context

walterbender · 2025-05-05T13:12:00Z

What are you basing the prompt(s) on?

ac-mmi · 2025-05-05T13:24:36Z

@walterbender I described the application then provided chunks of the JSON translation file to the GPT API. I asked it to generate context for each string based on the app’s purpose and usage. Since the translation file is large, I’m passing it in chunks to make the process manageable.

github-actions · 2025-07-08T19:30:59Z

This pull request has been open for more than 60 days without any activity. It will be closed in 3 days unless the stale label is removed or commented on.

github-actions · 2025-07-11T19:32:02Z

Closed pull request due to inactivity for more than 63 days.

ac-mmi and others added 5 commits February 19, 2025 23:53

Add files via upload

d78919c

Add files via upload

ac566eb

Merge branch 'sugarlabs:master' into master

29a4416

Merge branch 'sugarlabs:master' into master

1678672

Migrate internationalization from webL10n to i18next in JS files

5ead512

removed random encodings spoiling the .po files

b844c94

Add files via upload

5cf011e

ac-mmi added 2 commits February 24, 2025 22:47

Changed the function definition of _() suited for i18next

1680230

Merge branch 'internalization-js' of https://github.com/ac-mmi/musicb…

d8bbb85

…locks into internalization-js

updated the i18next json files

d6b1114

changed the language switch functions to i18next standard in activity…

e6e8c5a

….js so that renderLanguageSelectIcon() works correctly

ac-mmi closed this May 8, 2025

ac-mmi reopened this May 9, 2025

github-actions bot added the Stale label Jul 8, 2025

github-actions bot closed this Jul 11, 2025

Uh oh!

Migration from webL10n to i18next #4459

Migration from webL10n to i18next #4459

Uh oh!

Conversation

ac-mmi commented Feb 24, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

github-actions bot commented Feb 24, 2025

Uh oh!

walterbender commented Feb 24, 2025

Uh oh!

ac-mmi commented Feb 24, 2025

Uh oh!

ac-mmi commented Feb 24, 2025

Uh oh!

walterbender commented Feb 24, 2025

Uh oh!

walterbender commented Feb 24, 2025

Uh oh!

github-actions bot commented Feb 24, 2025

Uh oh!

walterbender commented Feb 24, 2025

Uh oh!

github-actions bot commented Feb 24, 2025

Uh oh!

ac-mmi commented Feb 24, 2025

Uh oh!

walterbender commented Feb 24, 2025

Uh oh!

omsuneri commented Feb 24, 2025

Uh oh!

omsuneri commented Feb 24, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

omsuneri commented Feb 24, 2025

Uh oh!

walterbender commented Feb 24, 2025

Uh oh!

github-actions bot commented Feb 24, 2025

Uh oh!

ac-mmi commented Feb 24, 2025

Uh oh!

walterbender commented Feb 24, 2025

Uh oh!

github-actions bot commented Feb 24, 2025

Uh oh!

walterbender commented Apr 4, 2025

Uh oh!

ac-mmi commented Apr 4, 2025

Uh oh!

walterbender commented Apr 4, 2025

Uh oh!

ac-mmi commented Apr 4, 2025

Uh oh!

ac-mmi commented Apr 4, 2025

Uh oh!

walterbender commented Apr 4, 2025

Uh oh!

ac-mmi commented Apr 4, 2025

Uh oh!

walterbender commented Apr 4, 2025

Uh oh!

ac-mmi commented Apr 5, 2025

Uh oh!

walterbender commented Apr 5, 2025

Uh oh!

ac-mmi commented Apr 6, 2025

Uh oh!

walterbender commented Apr 6, 2025

Uh oh!

ac-mmi commented Apr 6, 2025

Uh oh!

ac-mmi commented Apr 6, 2025

Uh oh!

ac-mmi commented Apr 6, 2025

Uh oh!

walterbender commented Apr 6, 2025

Uh oh!

ac-mmi commented Apr 6, 2025

ac-mmi commented Feb 24, 2025 •

edited

Loading

omsuneri commented Feb 24, 2025 •

edited

Loading

ac-mmi commented Apr 6, 2025 •

edited

Loading

ac-mmi commented May 5, 2025 •

edited

Loading