Skip to content

Latest commit

 

History

History
422 lines (315 loc) · 16.8 KB

I18N.adoc

File metadata and controls

422 lines (315 loc) · 16.8 KB
Raw
Table of Contents

1. Internationalization

⚠️
 I took a rough pass at updating this documentation, but nothing has been testing in a live app.

Fulcro has had internationalization support since inception. That original support worked well from the call-site; however, many lessons were learned about storing, transmitting, and structuring the actual data.

The support:

  • The ability to write UI strings in a default language/locale in the code. These are the defaults that are shown on the UI if no translation is available.

  • Supports string extraction from the compiled UI code.

    • The translator can use tools like POEdit to generate translation files

  • Leverages normal application state as the location to house loaded translations.

  • Allows the server to serve new locale translations via a normal query.

  • Includes server tools to directly read locales from GNU gettext PO files (no more code generation).

  • Supports a pluggable message formatting system.

1.1. The Core API

The following functions and macros make up the core of the API:

tr

A macro that looks up the given string and returns the translation based on the current locale.

trc

A macro that looks up the given string with a translation context (note to the translator) and returns the translation based on the current locale.

trf

A macro that looks up the given string, and passes it and additional options through to a message formatter.

tr-unsafe

All three of the above have an unsafe version. The main three require literal strings for their arguments because string extraction won’t work correctly otherwise. The unsafe versions are for situations where this is insufficient, but you still need some kind of marker to let you know where translation might be needed.

Examples:

(tr "Hello") ; might return "Hola"
(trc "Abbreviation for Male" "M") ; might return M (translator sees the other string as a note)
(trf "Hi {name}" {:name n}) ; passes a translated version of "Hi {name}" to the message formatter along with the options map.

(tr-unsafe current-selection) ; allows a non-literal to be sent through lookup. The possible values of current-selection will need to be extracted elsewhere.

1.1.1. Dealing with "Unsafe" Usage

When you use an unsafe variant, it simply means that GNU gettext is not going to be able to extract a string (because extraction is a static analysis of compiled code). One approach is simply to make notes for your translator. That approach isn’t very scalable.

Usually, this comes up when you have something like a dropdown that needs to display translated strings. Another approach is to simply call tr on the literal values in some unreachable code. Whitespace optimizations will not remove these, so extraction will find them, whereas advanced optimizations will see that they are not directly called and will remove them:

(defn do-not-call []
  (tr "Yes") ; these are here for extraction only
  (tr "No"))

(def options {:yes "Yes" :no "No"})

(defn render-option [o]
   (tr-unsafe (get options o)))
⚠️
 It is tempting to wrap the values of options in tr, but that is a bad idea. 
(def options {:yes (tr "Yes") :no (tr "No")}) ; BAD IDEA!

This is a problem because your program can break for unexpected reasons. If you changed the locale before that code executed then your options map might contain the translations instead of the default locale keys (e.g. {:yes "Oi" :no "Non"}) which are not the correct keys for the later calls to tr!

Remember that your string extraction is done against the real string you embed on the UI (your default locale), and those become the lookup keys for runtime. If you change those lookup keys based on the runtime locale, things will break.

1.2. Polyfills and Message Formatters

Fulcro’s i18n support is designed to make it easy to code and extract translatable strings in your UI. It is not, itself, interested in doing message, number, currency, or date formatting. There are plenty of libraries, including Google Closure, that can already fill that role.

The easiest pair to use for both server and client rendering are the FormatJS (client) and IBM ICU library (server). These two libraries follow the same formatting standards, and give good isomorphic rendering support.

Fulcro does one central task: it takes a string in the UI, looks up an alternate string (based on the locale) from a PO file, and pushes that alternate string through the rest of the i18n processing chain (which you define).

The macro:

(tr "Hello")

will use the combination of the current locale and loaded locale data to return the correct translation for "Hello". A call to:

(trf "Hi {name}" {:name "Joe"})

will look up "Hi {name}" in the translations, find something like "Hola, {name}", and will then pass that translation through to a message formatter that can substitute the supplied parameters for the placeholders.

For this to work you must load whatever polyfills and tools you need for message formatting, and then install your message formatter into Fulcro’s i18n system.

1.3. Configuring I18N

Your first step is to define a function that can format messages. If you want to use Yahoo’s FormatJS, then you’d add the FormatJS library as a script in your HTML, and then use something like this:

(ns appns
  (:require [com.fulcrologic.fulcro.components :as comp :refer [defsc]]
            [com.fulcrologic.fulcro-i18n.i18n :as i18n :refer [tr trc trf]]))

(defn message-formatter [{:keys [::i18n/localized-format-string ::i18n/locale ::i18n/format-options]}]
  (let [locale-str (name locale)
        formatter  (js/IntlMessageFormat. localized-format-string locale-str)]
    (.format formatter (clj->js format-options))))

The message formatter receives a single map with namespaced keys. The locale will be a keyword, the localized-format-string will be the already-translated base string, and the format-options will be whatever map was passed along to trf.

Fulcro’s i18n uses shared properties to communicate the current locale, message formatter, and translations to the UI components. This is a feature of the low-level reconciler.

When creating your client:

  1. Include these options on the client:

    (ns appns
  (:require
    [com.fulcrologic.fulcro.application :as app]
    [com.fulcrologic.fulcro-i18n.icu-formatter :as icu]
    [com.fulcrologic.fulcro-i18n.i18n :as i18n]))

(defonce app (app/fulcro-app
  {:shared    {::i18n/message-formatter icu/format}
   :shared-fn ::i18n/current-locale}))

  2. Your Root UI component MUST query for ::i18n/current-locale and should also set the initial locale in application state. The shared-fn extracts denormalized data from your UI root’s props. This also sets the "default" locale of your application.

1.4. Setting the "Default Locale"

Your root component should place a locale in the ::i18n/current-locale. This is normalized state, so the root component query should join on the Locale component:

(defsc Root [this props]
  {:query         [{::i18n/current-locale (comp/get-query i18n/Locale)}]
   :initial-state (fn [p] {::i18n/current-locale (comp/get-initial-state i18n/Locale {:locale :en :translations {}})})}

1.5. Accessing the Current Locale

Shared properties are visible to all UI components via (comp/shared this). You will find the property ::i18n/current-locale in there as well as your message formatter.

Mutations have the state database, and can simply look for the top-level key ::i18n/current-locale.

1.6. Changing the Locale

The are a few aspects to changing the locale:

  1. Ensuring that the locale’s translations are loaded.

  2. Changing the locale in app state.

  3. Force rendering the entire UI to refresh displayed strings.

All of these tasks are handled for you by the i18n/change-locale mutation, which you can embed anywhere in your application:

(comp/transact! this `[(i18n/change-locale {:locale :es})])

There is a pre-built locale selector for your convenience.

1.7. Responding to Locale Loading Queries on Your Server

Of course, triggering a change locale that tries to load missing translations will fail if your server doesn’t respond to the query! Fortunately, configuring your server to serve these is very easy!

  1. Place all of your .po files on disk or in your applications classpath. The names of the PO files must be LOCALE.po, where LOCALE matches the locale keyword (minus the :), case sensitive.

  2. Add a resolver like this (assuming you’re using Pathom)

(defresolver i18n-locale-resolver [env _]
  {::pc/output [::i18n/locale ::i18n/translations]}
  (let [{:keys [locale]} (:query-params env)]
    (when-let [translations (i18n/load-locale "po-files" locale)]
      ;; The result of `load-locale` is already a map with the proper keys.
      translations)))

You can augment this to log errors or whatever else you want it to do. The "po-files" argument is the location of the po files. If it is a relative path, the resources will be searched (i.e. CLASSPATH). If it is an absolute path, then the local disk will be searched instead.

1.8. Extracting Strings (Lisp method)

Since Clojure looks pretty much like Lisp, the xgettext utility can usually extract strings directly from you CLJ, CLJS, and CLJC files! Let’s say you wanted to lay out your i18n files like this:

src/main
├── config
│   ├── i18n
│   │   ├── Makefile
│   │   ├── es.mo
│   │   ├── es.po
│   │   └── messages.pot
...

so that your po files will be on the classpath in production for loading. The content of the Makefile can be:

ALL_SRC := $(shell find ../.. -type f -name '*.clj*')

i18n: es.po

messages.pot: $(ALL_SRC)
        xgettext --language=Lisp --from-code=UTF-8 -k -ktr:1 -ktrc:1c,2 -ktrf:1 -o messages.pot $(ALL_SRC)

es.po: messages.pot
        msgmerge --force-po --no-wrap -U es.po messages.pot

.PHONY: i18n

The Makefile assumes that you’ve generated es.po at least once. To do that run make messages.pot and run a translation app like POEdit.app to make your first set of translations.

Now every time you’re ready for doing a new release simply go to the src/main/config/i18n and run make. This will build an updated es.po file, and merge any existing translations with the new extractions in the pot file. Just make sure you refer the tr et al functions (don’t use a ns alias. The code should use (tr …​), not (i18n/tr …​).

Of course, you’ll need to re-run POEdit.app (or similar) on the resulting file(s) to fix any missing/changed translations.

To add more languages, just add more po targets. For example, to add German:

ALL_SRC := $(shell find ../.. -type f -name '*.clj*')

i18n: es.po de.po

messages.pot: $(ALL_SRC)
        xgettext --language=Lisp --from-code=UTF-8 -k -ktr:1 -ktrc:1c,2 -ktrf:1 -o messages.pot $(ALL_SRC)

es.po: messages.pot
        msgmerge --force-po --no-wrap -U es.po messages.pot

de.po: messages.pot
        msgmerge --force-po --no-wrap -U de.po messages.pot

.PHONY: i18n

1.9. Extracting Strings (js method)

The xgettext command has a Lisp mode, which is generally recommended, but if for some reason that crashes on your code and you cannot fix it, you can compile your cljs to js, and extract the strings in Javascript mode. If you can use Lisp mode, do it. The results are better because the source line attribution works there.

You can extract the strings from your UI for translation using GNU’s CLI utility xgettext (available via Brew, etc).

The steps are:

  1. Compile your application with whitespace optimizations.

  2. Run this on the resulting js file:

    $ xgettext --from-code=UTF-8 --debug -k -kfulcro_tr:1 -kfulcro_trc:1c,2 -kfulcro_trf:1 -o messages.pot application.js

1.10. Generating Locale Translation Files

See GNU’s gettext documentation for full details. Here are some basics:

Applications like POEdit can be used to generate a new locale from the messages.pot in the prior step. Once you have the output (a file like es.po) you simply copy that to your server’s PO directory as described in the section on serving locales.

When your application changes, you want to keep the existing translations. The gettext utility msgmerge is useful for this. It takes the new messages.pot file and old PO files and generates new PO files that include as many of the old translations as possible. This allows your translator to just deal with the changes.

Something like this will update a PO file:

$ msgmerge --force-po --no-wrap -U es.po messages.pot

Again send that off to your translator, and when they return it place the updated PO file on your server.

1.11. Prebuilt Locale Selector

The i18n support comes with a convenient LocaleSelector component that you can use. You can, of course, write your own and invoke the change-locale mutation, but the pre-written one can be used as follows:

(defsc Root [this {:keys [locale-selector]}]
  {:query         [{:locale-selector (comp/get-query i18n/LocaleSelector)}
                   {::i18n/current-locale (comp/get-query i18n/Locale)}]
   :initial-state (fn [p] {::i18n/current-locale (comp/get-initial-state Locale {:locale :en :translations {}})
                           :locale-selector      (comp/get-initial-state LocaleSelector
                                                   {:locales [(comp/get-initial-state Locale {:locale :en :name "English"})
                                                              (comp/get-initial-state Locale {:locale :es :name "Espanol"})
                                                              (comp/get-initial-state Locale {:locale :de :name "Deutsch"})]}}}
  (dom/div
    (i18n/ui-locale-selector locale-selector)
    ...))

The initialization parameters are a list of the locales that are available on your server. You could, of course, load these at startup and fill out app state; however, since you have to know what locales you’re supporting in order to work with translators, it’s probably just as easy to hard-code them.

Each locale must be given a name (UTF8) to be show in the resulting select drop-down. This renders as an HTML select with the CSS class "fulcro$i18n$locale_selector".

1.12. Server-Side Rendering

Server side rendering of the default locale require no additinal code, because the strings you need are already the strings in the code. If you wish to pre-render a page using a specific locale then there is just a little bit more to do.

The steps are:

  1. Load the locale from a po file.

  2. Generate initial db to embed in the HTML that contains the proper normalized ::i18n/current-locale.

  3. Use i18n/with-locale to wrap the server render.

(defn message-formatter ...) ; a server-side message formatter, e.g. use IBM's ICU library

(defn generate-index-html [state-db app-html]
  (let [initial-state-script (ssr/initial-state->script-tag state-db)]
    (str "<html><head>" initial-state-script "</head><body><div id='app'>" app-html "</div></body></html>")))

(defn index-html []
  (let [initial-tree     (comp/get-initial-state Root {})
        es-locale        (i18n/load-locale "po-directory" :es)
        tree-with-locale (assoc initial-tree ::i18n/current-locale es-locale)
        initial-db       (ssr/build-initial-state tree-with-locale Root)
        ui-root          (comp/factory Root)]
    (generate-index-html initial-db
      (i18n/with-locale message-formatter es-locale
        (dom/render-to-str (ui-root tree-with-locale))))))

If you use Yahoo’s FormatJS on the client, then a good choice on the server is com.ibm.icu/icu4j since it uses the same syntax for format strings.

The message formatter could be:

(ns your-server-ns
  (:import (com.ibm.icu.text MessageFormat)
           (java.util Locale)))

(defn message-formatter [{:keys [::i18n/localized-format-string
                                 ::i18n/locale ::i18n/format-options]}]
  (let [locale-str (name locale)]
    (try
      (let [formatter (new MessageFormat localized-format-string (Locale/forLanguageTag locale-str))]
        (.format formatter format-options))
      (catch Exception e
        (log/error "Formatting failed!" e)
        "???"))))