Discussion tracker: ICU4X "compile time" MessageFormat

[Manishearth]

This issue is a bit of a placeholder for now, but I am filing it to move discussion off of the MF2.o feedback doc.

This issue pertains to the sketch of a design in the MF2.0 ICU4X feedback document under "Compile time API".

The rough design is to have some way of declaring the upper bound of what data a message may need at compile time. The simplest design for this is to include a "source" message that does this.

// Preparses the message
declare_message!(MyMessage, "
.input {$date :datetime weekday=long}
.input {$num :number}
.match $num
 .......
");

let message = MyMessage::new(locale); // loads exactly the data the message needs: here, a weekday=long datetime formatter and plural rules info


// Does not load data
message.format(inputs);

There are i18n problems with this design (programmers should not necessarily be required to understand enough i18n to specify this stuff beforehand), but it can potentially be fixed with the right process/tooling, or by having other ways this data is computed and passed to ICU4X.

@robertbastian had better articulated thoughts around this, with examples. I would like to let him better specify the design, but I'm filing this issue as a place for discussion to be tracked.

cc @hsivonen @zbraniecki @sffc @robertbastian

[zbraniecki]

Responding to @robertbastian from the comment:

An attempt to reason about linguistic features of a given message in any locale forces the developer producing source message to anticipate (and set a limit?) on what linguistic features may appear in translation.

Linguists cannot come up with inputs, because inputs need to be set by the developer. If the developer doesn't inject a "gender" variable into the message during formatting, the information won't be there.

Correct, there's a social contract between localizers and developers which is gated on what variables developers expose to localizers. I tried to capture it in the following claim:

What is true are two statements:

1. the list of variables and their types is the same across all localizations of a single message. Implementations can reason based on that (for example, if there are no variables, no data is needed)
2. most localizations of a message will use at most the same formatters and selectors as the source one. This allows for optimizations.

The (1) is what developers bound localizers with. The "good" thing about that bound is that developers do not induce complexity by introducing a variable. A developer can expose user's name or gender or age without inducing complexity on any localizer who doesn't need those variables for a given message.

What I was commenting on is that the skeleton of the source message is not sufficient to reason about which formatters (and which data) is needed to format a given message. 99 locales may not use the age, but 100th locale may use the age to format proper honorific. 99 locales may not care about the name of the Japanese Era, or Hebrew Numerals, or ordinal rules for a given message, and the 100th may do.
I'm concerned about making a claim that if source message doesn't "hint" it then the 100th localization cannot use it.

Manish responded that we may need more sophisticated mechanisms to detect what is needed than just source message analysis.

the pattern he saw emerged from years of use of an insufficient localization system that mf2 is designed to directly solve.
How does MF2 solve this? If the source message doesn't contain a "gender" input, but the French translation needs one, where does the value come from?

I did not mean it that way. I'm sorry if I wasn't clear. As you pointed out the "gender" input has to be declared by the developer.
But what is possible is that based on a set of variables (you call them "inputs") the localizer may want to use a selector or formatter that no other locale used. And we need data for it.
If we are to enforce that no data is loaded during message formatting, then the data has to be loaded prior to that.

I want to also point out that addition of a variable/input to the message is not a breaking change and may happen after the message has been introduced to the corpus.
A scenario that is common is a sort of "negotiation" between developer and localizers over time in which the developer introduces a message, developers try to translate it, maybe they all succeed and then later, a new locale is added and the localizer points out that they can't translate that message without, say, gender, variable, and the developer then adds the variable to the message, without changing the source message, and then the localizers can use it.
This contract extension happens without any change to the message itself, any invalidation of existing translations etc.

It is compatible with the aspiration of using the set of variables to deduct what data to load, but it is not compatible with the aspiration of using the source message's skeleton to do so.

[zbraniecki]

btw. in Fluent, for that reason we use an idea of "l10n context" - a list of variables attached not to a single message, but to the a group of messages resolved in a single context:

let ctx = new L10nContext([
  "/res/menu.ftl",
  "/res/errors.ftl",
  "/res/data.ftl",
], {
  userName: "John",
  gender: "male",
  age: 26
});

let msg = ctx.resolveMessage("unreadEmails", { count: 5 });

This allowed us to resolve message references within a single "context" but also allowed developers to provide variables that are available to all messages in a given context - because the whole UI screen localized by this context is localized for John, who is a male age 26 and the developer shouldn't be the one to decide which of the messages needs this information, and which doesn't.
I think this model of thinking is quite alien to ICU4X and we should attempt to reconcile that.

[Manishearth]

A scenario that is common is a sort of "negotiation" between developer and localizers over time in which the developer introduces a message, developers try to translate it, maybe they all succeed and then later, a new locale is added and the localizer points out that they can't translate that message without, say, gender, variable, and the developer then adds the variable to the message, without changing the source message, and then the localizers can use it.

So I think one of the factors here is that the "skeleton" espouses a typed view of this: where not only the arity of the message, but the "types" of the parameters are relevant. A parameter may be a date, it may be a plain string, it may be a number of items, it may be a noun-that-can-be-inflected, it may be a it may be a user gender, etc. In theory it could be multiple of these at once. Moving the negotiation over to the typed view does not seem terrible, but it does introduce further back-and-forth in the common case which is not ideal. Tooling-extracted types may be better than letting the developer pick. Alternatively, perhaps by forcing the developer to select types, a well-designed set of types may require the right kind of introspection from the developer on the messages. It is unreasonable to expect a developer to understand where plurals/gender can apply, but "is this a count of an item" and "is this some noun" ... are actually answerable questions! So it may be possible to nudge the developer into getting this right. Maybe.