Localisation interface discussion and roadmap.

[TriangulumDesire]

This is a follow-up from a comment I made in #823; I created this separate issue to better discuss the plans for such an interface and determine what would be the best course of action.
I am positing the idea for a localisation interface as the most natural way to implement custom localisation and interntionalisation behaviour.

---

I've created several sections for potential functionality.

Translating strings.

Right now this is handled by the system interface, via TranslateString. This would be moved to the localisation inference. Obviously, this would be a breaking change, so I assume we'd have to wait for the next major release if this is the direction we go with.

Case-transforming strings.

These methods would be called to transform text with the text-transform property. I suggest the following signature:

virtual void TransformString(String& string, Style::TextTransform transform, const String& language);

where string is the string to be transformed in-place, transform is the value of the text-transform property, and language is the element's language (set by the lang attribute). I am suggesting in-place modification mainly due to efficiency.

The current implementation transforms strings in-place, which works fine for ASCII. However, some languages have special case rules depending on a letter's position in a string and what letters it is adjacent to (examples include the final sigma in Greek and IJ at the start of words in Dutch), so passing the whole string will be necessary in such cases.

Alternatively, we could make the signature similar to TranslateString's for consistency, where it is given an both an input string and output string (at the cost of some performance).

Also, after looking through the code, it seems that the capitalize value isn't actually implemented, and is synonymous with none; I'll get around to fixing that later.

Bidiretional text segmentation.

Has been partially discussed before, but it will be given a string and segment it based on text-flow direction. Perhaps it could be given an iterator that represents the beginning and returns an iterator corresponding to the end of the segment (and other information about the segment's properties)?

Another thing to handle with direction segmenting are mirror agents (for example, swapping ( and ) in certain places).

Word breaking.

Pretty self-explanatory. Basically splitting words better than only with whitespace (for example, by punctuation). Useful for text layout and determining where text can be wrapped.

Grapheme breaking.

Similar to word breaking, but for detecting which codepoint clusters correspond to a single visible glyph. Would be useful for proper text highlighting.

Codepoint properties.

Some extra methods for determining the property of a codepoint. For example, if a character is whitespace, uppercase, number, et cetera.
Example method:

virtual bool IsWhitespace(Character character);

---

To begin, we can create an implementation with basic functionality, and implement the other, more-advanced features at a later date. I've had a look through the code, and it shouldn't be too difficult to create an implementation that handles some of the above features. Here's a sample of what could be implemented to start:

class RMLUICORE_API LocalizationInferface : public NonCopyMoveable {
public:
    LocalizationInferface();
    virtual ~LocalizationInferface();

    virtual int TranslateString(String& translated, const String& input);

    virtual void TransformString(String& string, Style::TextTransform transform, const String& language);

    virtual bool IsWhitespace(Character character);
};

The default interface would assume that every string is ASCII-encoded. A possible stretch-goal would be to eventually include a localisation backend that uses ICU or similar Unicode algorithms library.

Anyway, these were just my thoughts for improving localisation/internationalisation in this project. I created this issue so that we to discuss improvements, suggestions, and nuances to this new potential interface API.

[mikke89]

I say, go for it! ;)

Looks very much reasonable to me. Some notes:

1. If we are moving / breaking the translate API, then I agree about doing in-place transformation of the string.
2. I'd prefer StringView over const String& for new API.
3. I am somewhat worried about the virtual IsWhitespace performance-wise.
4. I guess I am worried about the performance of virtual word breaking and grapheme breaking too.
5. Can we rule out language for the other functions, what about for IsWhitespace or TranslateString?
6. Just a note that we need to keep the ASCII-versions of these around for the CSS parsing at least (and I guess as a default).

Will this be sufficient to properly support right-to-left scripts and layout too?

[TriangulumDesire]

I put some more thought into this, and I've designed a basic pipeline that strings will go through:

Translation -> Case transformation -> Segment transformation -> Line breaking.

So, firstly, the string is translated as normal. Then, if there is a text-transform property, it will have its case transformed, and afterwards, it'll be segmented. Here is the idea I have for segmentation:

struct TextSegmentInfo
{
    size_t start_index;
    size_t length;

    Style::TextDirection text_direction;
}

virtual void SegmentText(String& string, Vector<TextSegmentInfo>& text_segments, Style::Direction main_text_direction, StringView language);

The SegmentText method is passed the current string, and it will transform the string in-place if needed (such as reversing RTL sections if needed, or swapping mirror agents). It will also output a list of segments within the string. However, these text segments would have to be modified after line breaking, since the start indices would change once the string is split, and a segment could span over multiple lines. These segments would be passed to the font engine since it is useful for text shaping (each segment will have to be shaped separately).

I based this design after having a look at how SheenBidi and ICU's Bidi APIs work. It might need some refinements to remove some oversights (I am sure I've missed something), but I think it's a good starting point. Let me know what you think.

I am somewhat worried about the virtual IsWhitespace performance-wise.

I put some more thought into this, and it might be feasible to support Unicode whitespace within the core itself. There are only twenty-five characters and the list hasn't had any new entries since 2002 (videlicet, new whitespace characters are few-and-far-between). Obviously the Unicode version would just be used for things like word-breaking. The RCSS parser will still just use the ASCII whitespace function.

I guess I am worried about the performance of virtual word breaking and grapheme breaking too.

Hmm, do you know what else we could do in this regard? We can forgo word/grapheme breaking for a first implementation. We could try benchmarking a basic ASCII implementation to see if there is any overhead from the vtable calls. From personal, anecdotal experience, I've never run into any issues with virtual function performance, even when calling thousands per frame, but, obviously, proper benchmarks would tell a more efficacious story.

Can we rule out language for the other functions, what about for IsWhitespace or TranslateString?

I think it would be good to add the language to all functions, now that I think about it. I don't think we'll need it for IsWhitespace though, since Unicode properties transcend language, for the most part. (At least, whitespace does, and I can't think of any other property functions needed within the core at this moment).

Will this be sufficient to properly support right-to-left scripts and layout too?

The aforementioned SegmentText function should take care of bidirectional text. As for layout, I don't think it's a job for the localisation interface, but rather the core itself. (I assume, by ‘layout’, you mean placing elements based on the dir attribute). The auto value for dir will need some localisation intervention, though, so perhaps something like virtual Rml::Style::Direction GetLanguageDirection(StringView language) once the layout modifications have been implemented?

[mikke89]

Yeah, I think this makes a lot of sense to me, very reasonable design.

One tiny concern perhaps, is that SegmentText does both string replacement and text segmenting at the same time, maybe this dual role can be somewhat confusing. Not sure if it makes sense to separate it. It's not as strong concern, so feel free to leave it as is.

For the whitespace. Yeah, for 25 characters, maybe it makes more sense to make this ourselves. An alternative perhaps, maybe we could keep a Set around from the interface on initialize, and make it accessible to those that need it.

Hmm, do you know what else we could do in this regard? We can forgo word/grapheme breaking for a first implementation. We could try benchmarking a basic ASCII implementation to see if there is any overhead from the vtable calls.

I think all we can do here is to implement it, and benchmark it. It could be completely negligible. But I think it's worth trying to design the API so we don't do virtual calls in hot loops. I.e. prefer to submit a string, rather than one and one code point, if possible. I think the SegmentText is a good example of this.

For bidirectional layout. Looks like there are some complicated CSS rules around this: unicode-bidi. I think it's material for another day, shouldn't really limit the design here it looks like.

[TriangulumDesire]

One tiny concern perhaps, is that SegmentText does both string replacement and text segmenting at the same time, maybe this dual role can be somewhat confusing.

I designed it like this for performance and to confine the BiDi algorithm calls to a single function. But I do wonder if it would be better to include the term ‘transform’ in its name.

How about:

• TransformTextCase for the text-transform property.
• TransformTextSegments for segmenting text. It is now obvious that is transforms the text, and the output parameter can be used to produce the segments as a side-effect.

An alternative perhaps, maybe we could keep a Set around from the interface on initialize, and make it accessible to those that need it.

This works fine, too! I'm thinking that the localisation interface would have a private set containing all of the whitespace codepoints, and a method (protected, perhaps) that would add codepoint(s) to this set (RegisterWhitespaceCodepoint[s]?). (The idea being to initialise the set in the constructor or other initialisation method). Then, the localisation interface would have a non-virtual public method that will determine if a codepoint belongs to this set (the IsWhitespace(Character character) mentioned above).

This could easily be expanded to support other properties if needed. For example:

enum class CodepointProperty
{
    Whitespace,
    Lowercase,
    Uppercase,
    // ...
    Count,
};

// Private.
Array<Set<Character>, static_cast<size_t>(CodepointProperty::Count)> codepoint_property_lists;

Then, the interface could have an array of sets, and the RegisterCodepoint method would also take a CodepointProperty value. However, it seems that whitespace is the only property needed at this stage, so this shouldn't be necessary right now.

But I think it's worth trying to design the API so we don't do virtual calls in hot loops. I.e. prefer to submit a string, rather than one and one code point, if possible.

I like the sound of that. Here's what I had in mind. (The returns can be replaced with output variables, if desired.)

// Returns a list of indices containing word boundaries.
virtual List<size_t> BreakStringWords(StringView string, StringView language);

// Returns a list of indices containing grapheme boundaries.
virtual List<size_t> BreakStringGraphemes(StringView string, StringView language);

[mikke89]

How about: TransformTextCase, [...] TransformTextSegments

Yup, I like those, very clear.

I'm thinking that the localisation interface would have a private set containing all of the whitespace codepoints, and a method

I think rather we should leave it as an interface in its purest form, like our other interfaces. It's about communicating with the library. Adding data members and methods changes it's responsibility. Access methods should be part of the internal library design details. In my view it could rather just return a list of code points (vector). Then we can store it and access it as we want based on our internal usage.

I like the idea of eventually generalizing it to different codepoint properties, but I agree, not needed to begin with.

On the BreakStringWords and BreakStringGraphemes. Yeah, that looks good to me overall. We should measure what performs best in terms of return type. Like list, or vector, or perhaps passing in something with an existing capacity to avoid some allocations, or the previous approach of iterating one at a time. Maybe none of this matters in the end.

[TriangulumDesire]

Alright, sounds good to me. I might have a go implementing this myself in the future, but it likely won't be any time soon since I'm working on something else at the moment.

We should measure what performs best in terms of return type. Like list, or vector, or perhaps passing in something with an existing capacity to avoid some allocations

Ah, I meant to have the example functions return Vector instead of List; that's just a carryover habit from another project I'm working on where I use List to refer to std::vector. I'd say the best idea is to have a Vector output parameter. That way, we can recycle a buffer to avoid repeated allocations if we want, and pass that (cleared) buffer as the argument for each invocation.

For future reference, here's an aggregated API reference from this discussion. To keep things simple for an initial implementation, we can implement the just the translation, transformation, and whitespace functions—and then implement the breaking methods at a later date.

struct TextSegmentInfo
{
    size_t start_index;
    size_t length;

    Style::TextDirection text_direction;
};

class RMLUICORE_API LocalizationInferface : public NonCopyMoveable {
public:
    LocalizationInferface();
    virtual ~LocalizationInferface();

    // Translate the input string into the translated string.
    virtual int TranslateString(String& translated, StringView input);

    // Transform the text case based on the text transform style.
    virtual void TransformTextCase(String& string, Style::TextTransform transform, StringView language);

    // Transform the text based on codepoint property segments (script, direction, et cetera).
    virtual void TransformTextSegments(String& string, Vector<TextSegmentInfo>& text_segments, Style::Direction main_text_direction, StringView language);

    // Create a list of indices containing word boundaries in a given string.
    virtual void BreakStringWords(StringView string, Vector<size_t>& word_boundary_indices, StringView language);

    // Create a list of indices containing grapheme boundaries in a given string.
    virtual void BreakStringGraphemes(StringView string, Vector<size_t>& grapheme_boundary_indices, StringView language);

    // Create a set containing all whitespace codepoints.
    virtual void GetWhitespaceCharacters(UnorderedSet<Character>& whitespace_characters);
};

[mikke89]

I understand, sounds good.

And that interface looks like a great starting point to me.