Implement Hangul NFD to NFC fast path

[hsivonen]

Note: This issue is not ready to be worked on in main. The implementation should be on top of https://github.com/hsivonen/icu4x/tree/nfdsinglemark , which hasn't landed, yet. I'm filing this now so that we have an issue on file in case I don't have the time to do this myself right away in the next few days.

Even with the optimizations present in https://github.com/hsivonen/icu4x/tree/nfdsinglemark , Hangul NFD to NFC UTF-16 is still slower than ICU4C.

In the composing normalizer in the slice mode (not in iterator mode), we should have a fast path that knows how to consume conjoining jamo and ASCII without normal trie lookups and exits back to the normal path when encountering something else.

We should enter the Hangul NFD to NFC fast path when the flag that we just formed an LV syllable from L and V without following ccc != 0 marks is true:
https://github.com/hsivonen/icu4x/blob/2e2611c6442737aa546ec16407136d31fb2b3e62/components/normalizer/src/lib.rs#L1913

The fast path should most likely be specialized for UTF-16 so that we don't do surrogate checks but surrogates exit the fast path due to being neither conjoining jamo nor ASCII. (In principle, we could even specialize what UTF-8 lead bytes to handle, but that's less likely to be worthwhile.)

[Jayant-kernel]

Hi @hsivonen,

I'm interested in working on this issue. I see it's marked "not ready to be worked on in main"
and depends on the nfdsinglemark branch.

Could you let me know:

1. When is the nfdsinglemark branch expected to be merged?
2. Is this issue ready to be worked on now, or should I wait?
3. If I should wait, are there related normalizer issues I could help with in the meantime?

[hsivonen]

When is the nfdsinglemark branch expected to be merged?

I don't have a specific timeline. The review is being done in #7528, so you can follow along there and draw your conclusions about how quickly it's going. That PR has a more recent branch that would now be a better base for this.

Is this issue ready to be worked on now, or should I wait?

This can be worked on now if you are OK with working on top of the code seen in #7528 instead of main for now.

If I should wait, are there related normalizer issues I could help with in the meantime?

The other open normalizer issues also involve working on top of the code seen in #7528 instead of working on main.

[Jayant-kernel]

Thanks for the clarification @hsivonen! I understand this can be worked on now based on the #7528 branch.

Before I start implementing, I want to make sure I understand the approach correctly:

I think :

1. Detect when we've just formed an LV syllable from L + V (with the flag that we formed it from L and V without following ccc being true)
2. Check if the next character is a Hangul T jamo
3. If yes, directly compute LV + T_index → LVT without trie lookup
4. If no (non-Hangul or ASCII), exit back to normal path

Questions:

1. Should the fast path be in the slice mode composition loop, right after we form an LV syllable?
2. Do you want me to maintain state tracking (like "just_formed_lv" flag) or infer from the character itself?
3. For the exit condition - should we handle ASCII continuation in the fast path, or immediately exit to normal path?

I'll study the nfdsinglemark branch code first, then implement based on your feedback. Let me know if my understanding is correct!

[hsivonen]

Should the fast path be in the slice mode composition loop, right after we form an LV syllable?

Yes

Do you want me to maintain state tracking (like "just_formed_lv" flag) or infer from the character itself?

Should not be necessary.

For the exit condition - should we handle ASCII continuation in the fast path, or immediately exit to normal path?

ASCII should stay on the Hangul fast path.

[Jayant-kernel]

@hsivonen sir ,@sffc,@robertbastian @echeran @Manishearth
just a quick update: I’ve finished implementing the Hangul fast path logic.

I added a dedicated test suite to verify the L+V+T composition and ASCII handling.

I built this directly on top of your nfdsinglemark branch (from PR #7528). Since that branch isn't in the upstream unicode-org repository yet, my PR will show a large diff if I target main , like my pr will also show the level code changes if done against the main
please guide what to do .

[hsivonen]

I took a look at the changeset in your repository that references this issue, and unfortunately it's not what is sought here. I don't have the time to guide the required changes. Sorry. (I wrote this issue down, because I don't have the time to write the code myself right now.)

[Jayant-kernel]

Thank for looking at it.
I understand the issue requires deeper familiarity with the normalizer internals and would like to study the code more carefully.
Will you reviewing a second attempt ?
If no, still fine

[Manishearth]

The first line of the issue clearly states that this is not yet ready to be worked on, and has been filed for tracking reasons.