The issues that occurred in this project in Chinese and Japanese environments

[quark-mac]

I used the English test audio for this project, and it looks exactly as expected

english.txt

But when using Chinese and Japanese speech, punctuation marks were completely removed, and spaces were added between each character, with the speech of multiple speakers all crammed together

chinese.txt
english.txt

The ctc-forced-aligner and romanize in the project seem to be having issues, and the nemo framework is also unable to correctly distinguish the speaker
Does the nemo framework use of titenet_large and other pre-trained models only support English?？
I want to know how to solve this tricky problem
I am using machine translation, so please bear with any misunderstandings

[MahmoudAshraf97]

we need to break this down because it's complicated.
as for the punctuation, make sure whisper is outputting punctuation in the first place, because this is not something that we can control
as for the spaces between characters, ctc-forced-aligner is not thoroughly tested with japanese and chinese so can you please test that alone and report any issues?
as for the nemo speaker distinguishing, it's known that it has a problem with long audios >=60mins, so keep that in mind

[quark-mac]

@MahmoudAshraf97
I have tested it individually and confirmed that the whisper configuration (large-v3+float32) can output punctuation in Chinese and Japanese environments
The tested speech is short, only about 20 seconds, and there is no issue of being too long
I will find time later to test the ctc-forced-aligner that comes with this project. What aspects of the issue I mentioned in this issue can be tested?

[quark-mac]

@MahmoudAshraf97
I've done some testing, and I believe the problem initially lies in the preprocess_text function in text_utils.py of ctc-forced-aligner. It's clear that it's not handling Chinese and Japanese effectively, despite having specific conditional logic.

For example, in Chinese, after converting each character to pinyin, the code separates each pinyin letter and adds spaces, causing subsequent processing to fail. (For example, "我的" should be spelled "wo de" normally, but becomes "w o d e" after code processing.) Even if the built-in romanize function is turned off, and the Chinese characters are manually converted to pinyin (in two cases: all pinyin letters are arranged closely in one line and the corresponding pinyin letters are separated by individual Chinese characters), the same problem persists.
And there are tags separating in strange places

It is clear that this function, and other parts of the ctc-forced-aligner code, do not provide good support for the spelling of non-Latin characters.

Since the alignment and romanize logic of ctc-forced-aligner seems quite complex, I need to clarify it further, and I'd appreciate your help. It seems this problem is beyond my ability to solve 😂