The current solution for the alignment doesn't work well with pauses in the sentence or between sentences
Let's say we have 2 sentences and a pause of 3 seconds between them, but merging united them together for a speech generation. The speech will have a miserable pause between sentences and the silence will be shifted to the right. Therefore, the whole speech will be extended during alignment to fill the gap. It doesn't sound good. And that is just one example.
Ideally, we should have an algorithm to generate speech n-gram to n-gram over time. But that might require an updating of the model architecture. Perhaps the inference of StyleTTS2 could be changed to achieve that.
The current solution for the alignment doesn't work well with pauses in the sentence or between sentences
Let's say we have 2 sentences and a pause of 3 seconds between them, but
mergingunited them together for a speech generation. The speech will have a miserable pause between sentences and the silence will be shifted to the right. Therefore, the whole speech will be extended during alignment to fill the gap. It doesn't sound good. And that is just one example.Ideally, we should have an algorithm to generate speech n-gram to n-gram over time. But that might require an updating of the model architecture. Perhaps the inference of StyleTTS2 could be changed to achieve that.