AlignerNet instead of MAS

[codeghees]

Is it possible to use AlignerNet (aligner.py in pflow-tts repo) instead of MAS in VITS2?

What should be changed in the code? I am a bit confused on what the inputs should be.

[p0p4k]

Check out pflow repo for guidance.

[codeghees]

Hey, thanks for replying!

I did take most of the code from that repo. I am trying to debug why my alignment curve looks like this:

align_loss is being added to duration loss.
Inputs:

 aln_hard, aln_soft, aln_log, aln_mask = self.aligner(
            m_p.transpose(1,2), x_mask, y, y_mask
            )
attn = aln_mask.transpose(1,2).unsqueeze(1)
align_loss = self.aligner_loss(aln_log, x_lengths, y_lengths)
m_p is returned by TextEncoder.

Appreciate any insights!

[p0p4k]

It might be that it is still learning the durations. It think MAS is good enough. What is important is the duration_predictor module.

[codeghees]

So this above graph is after training for several days and thousands of steps. It seems like some bug - maybe in shape size or sth.

The output is basically the first word is legible but the rest is basically gibberish.

[p0p4k]

I see. Maybe it needs some fixing to do, if you can start a PR we can debug this together. I am busy with other stuff.

[Tera2Space]

Probably same problem as we have in pflow)

[codeghees]

@Tera2Space What was the problem?

[Tera2Space]

@Tera2Space What was the problem?

Code geass nice, problem was that it generate wrong aligment, i just though of reason: p0p4k/pflowtts_pytorch#24 (comment)

in your model, what shape is input to alignernet?

[codeghees]

@Tera2Space @p0p4k I added a basic PR of the changes I have so far: #82

[codeghees]

For a single batch - shapes look something like this:

m_p torch.Size([32, 192, 74])
x_mask torch.Size([32, 1, 74])
y_mask torch.Size([32, 1, 293])
m_p torch.Size([32, 192, 74])
x torch.Size([32, 192, 74])
y torch.Size([32, 80, 293])
m_p.transpose(1,2) torch.Size([32, 74, 192])

[Tera2Space]

For a single batch - shapes look something like this:

m_p torch.Size([32, 192, 74])
x_mask torch.Size([32, 1, 74])
y_mask torch.Size([32, 1, 293])
m_p torch.Size([32, 192, 74])
x torch.Size([32, 192, 74])
y torch.Size([32, 80, 293])
m_p.transpose(1,2) torch.Size([32, 74, 192])

hm than my guess was wrong/there are more problems.
I tested my idea on pflow and it didn't work too, so probably we need to check for another problems, I think we should use something like https://github.com/lucidrains/naturalspeech2-pytorch as reference.

[p0p4k]

Iirc, I had yanked the alignernet from there.

[codeghees]

Yeah currently trying out various inputs to Aligner; possibly an input issue. It might be an issue with the masks.

[p0p4k]

still not convinced about putting efforts in aligner net, we must focus on a better TextEncoder instead.