Time-domain loss

[alexdemartos]

Hi,

I am just wondering wether the time-domain loss is working properly. I just noticed that the samples before the discriminator comes into play (>200K steps) are a bit muffled/noisy. After that, the GAN training scheme seems to be helping the audio quality and reducing this effect.

TFGAN_samples.zip

What is your experience? Is this muffled noise to be expected before the discriminator network comes in?

Thanks in advance, and thank you for sharing your work.

[rishikksh20]

@alexdemartos, I am also skeptic about time domain loss, but we expect muffled or metallic artifacts before discriminator kicks actually that's why we use discriminator to remove those artifacts.

[listener17]

@rishikksh20 and @alexdemartos: anyone has plots of how the time-domain loss is varying with training?

[listener17]

Also @rishikksh20: is the phase loss computed across different time scales (as in the paper)? I guess not.

[rishikksh20]

@Miralan implements the Time domain losses hope she might gives good input on this, I haven't rechecked this code for while as I am busy with other model implementation. Otherwise I will check myself within a week and let you know regarding this.

[listener17]

OK, thanks @rishikksh20. No worries, in my work (which is different from MelGAN) time-loss seems to behave as expected. So, I feel the code is OK as such.

[Miralan]

OK, thanks @rishikksh20. No worries, in my work (which is different from MelGAN) time-loss seems to behave as expected. So, I feel the code is OK as such.

Did my implemention of time domian loss have any problems?

[listener17]

Hi @Miralan: No it was OK. Only question I had was:
The phase loss seems to be not computed across multi-scale like the other two loss terms. But I guess, it doesn't matter.

[Miralan]

Hi @Miralan: No it was OK. Only question I had was:
The phase loss seems to be not computed across multi-scale like the other two loss terms. But I guess, it doesn't matter.

Thanks for your advice, I am sorry for that it seems like that I forgot to implement the case of frame size=1 and hop size=1.And phase loss can also apply on multi-scale, but I don't think it has much impact @rishikksh20 .
Just now, I updated a new version of timedomian loss, hope you can help correct error of it and merge it.

[rishikksh20]

@Miralan yes I think that not have much impact on overall quality, it just helps to removes minute artifacts and jitters.

[rishikksh20]

Also @Miralan have you worked with Wavenet architecture ?

[Miralan]

Also @Miralan have you worked with Wavenet architecture ?

No, I just used it on GAN based parallel vocoder.

[listener17]

@Miralan:

Thanks for including multi-scale phase loss.

Testing your code (https://github.com/rishikksh20/TFGAN/blob/main/utils/timeloss.py) with

def test():
    torch.manual_seed(1234)
    loss = TimeDomainLoss_v1(32, 16384)
    real = torch.randn(32, 1, 16384).to(device='cuda')
    fake = torch.randn(32, 1, 16384).to(device='cuda')
    final_loss = loss(real, fake)
    print(final_loss)

WORKS!

However, when I run the training in my framework it throws an error:

File "c:\git\ganMono\train.py", line 99, in main
    gan.train(opts, dloader, criterion, stft_criterion, time_criterion, opts.l1_weight, opts.sc_weight, opts.mag_weight,
  File "c:\git\ganMono\gan\models\model.py", line 397, in train
    g_l1_loss = l1_weight*time_criterion(fake, real)
  File "C:\Users\abisw\Anaconda3\envs\pytorch\lib\site-packages\torch\nn\modules\module.py", line 889, in _call_impl
    result = self.forward(*input, **kwargs)
  File "c:\git\ganMono\gan\models\timeloss.py", line 43, in forward
    y_tmp = torch.as_strided(y, self.shapes[i], self.strides[i])
RuntimeError: setStorage: sizes [32, 16384, 1], strides [16384, 1, 1], storage offset 0, and itemsize 4 requiring a storage size of 2097152 are out of bounds for storage of size 524288
PS C:\git\ganMono>

Any idea why it is?

[Miralan]

@Miralan:

Thanks for including multi-scale phase loss.

Testing your code (https://github.com/rishikksh20/TFGAN/blob/main/utils/timeloss.py) with

def test():
    torch.manual_seed(1234)
    loss = TimeDomainLoss_v1(32, 16384)
    real = torch.randn(32, 1, 16384).to(device='cuda')
    fake = torch.randn(32, 1, 16384).to(device='cuda')
    final_loss = loss(real, fake)
    print(final_loss)

WORKS!

However, when I run the training in my framework it throws an error:

File "c:\git\ganMono\train.py", line 99, in main
    gan.train(opts, dloader, criterion, stft_criterion, time_criterion, opts.l1_weight, opts.sc_weight, opts.mag_weight,
  File "c:\git\ganMono\gan\models\model.py", line 397, in train
    g_l1_loss = l1_weight*time_criterion(fake, real)
  File "C:\Users\abisw\Anaconda3\envs\pytorch\lib\site-packages\torch\nn\modules\module.py", line 889, in _call_impl
    result = self.forward(*input, **kwargs)
  File "c:\git\ganMono\gan\models\timeloss.py", line 43, in forward
    y_tmp = torch.as_strided(y, self.shapes[i], self.strides[i])
RuntimeError: setStorage: sizes [32, 16384, 1], strides [16384, 1, 1], storage offset 0, and itemsize 4 requiring a storage size of 2097152 are out of bounds for storage of size 524288
PS C:\git\ganMono>

Any idea why it is?

It seems like that out of memory and you can try to use batchsize 2 instead of 32.

[listener17]

@Miralan works :-)