Evaluating models

[anontiger]

Now that I can train the models, I decided to try to evaluate them. I want to evaluate the model on different stuff like SI-SDR and other stuff. However. However, I'm unsure of how to do this properly. I trained my model with the following code:

from torch import optim
from pytorch_lightning import Trainer
from asteroid.losses import pairwise_neg_sisdr, PITLossWrapper
from asteroid.data import LibriMix
from asteroid.engine import System
from asteroid.models import DPRNNTasNet
from customdata import customdata
import torch
import torchvision

#train_loader, val_loader = LibriMix.loaders_from_mini(task="sep_clean", batch_size=16)
train_loader, val_loader,test_loader = customdata.loaders_from_mini(task="sep_clean", n_src=3, batch_size=4)
model = DPRNNTasNet(n_src=3)

loss = PITLossWrapper(pairwise_neg_sisdr, pit_from="pw_mtx")
optimizer = optim.Adam(model.parameters(), lr=1e-3)
system = System(model, optimizer, loss, train_loader, val_loader)
trainer = Trainer(max_epochs=1, gpus=1)
trainer.fit(system)
model.separate("trboth1.wav", force_overwrite=True,resample=True)
torch.save(model, "model1.pth")

I tried using trainer.test(model=system,dataloaders=test_loader), but I got the following error:
test
Drop 0 utterances from 3000 (shorter than 3 seconds)
Drop 0 utterances from 1000 (shorter than 3 seconds)
Drop 0 utterances from 500 (shorter than 3 seconds)
GPU available: True, used: True
TPU available: False, using: 0 TPU cores
IPU available: False, using: 0 IPUs
HPU available: False, using: 0 HPUs
Traceback (most recent call last):
File "/mnt/c/asteroid/egs/librimix/DPRNNTasNet/pretrained1.py", line 43, in
trainer.test(model=system,dataloaders=test_loader)#dataloaders=test_loader ,ckpt_path="best"
File "/home/user/anaconda3/envs/asteroid3/lib/python3.10/site-packages/pytorch_lightning/trainer/trainer.py", line 938, in test
return self._call_and_handle_interrupt(self._test_impl, model, dataloaders, ckpt_path, verbose, datamodule)
File "/home/user/anaconda3/envs/asteroid3/lib/python3.10/site-packages/pytorch_lightning/trainer/trainer.py", line 723, in _call_and_handle_interrupt
return trainer_fn(*args, **kwargs)
File "/home/user/anaconda3/envs/asteroid3/lib/python3.10/site-packages/pytorch_lightning/trainer/trainer.py", line 985, in _test_impl
results = self._run(model, ckpt_path=self.ckpt_path)
File "/home/user/anaconda3/envs/asteroid3/lib/python3.10/site-packages/pytorch_lightning/trainer/trainer.py", line 1160, in _run
verify_loop_configurations(self)
File "/home/user/anaconda3/envs/asteroid3/lib/python3.10/site-packages/pytorch_lightning/trainer/configuration_validator.py", line 46, in verify_loop_configurations
__verify_eval_loop_configuration(trainer, model, "test")
File "/home/user/anaconda3/envs/asteroid3/lib/python3.10/site-packages/pytorch_lightning/trainer/configuration_validator.py", line 210, in __verify_eval_loop_configuration
raise MisconfigurationException(f"No {step_name}() method defined to run Trainer.{trainer_method}.")
pytorch_lightning.utilities.exceptions.MisconfigurationException: No test_step() method defined to run Trainer.test.

How can I make it so that I can evaluate my model?

[anontiger]

I tried to use the eval.py to evaluate it like the following:
python eval.py --test_dir ./MiniLibriMix/metadata/test --task sep_clean --out_dir ./result

But since I didn't use train.py to train my model, it gave me the following error:
Traceback (most recent call last):
File "/mnt/c/asteroid/egs/librimix/DPRNNTasNet/eval.py", line 190, in
with open(conf_path) as f:
FileNotFoundError: [Errno 2] No such file or directory: 'exp/tmp/conf.yml

Is the only way to use eval.py to train by train.py? I am having difficulty training this way because I can't seem to figure out how to fit my custom dataset into train.py.

[mpariente]

If you specify the --exp_dir of your trained model, the eval script should work.

Otherwise, you can use the function compute_metrics on examples directly if you prefer.

[anontiger]

Thank you for your reply. I tried what you said.
Even if I specify the --exp_dir of my trained model, I get the same error, because my model is still lacking a conf.yml. That isn't generated since I trained my model without the train.py. I tried using the conf.yaml, but got "No such file or directory: test_annotations.csv". I solved that by manually putting in the absolute path of the test csv into the eval.py, but after that, I get the following error:
AttributeError: 'DPRNNTasNet' object has no attribute 'keys'

I'm quite lost on how to solve this part. Did I make an error with the way I saved my model? Does some step in train.py somehow give DPRNNTasNet the attribute 'keys'?

[mpariente]

I'm quite lost on how to solve this part. Did I make an error with the way I saved my model? Does some step in train.py somehow give DPRNNTasNet the attribute 'keys'?

No. keys is an attribute of a dictionary, so it's not asking it from the good object.

I think you should learn to debug, read stack trace etc.. and get to the root of the problem yourself, you'll progress a lot from that.

And otherwise, using compute_metrics should work, and the usage is documented.