fix: make --continue_path work again

[eginhard]

TLDR: fixes loading of models via --continue_path

The issue

When resuming training via the --continue_path argument, first the following
error is logged, but training continues:

Traceback (most recent call last):
  File ".../lib/python3.10/logging/__init__.py", line 1100, in emit
    msg = self.format(record)
  File ".../lib/python3.10/logging/__init__.py", line 943, in format
    return fmt.format(record)
  File ".../lib/python3.10/logging/__init__.py", line 678, in format
    record.message = record.getMessage()
  File ".../lib/python3.10/logging/__init__.py", line 368, in getMessage
    msg = msg % self.args
TypeError: must be real number, not dict
Call stack:
  File ".../train_vits.py", line 120, in <module>
    trainer.fit()
  File ".../trainer/trainer.py", line 1826, in fit
    self._fit()
  File ".../trainer/trainer.py", line 1764, in _fit
    self._restore_best_loss()
  File ".../trainer/trainer.py", line 1728, in _restore_best_loss
    logger.info(" > Starting with loaded last best loss %f", self.best_loss)
Message: ' > Starting with loaded last best loss %f'
Arguments: {'train_loss': 18.22130616851475, 'eval_loss': None}

Then the following error occurs at the end of the epoch and training stops:

  File ".../trainer/io.py", line 183, in save_best_model
    if current_loss < best_loss:
TypeError: '<' not supported between instances of 'float' and 'dict'

This has been observed multiple times:

• Fixes [Bug]  #127
• Fixes [Bug] Logging error while continue training using --continue_path TTS#2880
• Fixes [Bug] Error when continuing training with vctk vits TTS#3069
• Fixes [Bug] Error occurs when resuming training a xtts model TTS#3131
• Fixes [Bug] Cannot restore from checkpoint TTS#3360

There are multiple open PRs to fix some aspects of this issue.

• Fix exception while saving a checkpoint after training was resumed #130
• made a change so that model can be saved when starting from checkpouint #128
• Update trainer.py #125

Others have fixed it in their Trainer forks:

• gattilorenz@8247031
• jmaty@dbc77f8

The reason

This error occurs because #121 changed

Trainer/trainer/trainer.py

Line 1924 in 47781f5

model_loss={"train_loss": train_loss, "eval_loss": eval_loss},

to save the model_loss as a dict instead of just a float.

Trainer/trainer/io.py

Line 195 in 47781f5

model_loss=current_loss,

still saves a float in model_loss, so loading the best model would still work fine. Loading a model via --restore-path also works fine because in that case the best loss is reset and not initialised from the saved model.

This fix

• changes save_best_model() to also save a dict with train and eval loss, so that this is consistent everywhere
• ensures that the model loader can handle both float and dict model_loss for backwards compatibility
• adds relevant test cases that would have previously failed

[erogol]

@eginhard thanks for the PR. There is a small fixup needed for a style issue. Else good to go.

[eginhard]

@erogol I simplified that if statement, should be good now.