Restoring Trainer State with Early Stop fails

[lersouza]

🐛 Bug

Whenever I try to restore the state of a previous (broken) run and I have Early Stop configured, I got the following error:

RuntimeError: Early stopping conditioned on metric `val/f1/default` which is not available. Pass in or modify your `EarlyStopping` callback to use any of the following: ``

It seems the metrics are not saved together with the Checkpoint data and Early Stop tries to be applied straight away.

To Reproduce

1. Start training a model with Checkpointing (best and last) and Early Stop on a validation metric. I have used Pytorch-Lightning Cli for this.
2. The model crashes after some checkpoints were saved
3. Try to re-run the training passing ckpt_path with the last checkpoint path to the trainer (with the same Early Stopping settings).
4. The error happens.

Expected behavior

The model should resume training, knowing the last best value, and apply Early Stopping as appropriate.

Environment

• CUDA:
  • GPU:
    • NVIDIA A100-SXM4-80GB
  • available: True
  • version: 11.4
• Packages:
  • numpy: 1.21.2
  • pyTorch_debug: False
  • pyTorch_version: 1.10.0a0+0aef44c
  • pytorch-lightning: 1.6.1
  • tqdm: 4.62.3
• System:
  • OS: Linux
  • architecture:
    • 64bit
    • ELF
  • processor: x86_64
  • python: 3.8.12
  • version: LR scheduler + train refactor #103-Ubuntu SMP Fri Nov 26 16:13:00 UTC 2021

Additional context

Running using the Pytorch Lightning Cli

cc @carmocca @awaelchli @rohitgr7

[DorianKodelja]

I face the same problem:

CUDA: no (training on cpu)

Packages:

numpy: 1.22.0
pyTorch_debug: False
pyTorch_version: 1.11.0
pytorch-lightning: 1.6.0
tqdm: 4.63.1

System:

OS: Linux
architecture:
    64bit
    ELF
processor: x86_64
python: 3.8.12

[SarBH]

I have the same issue:

I train a model with a lr_scheduler dict that looks like this:

def configure_optimizers(self):
        return {"optimizer": optimizer,
           "lr_scheduler: {
                "scheduler": scheduler,
                "monitor": "val_loss",
                  ...} }

is monitoring val_loss (needless to note validation_step is running self.log("val_loss",...))
I attempt to load via trainer.fit(model=model, datamodule=data, ckpt_path="/path/to.ckpt")
and I get an error:

  File "/usr/local/lib/python3.8/site-packages/pytorch_lightning/callbacks/early_stopping.py", line 145, in _validate_condition_metric
    raise RuntimeError(error_msg)
RuntimeError: Early stopping conditioned on metric `val_loss` which is not available. Pass in or modify your `EarlyStopping` callback to use any of the following: ``

But in fact EarlyStopping.monitor="val_loss" and why is the list of available options for metrics empty?

Thats why im forced to load via the MyLightningModule.load_from_checkpoint(ckpt_path) but then i lose access to hooks like on_load_checkpoint which I really need for my specific application.

# pip show pytorch-lightning
Name: pytorch-lightning
Version: 1.6.2

[carmocca]

Is this still an issue with version 1.7.0? I believe similar bugs have been fixed cc @rohitgr7

[thawro]

I had the same problem when saving the checkpoint with ModelCheckpoint initialized with every_n_train_steps=1. The problem disappeared when I used every_n_epochs=1 instead of the every_n_train_steps.

pytorch-lightning: 1.8.6

[carmocca]

Can you provide a reproducible bug snippet? You can update this script

[stale]

This issue has been automatically marked as stale because it hasn't had any recent activity. This issue will be closed in 7 days if no further activity occurs. Thank you for your contributions - the Lightning Team!

[thesofakillers]

Not stale, also just got this issue. Weirdly only happens when resuming a wandb run