How to use learning rate scheduler

[talhaanwarch]

Hi. In previous version of lightning i am using lr scheduler this way.

def configure_optimizers(self):
        opt=torch.optim.AdamW(params=self.parameters(),lr=self.lr )
        scheduler=CosineAnnealingLR(opt,T_max=10,  eta_min=1e-6, last_epoch=-1)
        return {'optimizer': opt,'lr_scheduler':scheduler}

but upon installing latest version, i am getting error.

MisconfigurationException: The provided lr scheduler `CosineAnnealingLR` doesn't follow PyTorch's LRScheduler API. You should override the `LightningModule.lr_scheduler_step` hook with your own logic if you are using a custom LR scheduler.

How i can i figure it out. I could find the example in documentation, if any one can provide a replacement code that would be great. I am not sure how to add lr_scheduler_step

[miladfa7]

You can add a lr_scheduler_step method inside the Lightning module class, which will be called by PyTorch Lightning at each step of the training loop to update the learning rate of the optimizer.

def configure_optimizers(self):
    opt=torch.optim.AdamW(params=self.parameters(),lr=self.lr )
    scheduler=CosineAnnealingLR(opt,T_max=10,  eta_min=1e-6, last_epoch=-1)

    # Define the lr_scheduler_step method to update the learning rate of the optimizer
    def lr_scheduler_step(self, epoch):
        scheduler.step()
        return {'lr': scheduler.get_last_lr()}
    
    return {'optimizer': opt, 'lr_scheduler': {'scheduler': scheduler, 'interval': 'step', 'monitor': 'val_loss'}}

[spolavar]

I tried your suggestion above, but still facing the same issue:

The provided lr scheduler CyclicLR doesn't follow PyTorch's LRScheduler API. You should override the 'LightningModule.lr_scedyler_step hook with your own logic if you are using a custom LR scheduler`

 if self.use_cyclic_lr:
            print(f"Using cyclic learning rate. min_lr: {self.learning_rate[0]}, max_lr: {self.learning_rate[1]}")
            lr_scheduler = torch.optim.lr_scheduler.CyclicLR(optimizer, base_lr=self.learning_rate[0], max_lr=self.learning_rate[1], mode='triangular2', cycle_momentum=False)
           return {'optimizer': opt, 'lr_scheduler': {'scheduler': scheduler, 'interval': 'step', 'monitor': 'val_loss'}}

Have been trying to get around this error since yesterday, didn't find help anywhere else