[Minor] Enable continuation of training

[weberpals]

🔬 Background

Enable to continue the training of a model as described in #828.

Also needed to address #1542

🔮 Key changes

Load model state from the last checkpoint and continue training with a new scheduler. The last learning rate is used as a starting point for the new scheduler.

ExponentialLR is used as new scheduler since continuing with OneCycleLR leads to issues. @ourownstory please let me know if you have an idea for a specific scheduler that make sense here.

📋 Review Checklist

• I have performed a self-review of my own code.
• I have commented my code, added docstrings and data types to function definitions.
• I have added pytests to check whether my feature / fix works.

Please make sure to follow our best practices in the Contributing guidelines.

[ourownstory]

@Constantin343 "The last learning rate is used as a starting point for the new scheduler."
Do you mean the LR found in the LR test, or the actual LR at the end of the cycle?

[ourownstory]

Looks like you got a first version running - good work!

I left a few comments aiming to robustify the user settings and to encapsulate code in a similar manner to existing practices.

Furthermore, I think we should avoid hard-coding the optimizer and scheduler for this functionality. I suggest you make them optionally user-settable (with reasonable defaults). And if so, then it may make sense to do the same for the regular optimizer and scheduler in the same manner.

[ourownstory]

Can you please explain what necessitates this (for my understanding)?

[weberpals]

My thinking was that it makes sense to continue from the checkpoint, but probably it's not necessary. All the necessary parameters should still be available in the model itself. I will adapt it

[weberpals]

I did some testing and it seems like checkpoint is indeed necessary to correctly continue training with the pytroch-lighting trainer. I can get it to run without checkpointing but fitting again always leads to a complete restart of the training.

Maybe there is some workaround, but I would suggest keeping it like this as continued training always goes hand in hand with checkpointing in pytroch-lighting.

[weberpals]

@Constantin343 "The last learning rate is used as a starting point for the new scheduler." Do you mean the LR found in the LR test, or the actual LR at the end of the cycle?

Yeah, the learning rate at the end of the cycle (actually from the last checkpoint, to be precise). The idea was to ensure a smooth transition

[weberpals]

Furthermore, I think we should avoid hard-coding the optimizer and scheduler for this functionality. I suggest you make them optionally user-settable (with reasonable defaults). And if so, then it may make sense to do the same for the regular optimizer and scheduler in the same manner.

Totally agree. I tried again to make it work with the OneCycleLR scheduler, but it just doesn't seem to work with this scheduler. I will try to make some adaptations over the weekend to accept scheduler_type as a parameter, so the user can choose among different schedulers.

[codecov]

Codecov Report

❌ Patch coverage is 81.08108% with 14 lines in your changes missing coverage. Please review.
✅ Project coverage is 88.09%. Comparing base (fa97742) to head (df74dc3).
⚠️ Report is 25 commits behind head on main.

Files with missing lines	Patch %	Lines
neuralprophet/configure.py	68.18%	7 Missing ⚠️
neuralprophet/forecaster.py	89.18%	4 Missing ⚠️
neuralprophet/time_net.py	80.00%	3 Missing ⚠️

Additional details and impacted files

@@            Coverage Diff             @@
##             main    #1605      +/-   ##
==========================================
- Coverage   88.27%   88.09%   -0.18%     
==========================================
  Files          41       41              
  Lines        5364     5428      +64     
==========================================
+ Hits         4735     4782      +47     
- Misses        629      646      +17     

☔ View full report in Codecov by Sentry.
📢 Have feedback on the report? Share it here.

🚀 New features to boost your workflow:

• ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.