Training time increase between 0.8.0 and 0.9.0 versions due to Lightning 2.0 upgrade

[quant-exchange]

We use 2 instances of NP trainings in our pipeline per result and running them back-to-back, historically, we noticed around a 7–8-minute runtime using 0.8.0. However, when upgrading to 0.9.0, we're around between 10-12-minute range. Not the biggest deal but does bring up questions around how future updates could affect training times and explainability around this =).

NOTE: no changes to frequency, data feed (history) nor hyperparameter tuning was done*.

[ourownstory]

Thank you for raising this.

Computation times between 0.8.0 and 0.9.0 should be identical when training related hyperparameters are manually set.
Please provide some additional detail on your setup - do you manually set learning rate, epochs and batch size?

Note: There is a significant change from 0.9.0 to 1.0.0 release candidates, where memory use is reduced significantly, but training times are adversely affected. We are working on finding a better trade-off with lesser increase in training time. The memory use was optimized due to very large datasets running out of memory on some machines.

[quant-exchange]

@ourownstory yes, we use a YAML to configure NP with custom learning rate, epochs and batch size (these were not altered between updating 0.8.0 -> 0.9.0).

[ourownstory]

Thank you - I just looked at the change set, and the only significant change that I saw was the upgrade to Lightning 2.0. I suspect it may be connected to some of their upstream changes.

Would you mind profiling the runtimes for your implementation with 0.8.0 and 0.9.0? That may help us identify potential sources.

[quant-exchange]

We will work on setting up a profiling for our NP pipeline and report back when we have results (we were assuming it is related to Lightning 2.0, but appreciate it being updated to that version either way).

[ourownstory]

Sounds good - FYI we are also planning to add a mini tutorial for profiling: #1628

[MaiBe-ctrl]

Opened an issue at Pytorch Lightning for further investigation: Link to Issue

[quant-exchange]

I need to add another note here***: I'm using an NVIDIA 4070 RTX laptop GPU for these trainings, but these processes are running 24/7.

Any profiling I do; wanted to make sure that my requirements are biased towards using that configuration (GPU acceleration in general). We'll start working on this next.

[MaiBe-ctrl]

Thank you for sharing! We've conducted profiling on various devices (CPU, GPU, MPS) and observed an average 25% increase in training time after upgrading to Lightning 2.0. It would be great to compare this with your profiling results to see if our findings align.

[ourownstory]

that configuration (GPU acceleration in general)

RE: GPU acceleration: We have also noticed that the training time is higher on GPU than CPU for small to regular size models. Moving data to device takes some extra time. GPU acceleration does however speed up training for larger, i.e. deep NN with many lags/forecasts, and larger batches. Keep this in mind and make sure you do your profiling on a similar setup as you deploy.
(This is independent of Lightning 2.0)