Follow PyTorch Guide of updating Learning Rate

[hyecheol123]

#1100

I modified the code slightly, adding conditional statement to avoid calling decaying function on the first epoch

[junyanz]

Thanks for the PR. If this condition is added, it seems that the learning rate will not change during the first epoch of contine_train. I am wondering if there is a workaround solution. @hyecheol123 @taesungp

[hyecheol123]

I did not considered continue training while writing that code and will take a look whether there is a good solution to handle it.

[hyecheol123]

I have one question regarding the starting epoch without continue_train option.

When continue_train does not specified, does it always starts train from epoch 1?
In this case, discussion from the PyTorch forum might be useful.

[junyanz]

The epoch_count is 1 by default. Without continue_train, it will start as 1.

[hyecheol123]

I will try to fix the code so that we can get proper learning rate decay for continued train.