Open
Description
(Created to ensure we keep track of a low priority issue.)
Status update emails are sent after every iteration if an email id is specified during training. However status information uses output of train/validation accuracy/log-probability computation jobs which are run in the background. In many cases these might be very slow to compute, depending on the model, as these are run on the CPU. This would mean that there are uninformative status mails which do not provide any info other than training time.
e.g.
%Iter duration train_loss valid_loss difference
Total training time is 0:59:53