(feature request) include tensorboard summary in major releases

[nicolaspanel]

The idea is to let people see how training goes for you on Fisher, LibriSpeech and Switchboard corpora and help identify weird training behaviors such as:

• over-fitting
• weird gradients
• etc.

[lissyx]

Didn't we said in a meeting it's here but just not enabled by default for performance reasons ? @reuben

[reuben]

Yes, full logging of variables and gradients slows down our training significantly. I'm planning to make the logging more useful by default but haven't had the time to work on it.

…

On 13 Jun 2019, at 11:02, lissyx ***@***.***> wrote: Didn't we said in a meeting it's here but just not enabled by default for performance reasons ? @reuben — You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub, or mute the thread.

[lissyx]

@reuben I think this was improved, maybe we could ship them now ?

[reuben]

It's still disabled by default, the only thing that changed is we have a finer trained loss progression graph (per step). We can share that, provided it was saved in the recent training runs, but ideally we need to add more useful information to diagnose training runs.

[reuben]

But Eren has told me that TensorBoard logging causes significant file system issues in the cluster, so I'm a bit cautious of increasing our logging there.