Support asynchronous evaluation during training

[athitten]

This is to better identify how the model is performing during the training/finetuning compared to the base model and report evaluation metrics from the eval benchmarks.
This is also a request from the Customizer team.