Add "best_trial": best_trial.number to results_to_serialize in _impl.py

[evansdoe]

Motivation

Adding the best_trial_number to the final optimization_results.yaml file provides several key benefits for analysis, debugging, and automation.

---

Traceability and Debugging

The trial number serves as a unique identifier that directly links the summary results to the specific, detailed logs for that single run. Hydra saves each trial's output (like logs, model checkpoints, and plots) in a numbered subdirectory. Knowing the best_trial_number tells you exactly which folder to inspect to reproduce or debug the best-performing run without having to manually search for it.

---

Analysis of the Optimization Process

Knowing when the best result occurred during the hyperparameter search offers valuable insights:

• An early best trial (e.g., trial number 5 out of 100) might suggest the search space was good, but the optimizer quickly stagnated.
• A late best trial (e.g., trial number 98 out of 100) suggests the search was still finding improvements and that more trials might yield even better results.

---

Simplified Post-Processing

For automated workflows, having a direct identifier is crucial. A script can parse the results file, grab the best_trial_number, and use it to reliably retrieve the correct model artifact from the corresponding trial directory for deployment, further evaluation, or archiving. This makes the entire MLOps pipeline more robust and less complex.

Have you read the Contributing Guidelines on pull requests?

Yes

Test Plan

(How should this PR be tested? Do you require special setup to run the test or repro the fixed bug?)

No. All you need is to run an experiment and check if the best_trial_number is added to the optimization_results.yaml. If this is, then that is exactly the purpose of this PR.

Related Issues and PRs

(Is this PR part of a group of changes? Link the other relevant PRs and Issues here. Use https://help.github.com/en/articles/closing-issues-using-keywords for help on GitHub syntax)

No.