Closed
Description
Hello,
I have an optimization task on the hyperparameters of a model with following constraints:
- The hparams are mostly int-type ranged parameters and it's impossible to try out every combination because the search space is very large.
- The metric is kind of noisy (high variance).
- The time consumption of running one evaluation is high.
Is there a way to deal with high variance in the observation? For now, I can only come up with:
- For an arm, run the evaluation k times and report the mean and sem to the BO model
- One evaluation for one arm and when the BO model suggests an existed trial, run another evaluation and update the mean and sem accordingly (which relates to [Sobol fallback needed] Repeated trials in experiment (and numerical errors they sometimes cause:
RuntimeError: cholesky_cpu: U(63,63) is zero, singular U.
) #228).
My questions are:
- Which way above is better?
- Is there a better way to deal with this kind of situation?
Thanks!
Activity