Skip to content

[Question] How to deal with high variance in the observation? #244

Closed
@showgood163

Description

@showgood163

Hello,

I have an optimization task on the hyperparameters of a model with following constraints:

  1. The hparams are mostly int-type ranged parameters and it's impossible to try out every combination because the search space is very large.
  2. The metric is kind of noisy (high variance).
  3. The time consumption of running one evaluation is high.

Is there a way to deal with high variance in the observation? For now, I can only come up with:

  1. For an arm, run the evaluation k times and report the mean and sem to the BO model
  2. One evaluation for one arm and when the BO model suggests an existed trial, run another evaluation and update the mean and sem accordingly (which relates to [Sobol fallback needed] Repeated trials in experiment (and numerical errors they sometimes cause: RuntimeError: cholesky_cpu: U(63,63) is zero, singular U.) #228).

My questions are:

  1. Which way above is better?
  2. Is there a better way to deal with this kind of situation?

Thanks!

Activity

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    questionFurther information is requested

    Type

    No type

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions