While running Priorband, the PriorbandSampler sometimes resamples configs. This would be fine as long as the configs don't already exist in all of the rungs of all the brackets (which is extremely unlikely). The issue here is that the same config is resampled and a lower rung (hence, fidelity value) is assigned to it. This would cause an issue when calculating continuations in the following manner: config: (f1, f2-f1, f3-f2-f1), where f3>f2>f1.
Answer the following questions as and when they're figured out:
-
Which sampling_policy inside the PriorbandSampler causes this issue?
Incumbent-based sampling
-
Are the sources of both the repeated samples the same? (which sampling_policy is used for each?)
- The config is sampled for the very first time by the
random sampling policy when the incumbent sampler is not yet active.
- The config is resampled during
incumbent-based sampling policy for a separate Hyperband bracket where the incumbent is not mutated at all and the same configuration is returned.
- This can happen for either of 2 reasons:
a. The mutate_config() function takes in the incumbent_config with the fidelity parameter and selects HPs for perturbation based on all of those. So, at times, the fidelity parameter is chosen but since the fidelity is not there in the mutation_prior which is generated from this incumbent_config, it's not really selected and the same configuration is passed back.
b. Sometimes, none of the HPs in the config might be selected if all the values randomly generated here is < mutation_rate. In that case the same config is returned.
-
Why is the resampled mutated_config assigned a lower rung than before?
-
How to assign a configuration sampled by the incumbent-based sampling policy to a rung? And thereafter, how to handle resamples?
-
Bookkeeping for continuations in Neps?
While running Priorband, the
PriorbandSamplersometimes resamples configs. This would be fine as long as the configs don't already exist in all of the rungs of all the brackets (which is extremely unlikely). The issue here is that the same config is resampled and a lower rung (hence, fidelity value) is assigned to it. This would cause an issue when calculating continuations in the following manner:config: (f1, f2-f1, f3-f2-f1), wheref3>f2>f1.Answer the following questions as and when they're figured out:
Which
sampling_policyinside thePriorbandSamplercauses this issue?Incumbent-basedsamplingAre the sources of both the repeated samples the same? (which
sampling_policyis used for each?)randomsampling policy when theincumbent sampleris not yet active.incumbent-basedsampling policy for a separateHyperbandbracket where the incumbent is not mutated at all and the same configuration is returned.a. The
mutate_config()function takes in theincumbent_configwith the fidelity parameter and selects HPs for perturbation based on all of those. So, at times, the fidelity parameter is chosen but since the fidelity is not there in themutation_priorwhich is generated from thisincumbent_config, it's not really selected and the same configuration is passed back.b. Sometimes, none of the HPs in the config might be selected if all the values randomly generated here is <
mutation_rate. In that case the same config is returned.Why is the resampled
mutated_configassigned a lower rung than before?How to assign a configuration sampled by the
incumbent-basedsampling policy to a rung? And thereafter, how to handle resamples?Bookkeeping for continuations in Neps?