Feat (benchmark): Allow easy specification of different search algorithms within benchmark scripts

[nickfraser]

Builds on #1356. Introduces a BenchmarkSearchMixin which determines how we generate experiments with the benchmark utils. The first one GridSearchMixin, replicates the behaviour of the benchmark utils before this PR was made.

The second one is a RandomSearchMixin, which allow specification of various search algorithms that can be used to specify parameters, for example:

act_equalization:
  rand_type: choices
  rand_values: [null, "layerwise", "fx"]
act_equalization_alpha:
  rand_type: linear
  rand_values: [0.05, 0.95]
gptq:
  rand_type: const
  rand_values: true
learned_round_lr:
  rand_type: log2
  rand_values: [0.0001, 0.1]
learned_round_scale_lr:
  rand_type: exp2
  rand_values: [0.0001, 0.1]

Running as follows:

python llm_rand_benchmark.py --config benchmark_rand_template.yaml --dry-run --seed 1 --num-experiments 10

Will give the following output:

Num. experiments: 10
Benchmark args.:
        config: benchmark_rand_template.yaml
        results_folder: ./
        gpus: 0
        num_gpus_per_process: 1
        max_num_retries: 1
        dry_run: True
        num_experiments: 10
        max_experimental_configs: 100000
        seed: 1
Non-default args.:
        --act-equalization: type: choices, values: [None, 'layerwise', 'fx']
        --act-equalization-alpha: type: linear, min: 0.05, max: 0.95
        --gptq: type: const, value: True
        --learned-round-lr: type: log2, min: 0.0001, max: 0.1
        --learned-round-scale-lr: type: exp2, min: 0.0001, max: 0.1

Note, a limitation with the current approach is that the worker queue generation (i.e., the random search space) and execution (i.e., the experiments) are two separate steps, meaning that it's difficult to integrate experiment feedback into the search space selection (e.g., for Bayesian optimization or simulated annealing).

[nickfraser]

Switch to Mixin.

[pablomlago]

Indicate that the concrete classes need to provide the abstract class variable argument_parser, e.g.

    @property
    @abstractmethod
    def argument_parser(self) -> ArgumentParser:
        pass

[nickfraser]

Not 100% sure about this... It's already specified in class BenchmarkUtils(ABC):. You want it in class BenchmarkSearchMixin(ABC): as well? Or instead of or as well as BenchmarkUtils?

[pablomlago]

There is a subset of the functionality of standardize_args which is shared for GridSearchMixin and RandomSearchMixing, e.g. YAML reading. Is it possible to extract the common functionality to the abstract parent class.

[pablomlago]

Most of the arguments are shared between GridSearchMixin and RandomSearchMixin, I would consider extracting the common logic.

[pablomlago]

Maybe it is worth renaming q to something more self-explanatory.