Create a benchmark based on PDEBench

[Deathn0t]

Let's define a benchmark suite based on the PDEBench: https://github.com/pdebench/PDEBench

[Deathn0t]

@iamyixuan advanced on this here: https://github.com/deephyper/benchmark/tree/main/lib/PINNBench/Diffusion-reaction

Next, we need to

• record learning curves metadata
• add skip connection capabilities
• add dataset selection capability DEEPHYPER_BENCHMARK_DATASET=...

[Deathn0t]

• document the possible values for DEEPHYPER_BENCHMARK_DATASET, what happens if another dataset is used? it looks like the data generation is happening in the installation.
• for the dataset document the storage size of the dataset
• fvcore.nn is not part of the installation procedure (not inside the requirements.txt) is it normal?
• the flops could be an additional objective in MOO
• would be great to have more parameters to tune such as:
  • regularization hyperparameters
    • dropout_rate
    • early_stopping_patience
    • batch_norm (batch normalization)
    • weight_decay
  • skip_co for skip connections (can be just a boolean with "regular skip co")
  • stochastic weight averaging
  • optimizer

[iamyixuan]

How to check the checkboxes above? Tried many times and only got to check two.

[Deathn0t]

• make weights as hyperparameters