add to docs page

Author: janfb

docs/how_to_guide.rst

@@ -46,6 +46,7 @@ Training
    how_to_guide/07_gpu_training.ipynb
    how_to_guide/07_save_and_load.ipynb
    how_to_guide/07_resume_training.ipynb
+   how_to_guide/21_hyperparameter_tuning.ipynb
 
 
 Sampling

docs/how_to_guide/21_hyperparameter_tuning.ipynb

@@ -11,22 +11,18 @@
    "cell_type": "markdown",
    "metadata": {},
    "source": [
-    "This guide shows a minimal [`optuna`](https://optuna.org/) loop for hyperparameter tuning in `sbi`. It uses a toy simulator, `NPE`, an embedding network, and the `posterior_nn` helper. We tune just two hyperparameters: the embedding dimension and the number of flow transforms in an `nsf` density estimator."
-   ]
-  },
-  {
-   "cell_type": "markdown",
-   "metadata": {},
-   "source": [
-    "Optuna is not a dependency of `sbi`, you need to install it yourself in your\n",
+    "This guide shows a minimal [`optuna`](https://optuna.org/) loop for hyperparameter\n",
+    "tuning in `sbi`. Optuna is a lightweight hyperparameter optimization library. You define\n",
+    "an objective function that trains a model (e.g., NPE) and returns a validation metric,\n",
+    "and Optuna runs multiple trials to explore the search space and track the best\n",
+    "configuration. As validation metric, we recommend using the negative log probability of\n",
+    "a held-out validation set `(theta, x)` under the current posterior estimate (see\n",
+    "Lueckmann et al. 2021 for details). \n",
+    "\n",
+    "Note that Optuna is not a dependency of `sbi`, you need to install it yourself in your\n",
     "environment. \n",
     "\n",
-    "Optuna is a lightweight hyperparameter optimization library. You define an objective\n",
-    "function that trains a model (e.g., NPE) and returns a validation metric, and Optuna runs multiple\n",
-    "trials to explore the search space and track the best configuration. As validation\n",
-    "metric, we recommend using the negative log probability of a held-out validation set\n",
-    "`(theta, x)` under the current posterior estimate (see Lueckmann et al. 2021 for\n",
-    "details). "
+    "Here, we use a toy simulator and do `NPE` with an embedding network built using the `posterior_nn` helper. We tune just two hyperparameters: the embedding dimension and the number of flow transforms in an `nsf` density estimator."
    ]
   },
   {
@@ -95,12 +91,27 @@
   },
   {
    "cell_type": "markdown",
+   "id": "aad395b1",
    "metadata": {},
    "source": [
     "## Run the study and retrain\n",
     "\n",
+    "Optuna defaults to the TPE sampler, which is a good starting point for many experiments.\n",
+    "TPE (Tree-structured Parzen Estimator) is a Bayesian optimization method that\n",
+    "models good vs. bad trials with nonparametric densities and samples new points\n",
+    "that are likely to improve the objective. You can swap in other samplers (random\n",
+    "search, GP-based, etc.) by passing a different sampler instance to `create_study`.\n",
+    "\n",
+    "The TPE sampler uses `n_startup_trials` random trials to seed the model. With\n",
+    "`n_trials=25` and `n_startup_trials=10`, the first 10 trials are random and the\n",
+    "remaining 15 are guided by the acquisition function. If you want to ensure to start at\n",
+    "the default configuration, _enqueue_ it before optimization.\n",
+    "\n",
     "```python\n",
-    "study = optuna.create_study(direction=\"minimize\")\n",
+    "sampler = optuna.samplers.TPESampler(n_startup_trials=10)\n",
+    "study = optuna.create_study(direction=\"minimize\", sampler=sampler)\n",
+    "# Optional: ensure the default config is evaluated\n",
+    "study.enqueue_trial({\"embedding_dim\": 32, \"num_transforms\": 4})\n",
     "# This will run the above NPE training up to 25 times\n",
     "study.optimize(objective, n_trials=25)\n",
     "\n",
@@ -121,16 +132,6 @@
     "posterior = inference.build_posterior(final_estimator)\n",
     "```"
    ]
-  },
-  {
-   "cell_type": "markdown",
-   "metadata": {},
-   "source": [
-    "## Notes\n",
-    "\n",
-    "- The toy simulator keeps the example short. Replace it with your simulator and prior.\n",
-    "- You can expand the search space with additional `posterior_nn` arguments (e.g., `hidden_features`)."
-   ]
   }
  ],
  "metadata": {