Skip to content

Commit 93d46da

Browse files
mwevermwever
committed
Added interface for optuna.
1 parent 639e2c6 commit 93d46da

9 files changed

Lines changed: 1124 additions & 63 deletions

File tree

‎.gitignore‎

Lines changed: 1 addition & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -212,3 +212,4 @@ __marimo__/
212212

213213
# IDE project files
214214
.idea
215+
.DS_Store

‎examples/ablation-example.ipynb‎

Lines changed: 4 additions & 25 deletions
Original file line numberDiff line numberDiff line change
@@ -18,9 +18,8 @@
1818
"id": "initial_id",
1919
"metadata": {
2020
"collapsed": true,
21-
"ExecuteTime": {
22-
"end_time": "2025-08-19T15:17:31.718520Z",
23-
"start_time": "2025-08-19T15:17:31.336310Z"
21+
"jupyter": {
22+
"is_executing": true
2423
}
2524
},
2625
"source": [
@@ -73,28 +72,8 @@
7372
"print(\"\")\n",
7473
"print(\"Config of interest\", config_of_interest, \"\\nValue\", eval_fun(config_of_interest))"
7574
],
76-
"outputs": [
77-
{
78-
"name": "stdout",
79-
"output_type": "stream",
80-
"text": [
81-
"Baseline Configuration(values={\n",
82-
" 'a': 0.6243561663863,\n",
83-
" 'b': 4,\n",
84-
" 'c': np.str_('Y'),\n",
85-
"}) \n",
86-
"Value 0.7004003054122951\n",
87-
"\n",
88-
"Config of interest Configuration(values={\n",
89-
" 'a': 1.45774946311,\n",
90-
" 'b': 10,\n",
91-
" 'c': np.str_('X'),\n",
92-
"}) \n",
93-
"Value 10.99361700532433\n"
94-
]
95-
}
96-
],
97-
"execution_count": 1
75+
"outputs": [],
76+
"execution_count": null
9877
},
9978
{
10079
"cell_type": "markdown",

‎examples/optuna-example.ipynb‎

Lines changed: 261 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,261 @@
1+
{
2+
"cells": [
3+
{
4+
"cell_type": "markdown",
5+
"id": "74b6c2461143b273",
6+
"metadata": {},
7+
"source": [
8+
"# HyperSHAP: Example for Optuna Integration\n",
9+
"\n",
10+
"In this example, we demonstrate how to load data from an [optuna](https://optuna.org/) study directly into HyperSHAP for downstream hyperparameter analysis.\n",
11+
"\n",
12+
"This is useful when you have already run an optuna HPO study and want to understand *why* certain hyperparameters matter more than others \u2014 without having to redefine a `ConfigSpace` manually.\n",
13+
"\n",
14+
"> **Prerequisites:** `optuna` must be installed.\n",
15+
"> ```bash\n",
16+
"> pip install optuna\n",
17+
"> # or\n",
18+
"> pip install hypershap[optuna]\n",
19+
"> ```\n",
20+
"\n",
21+
"## Step 1 \u2014 Run an optuna study\n",
22+
"\n",
23+
"We first set up a small synthetic objective and run an optuna study to mimic a realistic HPO scenario.\n",
24+
"The objective uses a float, an integer, and a categorical hyperparameter \u2014 the same types supported by the optuna integration."
25+
]
26+
},
27+
{
28+
"cell_type": "code",
29+
"execution_count": null,
30+
"id": "initial_id",
31+
"metadata": {},
32+
"outputs": [],
33+
"source": [
34+
"from __future__ import annotations\n",
35+
"\n",
36+
"import math\n",
37+
"\n",
38+
"import optuna\n",
39+
"\n",
40+
"optuna.logging.set_verbosity(optuna.logging.WARNING) # suppress per-trial logs\n",
41+
"\n",
42+
"\n",
43+
"def objective(trial: optuna.Trial) -> float:\n",
44+
" \"\"\"Synthetic objective that mimics a tunable ML algorithm.\n",
45+
"\n",
46+
" Hyperparameters\n",
47+
" ---------------\n",
48+
" a : float in [0.1, 1.5] \u2014 learning-rate-like continuous parameter\n",
49+
" b : int in [2, 10] \u2014 depth-like integer parameter\n",
50+
" c : str in {\"X\", \"Y\"} \u2014 algorithm variant (categorical)\n",
51+
" \"\"\"\n",
52+
" a = trial.suggest_float(\"a\", 0.1, 1.5)\n",
53+
" b = trial.suggest_int(\"b\", 2, 10)\n",
54+
" c = trial.suggest_categorical(\"c\", [\"X\", \"Y\"])\n",
55+
"\n",
56+
" # Variant X: performance mainly driven by b, slightly by a\n",
57+
" if c == \"X\":\n",
58+
" return math.sin(a) + b\n",
59+
" # Variant Y: interaction between a and b dominates\n",
60+
" return math.cos(a * b) + 1.5\n",
61+
"\n",
62+
"\n",
63+
"# Run a maximisation study with 200 trials\n",
64+
"study = optuna.create_study(direction=\"maximize\", sampler=optuna.samplers.TPESampler(seed=42))\n",
65+
"study.optimize(objective, n_trials=200)\n",
66+
"\n",
67+
"print(f\"Completed trials : {len(study.trials)}\")\n",
68+
"print(f\"Best value : {study.best_value:.4f}\")\n",
69+
"print(f\"Best params : {study.best_params}\")"
70+
]
71+
},
72+
{
73+
"cell_type": "markdown",
74+
"id": "97bf13e91a18e32e",
75+
"metadata": {},
76+
"source": [
77+
"## Step 2 \u2014 Load the study into HyperSHAP\n",
78+
"\n",
79+
"`from_optuna_study` is the main entry point for the optuna integration. It:\n",
80+
"\n",
81+
"1. Extracts a `ConfigurationSpace` from the trial distributions.\n",
82+
"2. Converts all completed trial results into `(Configuration, float)` pairs.\n",
83+
"3. Fits a surrogate model (default: `RandomForestRegressor`) on those pairs.\n",
84+
"4. Returns an `ExplanationTask` ready for HyperSHAP analysis.\n",
85+
"\n",
86+
"For **minimisation** studies (`direction=\"minimize\"`) the objective values are automatically negated so that HyperSHAP's *higher-is-better* convention is respected. Pass `negate=False` to disable this."
87+
]
88+
},
89+
{
90+
"cell_type": "code",
91+
"execution_count": null,
92+
"id": "31b290fdd2788b74",
93+
"metadata": {},
94+
"outputs": [],
95+
"source": [
96+
"from hypershap import HyperSHAP, from_optuna_study\n",
97+
"\n",
98+
"# One-liner: study \u2192 ExplanationTask\n",
99+
"explanation_task = from_optuna_study(study)\n",
100+
"\n",
101+
"print(\"Config space HPs :\", explanation_task.get_hyperparameter_names())\n",
102+
"print(\"Number of HPs :\", explanation_task.get_num_hyperparameters())\n",
103+
"\n",
104+
"hypershap = HyperSHAP(explanation_task=explanation_task)"
105+
]
106+
},
107+
{
108+
"cell_type": "markdown",
109+
"id": "6d398a1ada8f99a2",
110+
"metadata": {},
111+
"source": "## Step 3 \u2014 Tunability analysis\n\nFor tunability we need a **baseline configuration** \u2014 the starting point from which we measure how much tuning each hyperparameter can improve performance. A natural choice is the *default configuration* of the inferred `ConfigurationSpace` (i.e. the midpoint/default of each hyperparameter range), which represents the algorithm before any tuning has taken place."
112+
},
113+
{
114+
"cell_type": "code",
115+
"execution_count": null,
116+
"id": "a248ad7cf028ced8",
117+
"metadata": {},
118+
"outputs": [],
119+
"source": [
120+
"# Use the default configuration of the inferred ConfigSpace as the baseline \u2014\n",
121+
"# this represents the algorithm with no tuning applied.\n",
122+
"default_config = explanation_task.config_space.get_default_configuration()\n",
123+
"print(\"Default (baseline) config:\", default_config)\n",
124+
"\n",
125+
"iv_tunability = hypershap.tunability(baseline_config=default_config)\n",
126+
"print(iv_tunability)"
127+
]
128+
},
129+
{
130+
"cell_type": "markdown",
131+
"id": "9c0b58a6ff43f6f1",
132+
"metadata": {},
133+
"source": [
134+
"### Visualisations"
135+
]
136+
},
137+
{
138+
"cell_type": "code",
139+
"execution_count": null,
140+
"id": "27ff3fe327082acf",
141+
"metadata": {},
142+
"outputs": [],
143+
"source": [
144+
"hypershap.plot_si_graph()"
145+
]
146+
},
147+
{
148+
"cell_type": "code",
149+
"execution_count": null,
150+
"id": "87fcba5335b5aa7c",
151+
"metadata": {},
152+
"outputs": [],
153+
"source": [
154+
"hypershap.plot_stacked_bar()"
155+
]
156+
},
157+
{
158+
"cell_type": "markdown",
159+
"id": "c3a1b2d0e4f5a6b7",
160+
"metadata": {},
161+
"source": "## Step 4 \u2014 Ablation analysis\n\nWe can also run an ablation analysis to understand which hyperparameters are responsible for the performance gain from the default configuration to the best configuration found by optuna."
162+
},
163+
{
164+
"cell_type": "code",
165+
"execution_count": null,
166+
"id": "d5e6f7a8b9c0d1e2",
167+
"metadata": {},
168+
"outputs": [],
169+
"source": [
170+
"from ConfigSpace import Configuration\n",
171+
"\n",
172+
"best_config = Configuration(\n",
173+
" explanation_task.config_space,\n",
174+
" values=study.best_params,\n",
175+
")\n",
176+
"\n",
177+
"iv_ablation = hypershap.ablation(\n",
178+
" config_of_interest=best_config, # optimized config found by optuna\n",
179+
" baseline_config=default_config, # default / untuned starting point\n",
180+
")\n",
181+
"print(iv_ablation)"
182+
]
183+
},
184+
{
185+
"cell_type": "code",
186+
"execution_count": null,
187+
"id": "f0a1b2c3d4e5f6a7",
188+
"metadata": {},
189+
"outputs": [],
190+
"source": [
191+
"hypershap.plot_waterfall()"
192+
]
193+
},
194+
{
195+
"cell_type": "markdown",
196+
"id": "b8c9d0e1f2a3b4c5",
197+
"metadata": {},
198+
"source": [
199+
"## Step 5 \u2014 Advanced: using the lower-level helpers\n",
200+
"\n",
201+
"If you need more control \u2014 e.g. to inspect the inferred `ConfigurationSpace`, filter trials manually, or pass a custom surrogate model \u2014 you can use the lower-level helpers directly."
202+
]
203+
},
204+
{
205+
"cell_type": "code",
206+
"execution_count": null,
207+
"id": "d6e7f8a9b0c1d2e3",
208+
"metadata": {},
209+
"outputs": [],
210+
"source": [
211+
"from sklearn.ensemble import GradientBoostingRegressor\n",
212+
"\n",
213+
"from hypershap.optuna_task import study_to_config_space, study_to_data\n",
214+
"\n",
215+
"# 1. Inspect the inferred configuration space\n",
216+
"cs = study_to_config_space(study)\n",
217+
"print(\"Inferred ConfigurationSpace:\")\n",
218+
"print(cs)\n",
219+
"\n",
220+
"# 2. Convert trials to (Configuration, float) pairs \u2014 apply custom filtering if needed\n",
221+
"data = study_to_data(study, config_space=cs)\n",
222+
"print(f\"\\nConverted {len(data)} trials to (Configuration, float) pairs.\")\n",
223+
"\n",
224+
"# 3. Build an ExplanationTask with a custom surrogate model\n",
225+
"from hypershap.task import ExplanationTask\n",
226+
"\n",
227+
"custom_task = ExplanationTask.from_data(\n",
228+
" config_space=cs,\n",
229+
" data=data,\n",
230+
" base_model=GradientBoostingRegressor(n_estimators=200, random_state=0),\n",
231+
")\n",
232+
"\n",
233+
"hs_custom = HyperSHAP(explanation_task=custom_task)\n",
234+
"iv_custom = hs_custom.tunability(baseline_config=default_config)\n",
235+
"print(\"\\nTunability with GradientBoostingRegressor surrogate:\")\n",
236+
"print(iv_custom)"
237+
]
238+
}
239+
],
240+
"metadata": {
241+
"kernelspec": {
242+
"display_name": "Python 3",
243+
"language": "python",
244+
"name": "python3"
245+
},
246+
"language_info": {
247+
"codemirror_mode": {
248+
"name": "ipython",
249+
"version": 3
250+
},
251+
"file_extension": ".py",
252+
"mimetype": "text/x-python",
253+
"name": "python",
254+
"nbformat_minor": 5,
255+
"pygments_lexer": "ipython3",
256+
"version": "3.10.0"
257+
}
258+
},
259+
"nbformat": 4,
260+
"nbformat_minor": 5
261+
}

0 commit comments

Comments
 (0)