33Hyperparameter Optimization Benchmark with OpenML
44====================================================
55
6- In this tutorial, we walk through how to conduct hyperparameter optimization experiments using OpenML and OptunaHub.
6+ In this tutorial, we walk through how to conduct hyperparameter optimization experiments using
7+ OpenML and OptunaHub.
78"""
89
910############################################################################
1314
1415# License: BSD 3-Clause
1516
17+ import logging
18+
19+ import optuna
20+
1621import openml
1722from openml .extensions .sklearn import cat
1823from openml .extensions .sklearn import cont
19- import optuna
2024from sklearn .compose import ColumnTransformer
2125from sklearn .ensemble import RandomForestClassifier
2226from sklearn .impute import SimpleImputer
2327from sklearn .pipeline import Pipeline
2428from sklearn .preprocessing import OneHotEncoder
25- import logging
29+
2630
2731logger = logging .Logger (name = "Experiment Logger" , level = 1 )
2832
29- # Set your openml api key if you want to upload your results to OpenML (eg: https://openml.org/search?type=run&sort=date)
30- # To get one, simply make an account (you don't need one for anything else, just to upload your results), go to your profile
31- # and select the API-KEY. Or log in, and navigate to https://www.openml.org/auth/api-key
33+ # Set your openml api key if you want to upload your results to OpenML (eg:
34+ # https://openml.org/search?type=run&sort=date) . To get one, simply make an
35+ # account (you don't need one for anything else, just to upload your results),
36+ # go to your profile and select the API-KEY.
37+ # Or log in, and navigate to https://www.openml.org/auth/api-key
3238openml .config .apikey = ""
3339
3440############################################################################
3541# Prepare for preprocessors and an OpenML task
3642# ============================================
3743
3844# OpenML contains several key concepts which it needs to make machine learning research shareable.
39- # A machine learning experiment consists of one or several runs, which describe the performance of an algorithm (called a flow in OpenML),
40- # its hyperparameter settings (called a setup) on a task.
41- # A Task is the combination of a dataset, a split and an evaluation metric
42- # We choose a dataset from OpenML, (https://www.openml.org/d/1464) and a subsequent task (https://www.openml.org/t/10101)
43- # To make your own dataset and task, please refer to
44- # dataset - https://openml.github.io/openml-python/main/examples/30_extended/create_upload_tutorial.html#sphx-glr-examples-30-extended-create-upload-tutorial-py
45- # task - https://openml.github.io/openml-python/main/examples/30_extended/tasks_tutorial.html#sphx-glr-examples-30-extended-tasks-tutorial-py
45+ # A machine learning experiment consists of one or several runs, which describe the performance of
46+ # an algorithm (called a flow in OpenML), its hyperparameter settings (called a setup) on a task.
47+ # A Task is the combination of a dataset, a split and an evaluation metric We choose a dataset from
48+ # OpenML, (https://www.openml.org/d/1464) and a subsequent task (https://www.openml.org/t/10101) To
49+ # make your own dataset and task, please refer to
50+ # https://openml.github.io/openml-python/main/examples/30_extended/create_upload_tutorial.html
4651
4752# https://www.openml.org/search?type=study&study_type=task&id=218
4853task_id = 10101
5964# Define a pipeline for the hyperparameter optimization (this is standark for Optuna)
6065# =====================================================
6166
62- ## Optuna explanation
67+ # Optuna explanation
6368# we follow the `Optuna <https://github.com/optuna/optuna/>`__ search space design.
6469
65- ## OpenML runs
66- # We can simply pass the parametrized classifier to `run_model_on_task` to obtain the performance of the pipeline
70+ # OpenML runs
71+ # We can simply pass the parametrized classifier to `run_model_on_task` to obtain the performance
72+ # of the pipeline
6773# on the specified OpenML task.
68- # Do you want to share your results along with an easily reproducible pipeline, you can set an API key and just upload your results.
74+ # Do you want to share your results along with an easily reproducible pipeline, you can set an API
75+ # key and just upload your results.
6976# You can find more examples on https://www.openml.org/
7077
7178
@@ -90,7 +97,7 @@ def objective(trial: optuna.Trial) -> Pipeline:
9097 else :
9198 logger .log (
9299 0 ,
93- "If you want to publish your results to OpenML, please set an apikey using `openml.config.apikey = ''` " ,
100+ "If you want to publish your results to OpenML, please set an apikey" ,
94101 )
95102 accuracy = max (run .fold_evaluations ["predictive_accuracy" ][0 ].values ())
96103 logger .log (0 , f"Accuracy { accuracy } " )
0 commit comments