-
Notifications
You must be signed in to change notification settings - Fork 62
Add Trackio Callback for Optuna Registry #377
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Open
ParagEkbote
wants to merge
7
commits into
optuna:main
Choose a base branch
from
ParagEkbote:add-trackio-integration
base: main
Could not load branches
Branch not found: {{ refName }}
Loading
Could not load tags
Nothing to show
Loading
Are you sure you want to change the base?
Some commits from the old base branch may be removed from the timeline,
and old review comments may become outdated.
Open
Changes from all commits
Commits
Show all changes
7 commits
Select commit
Hold shift + click to select a range
a83d766
add the trackio callback.
ParagEkbote 01fd93f
lint the files and add license file.
ParagEkbote 022321c
lint files.
ParagEkbote b26842e
Update package/callbacks/trackio/LICENSE
ParagEkbote 60ca8e4
update readme to rm extra installation command.
ParagEkbote c0cd667
Merge branch 'optuna:main' into add-trackio-integration
ParagEkbote 112b971
lint the files and fix bug in the callback.
ParagEkbote File filter
Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
There are no files selected for viewing
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,21 @@ | ||
| MIT License | ||
|
|
||
| Copyright (c) 2026 y0z, neel04, nabenabe0928, and nzw0301 | ||
|
|
||
| Permission is hereby granted, free of charge, to any person obtaining a copy | ||
| of this software and associated documentation files (the "Software"), to deal | ||
| in the Software without restriction, including without limitation the rights | ||
| to use, copy, modify, merge, publish, distribute, sublicense, and/or sell | ||
| copies of the Software, and to permit persons to whom the Software is | ||
| furnished to do so, subject to the following conditions: | ||
|
|
||
| The above copyright notice and this permission notice shall be included in all | ||
| copies or substantial portions of the Software. | ||
|
|
||
| THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR | ||
| IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY, | ||
| FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE | ||
| AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER | ||
| LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM, | ||
| OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE | ||
| SOFTWARE. |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,191 @@ | ||
| --- | ||
| author: ParagEkbote | ||
| title: Trackio Callback | ||
| description: A callback to track Optuna trials with Trackio. | ||
| tags: [callback, trackio, logging, built-in] | ||
| optuna_versions: [4.9.0] | ||
| license: MIT License | ||
| --- | ||
|
|
||
| ## Installation | ||
|
|
||
| ```shell | ||
| pip install trackio | ||
| ``` | ||
|
|
||
| ## Abstract | ||
|
|
||
| This callback enables tracking of Optuna studies in Trackio. By default, the study is tracked as a single experiment run, where all suggested hyperparameters and optimized metrics are logged and visualized as a function of optimizer steps. | ||
|
|
||
| Trackio is offline-first and does not require authentication for local experiment tracking. Optionally, tracked experiments can be synchronized to Hugging Face Spaces for remote visualization and sharing. | ||
|
|
||
| The callback also supports multi-run mode, where each Optuna trial is tracked as an independent Trackio run. This is useful for sweep-style dashboards, parameter importance analysis, and per-trial experiment inspection. | ||
|
|
||
| ## APIs | ||
|
|
||
| - `TrackioCallback(project: str, metric_name: str | Sequence[str] = "value", as_multirun: bool = False, space_id: str | None = None, dataset_id: str | None = None, private: bool | None = None, resume: str = "allow", sync_on_finish: bool = False, sync_frequency: str = "study", sync_run_in_background: bool = False, trackio_kwargs: dict[str, Any] | None = None)` | ||
|
|
||
| - `project`: | ||
| Name of the Trackio project used for local storage and optional synchronization. | ||
|
|
||
| - `metric_name`: | ||
| Name assigned to the optimized metric. In case of multi-objective optimization, a list of names can be passed. These names will be assigned to objective values in the order returned by the objective function. | ||
|
|
||
| If a single name is provided, it will be broadcast to multiple objectives using numerical suffixes such as `value_0`, `value_1`. | ||
|
|
||
| - `as_multirun`: | ||
| Creates a new Trackio run for each Optuna trial. Useful for generating sweep-style dashboards and trial-level visualizations. | ||
|
|
||
| - `space_id`: | ||
| Optional Hugging Face Space ID (`"username/space-name"`) used for synchronization and remote visualization. | ||
|
|
||
| - `dataset_id`: | ||
| Optional Hugging Face Dataset ID used for exporting experiment metadata and metrics. | ||
|
|
||
| - `private`: | ||
| Whether synchronized Hugging Face artifacts should be private. | ||
|
|
||
| - `resume`: | ||
| Resume policy for Trackio runs. Accepted values are `"allow"`, `"must"`, and `"never"`. | ||
|
|
||
| - `sync_on_finish`: | ||
| Whether to synchronize the project to Hugging Face after study completion. | ||
|
|
||
| - `sync_frequency`: | ||
| Synchronization frequency strategy. | ||
|
|
||
| - `"study"` synchronizes once after the study completes. | ||
| - `"trial"` synchronizes after each completed trial in multirun mode. | ||
|
|
||
| - `sync_run_in_background`: | ||
| Whether synchronization should run asynchronously in a background thread. | ||
|
|
||
| - `trackio_kwargs`: | ||
| Additional keyword arguments passed directly to `trackio.init()`. | ||
|
|
||
| - `TrackioCallback.track_in_trackio() -> Callable` | ||
|
|
||
| - Decorator for enabling Trackio logging inside the objective function. | ||
|
|
||
| The decorator initializes and finalizes Trackio runs automatically. Additional metrics logged inside the objective function using `trackio.log()` are associated with the corresponding Optuna trial run. | ||
|
|
||
| Use as: | ||
|
|
||
| ```python | ||
| @trackioc.track_in_trackio() | ||
| ``` | ||
|
|
||
| - `TrackioCallback.finish() -> None` | ||
|
|
||
| - Explicitly finalizes synchronization and cleanup after `study.optimize()` completes. | ||
|
|
||
| ## Example | ||
|
|
||
| ### Add Trackio callback to Optuna optimization | ||
|
|
||
| ```python | ||
| import optuna | ||
| import optunahub | ||
|
|
||
|
|
||
| module = optunahub.load_module("callbacks/trackio") | ||
| TrackioCallback = module.TrackioCallback | ||
|
|
||
|
|
||
| def objective(trial): | ||
| x = trial.suggest_float("x", -10, 10) | ||
| return (x - 2) ** 2 | ||
|
|
||
|
|
||
| study = optuna.create_study( | ||
| study_name="trackio-demo", | ||
| ) | ||
|
|
||
| trackioc = TrackioCallback( | ||
| project="my-optuna-study", | ||
| ) | ||
|
|
||
| study.optimize( | ||
| objective, | ||
| n_trials=10, | ||
| callbacks=[trackioc], | ||
| ) | ||
|
|
||
| trackioc.finish() | ||
| ``` | ||
|
|
||
| ### Trackio logging in multirun mode | ||
|
|
||
| ```python | ||
| import optuna | ||
| import optunahub | ||
| import trackio | ||
|
|
||
|
|
||
| module = optunahub.load_module("callbacks/trackio") | ||
| TrackioCallback = module.TrackioCallback | ||
|
|
||
|
|
||
| trackioc = TrackioCallback( | ||
| project="my-optuna-study", | ||
| as_multirun=True, | ||
| ) | ||
|
|
||
|
|
||
| @trackioc.track_in_trackio() | ||
| def objective(trial): | ||
| x = trial.suggest_float("x", -10, 10) | ||
|
|
||
| # Additional logging inside Trackio | ||
| trackio.log( | ||
| { | ||
| "power": 2, | ||
| "base_of_metric": x - 2, | ||
| } | ||
| ) | ||
|
|
||
| return (x - 2) ** 2 | ||
|
|
||
|
|
||
| study = optuna.create_study( | ||
| study_name="trackio-multirun", | ||
| ) | ||
|
|
||
| study.optimize( | ||
| objective, | ||
| n_trials=10, | ||
| callbacks=[trackioc], | ||
| ) | ||
|
|
||
| trackioc.finish() | ||
| ``` | ||
|
|
||
| ## Notes | ||
|
|
||
| - Trackio synchronization to Hugging Face Spaces is eventually consistent and may take time to become remotely visible. | ||
|
|
||
| - For large studies or multirun experiments, it is strongly recommended to: | ||
|
|
||
| 1. complete the Optuna study locally first, | ||
| 1. verify local experiment tracking, | ||
| 1. then synchronize results to Hugging Face. | ||
|
|
||
| - In most cases, the recommended configuration is: | ||
|
|
||
| ```python | ||
| TrackioCallback( | ||
| ..., | ||
| sync_on_finish=True, | ||
| sync_frequency="study", | ||
| ) | ||
| ``` | ||
|
|
||
| instead of per-trial synchronization. | ||
|
|
||
| - Per-trial synchronization may significantly increase runtime due to repeated remote uploads and Hugging Face Space propagation delays. | ||
|
|
||
| - To ensure proper Trackio lifecycle management in multi-run mode, the objective function should always be wrapped with: | ||
|
|
||
| ```python | ||
| @trackioc.track_in_trackio() | ||
| ``` | ||
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,4 @@ | ||
| from .callback import TrackioCallback | ||
|
|
||
|
|
||
| __all__ = ["TrackioCallback"] |
Oops, something went wrong.
Oops, something went wrong.
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
I tried running the example locally, and got:
It works fine after adding
trackio.init(), but maybe it would be more user-friendly to include that setup in the example?Uh oh!
There was an error while loading. Please reload this page.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
There was an issue in the implementation of the callback where
__call__()did not correctly initialize the trials for trackio. I have updated the callback implementation to fix this and re-tested with the examples which now work correctly.