Skip to content
Open
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
21 changes: 21 additions & 0 deletions package/callbacks/trackio/LICENSE
Original file line number Diff line number Diff line change
@@ -0,0 +1,21 @@
MIT License

Copyright (c) 2026 y0z, neel04, nabenabe0928, and nzw0301

Permission is hereby granted, free of charge, to any person obtaining a copy
of this software and associated documentation files (the "Software"), to deal
in the Software without restriction, including without limitation the rights
to use, copy, modify, merge, publish, distribute, sublicense, and/or sell
copies of the Software, and to permit persons to whom the Software is
furnished to do so, subject to the following conditions:

The above copyright notice and this permission notice shall be included in all
copies or substantial portions of the Software.

THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR
IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,
FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE
AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER
LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM,
OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE
SOFTWARE.
191 changes: 191 additions & 0 deletions package/callbacks/trackio/README.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,191 @@
---
author: ParagEkbote
title: Trackio Callback
description: A callback to track Optuna trials with Trackio.
tags: [callback, trackio, logging, built-in]
optuna_versions: [4.9.0]
license: MIT License
---

## Installation

```shell
pip install trackio
```

## Abstract

This callback enables tracking of Optuna studies in Trackio. By default, the study is tracked as a single experiment run, where all suggested hyperparameters and optimized metrics are logged and visualized as a function of optimizer steps.

Trackio is offline-first and does not require authentication for local experiment tracking. Optionally, tracked experiments can be synchronized to Hugging Face Spaces for remote visualization and sharing.

The callback also supports multi-run mode, where each Optuna trial is tracked as an independent Trackio run. This is useful for sweep-style dashboards, parameter importance analysis, and per-trial experiment inspection.

## APIs

- `TrackioCallback(project: str, metric_name: str | Sequence[str] = "value", as_multirun: bool = False, space_id: str | None = None, dataset_id: str | None = None, private: bool | None = None, resume: str = "allow", sync_on_finish: bool = False, sync_frequency: str = "study", sync_run_in_background: bool = False, trackio_kwargs: dict[str, Any] | None = None)`

- `project`:
Name of the Trackio project used for local storage and optional synchronization.

- `metric_name`:
Name assigned to the optimized metric. In case of multi-objective optimization, a list of names can be passed. These names will be assigned to objective values in the order returned by the objective function.

If a single name is provided, it will be broadcast to multiple objectives using numerical suffixes such as `value_0`, `value_1`.

- `as_multirun`:
Creates a new Trackio run for each Optuna trial. Useful for generating sweep-style dashboards and trial-level visualizations.

- `space_id`:
Optional Hugging Face Space ID (`"username/space-name"`) used for synchronization and remote visualization.

- `dataset_id`:
Optional Hugging Face Dataset ID used for exporting experiment metadata and metrics.

- `private`:
Whether synchronized Hugging Face artifacts should be private.

- `resume`:
Resume policy for Trackio runs. Accepted values are `"allow"`, `"must"`, and `"never"`.

- `sync_on_finish`:
Whether to synchronize the project to Hugging Face after study completion.

- `sync_frequency`:
Synchronization frequency strategy.

- `"study"` synchronizes once after the study completes.
- `"trial"` synchronizes after each completed trial in multirun mode.

- `sync_run_in_background`:
Whether synchronization should run asynchronously in a background thread.

- `trackio_kwargs`:
Additional keyword arguments passed directly to `trackio.init()`.

- `TrackioCallback.track_in_trackio() -> Callable`

- Decorator for enabling Trackio logging inside the objective function.

The decorator initializes and finalizes Trackio runs automatically. Additional metrics logged inside the objective function using `trackio.log()` are associated with the corresponding Optuna trial run.

Use as:

```python
@trackioc.track_in_trackio()
```

- `TrackioCallback.finish() -> None`

- Explicitly finalizes synchronization and cleanup after `study.optimize()` completes.

## Example

### Add Trackio callback to Optuna optimization

```python
import optuna
import optunahub


module = optunahub.load_module("callbacks/trackio")
TrackioCallback = module.TrackioCallback


def objective(trial):
x = trial.suggest_float("x", -10, 10)
return (x - 2) ** 2


study = optuna.create_study(
study_name="trackio-demo",
)

trackioc = TrackioCallback(
project="my-optuna-study",
)

study.optimize(
objective,
n_trials=10,
callbacks=[trackioc],
)

trackioc.finish()
Comment on lines +87 to +114
Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I tried running the example locally, and got:

RuntimeError: Call trackio.init() before trackio.log().

It works fine after adding trackio.init(), but maybe it would be more user-friendly to include that setup in the example?

Copy link
Copy Markdown
Author

@ParagEkbote ParagEkbote May 31, 2026

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

There was an issue in the implementation of the callback where __call__() did not correctly initialize the trials for trackio. I have updated the callback implementation to fix this and re-tested with the examples which now work correctly.

```

### Trackio logging in multirun mode

```python
import optuna
import optunahub
import trackio


module = optunahub.load_module("callbacks/trackio")
TrackioCallback = module.TrackioCallback


trackioc = TrackioCallback(
project="my-optuna-study",
as_multirun=True,
)


@trackioc.track_in_trackio()
def objective(trial):
x = trial.suggest_float("x", -10, 10)

# Additional logging inside Trackio
trackio.log(
{
"power": 2,
"base_of_metric": x - 2,
}
)

return (x - 2) ** 2


study = optuna.create_study(
study_name="trackio-multirun",
)

study.optimize(
objective,
n_trials=10,
callbacks=[trackioc],
)

trackioc.finish()
```

## Notes

- Trackio synchronization to Hugging Face Spaces is eventually consistent and may take time to become remotely visible.

- For large studies or multirun experiments, it is strongly recommended to:

1. complete the Optuna study locally first,
1. verify local experiment tracking,
1. then synchronize results to Hugging Face.

- In most cases, the recommended configuration is:

```python
TrackioCallback(
...,
sync_on_finish=True,
sync_frequency="study",
)
```

instead of per-trial synchronization.

- Per-trial synchronization may significantly increase runtime due to repeated remote uploads and Hugging Face Space propagation delays.

- To ensure proper Trackio lifecycle management in multi-run mode, the objective function should always be wrapped with:

```python
@trackioc.track_in_trackio()
```
4 changes: 4 additions & 0 deletions package/callbacks/trackio/__init__.py
Original file line number Diff line number Diff line change
@@ -0,0 +1,4 @@
from .callback import TrackioCallback


__all__ = ["TrackioCallback"]
Loading
Loading