Disabled Experiment Version Control

Experiment Version Control (EVC) will now be disabled by default. When the EVC is disabled, any changes in the experiment will be saved to the DB, overriding the previous version. See configuration doc to enable the EVC: https://orion.readthedocs.io/en/stable/user/config.html#experiment-version-control.

🏗 Enhancements

• Feature/filter duplicate in evc @bouthilx (#630)
• Add enable option for EVC, with default = false @bouthilx (#626)

🐛 Bug Fixes

• Warn only if diffs exists during exp build @bouthilx (#638)
• Fix lost trials of parents @bouthilx (#637)
• Compute cardinality for loguniform with precision @bouthilx (#635)
• Duplicate pending trials from parent/child for exc @bouthilx (#631)
• Remove space upgrade from DB upgrade @bouthilx (#627)
• Allow branching when parent script isn't available @bouthilx (#625)
• fix benchmark ranking and regret visualization @donglinjy (#620)
• fix tpe space cardinality @donglinjy (#619)

📜 Documentation

• Change algo documentation @bouthilx (#623)

New Executor Backend for Parallelism with Joblib and Dask

🚀 Features

• Add Executor class and CLI arguments for parallelism @bouthilx (#608)
• Orion python api in parallel @donglinjy (#601)

🏗 Enhancements

• Add has_suggested/observed, new test-suite and re-implementation of ASHA @bouthilx (#604)
• Refactor hunt command to reuse ExperimentClient.workon @bouthilx (#605)

🐛 Bug Fixes

• Test different type of categorical dims @bouthilx (#609)
• Add has_suggested/observed, new test-suite and re-implementation of ASHA @bouthilx (#604)

📜 Documentation

• Documentation for the visualizations @bouthilx (#557)
• Add has_suggested/observed, new test-suite and re-implementation of ASHA @bouthilx (#604)
• Add presentations section in doc @bouthilx (#597)

PickledDB is now the default storage backend

🚀 Features

• Add plot commandline plot @gyom (#593)

🏗️ Enhancements

• Add where parameter to Storage.fetch_trials @breuleux (#584)
• Make command orion db setup ask for right arguments based on storage backend. @notoraptor (#586)
• Change the default database to PickledDB @abergeron (#585)
• Deterministic algorithm benchmark @donglinjy (#567)

🐛 Bug Fixes

• Add context-manager-enabled Trial subclass. @obilaniu (#587)
• Handle cases when trial.status is not reserved anymore locally. @bouthilx (#591)

🧰 Maintenance

• Ignore all test files in coverage @bouthilx (#592)
• Fix for coverage issue @abergeron (#581)
• Remove MongoDB dependency from core tests @breuleux (#582)
• Write generic tests for DB @satyaog (#590)

Benchmark, Partial dependency plot and Grid Search

🚀 Features

• Add hover to partial deps scatter @bouthilx (#565)
• Make plots use trials in EVC by default @bouthilx (#564)
• Add error bar to LPI @bouthilx (#563)
• Add benchmark support into orion @donglinjy (#513)
• Add partial dependency plot @bouthilx (#544)
• Add grid search algorithm @bouthilx (#539)

🐛 Bug Fixes

• Set ignore_code_changes in Consumer too @bouthilx (#541)

📜 Documentation

• Major rehaul of documentation @bouthilx (#549)
• Fix readthedocs build @bouthilx (#542)

Minor release: Read/Write/Execution modes for Experiments, branching on orion version, bug fixes

🚀 Features

• Add Read/Write/Execution modes for Experiment @bouthilx (#536)
• Branch if Oríon version changes @bouthilx (#530)

🐛 Bug Fixes

• Detect code change when no CMDLINE passed @bouthilx (#529)
• Support EVC in all algos @bouthilx (#523)
• Test that precision isn't messed up by linearization @bouthilx (#522)
• Ignore non-completed trials when computing LPI @bouthilx (#521)
• Add debugging verbosity @bouthilx (#520, #528)

📜 Documentation

• Add missing plotting doc in API section @bouthilx (#532)

🧰 Maintenance

• Automatically update backward test versions @bouthilx (#512)
• Use pypi Token to release @bouthilx (#511)

Minor release: Support for python 3.9, bug fixes, improved CI

Changes

• Sync back with master @bouthilx (#496)

🚀 Features

• Add support for python 3.9 @bouthilx (#497)

🐛 Bug Fixes

• Use SoftFileLock if FS doesn't support global lock @bouthilx (#506)
• Fix algorithm requirements backward @bouthilx (#498)

🧰 Maintenance

• Add isort @bouthilx (#505)
• Add black checks and remove flake8 @bouthilx (#504)
• Fix drafter config @bouthilx (#502)
• Test release drafter @bouthilx (#501)
• Add release drafter @bouthilx (#499)

Parallel Coordinates & Local Parameter Importance Plots

Detailed list of changes

New features

• Add max_broken in exp configuration (#473)
• Add Parallel Coordinates Plot (#480)
• Add ReshapedSpace and Linearize transformer (#482)
• Add LPI plot (#483)

Bug Fixes

• Add extensive functional testing for configuration (#474)
• Handle rm/add properly in EVC (#475)
• Override fields with set, not update (#479)
• Fix cardinality and integer cast (#481)

Other improvements

• Improve CI (#464, #472)

Plotting API and Database commands

Important changes

New plotting API

Plotting capability is being added to experiment clients. You can now plot the regret (curve of best objective found during optimization) with simply experiment.plot.regret(). You can find an example here.

Web API (REST server)

A web API was added for v0.1.9 in order to support the visualization dashboard that is currently under work. See full documentation here.

Database commands rm & set

Command line helpers have been added to simplify the process of deleting experiments and trials as well as modifying trials in the database. See full documentation for both commands here.

Detailed list of changes

New features

• New plotting API (#400, #426)
• REST server (#437, #438, #441, #446)
• New command orion db rm (#425)
• New command orion db set (#428)

Breaking changes

• Drop support for orion < 0.1.6 (#420)
• Deprecate init_only and add hunt --init-only (#436)

Bug Fixes

• Fix backward compatibility tests in master - Drop support for orion < 0.1.6 (#420)
• Fix test using 'in' to assert equality (#427)
• Avoid side effects in setup_storage (#444)
• Return onehot interval instead of categorical (#447)
• Adjust OneHotEncoding interval shape (#449)
• Support non existing user script conf files (#452)
• Support load yaml config for dict items (#453)
• Fix fetch_lost_trials (Causing random FailedUpdate when using many workers) (#445)

Other improvements

• Improve CI - Convert functional algo tests to python API (#418)
• Monitor code change between trial execution (#429)
• Add info about working_dir when using orion info CLI (#430)
• Enable is_broken for ExperimentView (#432)
• Specialized properties for Trials (#433)
• Add completed field to info cmd (#435)
• Add orion.client.get_experiment() (#439)
• Sort modules and extensions alphabetically (#443)
• Print user friendly err msg if script inexecutable (#445)
• Migrate from Travis to Github-actions (#458, #459, #461)

Documentation improvements

• Add a code of conduct (#353)
• Refresh database documentation (#413)
• Merging Sequential and Service API (#416)
• CLI documentation improvements (#440)
• Remove outdated branch_from parameter documentation in experiment_builder.build() (#442)
• Changed variable name in docs to reflect the correct name (#448)
• Fix config of EvolutionaryES (#451)

Python API and New Algorithms

Important changes

The python API is finally ready for release v0.1.8! 🎉

Python API

An API is now available to run experiments directly from python instead of using the commandline.

from orion.client import create_experiment

experiment = create_experiment(
   name='foo',
   space=dict(x='uniform(-50,50)'))

trial = experiment.suggest()

# Do something using trial.params['x']

results = [dict(
    name='dummy_objective',
    type='objective',
    value=dummy_objective)]

experiment.observe(trial, results)

Current API provides a simple function workon for cheap experiments that can be executed by a single worker, and a generic ExperimentClient (see example above) object for optimization with multiple workers.

See documentation for more details.

New Algorithms

Hyperband

Hyperband extends the Successive Halving algorithm by providing a way to exploit a fixed budget with different number of configurations for SuccessiveHalving algorithm to evaluate. It is especially useful when the trials are expensive to run and cheap noisy evaluations are possible. Think of it as using early evaluation during training to filter out bad candidates.

For more information on the algorithm, see original paper.

TPE

Tree-structured Parzen Estimator (TPE) algorithm is one of Sequential Model-Based Global Optimization (SMBO) algorithms, which will build models to propose new points based on the historical observed trials.

Instead of modeling p(y|x) like other SMBO algorithms, TPE models p(x|y) and p(y), and p(x|y) is modeled by transforming that generative process, replacing the distributions of the configuration prior with non-parametric densities.

TPE has the advantage of scaling particularly well compared to most Model-Based algorithm which are typically sequential. It does not model however dependencies between hyper-parameters, they
are assumed independent.

For more information on the algorithm, see original papers at:

• Algorithms for Hyper-Parameter Optimization
• Making a Science of Model Search: Hyperparameter Optimizationin Hundreds of Dimensions for Vision Architectures

Storage

To support integration with other tools and services such as MLFlow or Weight & Biases we wrapped our previous database backend with a storage backend. The database backends are now available within the Legagy storage backend. In addition, we now have a backend for Track. The latter is planned to serve as a bridge between Oríon and other experiment management platforms
or services. Track package development is on the ice for now, but contributions are very much welcomed. :)

Drop python 3.5, support 3.8

Although Oríon may still be compatible with python 3.5 we do not maintain it's support anymore. Python 3.8 is now officially supported.

Precision of real dimensions

By default Oríon now rounds hyperparameters to 4 decimals (ex 0.00041239123 would become 0.0004124). The rational is that little variations on continuous hyperparameters typically leads
to little variations in the in objective. When sharing hyperparameters (ex: in publications), one can now share the rounded values with the exact corresponding objectives instead of rounding the hyperparameters after the execution and risk sharing unreproducible results.

Rework of documentation

The documentation has been through a major rework.

• The introduction has been updated to better reflect current features.
• A section Getting Started and an Overview was added.
• The plugins documentation was updated with a new cookiecutter template to create algorithm plugins.
• A minimalist example of scikit-learn was added to serve as the simplest tutorial for Oríon.
• Extensive documentation on contributions has been added.

Detailed list of changes

New features

• Storage Backend with Track and Legacy (#289, #302, #316, #318)
• Python API (#297, #299, #300, #305, #307, #311, #312, #355, #399, #401)
• Add option to ignore some commandline options of user's script (#308)
• Add option to ignore code changes in user's script (#310)
• Add trial hash id that does not include fidelifty
  (useful to resume across fidelities)(#313, #322)
• Customize precision (#331)
• Support commandline call where first argument is not user's script (#333, #338)
• Add support for non-standard -args (#341)
• Add more helper functions for cmdline client (#343)
• Hyperband implementation (#354, #363, #383)
• TPE implementation (#381, #387, #389)

Breaking changes

• Drop python 3.5 support in favor of 3.8 (#303)
• Remove deprecated score_handle (#315)
• Make global, local and cmdline args coherent (#349, #404)

Bug Fixes

• Fix EphemeralDB index update after document deletion (#306)
• Support small fidelity scales for ASHA (#314)
• Algo redefinition should not drop algo config (#319)
• Random algorithm will hang if search space is smaller than specified trials number (#336)
• Inconsistent output in orion status and orion list (#342)
• Use EVC tree trials in producer (#347)
• Update fidelity sample method to honor required n_samples (#351)
• Add max_trials to algorithm.is_done (#360)
• Handle trials with corrupted status (#372)
• Making upper bound inclusive (#373)
• Fix --debug (#374)
• Round ASHA budget properly (#402)
• Update Plugins documentation section (#406)
• Print help when calling orion alone (#408)

Other improvements

• improve error message for ASHA (#317)
• Print important information at end of worker (#345)
• Add comprehensive error message for branching err (#346)
• Make global, local and cmdline args coherent (#349, #404)
• Improve tox configuration (#350, #398)
• Generic cardinality check in base algorithm (#352)
• Avoid duplicates in RandomSearch (#361)
• Handle timeout in DBs (PickledDB in particular) (#362)
• Add stress test (#364)
• Remove useless deprecation warnings (#376)
• Ignore test files for code coverage (#377)
• Show proper yaml syntax in resolve_config deprecation warnings containing None (#391)
• Raise exception when no prior is provided (#410)

Documentation improvements

• Add citation section (#301, #379)
• Fix doc for bayesopt's alpha (#309)
• Formatted commands and added a command description (#327)
• Clone via HTTPS instead of SSH (#328)
• Minimalist example with scikit-learn (#339)
• Full configuration documentation (#349)
• Contribution guidelines (#356)
• Update developer documentation (#365, #378)
• Release & Packaging (#371, #411)
• Update the documentation structure (#375)
• Refreshed introduction of Oríon's features (#380, #384)
• Overhaul of the documentation for new comers (#385, #403)
• Add Issue Template (#386)
• Add PR Template (#388)
• Add doc about label categories (#397)
• Update Plugins documentation section (#405)
• Document Windows compatibility (#409)
• General documentation updates (#332, #340, #344, #366, #367, #370)

Bug fix with database upgrade

Important changes

Database upgrade

There was an incompatibility introduced in v0.1.6 that would break pickleddb created with previous versions. This minor release introduces a new command orion db upgrade to upgrade the database scheme so that databases created with orion<v0.1.6 can still be used in new versions.

Detailed list of changes

New feature

• Add orion db upgrade command (#293)

Breaking changes

• Turn test-db into db test (#291)

Bug fixes

• Handle non-existing fields in EphemeralDB (#284)
• Fix deepcopy of Enumerate transform (#290)
• Instantiate adapter even if empty (#295)

Others improvements

• Add functional tests for backward-compatibility (#285)
• Save parser state in DB (#288)

Documentation improvements

• Add mention of windows in install doc (#287)
• Update skopt doc and prepare v0.1.7 (#292)