Auto-resolution and experiment versions

Important changes

Migrated to orion on PyPi

We finally have orion namespace: pypi/warehouse#4189!!! 🎉 🎊

Auto-resolution and experiment versions

The modification of experiments (code change, search space modification, etc) will now be automatically resolved and will no longer lead to the (confusing) branch resolver prompt. When there is any modification that leads to a branching, the version of the experiment will be incremented (starting at 1). The unique index of the experiments are now (name, version) instead of (name, user).

No more user index

The user name is no longer part of the experiment's index. This means that someone with a username A in one environment may retrieve the same experiments with a username B in another environment without any problem. Previously this would only be possible using the option --user B to override the default system's username. This was due to the fact that queries on the database was done using the username and it is no longer the case.

Improved ASHA

Interface

The new prior fidelity(low, high, base) now makes it more convenient to define the fidelity dimension. The different budget for each rungs can now be scaled between low and high according to a base logarithm.

Efficiency

The algorithm will now stop registering trials in the low rungs once the higher rungs are filled. This reduces the waste of resources spend on trials in low rungs that cannot be promoted to higher rungs since the latter will be completed shortly.

Detailed list of changes

New features

• Auto-resolution (#244)
• Add versioning to experiments (#240, #242, #243, #245, #246, #247, #248, #259, #268, #272, #279, #281 )
• Add to algo the ability to opt out (#266)
• Add env-vars to consumer (#280)

Breaking changes

• Remove user as an index for Experiment (#264, #273 )
• Add low, high, base to Fidelity (#269)

Bug Fixes

• Fix Trial ID mismatch with higher dimension (#237)
• Make Space Sorted instead of Ordered (#257, #263 )
• Fix bug in conda test (#258)
• Handle status --name when empty (#270)
• Fix ASHA termination condition and branching with fidelity() (#274 )
• Avoid branching in info (#277)

Other improvements

• Add a global configuration object orion.core.config (#239, #251, #265)
• Remove direct access to database backend in favor of storage backend (#241)
• Remove generic queries in favor of specific queries in storage backend (#250)
• Add best trial id to info command (#260)
• Make ASHA opt out when last rungs are filled (#267)
• Migrate to orion on PyPi (#271 )

Documentation improvements

• Fix invalid section names (#252)
• Make tutorial a bit cleaner (#262)
• Improve documentation of pool-size and fix experiment.API (#278)
• Document script integration with templates (#280 & #282)

New commands `list`, `status` and `info` and support interruption/resumption of trials

New commands

New commands list, status and info now enable simple monitoring and inspection of experiments. The command list prints out all the experiments' name so you can easily keep track of what is available in your database. The command status gives a summary of experiments' status, with number of trials in different status (completed, reserved, etc) and best objective found so far. Finally the command info prints out all details of a given experiment, command line used, search space, algorithm configuration and current statistics.

Trial interruption

So far interrupted trials would keep the status reserved and wouldn't be available for selection
again, so they would be lost somehow. They will now be set to interrupted and will be available
for resumption. If user's script does not support checkpointing internally, the process will simply be re-executed from scratch.

New features

• New command list (#208, #216 )
• New command status (#222, #229)
• New command info (#211)
• New StorageProtocol (#221)
• Handling of interrupted trials (#197, #202, #203, #235)
• Limit number of broken trials before experiment stop execution (#204)
• Add working_dir attribute to Trial (#205)

Breaking Changes

• Deprecate pack/unpack in orion.algo.space (#236)

Bug Fixes

• Fix code conflict detection when not using code versioning (#207)
• Add $ne and handle non-supported operations in EphemeralDB (#214)
• Fix ScriptConfigConflict for data with non-string values (#215)
• Fix race conditions during algo updates (#217)
• Fix experiment views (#218, #223, #234)
• Make sure higher shape points are not ndarray objects (#224, #236)

Other Improvements

• Mute (intense) logging of filelock (#201)
• Functional tests for user script config (#212)
• Use yaml.safe_load instead of yaml (#219)
• Remove upsert from write (#220)

Documentation Improvements

• Add documentation for new commands list, info and status (#226, #228)
• Add documentation on Search Space (#227)

Minor Release: OSX support, New Roadmap and flake8 fix

Bug Fixes

• Fix flake8 dependency because of pydocstyle's new release #198

Other Improvements

• Add continuous integration for OSX #196

Documentation Improvements

• Add RoadMap #195

ASHA, simpler DB setup and new DB backend for filesystem

New Features

• Add ASHA 🎉 #187
• Add command orion setup to create a database configuration file #192
• Add command orion test-db to test database setup #188
• Add PickledDB to support FileSystem backend #190
• Add port option for db backend #174
• Add multi-fidelity dimensions #177
• Add working-dir argument #184

Bug Fixes

• Handle bad socket.hostname (OSX) #179
• Handle bool arg in EVC Conflict #180
• Fix null comparison in EphemeralDB #182

Other Improvements

• Introduce OrionCmdlineParser, a specialized type of CmdlineParser for Oríon specific parsing needs like priors' expression and configuration files. #172
• Add error message for missing configuration #178
• Handle bad results in trials #181
• Add timeout for connection to Database #183
• Add state_dict/set_state to algos to serialize or synchronize states #185

Documentation Improvements

• Adapt documentation to new algos and database commands #193

Relax git constraint, make user name customizable and fix some bugs

Breaking Changes

• Relax git commit constraint for version control #157
• Set default pool-size value to 1 #170

Bug Fixes

• Properly set RNG state in algo #167
• Make worker count executed trials properly #171

Other Improvements

• Make user name in experiment customizable with --user #169

Documentation Improvements

• Fix some slips #158 #160 #163

Git commit detection, Strategies and Trials history

Breaking Changes

• Add git commit detection for version control #115
• Add Strategies with Trials history #152

Additional New Features

• Add hunt --worker-trials argument #127
• Add node property to experiment for the EVC system. #146
• Add a generic commandline parser #155

Bug Fixes

• Set verbosity >= 2 to DEBUG #123
• Fix github links #126

Deprecation

• Drop support for py34 and add py37 #140
• Fix mongodb deprecation #151

Other Improvements

• Add recipe for conda packages #141

Documentation Improvements

• Updated instructions for Atlas MongoDB. Added instructions for non-root user #131
• Clarify doc section name #133
• Add RL example #136
• Fix README utf-8 encoding #117
• Changed URLs from mila-udem to epistimio #139
• Add simple tutorial #153

Initial release

Initial Release 🎊 🎉

Includes

1. MongoDB backend
2. Commandline and configuration file agnostic support
3. Experiment Version Control
4. Manual insertion of new points in search space
5. Algorithms: Random Search, Wrapper of Scikit-optimize bayesian optimizer (plugin orion.algo.skopt)
6. Transformers of dimensions

Working on and soon available

1. Detection of code changes for scripts inside git repositories
2. Dynamic algorithms and schedulers
3. Visualizations
4. Parallel strategies
5. Full reproducibility with log of algorithm history and synchronized Random Number Generators