Plan for Optimas generator standardization

[shuds13]

Plan to support the generator standard in Optimas.

Step 1: Implement suggest/ingest (standardization PR).

• Attempt to make optimas generators compatible with generator standard #249
• Get current tests working with a provisional solution to the IDs.
• For Optimus generators:
  • Constructor is provided Optimus data structures. --- (How does this work with the multitask example?).
• For external (standardized) generators (e.g., libEnsemble):
  • ExternalGenerator provides Optimus data structures alongside the standardized generator.
• If a previous history is provided, still as a libEnsemble numpy file.
• Note: This stage does not require API change for Optimas (suggest/ingest are internal).

Step 2: Review approach to ID.

• see Attempt to make optimas generators compatible with generator standard #249 (comment)
• Do we need to maintain Optimas and Generator IDs or can they be combined into one.
• Does a generator in the standard need to be declared as one that manages IDs.
  • e.g., if you give it a history (or any unexpected points) are you expecting to give it IDs that it will use internally, so that when it generates new points it will enumerate from that point.
  • Most examples, do not give id to gen - but what about when using an Ax generator...
• Can we assume a single generator that will take charge of IDs (when using _id)
• Check consistent with unexpected points (previous history, ignored points etc).
• Note that in multitask ax id is shared for a set of points with each having an arm_name.

Step 3: Implement VOCS

• Summary. PR Implement VOCS interface #281 implements the vocs interface using conversions to Optimas internal parameters. This means external generators are still run through the ExternalGenerator wrapper so these internals get set. Optionally, a future pass could update the internals.
• Update Optimas generators to use VOCS constructors.
• In some cases a mapping might need to be given (e.g. variables to varying_parameters/trial_parameters).
• Or if these are always accessed through generators, do we remove the parameter structures and use the standard internally.
• How to handle trial features like ignored, status
• Can we lift more functions dealing with Optimas data structures outside of generators (e.g., incorporate_history)
• Or will Optimas data structures still be used inside the generators? - use internal bypass in Optimas?
• How to set Optimas data structures
  • from VOCS - from initialized gen)
  • In config step, by use of a wrapper (like ExternalGenerator) or in Exploration.
  • Note that in Exploration constructor it is assumed that Optimas data structures are set up (e.g., loading history of points).

Step 4: Check edge cases and details

• Ensure suggest correctly deals with case where num_points is not supplied, for each gen (gen decides).
• Handle MAXIMIZE / MINIMIZE / EXPLORE
• If suggest cannot return the number of points asked for - is correct error raised.

---

Ref:
Optimas data structures: e.g., varying_parameters, objectives, dedicated_resources, custom_trial_parameters
These are initialized with the generator and referenced through the generator throughout Optimas. They are also used in trial.

Optimas Exploration data structures:
The trial data structure is an Optimas data structure - where a list of trials are managed in persistent_generator with a unique trial.index. Each trial was generally created inside ask (now suggest) with more info filled in outside the gen (trial.index). It was also given to tell (now ingest). With the new standard the internal code in ingest to some extent can be retained by duplicating the structures internally, however, the elements can no longer be updated in place (as happens in some places (status / ignored). See the multitask generator as an example. Alternatively, the generators internal code could be re-written to no longer use these structures. It's also important not to confuse these trials with Ax trials which are also used inside the generators. Do we want these generators to be able to exist independently of Optimas (can they rely on importing Optimus data structures).

[shuds13]

Internal bypass

If Optimas data structures are already used inside the generator, then a back and forth conversion can be avoided when running Optimas, by having an internal function that takes Optimas data structures, and a wrapper that supports the standard. This is used in libEnsemble.

Optimas generator in standard.

ingest(list_of_dict)..
  ```Conversion wrapper calls ingest_internal```

ingest_internal(varying_parameters, objectives....)
  ```Called by ingest or directly by Optimas```

Optimas will call ingest_internal if a function of that name exists, giving Optimas data structures directly, otherwise, ingest is called. Hence this can be used for all Optimas generators or only some.

[shuds13]

Trial features like ignored, status - need to decide if they are features of Optimas (deal with outside gen) or the generator and if need to be passed back and forth (as variables in the standard). If just gen features that we may want to access them at the end or when we dump to a log then this is the kind of thing that we could use functions to extract the information from the generator rather than necessarily storing it in Optimas trials.

I would guess ignored can be handled outside of gen.

[shuds13]

Optimas trial ID and Ax trial ID analysis.

TLDR: When the generator produces an ID _id (according to the standard), this will be stored in Optimas trials as trial.gen_id, and returned to the generator when ingest is called. Optimas maintains a separate trial index (trial.index).

---

The standard says the generator can return an _id in addition to the declared variables when suggest is called. When we give back to the gen, if _id it's declared, it should match the value that came out originally.

Currently: In the Ax examples, the generator currently returns ax_trial_id, which Optimus stores in its own trial (trial.ax_trial_id). However, after new points have been generated, Optimus gives them all an index, which seems to simply enumerate from zero (trial.index). Before giving back to the generator this is automatically converted to _id. So the generator is given back both _id and ax_trial_id).

So we should move to having a single ID from the generator point of view. The key question is whether we can expect generator ID and Optimas ID to always match. In that case I could set the Optimas trial index to the generator ID if it exists and otherwise create the enumerated IDs.

OR

If we want to maintain separate IDs because they might differ, then I will do the following.

The _id that comes out of the generator is stored in the Optimas trial as trial.gen_id. The Optimas index will be generated as before (trial.index). However, the to_dict function that prepares trial data to be given back to the generator will use:

"_id": self.gen_id or self._index,

Or just

"_id": self.gen_id

which may be safer so as to avoid getting the two muddled - if unexpected points are given to the generator they should presumably come without IDs and, in general, if the generator doesn't produce an ID then you don't need to give one back to it.

That said, unexpected points currently have an Optimas index of None when ingest is called, and each Optimas trial is provided an enumerated index afterwards. Currently, all unexpected points are given to gens with _id of None, therefore.

What about incorporate_history?

When you attach a previous history, Optimas loads in trial IDs, whereas the gen ID (at least the current ax_trial_id) is only present for points created in this run. So this would create an offset between trial ID and gen ID.

So I think maintaining both is the way to go.

Other considerations:

If a previous history is being provided to the generator via ingest, or any unexpected points, then ingest will be receiving an ID (for an ID producing generator), but this could be inconsistent with what the generator produces. So an instantiated generator instance should always be in charge of assigning these IDs, meaning any supplied points with unknown IDs should get assigned a new ID. We also want something to indicate whether this is an ID producing gen, such as a query function in the standard.

Do we theoretically have to support multiple generators.

============================

Reason not to use single ID approach:

While a single ID approach could probably be made to work so long as strict rules are followed, there are potential scenarios where race conditions could occur between the generator setting IDs for points it generates, and Optimus setting IDs for unexpected points. Maintaining separate IDs is more robust.

Examples:

Suppose a generator is initialized with a set of points (outside of Optimas) and the generator internally assigns IDs to those points (0-99), so the next point it intends to generate will be point 100. Then Optimas starts running and receives unexpected points. It will send those to the generator with no ID, and set the Optimas ID to 0, 1, 2 etc…

In the APOSMM via Optimas approach, APOSMM internally is getting ahead in points produced - even though it does not need to give Optimas ids, its an example where this could happen.

To use this approach, rules would need to include - points only given to gen via ingest (none when gen set up).
Optimas sets ids only on suggest/ingest (via a wrapper):
Suggest sets to generator id (if gen producing ids) or creates a next id in Optimas
Ingest sends id to generator if exists, else sends None as id and creates an Optimas id.
No concurrency - when points given by ingest, there cannot be points already with ids to give in generator.

I will also note that the Ax trial id in the multitask example, which is not unique to each point (but rather a set of points) could also be handled as a VOCS variable, with integer values, as its currently handled as a TrialParameter, without using _id.