Track cost; order oracle trace by completion order

[esantorella]

Summary:
I am not sure if this is what we will want in the long run, but it will unblock benchmarking early stopping.

What's wrong with the current behavior

Ordering by start order vs. completion order:

Currently, the oracle trace is ordered by trial order and has one entry for each trial. The inference trace has always been ordered by completion order because it is updated every time a trial ceases running. The order of completion (including early stopping) seems preferable for both, and it's a little weird for the oracle trace to have a different ordering than the inference trace. See here for discussion on this: https://fb.workplace.com/groups/1294299434097422/posts/2563368300523856

Inability to compare more costly vs. less costly strategies:
Separately, tracking cost is necessary to fairly compare more aggressive vs. less aggressive early-stopping strategies or to compare stopping early against not.

I am bundling these two changes (reordering the oracle trace and introducing cost) because the oracle trace should now only be compared against the cost. Ordering by completion order doesn't make a lot of sense without a notion of cost when multiple trials can complete at the same time.

New behavior

time	first trial running	second trial running	objective values	best point
0	0	1
1	0	2	y_1	y_a
2	0	2	y_1	not computed
3	0	2	y_1, y_0, y_2	y_b

Assuming higher is better, this produces

  cost_trace: [1, 3]
  oracle_trace: [y_1, max(y_1, y_0, y_2)]
  inference_trace: [y_a, y_b]

Now traces are only updated when a trial completes, so there are 2 trace elements with 3 trials. (We could also just duplicate elements when multiple trials complete at the same time to preserve the length.) See docstrings for more detail.

What's not ideal about this

I want to flag that a few things are not great about this setup.

• It makes plotting hard: One one replication produces a cost_trace of [3, 5] and another one produces a cost_trace of [2, 6], how do we aggregate their optimization traces? We can do this by left-interpolating the optimization traces onto [2, 3, .., 6] and then aggregating as usual, but it is clunky.
• Even aside from the issue of different replications producing different cost traces, plotting is harder because plotting must be against cost now.
• People typically are interested in epoch-by-epoch results for early stopping, and those are not available here.

Better long-term solution

Two alternatives are

• Storing trace values for each time step, which would remove the need to track cost at all: element i of the trace would have happened at virtual second i.
• Storing cost/time information at each step in MapData, and then deriving a proper trace from there (we may already have this -- need to check)

Internal:

Differential Revision: D69489720

[facebook-github-bot]

This pull request was exported from Phabricator. Differential Revision: D69489720

[codecov-commenter]

Codecov Report

All modified and coverable lines are covered by tests ✅

Project coverage is 95.98%. Comparing base (63a1eaf) to head (8091386).

Additional details and impacted files

@@            Coverage Diff             @@
##             main    #3411      +/-   ##
==========================================
- Coverage   95.99%   95.98%   -0.01%     
==========================================
  Files         539      539              
  Lines       52794    52791       -3     
==========================================
- Hits        50677    50674       -3     
  Misses       2117     2117              

☔ View full report in Codecov by Sentry.
📢 Have feedback on the report? Share it here.