WIP: Experimental parallel

[bruAristimunha]

No description provided.

[bruAristimunha]

This is so hard >.<

[PierreGtch]

Discussion with @bruAristimunha, action plan for the refactoring:

• We will have one shared evaluation method in the base evaluation class which will:
  • Load only the metadata
  • Split the metadata into different cv folds (calling methods defined in the subclasses)
  • Check which cv folds already have a result and can be skipped (except if overwrite=True)
  • Then, two options:
    • If lazy_loading=False, load the data of the cv folds that still need to be computed
    • If lazy_loading=True, do nothing
  • Launch, in parallel, a shared train_and_evaluate function on all cv folds (if lazy_loading=False, pass the pre-loaded data)
  • Gather the results in a dataframe and return
• The shared train_and_evaluate method defined in the base evaluation class:
  • (It only handles one cv fold and one pipeline)
  • Loads the data:
    • If lazy_loading=False (data+metadata provided), selects only the epochs corresponding to this fold
    • If lazy_loading=True (only metadata provided), loads the data. In this case, we recommend the users to activate the BIDS cache mechanism to avoid pre-processing the same data multiple times.
  • Trains the pipeline on the cv fold
  • computes codecarbone, training time, etc.
  • Groups the test data by session
  • For each test session, computes the pipeline's score
  • Pushes all the test session scores
  • Returns the list of test session scores