Cross Dataset Evaluation #703

ali-sehar · 2025-02-24T17:50:17Z

This adds a new type of evaluation to be able to validate models across several datasets. This particularly relevant for deep learning models as it allows MOABB to be used for benchmarking transfer learning.

Some examples are also added, one of which uses braindecode.

…se improve on spline channel interpolation and handling of events. Please add dataset augmentation and load balancing if necessary

gcattan

Thank you for your contribution!

We have some way to do transfer learning accross dataset in MOABB, using the compound dataset.

I see a different use case however, as:

The compound_dataset considers all subjects identically (so one of the split can contain a mix of subjects from different datasets). So you cannot run one evaluation with only subjects from one dataset for training and another dataset for testing.
And at the inverse, the new cross dataset evaluation is agnostic of the number of subjects/sessions/runs.

I guess the main point is rather how to align with the new splitter API.

gcattan · 2025-03-04T09:10:24Z

examples/cross_dataset.py

+logging.getLogger("mne").setLevel(logging.ERROR)
+
+
+def get_common_channels(datasets: List[Any]) -> List[str]:


There is a match_all method in base paradigm:

moabb/moabb/paradigms/base.py

Line 429 in 357cd12

def match_all(

please use this method @ali-sehar

gcattan · 2025-03-04T09:11:57Z

examples/cross_dataset_braindecode.py

+logging.basicConfig(level=logging.WARNING)
+
+
+def get_common_channels(train_dataset, test_dataset):


Same here (match_all method)

gcattan · 2025-03-04T09:12:05Z

examples/cross_dataset_braindecode.py

+    return event_id
+
+
+def interpolate_missing_channels(


match_all method.

gcattan · 2025-03-04T09:21:55Z

moabb/evaluations/evaluations.py

@@ -780,3 +780,132 @@ def evaluate(

    def is_valid(self, dataset):
        return len(dataset.subject_list) > 1
+
+
+class CrossDatasetEvaluation(BaseEvaluation):


Hm. I think there is a plan to refactor the existing evaluation.
The recommended way to go will be to use the new Splitter API (see: #612 (comment)).

@bruAristimunha can probably advise you better than me what refactoring is necessary in this case.

Hey @gcattan, thanks for all your feedback!

@bruAristimunha - if you could comment on the best way to move forward :)

We can implement this one and we migrate later

gcattan · 2025-03-04T09:24:16Z

moabb/evaluations/evaluations.py

+    train_dataset : Dataset or list of Dataset
+        Dataset(s) to use for training
+    test_dataset : Dataset or list of Dataset
+        Dataset(s) to use for testing


Probably you want to have a cross-evaluation.
So provide a list of datasets, and then, keep one for training and the other for testing. and then rotate.

@ali-sehar @EazyAl Please implement this suggestion too.

Pass a list of datasets

And implement cross-validation

gcattan · 2025-03-04T09:35:36Z

moabb/evaluations/evaluations.py

+                    model = clone(pipeline).fit(train_X[0], train_y)
+                    score = model.score(test_X, test_y)


Ok, so you train on the whole subjects/sessions/runs, and then test on the whole subjects/sessions/run of the second dataset?

gcattan · 2025-03-12T12:15:55Z

examples/advanced_examples/plot_cross_dataset.py

+# Get the list of channels from each dataset before matching
+print("\nChannels before matching:")
+for ds_name, ds in datasets_dict.items():
+    try:
+        # Load data for first subject to get channel information
+        data = ds.get_data([ds.subject_list[0]])  # Get data for first subject
+        first_subject = list(data.keys())[0]
+        first_session = list(data[first_subject].keys())[0]
+        first_run = list(data[first_subject][first_session].keys())[0]
+        run_data = data[first_subject][first_session][first_run]
+
+        if isinstance(run_data, (RawArray, RawCNT)):
+            channels = run_data.info["ch_names"]
+        else:
+            # Assuming the channels are stored in the dataset class after loading
+            channels = ds.channels
+        print(f"{ds_name}: {channels}")
+    except Exception as e:
+        print(f"Error getting channels for {ds_name}: {str(e)}")


remove this

gcattan · 2025-03-12T12:23:22Z

examples/advanced_examples/plot_cross_dataset.py

+# Get channels from all datasets after matching to ensure we have the correct intersection
+all_channels_after_matching = []
+print("\nChannels after matching:")
+for i, (ds_name, _) in enumerate(datasets_dict.items()):
+    ds = all_datasets[i]  # Get the matched dataset
+    try:
+        data = ds.get_data([ds.subject_list[0]])
+        subject = list(data.keys())[0]
+        session = list(data[subject].keys())[0]
+        run = list(data[subject][session].keys())[0]
+        run_data = data[subject][session][run]
+
+        if isinstance(run_data, (RawArray, RawCNT)):
+            channels = run_data.info["ch_names"]
+        else:
+            channels = ds.channels
+        all_channels_after_matching.append(set(channels))
+        print(f"{ds_name}: {channels}")
+    except Exception as e:
+        print(f"Error getting channels for {ds_name} after matching: {str(e)}")
+
+# Get the intersection of all channel sets
+common_channels = sorted(list(set.intersection(*all_channels_after_matching)))
+print(f"\nCommon channels after matching: {common_channels}")
+print(f"Number of common channels: {len(common_channels)}")
+
+# Update the datasets_dict with the matched datasets
+for i, (name, _) in enumerate(datasets_dict.items()):
+    datasets_dict[name] = all_datasets[i]
+
+train_dataset = datasets_dict["train_dataset"]
+test_dataset = datasets_dict["test_dataset"]
+
+# Initialize the paradigm with common channels
+paradigm = MotorImagery(channels=common_channels, n_classes=2, fmin=8, fmax=32)


Remove this.
match_all don't change the number of channels in the dataset,
it just automatically set the filter in the paradigm.

gcattan · 2025-03-12T12:25:08Z

examples/advanced_examples/plot_cross_dataset_braindecode.py

@@ -0,0 +1,691 @@
+"""


same comments about match_all here. Please apply.

gcattan · 2025-03-12T12:26:03Z

moabb/evaluations/evaluations.py

+    train_dataset : Dataset or list of Dataset
+        Dataset(s) to use for training
+    test_dataset : Dataset or list of Dataset
+        Dataset(s) to use for testing


@ali-sehar @EazyAl Please implement this suggestion too.

Pass a list of datasets

And implement cross-validation

EazyAl and others added 15 commits February 24, 2025 18:04

cross dataset working with pyriemann

fec5185

cross dataset eval

6812f12

changes to match MOABB syntax and format

c203219

deep learning example working, pls make it clean

5228cbc

multiple dataset training and testing with braindecode working - plea…

fecf06c

…se improve on spline channel interpolation and handling of events. Please add dataset augmentation and load balancing if necessary

Few changes

d9f598c

cross dataset eval with examples

44cd428

[pre-commit.ci] auto fixes from pre-commit.com hooks

6369eed

cross dataset eval with examples

6f83dcb

[pre-commit.ci] auto fixes from pre-commit.com hooks

2cec69c

Merge branch 'NeuroTechX:develop' into develop

4663226

fix: resolve merge conflicts in cross dataset example

023a0a7

[pre-commit.ci] auto fixes from pre-commit.com hooks

4a8bcb9

added tests and edited changelog

96a678a

[pre-commit.ci] auto fixes from pre-commit.com hooks

d6d1d00

gcattan requested changes Mar 4, 2025

View reviewed changes

EazyAl and others added 4 commits March 11, 2025 19:32

merge

669b8a9

merging

e6d5123

Using match all

0f90c4b

[pre-commit.ci] auto fixes from pre-commit.com hooks

ae8ac23

gcattan requested changes Mar 12, 2025

View reviewed changes

Merge branch 'develop' into develop

9626ca5

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Cross Dataset Evaluation #703

Cross Dataset Evaluation #703

ali-sehar commented Feb 24, 2025

gcattan left a comment

gcattan Mar 4, 2025

bruAristimunha Mar 7, 2025

gcattan Mar 4, 2025

gcattan Mar 4, 2025

gcattan Mar 4, 2025

ali-sehar Mar 7, 2025

bruAristimunha Mar 7, 2025

gcattan Mar 4, 2025

gcattan Mar 12, 2025

gcattan Mar 4, 2025

gcattan Mar 12, 2025

gcattan Mar 12, 2025

gcattan Mar 12, 2025

gcattan Mar 12, 2025

		logging.getLogger("mne").setLevel(logging.ERROR)


		def get_common_channels(datasets: List[Any]) -> List[str]:

		logging.basicConfig(level=logging.WARNING)


		def get_common_channels(train_dataset, test_dataset):

		model = clone(pipeline).fit(train_X[0], train_y)
		score = model.score(test_X, test_y)

Cross Dataset Evaluation #703

Are you sure you want to change the base?

Cross Dataset Evaluation #703

Conversation

ali-sehar commented Feb 24, 2025

gcattan left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment