Having brainstorm database self contained, by default

[Edouard2laire]

Currently, managing files linked to a brainstorm database is a pain. Some file are in brainstorm, some files are link to files outside brainstorm: .snirf are converted and stored in the brainstorm database, .eeg are link. .edf are linked, except if imported using FieldTrip when they are stored in the brainstorm database.

-> So that if you move your database, it break.
-> Move the location of the data, it break
-> Rename the data to follow bids standard, it break.

Yeah, there is the copy raw file to database, but this will convert all file to .bst and import them in the database, add an ugly tag saying 'copy' to the file, and doesn't solve the underlying problem. Knowing that every time, I receive a database from someone, half of the links are broken, because they didn't copied the link in the database.

Proposition

-> For each protocol, add a folder called 'source_data', in addition to anat, and data
-> Every time, as file* is linked to the database, copy the file to source_data and then link the file to the database, using a relative path

*Note: also works when importing anatomical file, freesurfer folder ...

Advantage

1. The database can now be moved. No pain about the broken link. No pain associated with the 'Copy raw file to database'

2. Continious file can now be renamed. Want to tidy up the database, and have file properly named? We can now change their name in brainstorm, updating the name of the underlying file. Because the file is a copy, there is no risk of breaking another person's database.

Regards,
Edouard

[rcassani]

Hi @Edouard2laire, apologies for the extensive message.

Managing the original raw files in Brainstorm should not painful

Currently, managing files linked to a brainstorm database is a pain. Some file are in brainstorm, some files are link to files outside brainstorm: .snirf are converted and stored in the brainstorm database, .eeg are link. .edf are linked, except if imported using FieldTrip when they are stored in the brainstorm database.

-> So that if you move your database, it break.
-> Move the location of the data, it break
-> Rename the data to follow bids standard, it break.

1. Some files are links to files outside of Brainstorm

The reasoning of creating links to original raw files is to avoid having unnecessary copies of files. The original raw files is read-only, are only read, and any change (new events, projector, etc) are saved in the link file. Copying every time the original raw files is problematic as it introduces an overhead in storage and copy time, not to mention the possibility or creating multiple versions of the dataset. Consider these scenarios: (1) Multiple Protocols using the same original raw files, (2) multiple users (and Brainstorm DBs) using the same original raw files.

2. Some files are in the Brainstorm DB dir

In the case of recordings, many file formats allow random access, so it is not needed to load the entire data to read specific sections. When this is the case, only the link to that file is created (as indicated above). However, if the file format or the function that reads it do not support random access (as in the mentioned cases .snirf and .edf (through FieldTrip reader)), Brainstorm saves a copy of the entire data as a .bst binary file that allows random access, optimizing the use of computing resources. This .bst file is saved in the database to avoid changes on the original raw data dir, and a link to to this .bst file is created.

Note: Anatomical files that are imported in the Brainstorm DB are also converted to Brainstorm format, and stored in the Brainstorm DB dir.

3. When links get broken

If the original raw files are moved or renamed, the expected behaviour is that the links get broken.

Moving the Brainstorm DB inside the same computer, should not break those links as these as stored as absolute paths.
However, if the Brainstorm DB is moved to another computer, and the original raw data is not in the same absolute path, the link will be broken.
In the past, this portability between computers was improved, thought it can fail as it guesses the new location of data:
83e2591

Option "Copy raw files to database"

Yeah, there is the copy raw file to database, but this will convert all file to .bst and import them in the database, add an ugly tag saying 'copy' to the file, and doesn't solve the underlying problem. Knowing that every time, I receive a database from someone, half of the links are broken, because they didn't copied the link in the database.

This conversion to .bst is done so the copied raw file contains the recordings of the original raw, plus extra data such channel localization, projectors, headpoints, fiducials and events. If the users forget to include their raw files while sharing a Brainstorm database, it sounds more a trouble on the user side rather than on Brainstorm side.

Troubles with proposed solution

The proposed solution creates unnecessary copies of files

• Not optimal nor practical: large overhead in storage, and copy time, often databases consist in 100s of GB
• Potential risk of diverging versions on original raw files
• Ignores cases where the recipient has access already to the original raw files. E.g., original raw data was shared beforehand, or its stored in a network disk where both users have access
• It does saves the user one click in the option "Copy raw files to database", user should understand what is what they are sharing

How to improve it?

An option to check a protocol for broken links could be added.

So, the recipient of a Brainstorm DB does not get surprises later. This action can be executed when a Protocol is first time loaded and/or on request.

---

@tmedani, do you have some additional thoughts on this?

[Edouard2laire]

Hi @Edouard2laire, apologies for the extensive message.

No worries. If anything can help with my current practice of Brainstorm, it is welcome. Sorry for my also extensive message :) If you don't have time, you can skip to #811 (comment)

Copying every time the original raw files is problematic as it introduces an overhead in storage and copy time.

I don't believe this is problematic. Storage is cheap. I just checked on Amazon, and you can find a 20 TB hard drive for ~300$*. Also, copying the file at import time will feel less time consuming than creating a full database, and having to copy the files using 'copy raw file to database' at the end. Indeed, the copy is just calling cp from the system, whereas 'copy raw file to database', converts the file to .bst in addition to performing the copy, creating extra time overhead.

The possibility or creating multiple versions of the dataset.

This is already the case. Let's say you have an edf. You link it to your database, and start making artifacts. You created a version of the dataset. Then someone else link the file to their database, they won't see the same file as you. The artifact are not marked.

Let's say you link the same edf file in your database, then a neurologist has a second look at the edf and mark additional epileptiform activity, you wo't see them in Brainstorm because the event are not read from the edf, and you in fact already created a second version of the dataset. If someone re-link the edf file, they won't see the same events as you: in this case, the additional markers will be present in the file.

The same thing happens for all the fields that are stored in brainstorm. The most obvious ones are the events and co-registration. So we are already making a separate dataset, but it's hidden.

By making a copy of the file, we make it clear that any change made to the file are only local, and that change to the external file are not reported in Brainstorm.

Moving the Brainstorm DB inside the same computer should not break those links as these as stored as absolute paths.

Yes, and I agree. But the main issue is when working from multiple sites and different computers. Most of the week, I am working from Concordia, and the data is on Concordia NAS. But when we have a lab meeting at McGill, I don't have access to the NAS, so it means connecting remotely through slow internet. Similarly, if a new student want to work from mcgill, we could transfert the raw data, but they could not easily use my brainstorm database with all the file already loaded and prepped for analysis: because then all the link would be broken and the student would have to reimport everything wasting a lot of time and potentially making mistakes.

Similarly, let's say I am going to a conference and won't have internet, I currently copy beforehand the files from the brainstorm database to an SSD setup on an external SSD that I can then load from my laptop. It's working ok for the pre-process file, but it doesn't work for the raw file as the link would be broken (no access to the NAS).

And actually, I do have at least 3 versions of my database that are not fully synced: one in McGill, one on my laptop (SSD), and one at Concordia.

Now let's say all the files, including raw, are in the database. I can set a job to synchronize the database from Concordia and McGill NAS every day. I can also regularly download this database on my laptop to either work from home or remotely and easily push back the change to either server and keep everything synchronized.

If a new student joins the lab, they can just copy the database and start working without any issues of broken links. Because the entire protocol is in one folder, this could even be managed by git, allowing to have some kind of versioning of the database.

This conversion to .bst is done so the copied raw file contains the recordings of the original raw, plus extra data such as channel localization, projectors, headpoints, fiducials, and events. If the users forget to include their raw files while sharing a Brainstorm database, it sounds more like a trouble on the user side rather than on the Brainstorm side.

So, by doing that, you are creating the possibility of creating multiple versions of the dataset, which was something you said you wanted to avoid. And I agree that users should be careful when exporting, but they are not. Maybe we can add an error (warning?) when exporting links that are not in the database.

How to improve it?

Maybe we don't put this as the default for everyone, but maybe this could be an option. We could also have a button 'Make this dataset independent', making the copy so the dataset can be sent.

Other solution

Technically, it can already be done actually by the user as nothing is preventing to creation of a folder in the brainstorm database called 'source_data' and linking data from there.

The data would not even need to be copied there, as it could be a link to the folder storing the data, meaning that if you copy the database and data to another computer, the only thing to fix would be the link to that one folder, and brainstorm would not see the difference:

[edelaire1@home-imglab01:/Volumes/CrucialX8/bst_db/MEG_simulations]$ ls -la
total 48
drwx------  1 edelaire1  staff  4096 Nov 11  2024 .
drwx------  1 edelaire1  staff  8192 Nov  5  2024 ..
drwx------  1 edelaire1  staff  4096 Nov 11  2024 anat
drwx------  1 edelaire1  staff  4096 Nov 11  2024 data
lrwx------  1 edelaire1  staff    30 Jul  4 15:32 source_data -> /Users/edelaire1/Desktop/data/

Edit: Actually, i really love this solution. There is no need to change anything in brainstorm and it solve everything. Do you see any danger in doing that or can i start doing like that ?

[Edouard2laire]

Hello,
So here is a solution, based on my previous message, that i think will make everyone happy.

The solution

From the user's perspective, create a folder in the project folder called 'source_data'. This is a link to the folder storing the data:

Example:

[edelaire1@home-imglab01:/Volumes/CrucialX8/bst_db/MEG_simulations]$ ls -la
total 48
drwx------  1 edelaire1  staff  4096 Nov 11  2024 .
drwx------  1 edelaire1  staff  8192 Nov  5  2024 ..
drwx------  1 edelaire1  staff  4096 Nov 11  2024 anat
drwx------  1 edelaire1  staff  4096 Nov 11  2024 data
lrwx------  1 edelaire1  staff    30 Jul  4 15:32 source_data -> /Users/edelaire1/Desktop/data/

From Brainstorm, when linking a file, that is in the form of ${BrainstormDbDir}/{protocolName}/source_data/path_to_data then only store the relative path to the protocol folder, eg: source_data/path_to_data.

When the user tries to access a file, reconstructs the full path using the path to the brainstorm database and the protocol name. Similarly to how we recover the path to any file stored in the brainstorm database.

Does it solve the problem ? yes

It solves two case scenarios: if the database is moved, and if the data is moved.

1/ If the data is moved, then the link from the source data to the data folder is no longer valid. However, this link can be easily updated using ln -sfn. Only link needs to be updated instead of updating every link in the database

2/ If the database is moved, then there is nothing to do. Since the path is relative to the database, then there is no issue. As long as the data are still accessible.

When exporting the database, the user just needs to make sure to send both the brainstorm database and the data folder. The recipient just needs to adjust the link.

Time and storage overhead?

I understand your concern relative to the time and space overhead caused by my previous solution. With this new system, we are only creating a link from the brainstorm database to the folder containing the data. Thus creating no overhead.

Future extension?

For now, we can keep the creation of this link manual. But I think in the future it would be nice to have this link managed by Brainstorm so that, when a protocol is loaded, Brainstorm could check that the link to the data is still ok, or ask the user to locate the data ands add a bunch of warning for example if linking data outside this source folder.

When exporting data, we could then remind the user to transfer this data folder or pinpoint the specific file that needs to be transferred. Removing the time cost associated with copying raw data to the database.

Finally, and in a distant future, we could imagine having much richer interaction between brainstorm and this source_data folder. The way I see Brainstorm is that it is an interface between the user and a BIDS dataset. Brainstorm is pulling data from a BIDS dataset, to then allow the user to apply a process on it.

But right now, Brainstorm is mainly around the idea of pulling the data at a specific instant in time. As I said earlier, if you link a file in the database, and this file is changed (by adding events, for example), then the user in brainstorm would not be aware of it But we could imagine a system where brainstorm would check at each load that the file hadn't changed (checksum for example) and eventually reload the file if it did. Actually, until then, we should suggest that files linked in Braisntorm to be read-only to avoid having them be modified while being linked to Brainstorm.

But change could also be pushed back to this BIDS dataset, for example, updating the opodes coordinates once the co-registration has been done. Or updating events when modified in Brainstorm to keep the view of Brainstorm and of the dataset in sync.

I have other ideas on how brainstorm could interact with BIDS dataset, notably when it comes to all the transformations we apply on anatomical data, like segmentation, remesh ... but let's keep if for the future :)

Conclusion

I think this can be really helpful, especially when dealing with data being transferred across multiple sites (McGill/Concordia) in my case, and really ensure that the database is portable. If this is done, I feel pretty confident that I can build a brainstorm database that I could then share with the next students and that they could just load it back in their brainstorm and start working without any issue of missing data.

Also, i think, this would represent a very minor change to Brainstorm :)

[rcassani]

• This solution is that it assumes there is a data/ dire with the data files. However, data can be distributed in different folders.
  E.g., a protocol that analysis subjects from two different datasets, that may not be in the same disk.

• It is not really encouraged to have files other than the ones created by Brainstorm in the Protocol dir.

---

When exporting the database, the user just needs to make sure to send both the brainstorm database and the data folder. The recipient just needs to adjust the link.

For now, we can keep the creation of this link manual. But I think in the future it would be nice to have this link managed by Brainstorm so that, when a protocol is loaded, Brainstorm could check that the link to the data is still ok, or ask the user to locate the data ands add a bunch of warning for example if linking data outside this source folder.

This approach sounds good to me. At first glance it my not seem like that, but it's an extension of the automatic fixing of links, that asks the user the path in case it was not able to guess it. So, every time a Protocol is loaded (or on request) perform a check of the links, if any link broken, an automatic fix is attempted, if it fails, the user will be asked if they want to fix them now (as they may not need it), if yes the user will be asked the new path of one of the broken links, with this answer Brainstorm will try to fix all the other links in the database (as it is done in case the automatic fix is able to figure out the new paths).

Apologies as I did not convey better my message, but this is en essence the last section of this message.

---

BIDS-Brainstorm interactions are certainly something that we have in our roadmap.

---

@Moo-Marc, I wonder if you have some additional thoughts on this topic? Maybe some recommendations and good practice points regarding handling large datasets.

---

EDIT: One advantage of using the "Copy raw files to database" option is that it minimizes the size of the data to be shared. No need to send the entire dataset if only a small part if being used in the Protocol.

[Edouard2laire]

This approach sounds good to me. At first glance it my not seem like that, but it's an extension of the automatic fixing of links, that asks the user the path in case it was not able to guess it.

so is it ok for you if I open a PR for that ?

• This solution is that it assumes there is a data/ dire with the data files. However, data can be distributed in different folders.

You can always have them in a folder. Let's say you're data are in /NAS1/data/EEG_dataset and the other in /NAS/data/MEG_dataset, then you can create a folder data containing two folder each being a link to each dataset.

[Moo-Marc]

Hello,

Commenting from the original idea, I agree that duplication of raw data should not be a default. Easy to say storage is cheap, but that's relative and simply not practical in every situation. It's common for users to already struggle to manage disk usage in Brainstorm, so it is important to do our best to help keep it at a minimum. I also agree with Raymundo that this is primarily a user issue, that they would need to be mindful to share the raw data when sharing datasets. It shouldn't be much of a problem to ask for the raw data folder if someone forgot to share it... That said, I like the idea of simply linking files (which would rely on the underlying OS & file system) in a "sourcedata" folder for the reasons Edouard indicated. The user still needs to mind these links when copying, using the right tools and options so that the files are copied and not the links, especially for soft links. Note that I'd rename it to "sourcedata", without underscore, to match BIDS naming.

I also like the idea of adding a few things to Bst. One process could create these links as a preparation to share a dataset, as an alternative to "copy raw files to db". Such a process would still only include data that is in the Bst protocol, as noted by Raymundo, so a user would still need to do it manually if they want to share more than what's already in Bst. It could also be an option, either a general preference or each time a raw file is "reviewed/linked" in Bst, to first link the files under sourcedata, and have Bst open it from there. A second process could verify all linked files and conveniently list the missing ones - though maybe that already exists, e.g. when reloading the entire protocol? When exporting a database, is there a warning about linked files? That would also help avoid people sharing without raw data (when it's needed).

Finally, worth mentioning that we're working on a major update to allow sharing protocols between users, which would introduce a real central database. This would be an alternative to the idea here of having multiple copies and syncing them, in the case where the various users/devices have access to the same database. @rcassani, I guess this is something we could think about (I don't recall discussing this before), how we could eventually extend the new database system to run across multiple sites with a mirroring system, or a way to merge databases with changes on both sides.

[rcassani]

PR #818 is a WIP to automate fixing broken links to address the situations described in this issue