Multiple issues while reading and resampling raw mff

[Genuster]

Description of the problem

I've seen the issue #11188, it seems similar to the error I get, but I haven't quite understand where does the problem come from and if there are any existing solutions.
I tried to find some workaround for the problem, that's where I've got so far:

1. First issue (see first screenshot in actual results) was solved by commenting out the following code in the /mne/io/egi/egimff:

if bad:
    raise RuntimeError('EGI epoch first/last samps could not be parsed:\n'
                       '%s\n%s' % (list(epochs['first_samps']),
                                   list(epochs['last_samps'])))

2. The second issue (second screenshot) seems unrelated to the first one. The workaround is to remove Events_User Markup.xml file from the .mff. After that mne reads .mff successfully.
3. The third issue appears while trying to resample. I'm not sure if there is any connection to the previous ones, but I'm stuck with it currently.

I'm not sure if it's an MNE bug, so I apologise in advance if it's not.
Do you have any insight about roots of the issues? Could you please advise how to proceed?

Steps to reproduce

raw = mne.io.read_raw_egi(path_to_uploaded_mff)
raw.resample(250)

Link to data

Link to the mff (google drive)

Expected results

Load and resample the mff file successfully.

Actual results

1. 
2. 
3. 

Additional information

Platform:         Windows-10-10.0.20348-SP0
Python:           3.10.9 (tags/v3.10.9:1dd9be6, Dec  6 2022, 20:01:21) [MSC v.1934 64 bit (AMD64)]
Executable:       D:\!eeg-processing-pipeline\code\.venv\Scripts\python.exe
CPU:              Intel64 Family 6 Model 167 Stepping 1, GenuineIntel: 16 cores
Memory:           63.9 GB

mne:              1.2.3
numpy:            1.23.5 {OpenBLAS 0.3.20 with 16 threads}
scipy:            1.9.3
matplotlib:       3.6.2 {backend=TkAgg}

sklearn:          1.2.0
numba:            0.56.4
nibabel:          Not found
nilearn:          Not found
dipy:             Not found
openmeeg:         Not found
cupy:             11.4.0
pandas:           1.5.2
pyvista:          Not found
pyvistaqt:        Not found
ipyvtklink:       Not found
vtk:              Not found
qtpy:             Not found
ipympl:           Not found
pyqtgraph:        Not found
pooch:            v1.6.0

mne_bids:         Not found
mne_nirs:         Not found
mne_features:     Not found
mne_qt_browser:   Not found
mne_connectivity: Not found
mne_icalabel:     Not found

[welcome]

Hello! 👋 Thanks for opening your first issue here! ❤️ We will try to get back to you soon. 🚴🏽‍♂️

[larsoner]

First issue (see first screenshot in actual results) was solved by commenting out the following code in the /mne/io/egi/egimff:

I think commenting this out likely causes all the other problems -- this error means we haven't parsed/understood the file correctly, so any reading operation after that is likely to fail.

Hopefully someone can try the file soon!

[Genuster]

I think commenting this out likely causes all the other problems

Yeah, I completely understand that this is a poor way to solve the problem. It's just too complicated to debug it with my current level of understanding MNE code and mff structure.

Hopefully someone can try the file soon!

Thank you! I'm looking forward to it. If I can be of any help, please tell me.

[drammock]

I cannot test the file because of the file sharing system you're using. It says "firefox doesn't provide a big enough buffer, use chrome" but then in chrome it says "not enough disk space, you have to use our app or chrome plugin". I'm not going to install an unknown app or browser plugin so please find another way to share the file, or else crop the data so that it's small enough to not have these problems.

[Genuster]

@drammock I'm sorry about that, it sounds really nasty. I've uploaded it to the google drive, here is the link. I hope it will work well.

[mscheltienne]

For the first issue, it comes from a mismatch of 2 samples between those 2 variables

mne-python/mne/io/egi/egimff.py

Lines 85 to 86 in e5086fc

	n_samps_block = signal_blocks['samples_block'].sum()
	n_samps_epochs = (epochs['last_samps'] - epochs['first_samps']).sum()

I do recall having some mismatch issues in those 2 lines a while ago, probably while working on mne-tools/mne-bids#1006 but I don't remember what the reason was.

I'll try to have a deeper look in early January, as we are still using heavily the EGI system both for awake and sleep studies in Geneva. FYI, it seems like mffpy is able to read the file. I was able to access the data array and the main information easily; although I could not load entirely this large array in RAM.

When the reader for MFF files was added, it was not based on mffpy (#4017). This additional dependency was added to support Evoked (#8354). What do you think of a rework of this reader based on mffpy which seems to support better the many different version of the MFF format?

[larsoner]

I think that would be great!

[agramfort]

+1