"Reading chunk length error" after a Python crash at the end of the recording, any solution ?

[Jbelo-Inria]

Hi guys,
I'm trying to read .xdf files with the pyxdf.load_xdf() function but I have this error for some of my files :

Error reading chunk length
Traceback (most recent call last):
File "/user/jbelo/home/anaconda3/lib/python3.7/site-packages/pyxdf/pyxdf.py", line 237, in load_xdf
chunklen = _read_varlen_int(f)
File "/user/jbelo/home/anaconda3/lib/python3.7/site-packages/pyxdf/pyxdf.py", line 487, in _read_varlen_int
raise RuntimeError("invalid variable-length integer encountered.")
RuntimeError: invalid variable-length integer encountered.
got zero-length chunk, scanning forward to next boundary chunk.

For those files, Python crash during the recording at the very end of the experiment and even if the experiment was already finished it seems that the files were corrupted by it.
I check in the files and there is no "Footer" dict.
And the second problem is when I want to transform my .xdf files in .fif ones (using MNElab), it doesn't work probably because of this error.

Do you have any suggestion to solve this problems or those files are definitely unusable ?

Thanks in advance,
Joan

[cbrnr]

You don't need a footer for a valid XDF file, but if the recording crashes without properly closing, you will end up with invalid data. I don't know if there are any repair tools available, but in principle it should be possible to use all chunks up until the corrupted one.

[agricolab]

You can use XDFBrowser to inspect the file: https://github.com/xdf-modules/XDFBrowser, which can give you a little bit more info about the misbehaving chunk.

Considering the recording stopped, the warning you see is likely simply the data loss, and the recorder stopped writing to the file before the chunk (and the footer) was correctly finished. I also don't know about any tool for repairing but that's certainly doable and not too hard. I don't know about your pipeline for fif-conversion. If you share it, we can give more advice.

Following these two example scripts, wouldn't first loading with pyxdf, then conversion to mne,Raw and then saving not work?
https://mne.tools/dev/auto_examples/io/read_xdf.html
https://mne.tools/stable/generated/mne.io.Raw.html#mne.io.Raw.save

[Jbelo-Inria]

Thanks for the comments and advices guys.
Ok, so I can use the files up until the corrupted chunk, that's good news.

To give you more details, I'm using LSL to record multiple streams (EEG, Eye tracker, Markers) synchronously and a modified version of Open Sesame to run the experiment.
This way of recording things allow me, at the end, to synchronise the markers with the EEG (and Eye tracker) data. From the files (and also the corrupted ones), I can extract the marker stream and the EEG stream, create a Raw file using mne raw data structure creation pipeline and add the markers extracted as annotation and then record this raw file as a .fif file.
But the weird think with the corrupted files is that the annotation starts very lately (while they should start at the beginning of the experiment). It's like an information about the real start time of the annotations is missing in the .xdf file. This real start time probably correspond to the "orig time" variable of the mne annotations. But I'm not sure how to obtain this information and if it's at least present in the my corrupted .xdf file.
So my question is : is a solution exist to fix this issue ?

Thanks,
Joan

Unfortunately I cannot attached one of those .xdf files because this file type is not supported...

[agricolab]

I struggle a little bit with understanding your issue here. What i understand is that there is an additional marker stream, and you want to use that one to create annotations for the mne.Raw. But, the timestamps of this marker stream are not aligned well?

What i usually do is create a plot of the timestamps to inspect the distribution. If they are only shifted (but not clustered due to a network issue), you might be able to recover them. Did you start recording with LabRecorder and your experimental task via a script, or was everything started separate and manually?

This seems to be in addition to the file recording having been aborted prematurely. Might be that both was caused by a common error in the recording PC, but the fix is probably different.

[cbrnr]

I agree that these might be two separate issues.

FYI, I've just added an option to view XDF chunks to MNELAB. It's pretty much what XDFBrowser does, but it should be easier to deploy (not everyone is familiar with compiling a C++ program for those who cannot use the provided Windows binary). If you want to use this feature right now (i.e. before the next release), you can install the dev version with:

pip install git+https://github.com/cbrnr/mnelab

The option is then available under "File" – "Show XDF chunks...".