[draft] new reader for curry files, using curry-python-reader code

[dominikwelke]

hi all
as discussed in #13033 here a draft PR to use the official curry reader code.

in a first step i just use the reader as a module (only fixed formatting to force it past pre-commit).
it has some drawbacks (e.g. data is always loaded) and i did not implement all possible data yet (eg hpi, or epoched recordings) but in general it already works pretty well. tested it with all their example data, and one of my own recordings that didnt work in mne before.

it would be great to get some feedback how you want me to proceed with this @drammock @larsoner:

• do we want to stick with the module approach, leave their code untouched and work with the output (would allow easier updating when they push changes)
• or should i merge the code more thoroughly. making it easier to maintain and in terms of clarity

BACKGROUND:
the curry data reader currently cant handle all/newer curry files
plan is port code from the official curry reader into mne-python

for permission see #12855

closes #12795
closes #13033
closes #12855

[drammock]

do we want to stick with the module approach, leave their code untouched and work with the output (would allow easier updating when they push changes), or should i merge the code more thoroughly. making it easier to maintain and in terms of clarity

Given that @CurryKaiser refused our offer to help them package up their reader for PyPI / conda-forge, I see two remaining options:

1. "vendor" their code. To make it slightly future-proof, we could write a script (in tools/ I guess) that fetches the latest code from their repo, auto-formats it to make it compatible with MNE's pre-commit requirements, and puts the (formatted but otherwise unmodified) code in mne/io/curry/_vendored.py. (This is basically a manual version of git submodule update because I don't think we should invoke git submodule for this use case.) We then adapt our code in mne/io/curry.py to be a wrapper around their code that basically just gets things into proper MNE container objects; and know that we might need to tweak our wrappers any time the vendored code is updated.

2. Fully incorporate their code. Re-write their reader to align better with our codebase, in terms of variable names, idioms like _check_option or _validate_type, features like preload=False, etc.

Personally I lean toward option 2. I say this because if we're going to try to support curry files, at a minimum we need to be able to fix bugs when they arise, and ideally we should be willing/able to incorporate new features that have high demand from our users (preload=False is an obvious first example). But if we're fixing bugs, do we open PRs to upstream (with no guarantee of responsiveness), or tweak our "thin" wrapper to handle more and more corner cases? Neither option is appealing, so at that point it starts to seem easier to me to just maintain the entire reader ourselves.

[agramfort]

hum... my first reaction is to push a version 0.1 of their package on pypi and rely on this. Basically we maintain a fork and hope that the fork changes are accepted upstream... it feels less hacky and they also have a CI and some testing setup with test_data that I would not duplicate in mne-python...

…

Message ID: ***@***.***>

[drammock]

hum... my first reaction is to push a version 0.1 of their package on pypi and rely on this. Basically we maintain a fork and hope that the fork changes are accepted upstream... it feels less hacky and they also have a CI and some testing setup with test_data that I would not duplicate in mne-python...

that indeed is less hacky than my approach to vendoring. I'd be OK with that outcome, though curious what @larsoner will think.

[larsoner]

I'm fine with that idea but it would be good to get some blessing/permission from them to do this

[drammock]

@CurryKaiser

I'm fine with that idea but it would be good to get some blessing/permission from them to do this

xref to #12855 (comment) where I've asked for confirmation that Compumedics really doesn't want to be the packager and they're OK with us doing it.

[CurryKaiser]

@CurryKaiser

I'm fine with that idea but it would be good to get some blessing/permission from them to do this

xref to #12855 (comment) where I've asked for confirmation that Compumedics really doesn't want to be the packager and they're OK with us doing it.

And nothing has changed, so all good from our side. Sorry we couldn't package it for you. And thank you for working on this!

[dominikwelke]

thanks @CurryKaiser !

ok, sounds like a plan.. I can start working on this again soon, if you give a go @agramfort @drammock @larsoner

I guess the fork should live in the mne-tools org? I have the necessary rights to create it

[larsoner]

Yeah I think so

[drammock]

Yeah I think so

I already made the fork

[agramfort]

ok, sounds like a plan.. I can start working on this again soon, if you give a go @agramfort <https://github.com/agramfort> @drammock <https://github.com/drammock> @larsoner <https://github.com/larsoner>Message ID: ***@***.***>

+1

[drammock]

xref to mne-tools/curry-python-reader#1

[dominikwelke]

i could need some guidance on 2 things:

1. channel locations:
   curry files come with channel locations, and for EEG it was straight forward to build a montage and apply.
   but for MEG it seems i need to use other functions. any pointers would help!
   do i need to populate info["dig"] directly?

2. HPI/CHPI data:
   some MEG files seem to come with these data. how do i store them in the raw object?

a few other things to discuss:

• preload
  easiest would be to not offer preload=False and just load the data to memory.
  a single load_data() call would also be doable with the official reader, but a chunk reader not really (if we dont want to hack it; e.g. load all data and discard large parts). not sure i'm deep enough in the mne codebase to know what the implications are (e.g. computations, plots etc with unloaded data)
  what are your thought?

• epoched files
  the reader code looks as if there could be files with epoched recordings, but there are none among their sample files. do any of you know more about this? otherwise ill ask the curry devs

[dominikwelke]

p.s. and could you remind me how to switch off the CIs when pushing these early commits?

[larsoner]

Push commits with [ci skip] in the commit message and they long / expensive CIs shouldn't run (a few quick ones still will I think)

[larsoner]

... for the cHPI stuff it's probably easiest to load a Neuromag raw FIF file with cHPI info and look at how the info is stored for example in info["hpi_subsystem"]. You can also look at the Info docs, especially the notes. It's not complete but it will help.

For preload, since preload=False is in main it would be a functionality regression to remove it. Once you know how the data are stored on disk and how to read an arbitrary time slice from it, it's really not bad to do the work to make preload=False work. So if you can figure this part out in some function, I can help you fit it into the _read_segment_file code. Since the .py file is only a few hundred lines (a lot of which seems like plotting etc. that we won't use), I'm cautiously optimistic we can figure it out and make it work. And then the python-curry-reader code can really be for reading metadata, annotations, sensor locs, etc. plus knowing where to read the data from disk. We can probably even keep the existing _read_segment_file, it should work in theory...

[dominikwelke]

ok, _read_segment_file does indeed work unchanged.
the reader should now be more/less functional

• I'd still need some guidance on handling/storing the channel locations, esp. for MEG data
• HPI data - looks like I got it wrong - there might not be cHPI data after all, only HPI marker locations provided in different formats depending on the system

[dominikwelke]

@CurryKaiser
thanks for the permission to use the code, also from my side!

in another place you said you might be able to provide us with test files - could we perhaps get a small one with epoched recordings in it (format version shouldn't matter)?
your repository for the python reader contains some test files that the reader interprets as epoched, but they dont seem to really be (perhaps the files were truncated for size)

[CurryKaiser]

Could be that they were truncated, let me check.

[CurryKaiser]

Ok, try these:
EpochedData

[dominikwelke]

thanks for the file @CurryKaiser
fyi, we have now packaged and published the curryreader on PyPI.
it can be installed via pip install curryreader

[dominikwelke]

@drammock @larsoner @agramfort
it is on PyPI but not on conda-forge - how is this case dealt with in MNE? should we also submit it to conda forge?

currently pip install mne[full] fetches it, but conda env create --file environment.yml doesnt

related question:
which pip dependency level in pyproject.toml should this go to? i treated curryreader like the antio package (for ANT neuro files) but this makes it an optional requirement (in mne[full]). i believe this mean it wont be automatically installed when calling pip install mne?

[larsoner]

it is on PyPI but not on conda-forge - how is this case dealt with in MNE? should we also submit it to conda forge?

Yeah use grayskull, it's not too painful, see for example conda-forge/staged-recipes#28279

[dominikwelke]

Yeah use grayskull,

see conda-forge PR: conda-forge/staged-recipes#29754

[drammock]

fwiw, the sensor alignment looks better with this transform, but probably not great yet? idk if this is due to the sample mri..

yikes, looks like the subject's nose is intersecting with the helmet surface!

[dominikwelke]

yeah, the nasion landmark definitely doesnt align with the sample subject's nasion :)

[dominikwelke]

just pushed the version with updated coil type and dev_head_t that produces the plots above.

for you to make a call @larsoner :

1. comment on the alignment plots above
2. violated tolerance in HPI point matching:

this currently breaks the reader as _quaternion_align simply raises a RuntimeError

• should i remove/crank up the tolerance and replace by a warning if the matching is bad?
• ..or keep the 10mm tolerance, and fall back to `dev_head_t=None' if matching is too bad?

---

on another note:
the failing tests are expected, as explained in another response above.
there were questions, too @larsoner - tl;dr: more info on the used testfiles (c,rfDC Curry 7.dat / 8.cdt) would help! they dont seem to be official curry files?
i can "fix" the header files, and then all tests pass

[larsoner]

should i remove/crank up the tolerance and replace by a warning if the matching is bad?

Let's add a on_bad_match="warn" using _on_missing internally

fwiw, the sensor alignment looks better with this transform, but probably not great yet? idk if this is due to the sample mri..

I think there is a bug where the helmet surface is in the wrong place. If you look just at the MEG sensors (or plot with meg="sensors") it seems reasonable. The EEG electrodes (or dig points?) in pink look like they might be spun 180 degrees, or flipped forward-backward or something. The trans is also wrong which makes the MRI head surface poorly aligned with the head coordinate frame...

Can you paste a minimal example to reproduce the plot_alignment? Image? Don't need every angle, just one plot I can click around etc.

there were questions, too @larsoner - tl;dr: more info on the used testfiles (c,rfDC Curry 7.dat / 8.cdt) would help! they dont seem to be official curry files?

I am not sure about these... can you browse through git blame to find a PR where they're first used? Ideally there it would say where the files came from. If you can't figure it out I can help

[drammock]

I think there is a bug where the helmet surface is in the wrong place. If you look just at the MEG sensors (or plot with meg="sensors") it seems reasonable.

Does that mean that https://mne.tools/dev/auto_examples/visualization/meg_sensors.html#ctf is incorrect? Locally interacting with that 3D plot it looks OK (though I've not used a CTF system so just using instinct about what looks "reasonable")

[larsoner]

Does that mean that https://mne.tools/dev/auto_examples/visualization/meg_sensors.html#ctf is incorrect? Locally interacting with that 3D plot it looks OK (though I've not used a CTF system so just using instinct about what looks "reasonable")

I think that one is okay... the sensors there look properly aligned with the helmet. I suspect the curry code modifies the MEG sensor coordinates or something -- at least compared to how we represent them when reading native files from the CTF system -- so then when we put the helmet in assuming the MEG sensors are in the same place, it's wrong. At least that's what I'm assuming is going on.

Maybe a dumb question but are those CTF sensors? In the code we have ch["coil_type"] = FIFF.FIFFV_COIL_CTF_GRAD hard-coded, but if it's actually from a different system (like KIT) then it'll pull the wrong helmet to show...

[dominikwelke]

@larsoner -

Can you paste a minimal example to reproduce the plot_alignment? Image? Don't need every angle, just one plot I can click around etc.

yeah, i just used your minimal working example here

unfortunately i've not really worked with MEG myself before :D

Maybe a dumb question but are those CTF sensors?

I reuse the legacy MNE code - it was hardcoded there.
i did some research and believe we should mostly encounter data from compumedics orion devices.
what this now means, no clue.. do we need to design a new helmet shape?

perhaps ppl can use curry software to save data from other MEG devices (maybe that's what these rf,dc testfiles are), but not sure we can really cater for that:

1. i didnt see any info in the header files as to which MEG device is used
2. in the curryreader code by the compumedics team there is a comment that HPI locations (those that we need for point matching for the transform) are there for orion devices only ("HPI-coil measurements matrix (Orion-MEG only) ..") - as opposed to older compumedics MEGs or other manufactures? dont know..

code wise, i guess it is possible to let ppl specify the MEG device via input argument, but really not sure if this a plausible scenario..

[dominikwelke]

re: rfDC test files:

can you browse through git blame to find a PR where they're first used?

ok - these files were uploaded by @DiGyt in 2019 - he seems to be the author of most our legacy curry code.
EDIT: oh yeah, and if you haven't looked into the test code yet - they seem to correspond to BTi testfiles bti_rfDC_file = data_dir / "BTi" / "erm_HFH" / "c,rfDC", thats why i thought they might be fabricated..

can you also have a look at the test_read_raw_curry_rfDC test @larsoner ?
this is also legacy code and seems supposed to explicitly check if MEG sensors are appropriately oriented; this test still passes on my system

also fyi, on my system the legacy code and my new reader version produce identical (distorted) plots for the files that i checked and that both can read.

unfortunately it wont work for you, as the rfdc files cannot be read by curryreader, as described above and the HPI.cdt file i sent you cannot be read by the legacy MNE reader.

if you replace the header file of the rfDC testfile with this version it should work (i stripped away the surplus channels that are not in the binary file):
c,rfDC Curry 8.cdt.dpa.zip

[dominikwelke]

Let's add a on_bad_match="warn" using _on_missing internally

implemented

[larsoner]

code wise, i guess it is possible to let ppl specify the MEG device via input argument, but really not sure if this a plausible scenario..

In this case, we should change the coil type to a compumedics one and consider it a bugfix probably. I'll take a look and probably push a commit...

[larsoner]

Okay changing the coil type (commit pushed) and putting in a plausible trans for head<->MRI we get something pretty reasonable:

Code

import mne
subjects_dir = mne.datasets.sample.data_path() / "subjects"
raw = mne.io.read_raw_curry("~/Desktop/HPI/HPI.cdt")
print(raw.info["dev_head_t"])
subject = "sample"
trans = mne.coreg.estimate_head_mri_t(subject, subjects_dir=subjects_dir)
# eyeballed!
trans["trans"][1, 3] += 0.01  # move MRI back relative to dig
fig = mne.viz.create_3d_figure((800, 800), bgcolor="w")
mne.viz.plot_alignment(raw.info, coord_frame="meg", dig=True, meg=dict(helmet=0.2, sensors=0.1), subject=subject, subjects_dir=subjects_dir, surfaces=dict(head=0.2), trans=trans, fig=fig)
mne.viz.set_3d_view(fig, azimuth=180, distance=0.7)

if you replace the header file of the rfDC testfile with this version it should work (i stripped away the surplus channels that are not in the binary file):

@dominikwelke it seems like these test files are probably incorrect... you want to push a fix to mne-testing-data and then re-enable the tests here? For now I've marked them with xfail but hopefully after your fix you can remove those...

I think that's all you asked about but let me know if I missed something @dominikwelke !

[dominikwelke]

great, thats looking much better.. thanks!
will fix the testfiles etc.

[dominikwelke]

this is what i asked so far, but now that this seems fixed i have two more API questions @larsoner @drammock -

1. legacy montage reader (read_dig_dat)

this only works with curry 7 format, and the name is suboptimal as newer curry files don't have a .dat extension (plus .dat is pretty generic, anyway).
i already wrote a new version that i call read_dig_curry

shall we deprecate read_raw_dat?
..keep the API unchanged (migrating the new code into read_raw_dat even though its a bad name)
..or keep both read_raw_dat and read_raw_curry?

[larsoner]

The lowest-effort solution (for us and users) is to mark read_dig_dat as @legacy, and make it call read_dig_curry under the hood so it's essentially a one-liner, keeping read_dig_curry as the new/preferred/documented solution (and remove read_dig_dat from the docs).

[drammock]

• I'd be OK with deprecating read_dig_dat and recommending users use read_dig_curry instead.
• I'd be OK with deprecating read_raw_dat and recommending users use read_raw_curry instead. If it's helpful you can keep read_raw_dat around but make it private, and triage reading of old .dat format to that function.

Another option instead of deprecating is using our @legacy decorator, which keeps the function in the API but adds prominent warnings that it shouldn't be used for new code (and what should be used instead). I defer to @larsoner on whether legacy or deprecation is the better choice here; I don't have a clear sense of how disruptive deprecation would be, since IDK how many curry files there are out there in the wild.

[dominikwelke]

2. maybe more of a comment..

I think it makes sense to use the curryreader from the compumedics crowd, but it does have some negative side effects:

• the full recordings are always loaded to memory under the hood, even though we ignore the data and soon remove them again from memory.
  this is suboptimal for multiple reasons - importantly, you always need the full file suite on disk, even for the montage reader the binary file needs to be there (not just the text header that holds the channel information).
• i needed to add some additional readers for relevant bits and pieces, so curry files are accessed multiple times

i just wanted to flag that again @larsoner @drammock

now we maintain the curryreader ourselves and already fixed quite a few things that are not merged upstream.
one option would be to go further, and add a preload flag, but it would be more substantial changes

[drammock]

one option would be to go further, and add a preload flag, but it would be more substantial changes

let's focus first on getting it working, even if it's inefficient / slow. Adding preload support can come later if folks really want it / complain about speed.