ENH: Support TD data #11064

larsoner · 2022-08-21T19:50:09Z

@rob-luke how about this plan:

For now I'm inclined to make it so if you get a SNIRF_PROCESSED data type that we don't know how to handle, we should maybe raise an error in this case...? For now I've removed it as a coil_type because this _PROCESSED really seems to mean something else specific, and we should perhaps add those one by one as we find use cases for them.

I have some collaborators who have seen Kernel help files telling them to use this very out of date branch, so it would be nice to get this in!

…a type as outlined in the SNIRF specification document

…s, moments and processed (hbo/hbr)

rob-luke · 2022-08-23T11:04:27Z

Thanks for jumping in to finish this off @larsoner

For now I'm inclined to make it so if you get a SNIRF_PROCESSED data type that we don't know how to handle, we should maybe raise an error in this case...?

Agreed

Ping me when you want a review, it doesn't need to be a final review, I am happy to take a look early if that helps.

larsoner · 2022-08-23T11:26:53Z

Feel free to look now @rob-luke to see if you're happy with the overall design, the rest of the work is "just" some remaining details (adding unit tests, playing with the data a little bit to set scale factors, etc.) that I think we can rely on CIs to check, or that we can iterate on in follow-ups (e.g., tweaking plot scale factors)

rob-luke

Looking good, it will be great to have support for TD in MNE! I will review again once there are tests.

mne/defaults.py

* upstream/main: (264 commits) BUG: Fix deprecated API usage in example (mne-tools#11512) Deprecate 'kind' and 'path' in favor of 'fname' in the layout reader (mne-tools#11500) EGI/MFF events outside EEG recording should not break the code (mne-tools#11505) fixed annotations error on export (mne-tools#11435) DOC: Update installer links [skip azp] [skip actions] [skip cirrus] (mne-tools#11506) BUG: updates for MPL 3.7 compatibility (mne-tools#11409) Fix docstrings by replacing str with path-like and fix double backticks for formatting (mne-tools#11499) Use pathlib.Path instead of os.path to handle files and folders [circle deploy] (mne-tools#11473) MAINT: Fix Circle [circle deploy] (mne-tools#11497) MAINT: Use mamba in CIs (mne-tools#11471) Updating documentation to clarify full vs half-bandwidth and defaults in time_frequency.multitaper.py (mne-tools#11479) Fix typo in tutorial (mne-tools#11498) Typo fix and added colons before code (mne-tools#11496) [MRG] ENH/DOC: demo custom spectrum creation (mne-tools#11493) Accept only left-clicks for adding annotations (mne-tools#11491) [BUG, MRG] Fix pial surface loading, logging in locate_ieeg (mne-tools#11489) [ENH] Added unit_role to add non-breaking space between magnitude and units (mne-tools#11469) MAINT: Fix CircleCI build (mne-tools#11488) [DOC] Updated decoding.SSD documentation and internal variable naming (mne-tools#11475) Typo fix (mne-tools#11485) ...

larsoner

Okay code has been updated. I suspect the scaling could be wrong as I mention in a comment ~~above ☝️~~ below 👇 but in any case people can test it with some data!

It's possible/likely that in addition to the possible scaling bug, we'll want to adjust the default plot scaling limits.

It would also be great to have an example added (maybe using mne-misc-data) showing that this data can at least be read!

mne/io/snirf/_snirf.py

mne/io/snirf/tests/test_snirf.py

* upstream/main: eegbci api: allow downloading multiple subjects (mne-tools#12918) DOC: Document Linux desktop workaround (mne-tools#12900) Allow exporting edf where a channel contains only constant values (mne-tools#12911) Autogenerate environment.yml file from pyproject.toml (mne-tools#12914)

JohnGriffiths

confirming that the example runs here, and also that it seems to load other data files properly. the latter point I will can make and share a separate NB demonstrating that for 'peace of mind', in lieu of an analysis example discussed on the issue thread. Also noting that given @julien-dubois-k 's comment re some .snirf attributes not being read, so those steps may also need to be added either to this PR or (probably better) a subsequent one. I will be looking at this next.

Afraid I don't know the answer to the uM units Q off hand.

JohnGriffiths · 2024-11-07T01:56:31Z

@larsoner thanks for this. Have gone through, left review above.

Not quite sure if I missed something specific other than the units Qs and the time_delay_widths Qs that you needed an immediate answer to; LMK

I don't not know to these off the top of my head, but I imagine @julien-dubois-k does. I can take a look at relevant documentation if that is useful.

Re: datasets

LMK if I have got this right re next steps

You have two small .snirf files in the example data that is loaded here so new ones of those aren't needed from me now for this test function
But you are suggesting to add some other, larger files to mne-misc-data for an analysis example (?)
So, would you like me to do a PR to that repo with a .snirf data file for a more fulsome analysis example?
If so yes I can get on that. What I would do for the example code is use the colab notebook example @julien-dubois-k linked to in his above comment, and the example file the one used in that nb.
( I can add other finger tapping .snirf files to this if warranted )
As I work on that I will take a look at the comments from @julien-dubois-k noted in my review comment above about some .snirf attributes not being read from the .hdf properly by the mne reader. Which may lead to another .io.snirf PR before the analysis example PR.
But I think it's prob best not to muddy the current PR with that Q (unless you disagree?)

julien-dubois-k · 2024-11-07T03:06:50Z

thank you @larsoner and @JohnGriffiths for your work pushing this PR forward. I’ll review tomorrow and answer any remaining questions that I can answer!

…

On Wed, Nov 6, 2024 at 17:56 John Griffiths ***@***.***> wrote: @larsoner <https://github.com/larsoner> thanks for this. Have gone through, left review above. Not quite sure if I missed something specific other than the units Qs and the time_delay_widths Qs that you needed an immediate answer to; LMK I don't not know to these off the top of my head, but I imagine @julien-dubois-k <https://github.com/julien-dubois-k> does. I can take a look at relevant documentation if that is useful. Re: datasets LMK if I have got this right re next steps - You have two small .snirf files in the example data that is loaded here so those aren't needed now for this test function - But you are suggesting to add some other, larger files to mne-misc-data for an analysis example (?) - So, would you like me to do a PR to that repo with a .snirf data file a a more fulsome analysis example? - If so yes I can get on that. What I would do for the example code is use the colab notebook example @julien-dubois-k <https://github.com/julien-dubois-k> linked to in his above comment <#11064 (comment)>, and the example file the one used in that nb. - ( I can add other finger tapping .snirf files to this if warranted ) - As I work on that I will take a look at the comments from @julien-dubois-k <https://github.com/julien-dubois-k> noted in my review comment above about some .snirf attributes not being read from the .hdf properly by the mne reader. Which may lead to another .io.snirf PR before the analysis example PR. - But I think it's prob best not to muddy the current PR with that Q (unless you disagree?) — Reply to this email directly, view it on GitHub <#11064 (comment)>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/AWIKIDKMADYHYTI6KHYECT3Z7LCGNAVCNFSM57FOV3OKU5DIOJSWCZC7NNSXTN2JONZXKZKDN5WW2ZLOOQ5TENBWGEYTKNZRG44A> . You are receiving this because you were mentioned.Message ID: ***@***.***>

larsoner · 2024-11-07T19:02:57Z

You have two small .snirf files in the example data that is loaded here so new ones of those aren't needed from me now for this test function

Well related to the time_delay_widths issue, it would be good to have some small SNIRF test files that actually have this field. I'm not sure if we're using it properly, and if we had test files with this field then we could probably figure it out. @julien-dubois-k if you have such a file we could add it to mne-testing-data!

But you are suggesting to add some other, larger files to mne-misc-data for an analysis example (?) ... What I would do for the example code is use the colab notebook example @julien-dubois-k linked to in his above comment, and the example file the one used in that nb.

Sure, then we modify an example here or more likely in mne-nirs to use it. Doesn't have to be a full-blown example with regression and stats. But if that notebook runs in less than, say, a minute, we could convert it, add some narrative docs if needed, and be good to go. (Examples are meant to run fairly quickly and not require a ton of data to be downloaded!)

( I can add other finger tapping .snirf files to this if warranted )

Sure if it adds something complementary from a documentation perspective. But if it's just additional subjects or a different paradigm from what we currently have, might not be worth it.

(Just an aside, not sure if you've looked at mne-bids-pipeline but there is mne-tools/mne-bids-pipeline#365 that we could resurrect if you're interested in a BIDS pipeline for your data.)

But I think it's prob best not to muddy the current PR with that Q (unless you disagree?)

I don't mind adding more stuff here, a +305/-85 PR is fairly small actually. So based on discussion I think the next steps would be:

@JohnGriffiths open PR to add example data in mne-misc-data that we need for the notebook
I modify mne.datasets code in this PR to reflect the new data
@JohnGriffiths open a PR to mne-nirs to use the new example data based on the notebook
I modify the mne-nirs PR to use this branch temporarily (or you can if you already know how @JohnGriffiths)
@julien-dubois-k supply some tiny files we can add to mne-testing-data that would shed light on the time_delay_widths issue
@julien-dubois-k comment on the unit issue above
I push commits to update tests with new data and/or unit expectations
I make sure all plotting scales etc. are correct based on the MNE-NIRS example
We merge this PR
I remove "use larsoner:feature/TD-nirs_snirf branch" shim from MNE-NIRS PR
We merge the MNE-NIRS PR

JohnGriffiths · 2024-11-07T21:30:50Z

@JohnGriffiths open a PR to mne-nirs to use the new example data based on the notebook

Ok I'm on it.

@JohnGriffiths open PR to add example data in mne-misc-data that we need for the notebook

Will do. What is the heuristic on acceptable size for a file in mne-misc-data to use in an example?

I modify the mne-nirs PR to use this branch temporarily (or you can if you already know how @JohnGriffiths)

So just temporarily change this requirements line to the commit url for the current PR, right?

(Just an aside, not sure if you've looked at mne-bids-pipeline but there is mne-tools/mne-bids-pipeline#365 that we could resurrect if you're interested in a BIDS pipeline for your data.)

That's great to know. thanks. We are actually currently in the process of getting an mne-bids-pipeline auto QC in XNAT for our EEG data. Would like to do the same for fNIRS. That would slot in nicely.

larsoner · 2024-11-07T21:36:14Z

Will do. What is the heuristic on acceptable size for a file in mne-misc-data to use in an example?

We try to keep our datasets to < 1GB if possible. Looks like mne-misc-data is around 700 MB so it would be good not to add too much more. Maybe up to 100MB is okay for the purposes here? If you need more than that then we can add it as a new dataset on OSF or somewhere else, and add a mne.datasets.<something>.data_path() downloader. That's a little more work but can be worth it.

So just temporarily change this requirements line to the commit url for the current PR, right?

CircleCI is the CI that builds the doc so I would do it here actually

https://github.com/mne-tools/mne-nirs/blob/5c63e855b60d6f7aab907477e582acb323a69d81/tools/circleci_dependencies.sh#L13

mne/defaults.py

mne/io/snirf/tests/test_snirf.py

larsoner · 2024-11-18T14:14:09Z

I am a bit busy for the next two weeks but then should be able to come back to this!

larsoner · 2025-01-24T19:28:48Z

Okay two weeks turned into two months 😅 But I should be able to look next week

* upstream/main: (149 commits) FIX make_watershed_bem to handle missing talairach_with_skull.lta courtesy Freesurfer 8 (mne-tools#13172) ENH: Add upsampling for MEG helmet surface (mne-tools#13179) MAINT: Update code credit (mne-tools#13180) BUG: Fix bug with least-squares sphere fit (mne-tools#13178) fix EDF export (mne-tools#13174) fix typo (mne-tools#13171) [pre-commit.ci] pre-commit autoupdate (mne-tools#13164) Fix dev installation guide (mne-tools#13163) expose 'mode' for plotting dipole on brain (mne-tools#13162) turn dipole attrs into properties (mne-tools#13153) remove misformatted (and unused) crossref anchor (mne-tools#13155) doc: point to read_dipole (mne-tools#13149) [pre-commit.ci] pre-commit autoupdate (mne-tools#13152) BUG: Fix bug with not short-circuiting n_jobs=1 (mne-tools#13147) FIX: Missing coordinates.xml in MFF file (mne-tools#13148) FIX: Gracefully handle bad XML files in EGI reader (mne-tools#13145) Fixes for Latest IPython (9.0.1) (mne-tools#13146) Fix intersphinx (mne-tools#13143) BUG: Fix bug with parallel doc build (mne-tools#13140) [pre-commit.ci] pre-commit autoupdate (mne-tools#13141) ...

larsoner · 2025-04-04T18:22:28Z

mne/defaults.py

+        fnirs_td_gated_amplitude="AU",  # counts
+        fnirs_td_moments_intensity="AU",  # counts
+        fnirs_td_moments_mean="s",
+        fnirs_td_moments_variance="s²",


@julien-dubois-k better? And in particular the S you suggested for the unit is for Siemens?

yes these are good. s is seconds

hah, whoops! Will fix

larsoner · 2025-04-04T18:23:52Z

Apologies for the embarassingly long delay on this one 🥵 Pushing some commits to make sure things are green, then I'll add the test files, etc!

larsoner

@julien-dubois-k okay I think I have it roughly where it could use another look from you to see if values are in the correct ranges, etc.!

larsoner · 2025-04-04T19:22:56Z

mne/io/snirf/_snirf.py

+_TD_MOMENT_ORDER_MAP = {
+    0: "intensity",
+    1: "mean",
+    2: "variance",
+}


@julien-dubois-k I just took a stab at this, not sure if it's actually correct?

larsoner · 2025-04-04T19:23:41Z

mne/io/snirf/tests/test_snirf.py

+        assert hbo_data.shape == hbr_data.shape == (shape[0] // 2, shape[1])
+        hbo_norm = np.nanmedian(np.linalg.norm(hbo_data, axis=-1))
+        hbr_norm = np.nanmedian(np.linalg.norm(hbr_data, axis=-1))
+        # TODO: Old file vs new file scaling, one is wrong!


Would be good to know if the numbers for the newer one are correct or not...

sorry, what do you mean?

The way the code works now the old test file and the new test file differ by several orders of magnitude (like 1e5) so I think there is uM vs M error scaling either in the file or in the code. Based on It looks like they should be closer to ~1e-6, right? If so then there is some bug in how we're reading the hbo/hbr values for the old file

larsoner · 2025-04-04T19:24:03Z

mne/io/snirf/tests/test_snirf.py

+        # TODO: Reasonable values here???
+        lims = dict(intensity=(1e4, 1e7), mean=(1e3, 1e4), variance=(1e5, 1e7))


I also don't know if these are reasonable values for the counts (intensity), mean (s), or variance (s**2)

these are reasonable if mean is in picoseconds and variance in ps^2.

julien-dubois-k · 2025-04-07T15:53:51Z

mne/defaults.py

-        fnirs_td_moments_amplitude=1e6,
+        fnirs_td_gated_amplitude=1.0,
+        fnirs_td_moments_intensity=1.0,
+        fnirs_td_moments_mean=1.0,


the values are in the range of nanoseconds, so this should be 1e-9 for mean and 1e-18 for variance maybe?

Zahra-M-Aghajan and others added 10 commits August 10, 2021 22:45

adding the new snirf data types to the constants

3f875d3

more places to add the data types and constants

cc92adc

adding constants for the allowed dataTypeLabels for the processed dat…

0d2fe87

…a type as outlined in the SNIRF specification document

updating the snirf reader to accept other data types: gated histogram…

21ecb90

…s, moments and processed (hbo/hbr)

setting the formatter line length

4491064

the reordering of the channels does not make much sense, need to inquire

dbdcebc

Merge remote-tracking branch 'upstream/main' into feature/TD-nirs_snirf

80de954

Merge remote-tracking branch 'upstream/main' into feature/TD-nirs_snirf

32c5496

MAINT: Simpler with .item

a35822a

WIP: Constants

c2c217c

larsoner mentioned this pull request Aug 21, 2022

Feature/td nirs snirf #9661

Open

larsoner added 9 commits August 21, 2022 20:54

FIX: Flake

ac52bc0

FIX: Consistent

da275f1

FIX: Missed

3986b96

FIX: Better code

ce8b3af

FIX: Scale

1244add

FIX: Missing

3944254

FIX: Flake

967dddd

FIX: Syntax

e97367d

Merge remote-tracking branch 'upstream/main' into feature/TD-nirs_snirf

4b963cd

larsoner added the fNIRS label Aug 23, 2022

rob-luke reviewed Aug 26, 2022

View reviewed changes

mne/defaults.py Outdated Show resolved Hide resolved

JohnGriffiths mentioned this pull request Sep 6, 2022

fix fnirs channel name cropping in topomap combination #11138

Open

larsoner mentioned this pull request Feb 2, 2023

add ch_type and coils for eye-tracker data mne-tools/fiff-constants#39

Merged

larsoner added 4 commits March 1, 2023 12:46

FIX: shape

d26ff8a

FIX: Slash

f84fe27

FIX: Bad merge

50f3064

larsoner added 6 commits October 28, 2024 13:51

FIX: SI

880356e

FIX: Oops

4f0e8e6

FIX: Oops

a5c97b8

FIX: idx

fba1ffa

WIP

9f4234d

FIX: Test

b3b3f2e

larsoner commented Oct 28, 2024

View reviewed changes

mne/io/snirf/_snirf.py Show resolved Hide resolved

mne/io/snirf/tests/test_snirf.py Outdated Show resolved Hide resolved

larsoner marked this pull request as draft October 28, 2024 18:34

JohnGriffiths approved these changes Nov 7, 2024

View reviewed changes

julien-dubois-k reviewed Nov 12, 2024

View reviewed changes

mne/defaults.py Outdated Show resolved Hide resolved

julien-dubois-k reviewed Nov 12, 2024

View reviewed changes

mne/defaults.py Outdated Show resolved Hide resolved

mne/defaults.py Outdated Show resolved Hide resolved

mne/defaults.py Outdated Show resolved Hide resolved

mne/io/snirf/tests/test_snirf.py Outdated Show resolved Hide resolved

JohnGriffiths mentioned this pull request Nov 13, 2024

initial draft of new TD-fNIRS KF2 example mne-tools/mne-nirs#586

Open

3 tasks

larsoner modified the milestones: 1.9, 1.10 Dec 9, 2024

larsoner added 2 commits April 4, 2025 10:19

WIP: More constants

e9403d8

larsoner commented Apr 4, 2025

View reviewed changes

larsoner added 2 commits April 4, 2025 14:29

FIX: Caps

6ee5068

FIX: More

b2b3716

larsoner commented Apr 4, 2025

View reviewed changes

julien-dubois-k reviewed Apr 7, 2025

View reviewed changes

		# TODO: Reasonable values here???
		lims = dict(intensity=(1e4, 1e7), mean=(1e3, 1e4), variance=(1e5, 1e7))

ENH: Support TD data #11064

Are you sure you want to change the base?

ENH: Support TD data #11064

Uh oh!

Conversation

larsoner commented Aug 21, 2022 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

rob-luke commented Aug 23, 2022

Uh oh!

larsoner commented Aug 23, 2022

Uh oh!

rob-luke left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

larsoner left a comment • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

JohnGriffiths left a comment • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

JohnGriffiths commented Nov 7, 2024 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

julien-dubois-k commented Nov 7, 2024 via email

Uh oh!

larsoner commented Nov 7, 2024

Uh oh!

JohnGriffiths commented Nov 7, 2024

Uh oh!

larsoner commented Nov 7, 2024

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

larsoner commented Nov 18, 2024

Uh oh!

larsoner commented Jan 24, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

larsoner commented Apr 4, 2025

Uh oh!

larsoner left a comment

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

larsoner commented Aug 21, 2022 •

edited

Loading

larsoner left a comment •

edited

Loading

JohnGriffiths left a comment •

edited

Loading

JohnGriffiths commented Nov 7, 2024 •

edited

Loading

larsoner commented Jan 24, 2025 •

edited

Loading