Case mismatches in entity values between data files and sidecars in real-world datasets

[bruAristimunha]

Context

In ds003751, data files use task-Emotion while their JSON sidecars use task-emotion. Strict case-sensitive matching fails to associate them.

The BIDS spec states that labels "MUST not collide when casing is ignored", implying that within a dataset, labels differing only in case refer to the same entity. ds003751 violates this by using inconsistent casing for the same task across files.

Question

Should find_matching_paths() or similar functions warn when near-matches differ only in case? This would help users identify non-compliant datasets without changing the default strict matching behavior.

Additional context

• MNE-BIDS 0.18.0
• Encountered during automated processing of ~200 OpenNeuro datasets.

[larsoner]

I would comment on the dataset on OpenNeuro to see if the authors can update it

The dataset also should not have passed a validator check, which I think OpenNeuro should have run, so I think there is possibly some bug there as well. If you try to validate locally does it warn or error (or neither)?

[bruAristimunha]

I have systematically contacted the authors and even contacted the openneuro itself about one dataset that wasn't even a valid format (*.tar instead of edf), for example. What I've noticed is that the authors ignore it or say they will fix it and never do.

Many of the errors I've encountered are extremely simple to resolve, but I have little hope that the authors will be able to fix them.

Maybe we can run something that use mne bids as validator within open neuro, and fix everything that needs to be fix.

My feeling is that they correct several MRI and fMRI datasets according to the issues I found, but the EEG/IEEG/MEG part is more complicated to have that iterability, so several of these errors prevent the datasets from being bid on.

[bruAristimunha]

@sappelhoff and @effigies, any input here?

[drammock]

In ds003751, data files use task-Emotion while their JSON sidecars use task-emotion. Strict case-sensitive matching fails to associate them.

The BIDS spec states that labels "MUST not collide when casing is ignored", implying that within a dataset, labels differing only in case refer to the same entity. ds003751 violates this by using inconsistent casing for the same task across files.

I don't think your interpretation is quite correct here. The main point of that paragraph in the BIDS standard is you cannot use "Emotion" and "emotion" as distinct labels, and IIUC ds003751 is not trying to do that. Now, the paragraph also says that "filename components are case-sensitive" but that doesn't seem to be the main point, and could be interpreted a couple different ways (e.g., it might be simply acknowledging that on some OSes filename uniqueness does/does not extend to case mismatch, or it might mean "according to BIDS, you should be consistent about case usage in your filename components"). And even if it's meant to convey the latter, stronger meaning, it's clear (to me anyway) that the non-collision point is more important to respect, so I'd argue that it would be better for MNE-BIDS to treat "Emotion" and "emotion" as the same entity value (perhaps with a warning) rather than being maximally strict about case matching. EDIT: in other words, by NOT treating "Emotion" and "emotion" as the same value, MNE-BIDS is effectively treating them as if they are different, which the BIDS standard forbids.

It would be nice if the standard were a bit more prescriptive about how the tools should handle such cases, e.g., "tools MUST treat entity values differing only by case-mismatch as the same" or alternatively "tools MAY handle case-mismatch as they like" or whatever EDIT: this latter possibility actually doesn't make sense; the tool cannot both adhere to the non-collision rule and also be strict about case-matching.

[drammock]

in case you're viewing on email: prior comment edited, please click through to read it.

[effigies]

The "MUST not collide" was added so you couldn't have sub-a_T1w.nii.gz and sub-A_T1w.nii.gz, which would cause technical problems on systems with case-insensitive filesystems. We did not really consider whether someone might use task-Emotion on the data and task-emotion on the sidecar, and I'm surprised that the validator did not catch it when this dataset was uploaded.

For what it's worth, the schema-based validator does error on this case, raising SIDECAR_KEY_REQUIRED and SIDECAR_WITHOUT_DATAFILE errors. It would be ideal if the author would resolve the issue by selecting a spelling and making a new release.

OpenNeuro does not request editorial control over datasets, and only requires that datasets validate at the time they are uploaded or a new version is created. Ultimately, the dataset owner is responsible for maintaining their dataset, and I think it is reasonable to choose not to support a dataset that has problems that go unfixed.

One thing we have done when we need to work with a dataset that has issues is to fork the dataset and publish it on GitHub: https://github.com/orgs/OpenNeuroForks/repositories. This allows us to have clear provenance without modifying the dataset in-place. This doesn't work if you need to modify the contents of data files, but if it's just renames, or patches to metadata, it's a practical solution.

[bruAristimunha]

oh, nice @effigies!

Can I create PRs across the repo that rename very small things that are failing? It would be perfect. I have a list of a few datasets from EEG+MEG+iEEG+fNRIS with micro problems that are preventing correct reading when I use mne-bids to read:

Thinking in terms of macro categories, I think I have these errors:

Problem	Example	Rename fix
Case-mismatched task entity	sub-01_task-Emotion_eeg.set + sub-01_task-emotion_eeg.json	Rename sidecar → sub-01_task-Emotion_eeg.json
Case-mismatched channels.tsv	sub-01_task-Emotion_channels.tsv vs data with task-Emotion	Rename TSV to match data file casing
Mismatched between participates.tsv and folder name	sub-01 vs data with sub-001	Rename to match the folder name it would solve
VHDR companion file typos	VHDR points to rsub-01.eeg but file is sub-01.eeg	fix VHDR pointer (metadata patch)
Modality directory naming	fnirs/ instead of nirs/	Rename directory to BIDS-standard nirs/
Malformed entity separators	task-ECONrun-1_eeg.set (missing _ before run)	Rename → task-ECON_run-1_eeg.set
Non-UTF-8 encoding	Latin-1/cp1252 TSV files	Need to re-encode file content of tsv metadata
Non-numeric run entities	run-5F	Rename → run-05
Duplicate participant_id rows	Same sub-ID appears twice in participants.tsv	Need to edit metadata TSV
n/a in electrode coordinates	x=n/a, y=n/a, z=n/a in *_electrodes.tsv	Need to edit metadata TSV cell values
Comma decimal separators	0,5 instead of 0.5 in TSVs	Need to edit metadata TSV cell values
n/a in channels.tsv numerics	sampling_frequency=n/a	Need to edit metadata TSV cell values
NaN onsets in events.tsv	onset=n/a rows	Need to edit/delete metadata TSV rows

Would it be okay if I created the PRs?

I think there are 40 datasets.

[bruAristimunha]

And many thanks to you very much for your understanding, @drammock. I will proceed with the PR that treats it as the same entity value, with some warning. I think the PR will be quite small, just improving the regex matching system.

I think that by combining this, we solve the problem at the root with OpenNeuro help 🙏🏽 , and we also solve it for the people who will consume the data.

[larsoner]

@bruAristimunha if you get on a roll fixing datasets and want extra credit... MNE-BIDS-Pipeline has a few workarounds for datasets we test with as well

https://github.com/mne-tools/mne-bids-pipeline/blob/035eafc1b412410f5ae666d00aaa43b9fc626c23/mne_bids_pipeline/tests/test_run.py#L192-L214

[effigies]

Can I create PRs across the repo that rename very small things that are failing?

You can create PRs, but they cannot be easily merged. The GitHub interface is just a mirror and a PR just provides a convenient diff view. Improving that workflow is on our roadmap. If you do create a PR and get in touch with the dataset owners and they want to adopt your changes, there is a method.

Method for the interested

git clone https://openneuro.org/api/git/$DS
git remote add github https://github.com/openneurodatasets/$DS
git fetch github refs/pull/$PR/head
git merge --squash $FETCH_HEAD
git push

That will update their draft, and they can then create a new version once it validates.

Docs: https://docs.openneuro.org/git.html

Anyway, for now I would just have a place where you put patched datasets.