stimuli BEP

[yarikoptic]

• edit: at large being addressed by BEP044 lead by @neuromechanist in [ENH] BEP044 - Stim-BIDS #2022 ; with some ideas slated for other efforts in bids-2.0 etc

As @Remi-Gau hinted by #695 , we still lack total clarity on original stimuli storage and annotation.

We do have

• stimuli/ folder which, like sourcedata/ is nohow "prescribed" for a specific structure.
• stim_file column in _events.tsv as to point to a (unregulated) location under stimuli/ and then populating that stim_file description/HED tags within _events.json (bless the inheritance principle).
• "human wording" to point to the origin of a stimuli within _events.json as possibly coming from some DB
• _stim.tsv.gz files for "signals related to the stimulus" (but not necessarily stimulus)

In respect to the first 3 items, and in conjunction with

• stimuli collections/datasets should be self-sufficient/described
• incoming requests to store stimuli datasets on DANDI archive

I wondered if there either an ongoing effort to standardize "stimuli datasets" so

• they could be readily reusable across studies by simply placing them under stimuli/<name> and avoiding necessity to describe stimuli in _events.json since information could be picked from their standardized layout
• derivatives of them could be created and possibly shared along (e.g. all the feature extractions done by pliers (thanks to @tyarkoni , @qmac, et al)

With that in mind I am even thinking such datasets could follow BIDS mantra and just get "participant/subject" and sub- renamed to "stimulus"/stim-, and preserve README.md, dataset_description.json, stimuli.tsv etc

Worth a BEP/effort or may be it is already a "solved problem"? ;) WDYT?

Related:

• fresh [ENH] allow for _stim.{mp[34],mkv,avi} to provide stimuli files for func data #750 for storing _stim.{mp3.mp4,...} along side with neural data. sidecar fields in that file could come handy why instrumenting specification of stimuli as presented from the shared across subjects sourcedata/ (e.g. StartTime, possibly TimeDrift between scanner and stimuli delivery for lengthy (an hour) presentation, etc)

[effigies]

At one point I talked to @Gilles86 about how he was storing stimuli, but don't clearly recall how deep we went. He might have some thoughts here.

Just to comment on one thing, I'm not sure stim-<label> buys much over <label>. It would be worth thinking about what are the orthogonalish dimensions that it would make sense to have entities for. A stimulus class name and an index to distinguish instances of that class are going to be most common. Then you may have some within class paramterization, but that's going to be really specific to the type of stimulus. For example, if you're interested in speaker-invariant speech representations, you might split your stimuli by speaker, but I don't see an entity that could cover all such parameterizations.

Something like <label>[_desc-<label>][_<index>].<ext> might cover most use cases without unnecessarily adding boilerplate.

[yarikoptic]

Just to comment on one thing, I'm not sure stim-<label> buys much over <label>.

• Having clean prefix allows to avoid collisions of <label>s conflicting with other possible directories (e.g. code/, sourcedata/ etc).
• With similar argument sub-<label>/ and sub-<label>_ IMHO also buy us nothing really, but that is what we have, and likely because they provide immediate "metadata" about the domain of <label> we are talking about,
• Having stim-<label> directory and stim-<label>_ filenames prefix allows to generalize BIDS dataset layout to cover "stimuli BIDS datasets", where stim entity serves analog to sub entity we have now.

[yarikoptic]

Something like <label>[_desc-<label>][_<index>].<ext> might cover most use cases without unnecessarily adding boilerplate.

I also hate boilerplate , and indeed in many use cases which might not even really need directories needed at all. BUT I can see stimuli collections where each stimuli could have a good number of files (audio, audio/video, images, etc) associated with that stimuli category; thus would be beneficial for organization and also navigation and reuse (clear "module" for a stimuli at the directory level). So, again, similarly to neuroimaging datasets where having just a single T1w image per subject, it might be sensible to have per-label directories. (moreover there could be multiple samples of the same label -- so semantically similar to _run- but that entity is really not a good fit for that, and indeed _desc- could be better)

PS although even may be run could have sense for some stimuli recordings of the same scene/action taken in sequence and otherwise having no immediate qualitative difference!

[effigies]

Hmm. Okay, fair enough. I guess the question is how much is this supposed to be BIDS-like or is it supposed to be BIDS? That matters for what entity names are chosen, since if it is BIDS, then we can't change the meaning of an entity too far. If it's just BIDS-like, then we can choose entities that are appropriate for stimuli with little regard for BIDS' existing definitions or ones that are likely to be claimed by future BEPs.

I would probably prefer BIDS-like, since a subject or a recording session is integral to a lot of definitions.

So here's a notion:

stimuli/
  dataset_description.json
  stim-<label>/
    stim-<label>[_desc-<label>][_item-<index>].<ext>
    stim-<label>[_desc-<label>][_item-<index>].json

• stim-<label> would be the task-relevant class
• desc-<label> would be within-class parameterization
• item-<index> would be like run- with no qualitative difference
• <ext> can probably indicate data type without resorting to an additional _<suffix>, which is another reason to be BIDS-like, instead of BIDS, where suffix is required.

An alternative (or addition) to desc- could be a stims.tsv that allowed you to explicitly say that here are relevant factors:

stimulus                                speaker_id  speaker_gender  tone
stim-word1_desc-sp1normal_item-1.wav    1           M               normal
stim-word1_desc-sp1normal_item-2.wav    1           M               normal
stim-word1_desc-sp1strained_item-1.wav  1           M               strained
stim-word1_desc-sp1strained_item-2.wav  1           M               strained

Then we presumably need a stims.json to define columns.

[yarikoptic]

Thank you @effigies !!! I feel like we are on the same page and progressing leaping forward ;)

I guess the question is how much is this supposed to be BIDS-like or is it supposed to be BIDS? That matters for what entity names are chosen, since if it is BIDS, then we can't change the meaning of an entity too far.

what entities meaning you see needing much of adjustment? Even for run I feel we would not need much of adjustment although some might already be a bit overdue: filed #760 . So not sure if we really need to introduce _item in favor of _run just yet

An alternative (or addition) to desc- could be a stims.tsv

+1 on that. Additional thoughts:
If we are to retain scans as a term, and aim for "BIDS" (not just "BIDS-like") then such a file would be analogous to _scans.tsv we already have, thus be stim-word1/stim-word1_scans.tsv...
with aforementioned #760 in mind, I wonder if with this "stimuli BEP" we could indeed be the first to generalize that into samples (from scans) or some other good generic term?

But then I see the point of having top level stims.tsv analogous to participants.tsv to describe common high level attributes for each stimuli <label> such as

stimulus_id  language word_class  ...
word1          english     noun

[effigies]

what entities meaning you see needing much of adjustment? Even for run I feel we would not need much of adjustment although some might already be a bit overdue: filed #760 . So not sure if we really need to introduce _item in favor of _run just yet

It feels like shoehorning an experimental notion into a corpus description. I would rather step back and think about what would make a good corpus standard with minimal reference to BIDS.

Maybe if you're thinking of the generation of the stimuli as a procedure that is repeated multiple times, run works. But perhaps I'm sampling from a larger corpus where the notion doesn't apply (e.g., going back through BBC archives for different pronunciations of words).

An alternative (or addition) to desc- could be a stims.tsv

+1 on that. Additional thoughts:
If we are to retain scans as a term, and aim for "BIDS" (not just "BIDS-like") then such a file would be analogous to _scans.tsv we already have, thus be stim-word1/stim-word1_scans.tsv...
with aforementioned #760 in mind, I wonder if with this "stimuli BEP" we could indeed be the first to generalize that into samples (from scans) or some other good generic term?

But then I see the point of having top level stims.tsv analogous to participants.tsv to describe common high level attributes for each stimuli <label> such as

stimulus_id  language word_class  ...
word1          english     noun

Yeah stims.tsv and samples.tsv makes sense to me.

[Remi-Gau]

Quick thought to point out what @sappelhoff mentioned regarding subject specific stimuli here: #750 (comment)

I don't think that it will such a rare case and we should probably give that some thought.

If it is just a matter of a raw stimulus being adapted to each participant, this could be treated as derivatives but having a way to describe the subject the stimulus is for would be a good thing.

Use the desc label to do that?

stimuli/
  dataset_description.json
  stim-<label>/
    stim-<label>[_desc-<label>][_item-<index>].<ext>

Reuse the sub entity?

stimuli/
  dataset_description.json
  stim-<label>/
    stim-<label>[_sub-<label>][_desc-<label>][_item-<index>].<ext>

[Remi-Gau]

Also does it make sense to have "prefix" or is it really shoehorning too much BIDS into this?

[effigies]

RE: sub entities, is the stimulus truly related to the subject, or is it that each subject gets a different stimulus? For a stimulus dataset that needs to be able to be understood in isolation, I'm wary of infecting with a separate notion. For example, maybe I created the stimuli for the subjects in a particular study, but then I want to perform a second study with the same stimuli, and the stim-movie_sub-01.mp4 no longer is viewed by sub-01 in my new study. Or maybe it's viewed by sub-01 and sub-38.

I would suggest that this would be a good use case for item. If sub-01 watches stim-movie_item-01.mp4, sub-02 watches stim-movie_item-02.mp4, and so on, then there's a straightforward mapping, but it is not confusing if it doesn't apply when the same stimulus set is used in a different study.

Also does it make sense to have "prefix" or is it really shoehorning too much BIDS into this?

I don't really understand this question. Could you clarify?

[Remi-Gau]

RE: sub entities, is the stimulus truly related to the subject, or is it that each subject gets a different stimulus?

Yes but...

For a stimulus dataset that needs to be able to be understood in isolation, I'm wary of infecting with a separate notion.

I will get specific to better explain.

So the case I have in mind the stimuli are literally made for each participant: participants are presented with sounds played from different locations, the sounds are recorded with microphones placed next to their ears so that the sound can replayed to them in the scanner as if they were listening to sound coming from that very specific location. Each person has their own "head related transfer function" that filters the sound in a given way, so each participant has their own set of sounds.

This is very much related to a given dataset so in most cases it won't work in isolation from the data.

But even if you "ship" the stimuli with the BIDS dataset I am wondering if it would make sense to worry about this to the level of having an entity that "pairs" a stimulus to a subject. Sort of thinking this is in the 20% of our pareto principle.

[tyarkoni]

My two cents: this feels to me like way more trouble than it's worth. Just give each stimulus a unique stim and/or item label, and then you can map between stimuli and subjects using a .tsv file that maps between them, or by adding a sidecar to each stimulus that indicates which subject they're for. Otherwise it gets very messy because everywhere else in BIDS that sub occurs, it's mandatory. I would honestly even consider sticking with just stim in the filename and doing everything else with a stims.tsv file. The space of potential stimuli and their applications seems to me too wide to plausibly encode in a meaningful way in filenames.

I also think this (i.e., stimulus naming/encoding) is a big and important enough problem it could easily be spun off into its own non-BIDS spec, and just be wrapped later by BIDS.

[effigies]

I also think this (i.e., stimulus naming/encoding) is a big and important enough problem it could easily be spun off into its own non-BIDS spec, and just be wrapped later by BIDS.

ReCorDS (Research Corpus Data Structure)?

[Remi-Gau]

I don't really understand this question. Could you clarify?

as many files in BIDS are of the form [entity1-<label>][_entityX-<label>]*_<suffix>.<ext> I was just thinking if having a suffix in there made any sense.

[Remi-Gau]

My two cents: this feels to me like way more trouble than it's worth. Just give each stimulus a unique stim and/or item label, and then you can map between stimuli and subjects using a .tsv file that maps between them, or by adding a sidecar to each stimulus that indicates which subject they're for.

Makes sense: when I wrote my last reply, I started thinking of an optional "intended for" or something equivalent instead of an entity

I also think this (i.e., stimulus naming/encoding) is a big and important enough problem it could easily be spun off into its own non-BIDS spec, and just be wrapped later by BIDS.

Agreed. That is definitely something where I would like to hear the opinion of the psych-DS folks for example.