BEP044: Within-stimuli conditions

[adelavega]

stim_file columns in event files allow users to specify which stimuli files are associated with an event onset:

stim_file | OPTIONAL. Represents the location of the stimulus file (image, video, sound etc.) presented at the given onset time. ...

However, what this does not allow for is the specification of sub-conditions that occur during a long-running stimulus.

For example, in ds001545 a video file is presented which spans the entirety of the run. However, within each run/video there are 6 distinct conditions.

For example:

onset	duration	trial_type	stim_file
6	90	Intact A	cond1_run-01.mp4
105	90	Scramble Fix C	cond1_run-01.mp4
204	90	Scramble Rnd B V1	cond1_run-01.mp4
303	90	Scramble Fix C	cond1_run-01.mp4
402	90	Intact A	cond1_run-01.mp4
501	90	Scramble Rnd B V2	cond1_run-01.mp4

IMO, the above example is invalid as the stim_file only has a single onset.
The following is an event file which has all the necessary information (note I'm having to guess when the onset of the stim_file is, it could actually be 0).

onset	duration	trial_type	stim_file
6	540	n/a	cond1_run-01.mp4
6	90	Intact A	n/a
105	90	Scramble Fix C	n/a
204	90	Scramble Rnd B V1	n/a
303	90	Scramble Fix C	n/a
402	90	Intact A	n/a
501	90	Scramble Rnd B V2	n/a

However, this is ambiguous as the conditions are only implied to occur during stimulus presentation due to the duration of the first row.

@tyarkoni suggests adding optional but strongly encouraged stim_onset and stim_offset columns. These would denote onsets within a stimulus.

[yarikoptic]

I would have made it

onset	duration	trial_type	stim_file
6	540	Movie starts	cond1_run-01.mp4
6	90	Intact A	cond1_run-01.mp4
105	90	Scramble Fix C	cond1_run-01.mp4
204	90	Scramble Rnd B V1	cond1_run-01.mp4
303	90	Scramble Fix C	cond1_run-01.mp4
402	90	Intact A	cond1_run-01.mp4
501	90	Scramble Rnd B V2	cond1_run-01.mp4

stim_onset/stim_offset - I guess could be added but would have redundant information which could be computed (and validated to not go beyond stimuli duration) from "Movie starts" for that stimuli and corresponding onset and duration. And we all know what happens when there is redundancy ;)

As for the hierarchical description of events -- isn't there https://bids-specification.readthedocs.io/en/latest/99-appendices/03-hed.html ? (never used it myself though)

[tyarkoni]

I'm not crazy about either of the solutions proposed above because, while both compliant with the current spec, neither one eliminates the fundamental ambiguity here, which is that you don't know which part of the clip is being presented. It also is kind of problematic from a BIDS-StatsModel standpoint, because it will cause almost all users to have to drop a Filter transformation into their model just to weed out the first row, since nobody is going to want that in their model.

The benefit of having optional stim_onsetand stim_offset columns is those would eliminate the ambiguity in question without making most model specifications more complex. What I don't like about this proposal is that the extra columns are essentially metadata—there's virtually no situation under which they would be treated like other non-mandatory columns (i.e., as containing design-relevant information).

The more I think about this, the more I lean towards maybe keeping the current approach and not codifying this at all in the _events.tsv files. Maybe the solution is to require a supplementary metadata file for the stimulus files that contains the onsets. I.e., cond1_run-01.mp4 would have to have a cond1_run-01.json file that has fields PresentationOnset and PresentationOffset. But even that isn't sufficient, because presentation onset/offset can vary not just by stimulus, but also by event...

Should we just say this is in the 20% (really more like 1%) and not worry about it?

[yarikoptic]

BTW, ... Do they actually would need to filter then out? Why don't you want them to model that entire "super" condition as well? If there are different movie cuts, you might want them explicitly in the model, even if only to absorb transition (if it visible) between different stimuli. If there is only one big one for the entire run - well, it will largely be your constant. If there design disbalance and stimuli files have subtle unique features to them (differently trimmed, color scheme, audio volume level), having them modeled might save us from one other possible retraction.

The only problem I see is if all the trials follow each other in such a way that model becomes degenerate if the whole stim file condition is present too.
So, overall, it might be specific design related.

The only cons is that may be those stimuli onset and duration are actually of interest to other tools, not just the linear model, so they would need to recompute them as well. But it shouldn't be too hard.

As for extra unused meta data - I would say the more the merrier. My main concern is the fear of it being redundant and this requiring "manual" recomputation if I find that eg I need to fix onset. Then I will forget and the stimuli onset value will no longer be valid

[adelavega]

I would agree this probably falls into the 1% as the majority of experiments don't have sub-conditions within a stimuli. And so in 90% of cases, the mention of a stimulus indicates a complete presentation, so this is such a rare situation its probably not worth putting in the spec itself.

I still think it might be worth clarifying that including a stimulus in stim_file does not necessary indicate that the stimulus is played from the beginning (which is what I thought on first read).

[satra]

i'm not sure this is 1%. in many standard experiments, there are sub conditions. for example in experiments that involve showing faces/objects there are often sub categories: emotions, types of objects, types of faces (human faces/animal faces). in fact the modified hariri task is a perfect example of this, and gets used by emotion/mood researchers a lot.

i don't think we should reinvent ontologies of stimuli (e.g., paradigms - https://www.ncbi.nlm.nih.gov/pmc/articles/PMC3682219/, audio - https://research.google.com/audioset/ontology/index.html, images - https://bioportal.bioontology.org/ontologies/BIM).

but provide a way where stimulus properties can be encoded appropriately.

The more I think about this, the more I lean towards maybe keeping the current approach and not codifying this at all in the _events.tsv files. Maybe the solution is to require a supplementary metadata file for the stimulus files that contains the onsets. I.e., cond1_run-01.mp4 would have to have a cond1_run-01.json file that has fields PresentationOnset and PresentationOffset.

i like the idea of a json going alongside a stimulus file, but this json should be able to reflect timed objects inside it.

[satra]

just to follow up:

1. so in case of the hariri task, trial_type can represent the most dominant trial_type (faces, objects for example, or for mood researchers neutral/angry/etc). then stimulus properties could somehow represent not only details like cropping/full frame, colorspace, etc., but also ontological objects like this is an image/video of a face.

2. so in case of a movie, events.tsv could simply say i showed this clip for 240s. the stimulus event should have a json file that can encode many different types of extracted events within the clip.

3. another option is to allow multiple events files, and any model has to refer to a specific event file (may be we allow composition of event files).

[tyarkoni]

@satra by "sub-conditions" here we're not talking about hierarchical organization, we're talking about a temporal subset of a single file. Codifying hierarchical structures is IMO not in scope, but in any case presents no particular challenge from an events.tsv perspective, because you can just put the filename for each event in the stim_file column, and the analyst is welcome to do whatever they want with that. The case we're talking about is where you have, say, an 8-minute movie file identified as the stim_file, but the presentation starts halfway through that clip. In such cases the analyst needs to have some way to know that the onset of the presentation isn't synced with the onset of the event. But this seems like an edge case (indeed, I'm pretty sure this is the first BIDS dataset we've run into where it's an issue), so the proposal is to just let it be.

[tyarkoni]

1. then stimulus properties could somehow represent not only details like cropping/full frame, colorspace, etc.

I think this is analogous to the movie example, but I still think it's an edge case. Situations where researchers dynamically crop images are likely to be pretty rare; in most cases, the cropping will have been done in advance, and what's in the stimuli/ folder will be what was presented to the subject.

I think a reasonable way to update the spec is to strongly encourage users to provide files in stimuli/ that are as close as possible to the ones participants actually experienced. That means temporally or spatially cropping movies and images if needed. But I agree with @adelavega that we should also explicitly say that there is no actual guarantee that the contents of stimuli map perfectly onto what participants experienced.

[satra]

@tyarkoni - sorry i misunderstood the within stimuli conditions, so please ignore the ontological variations (although see last paragraph below).

for movie, i'm thinking of things like commercial clips that are shown, and i'm sure that certain clips cannot be shared.

for movies as an example are you saying i can extract faces, then specific emotions on those faces, and then encode both face and face+emotion in the events file, kind of a redundant stimulus list. all possible events in trial_type and then the analyst figures out which trials are of interest? for many of our tasks, that would work pretty well.

[tyarkoni]

for movie, i'm thinking of things like commercial clips that are shown, and i'm sure that certain clips cannot be shared.

I don't know that we can do anything about this, short of asking people to provide a description of where/how to obtain stimuli that can't be publicly shared. I don't think it's worth trying to codify this—there's too much variability in what that procurement process could look like.

for movies as an example are you saying i can extract faces, then specific emotions on those faces, and then encode both face and face+emotion in the events file, kind of a redundant stimulus list.

Sure, you can create arbitrary columns in events.tsv that code anything you like. Aside from stim_file, you could add columns for face_id, face_gender, face_age, face_emotion_rater1, face_emotion_rater2, face_emotion_avg, and anything else you like. The expectation is that you then put descriptions of columns in the data dictionary in the JSON sidecar, though I believe this is non-mandatory right now.

[Remi-Gau]

This is an old one.

I wonder if HED tags can help with such issue. @VisLab do you have some opinion on this?

[VisLab]

As it turns out the HED Working Group has been discussing this very issue and some of our members will weigh in shortly with a concrete proposal --- @neuromechanist @dorahermes @tpatpa @dungscout96 @monique2208 @makeig

[dorahermes]

Yes, I agree that HED tags can come in useful here and probably tackle this issue. When working through an example it seems like this may be a relatively larger contribution with some added machine readable files in the /stimuli/ folder. When starting to work a visual images and movie example with @neuromechanist it seems that there would be a need for community input for review and other examples such as e.g. auditory, motor, electrical stimulation, etc as well. This seems to perhaps go to the scope of a potential BEP. Should we open a separate GitHub issue to discuss whether to open a BEP or continue here?

@neuromechanist could share a preliminary google doc (not BEP yet, just the examples we were working through) if that would help give an idea?

Tagging some people who previously contributed to this discussion for input: @adelavega @tyarkoni @yarikoptic @satra @Remi-Gau