[ENH] BEP036 - Phenotypic Data Guidelines #2123

ericearl · 2025-05-30T14:09:19Z

The BEP leads can meet as-needed to discuss this BEP PR

Coordinate a meeting by emailing Eric Earl: [email protected].

Communicate on this PR to provide feedback otherwise.

BEP036 brings guidelines for best tabular phenotypic data to the BIDS specification.

Includes an appendix called phenotype.md
Includes a new AdditionalValidation key for the dataset_description.json, for which the usage is described in the modality agnostic files sections
Includes the new option to store session_id as the second column in the participants.tsv

Additional Links

Co-authored-by: Eric Earl [email protected] @ericearl
Co-authored-by: Samuel Guay [email protected] @SamGuay
Co-authored-by: Sebastian Urchs [email protected] @surchs
Co-authored-by: Arshitha Basavaraj [email protected] @Arshitha

Upstream PR

Quick update before merging our PR on surchs fork

BEP036 brings guidelines for best tabular phenotypic data to the BIDS specification. - Includes an appendix called `phenotype.md` - Includes admonitions for the guidelines in-line with modality agnostic files sections --------- Co-authored-by: Eric Earl <[email protected]> Co-authored-by: Samuel Guay <[email protected]> Co-authored-by: Sebastian Urchs <[email protected]> Co-authored-by: Arshitha B <[email protected]>

Changed "e.g." to "for example" to follow contributing style guidelines.

for more information, see https://pre-commit.ci

src/modality-agnostic-files/data-summary-files.md

surchs · 2025-05-30T14:57:31Z

src/appendices/phenotype.md

+each `phenotype/<measurement_tool_name>.json` data dictionary.
+This improves reusability and provides clarity about the measurement tool.
+
+### 5. Use the demographics file for common variables about participants


Copying from https://github.com/surchs/bids-specification/pull/1/files#r2103117486

For this section, would it make sense to suggest that demo-like information be prioritized in this file rather than participants.tsv, making the latter primarily a list of subject IDs? I haven't seen this explicitly addressed anywhere, though I'm unsure if it's something we want to formalize 😬
Something like this could follow the paragraph?:

When all demographic data is stored in phenotype/demographics.tsv, participants.tsv may serve primarily as a minimal listing of subject identifiers with only the participant_id column.

I agree. It'd be good to mention this.

src/appendices/phenotype.md

src/modality-agnostic-files/data-summary-files.md

Put the phenotypic and assessment data content where it belongs.

src/modality-agnostic-files/data-summary-files.md

src/modality-agnostic-files/phenotypic-and-assessment-data.md

src/appendices/phenotype.md

- Added in a new guideline 7 to encourage the use of participants and sessions files for different uses. - Re-numbered old guidelines 7-9 to 8-10.

Removing excess line I forgot to remove earlier. Thanks remark CI!

surchs · 2025-09-24T22:20:14Z

.vscode/settings.json

I think this is added by accident in surchs@0eba71d @ericearl

Agreed. I will try to get it out of there.

src/modality-agnostic-files/phenotypic-and-assessment-data.md

src/appendices/phenotype.md

surchs · 2025-09-25T00:49:06Z

src/appendices/phenotype.md

+1.  Each row MUST start with `participant_id`.
+
+1.  Each TSV file MUST contain a `session_id` column when
+    multiple [sessions](../glossary.md#session-entities)[^1] are present
+    in the data set regardless of whether those sessions are in
+    the `phenotype/` data, `sub-<label>/` data, or a combination of the two.
+
+1.  If more than one of the same measurement tool is acquired within
+    the same `session_id`, a `run_id` column MUST be added.
+
+1.  To encode the acquisition time for a measurement tool’s `session_id`,
+    add the `session_id` to the sessions file and
+    include the OPTIONAL `acq_time` column.


Suggested change

1. Each row MUST start with `participant_id`.

1. Each TSV file MUST contain a `session_id` column when

multiple [sessions](../glossary.md#session-entities)[^1] are present

in the data set regardless of whether those sessions are in

the `phenotype/` data, `sub-<label>/` data, or a combination of the two.

1. If more than one of the same measurement tool is acquired within

the same `session_id`, a `run_id` column MUST be added.

1. To encode the acquisition time for a measurement tool’s `session_id`,

add the `session_id` to the sessions file and

include the OPTIONAL `acq_time` column.

a. Each row MUST start with `participant_id`.

b. Each TSV file MUST contain a `session_id` column when

multiple [sessions](../glossary.md#session-entities)[^1] are present

in the data set regardless of whether those sessions are in

the `phenotype/` data, `sub-<label>/` data, or a combination of the two.

c. If more than one of the same measurement tool is acquired within

the same `session_id`, a `run_id` column MUST be added.

d. To encode the acquisition time for a measurement tool’s `session_id`,

add the `session_id` to the sessions file and

include the OPTIONAL `acq_time` column.

Or just regular list? I wouldn't nest one top-level enumeration in another.

surchs · 2025-09-25T02:20:34Z

src/modality-agnostic-files/data-summary-files.md

+Optional: Yes
+
+An aggregated sessions file CAN be provided at the dataset root.


It's not really optional though, right? Or rather, it's only optional in the sense that you could chose the other option and make subject-level sessions.tsv files.

As mentioned above: I would take a stance here and make one of the two options the recommended one. To me that would be the root-level file. And then we can say a word on why we recommend the option.

surchs · 2025-09-25T02:23:00Z

src/modality-agnostic-files/data-summary-files.md

 ## Sessions file

-Template:
+### Option 1: Segregated sessions files


Can't comment on the main heading for Sessions file:

There is a lot of commonality b/w the root-level and subject-level sessions.tsv. i.e. everything about what kind of info should go in them and what they are for. So how about we pull that info up under the heading. And then only explain the differences in the two options sections

surchs · 2025-09-25T02:24:18Z

src/modality-agnostic-files/data-summary-files.md

+`sessions.json` example:
+
+```JSON
+{
+    "participant_id": {
+        "Description": "Participant identifier"
+    },
+    "session_id": {
+        "Description": "Session identifier for the session",
+        "Levels": {
+            "ses-predrug": "session before drug administration",
+            "ses-postdrug": "session after drug administration",
+            "ses-followup": "follow-up session"
+        }
+    },
+    "acq_time": {
+        "Description": "Acquisition time of the session"
+    },
+    "systolic_blood_pressure": {
+        "Description": "Systolic blood pressure measured at the beginning of the session in mmHg"
+    }
+}
+```


as mentioned above: this should go under the Sessions File heading - right now the example table of which columns to put in a sessions.tsv is listed under the "segregated" option, but the data dictionary under the "aggregated" option

src/modality-agnostic-files/data-summary-files.md

src/schema/objects/files.yaml

Accidental file.

Added in easily-agreeable suggestions in a batch. Co-authored-by: Sebastian Urchs <[email protected]>

for more information, see https://pre-commit.ci

src/schema/rules/tabular_data/modality_agnostic.yaml

@surchs

Attempt to address more of @surchs comments.

Thanks for catching that excess newline, remark!

Remove acq_time as a phenotype column recommendation/option, as it should go into the sessions file instead.

src/schema/objects/columns.yaml

Remove acq_time__phenotype from columns.yaml since it was removed from the rest of the schema.

Accept Sebastian's suggestion about the phrasing of guideline 8. Co-authored-by: Sebastian Urchs <[email protected]>

for more information, see https://pre-commit.ci

src/modality-agnostic-files/data-summary-files.md

Changing "subject-level" to "participant-level" in sessions files section.

To better differentiate demographic data from phenotypic data

Made changes to align with final feedback prior to community review.

ericearl · 2025-10-12T13:05:53Z

@effigies @rwblair Here is a blurb for the community review period to make announcements easier. If edits are needed, I will apply them directly to this comment before tomorrow.

Community Review: BEP036 - Phenotypic Data Guidelines

We are pleased to announce the community review period for BIDS Extension Proposal (BEP) 036!

BEP036 extends the BIDS standard to include an appendix with 10 tabular phenotypic data guidelines you can opt into for the BIDS validator. We have developed the extension to allow everyone to follow good practices in preparing their tabular phenotypic data. Additionally, this BEP introduces the ability to include session_id as a second column in participants files and to aggregate sessions files to the root-level, allowing you to store longitudinal tabular data about participants and sessions, respectively, inside those files.

The draft specification may be found at: https://bids-specification--2123.org.readthedocs.build/en/2123/
The proposed changes may be found at bids-specification pull request #2123.
Example datasets may be found under the titles pheno001 through pheno006 in the bids-examples pull request #465.

To view the file differences in either pull request, click the "Files changed" tab.

effigies · 2025-10-16T21:01:16Z

Encoding the acquisition time for a measurement tool’s session_id, is RECOMMENDED. This information MUST be stored in the sessions.tsv file at the root level of the dataset in the acq_time column.

This is logically equivalent to "the acq_time column MUST NOT appear in a phenotype TSV file", but it takes some thinking about to get there. The spec should just say that.

"if anyone uses sessions, everyone uses sessions."

This is extremely difficult to do without requiring a root-level /sessions.tsv to the exclusion of subject-level sub-<label>_sessions.tsv files. The reason is that sessions columns in phenotype are analyzed on their own. If we can depend on the presence or absence of sessions.tsv as an indication of whether there are any sessions in the dataset, then when we visit a phenotype file, we can check that length(columns.session_id) > 0 iff exists('/sessions.tsv'). Similarly when visiting a subject directory, we can check that length(subject.sessions.ses_dirs) > 0 iff exists('/sessions.tsv').

7. Use the sessions file at the root-level

If there is more than one session for any one participant, then it is RECOMMENDED to provide a sessions file at the dataset root. The sessions file MUST list all sessions for all subjects across imaging and tabular phenotypic data. The data dictionary JSON file’s session_id field MUST include Levels with the description of each session_id.

The bolded text is not doable in the current schema. This would need access to all the (subject, session) pairs in /sessions.tsv and in each phenotype file. I think it's tractable, but we will need to extend the validation context and implement those changes in the validator.

10. Respect participant privacy when recording acquisition times

When needed to preserve participant privacy, you SHOULD record relative acquisition times with respect to the earliest session. Relative session acquisition times MAY be listed as durations from the earliest session (baseline) in days, months, or years using the acq_time column.

Unvalidatable and ambiguous. I think this should just piggy-back off of common principles:

Dates can be shifted by a random number of days for privacy protection reasons. To distinguish real dates from shifted dates, is is RECOMMENDED to set shifted dates to the year 1925 or earlier. Note that some data formats do not support arbitrary recording dates. [...] For longitudinal studies dates MUST be shifted by the same number of days within each subject to maintain the interval information. For example: 1867-06-15T13:45:30

rwblair · 2025-10-16T21:14:42Z

src/appendices/phenotype.md

+
+Aggregate participant information across all sessions into one tabular TSV file per
+measurement or phenotypic assessment and store this file in the `/phenotype` directory.
+Demographic information is a special case and  MUST be aggregated


As of right now there are suggestions of what counts as demographic data, from a validation perspective this is hard to enforce without specific field names being listed in the schema. My interpretation is that these are then to become forbidden columns in any pheno/*.tsv? Are there any other demographic fields we'd like to enforce that on beyond sex age, gender, race, household_income?

rwblair · 2025-10-16T21:18:29Z

src/appendices/phenotype.md

+measurement or phenotypic assessment and store this file in the `/phenotype` directory.
+Demographic information is a special case and  MUST be aggregated
+in the `participants.tsv` file at the root level of the dataset.
+It is RECOMMENDED to use the `age` column in the `participants.tsv` file


Theoretically we could validate the appropriate age being used in each session based on the relative acq_times if present but I don't think its worth the effort. Maybe monotonically increasing age like schema.rules.checks.mri.VolumeTimingNotMonotonicallyIncreasing would be a compromise?

rwblair · 2025-10-16T21:26:24Z

src/appendices/phenotype.md

+
+### 3. Add `MeasurementToolMetadata` to each tabular phenotypic measurement tool
+
+Whenever possible, it is RECOMMENDED to add `MeasurementToolMetadata` to


Not an issue for this bep: In this and in the main phenotype article its implied that every tsv in the phenotype directory is a "Measurement Tool", but never explicitly stated that this is the only kind of tsv. Gave me pause when reviewing this, but it may be obvious to everyone else.

rwblair · 2025-10-16T21:35:53Z

src/appendices/phenotype.md

+-   If more than one of the same measurement tool is acquired within
+    the same `session_id`, a `run_id` column MUST be added.


Suggested change

- If more than one of the same measurement tool is acquired within

the same `session_id`, a `run_id` column MUST be added.

- If a measurement tool is acquired multiple times within a single session, a `run_id` column must be added to disambiguate the separate acquisitions.

Note: This MUST is implicitly enforced by the combined index columns for phenotype tsv, If multiple results are acquired for the same subject and session with no run_id column the index check will error out.

rwblair · 2025-10-16T21:39:37Z

src/appendices/phenotype.md

+-   Encoding the acquisition time for a measurement tool’s `session_id`,
+    is RECOMMENDED. This information MUST be stored in the `sessions.tsv`
+    file at the root level of the dataset in the `acq_time` column.


@effigies mentioned this as "This is logically equivalent to "the acq_time column MUST NOT appear in a phenotype TSV file", but it takes some thinking about to get there. The spec should just say that."

I agree with the explicit "MUST NOT", But it also goes a step further in enforcing a root level sessions.tsv.

rwblair · 2025-10-16T21:45:31Z

src/appendices/phenotype.md

+The combination of values in the `participant_id`, `session_id`, and `run_id` (if present)
+columns MUST be unique for the entire tabular file.
+
+### 5. Store demographic data in the participants file and instrument data in the phenotype directory


This is mentioned in ### 1. Aggregate data across sessions, moving the two closer together or combining them would be nice.

rwblair · 2025-10-16T21:53:54Z

src/appendices/phenotype.md

+Create one tabular file for each instrument
+in the phenotypic and assessment data directory.
+
+### 6. Record participant properties in the participants file and session properties in the sessions file


Phenotypes aren't properties of participants? ;)

https://bids-specification.readthedocs.io/en/stable/modality-agnostic-files/data-summary-files.html#sessions-file
For pathology states When different from healthy, pathology SHOULD be specified.

Should this entry be taken as overriding the main spec, and this field should go in participants.tsv instead?

rwblair · 2025-10-16T21:55:51Z

src/appendices/phenotype.md

+Properties of participants MAY include things like
+age, sex, race, or household income.
+Properties of sessions MAY include things like
+acquisition time, measurement device properties,


Like participant properties, any way to explicitly list what is session appropriate metadata in the schema will make enforcing these rules easier/allow for making strong requirement claims. There is much less consistency here, so it may not be feasible.

ericearl and others added 4 commits May 20, 2025 08:24

Merge pull request #2 from bids-standard/master

3cedc86

Upstream PR

Merge pull request #3 from bids-standard/master

11fbb47

Quick update before merging our PR on surchs fork

Update phenotype.md and data-summary-files.md

0a640e6

Changed "e.g." to "for example" to follow contributing style guidelines.

ericearl requested review from effigies and rwblair May 30, 2025 14:09

ericearl assigned surchs, ericearl and SamGuay May 30, 2025

ericearl requested review from DimitriPapadopoulos and erdalkaraca as code owners May 30, 2025 14:09

ericearl added enhancement New feature or request BEP phenotype labels May 30, 2025

[pre-commit.ci] auto fixes from pre-commit.com hooks

a19512b

for more information, see https://pre-commit.ci

effigies reviewed May 30, 2025

View reviewed changes

src/modality-agnostic-files/data-summary-files.md Show resolved Hide resolved

src/modality-agnostic-files/data-summary-files.md Outdated Show resolved Hide resolved

surchs reviewed May 30, 2025

View reviewed changes