Quality ratings in BIDS

[dorianps]

Hello everyone.

We plan to put manual quality control ratings in the BIDS dataset. Currently, we need it for the Freesurfer derived data, but this can be thought of for any type of BIDS folder. We have devised an ad hoc .json structure to do this, going from general to specific and including info of the rater name, duration, time, files, etc. We place this json file in each Freesurfer datafolder.

Before we go ahead with this process with our arbitrary choices, I wanted to ask whether there is any BIDS directive on how to integrate manual quality inspections in the BIDS dataset. Has there been any discussion about this?

We are happy to share our thoughts on how we have thought QC ratings should be integrated in BIDS, and what json structure we have planned to follow.

[raamana]

As far as I am aware, there is not any standard yet. It would be good to start thinking about, as we also perform a lot of FS QC, esp. via VisualQC, which outputs a CSV file at this point containing the ID, rating and user notes.

CSV was chosen for ease of reuse and portability - JSON would work just as well.

[effigies]

We are happy to share our thoughts on how we have thought QC ratings should be integrated in BIDS, and what json structure we have planned to follow.

That sounds like a good idea. Once a consensus starts to emerge, you can either propose a PR or start a BEP to workshop it as a document.

This would be a derivative, so it would need to wait on (and be consistent with) #265.

[dorianps]

Ok, we will post here an example QC json file. For now, I just wanted to make sure our work is compliant with BIDS thoughts.

From what @effigies says looks like this needs to be considered a derivative. This means that at least for raw data this info needs to be in a separate folder, e.g., called QC. I guess this makes sense. For Freesurfer though, I had envisioned to put the json file directly on the Freesurfer output folder itself. Do you advise to rather create another derived folder, something like Freesurfer_QC, or perhaps just keep a generic Manual_QC derived folder where everything goes, from QC of raw data to QC of Freesurfer, fmriprep output, etc.,? I find it complicated to keep things separate, that is why I think QC can go within each folder containing data as a single json file regarding QC of the data in that folder/derivation.

[effigies]

Freesurfer output would also be a derivative, so at least from the bids perspective there would not be problems in mixing them. There might be provenance and communication benefits to separating it though.

[oesteban]

Related discussion: https://groups.google.com/d/msg/bids-discussion/21NOkxDSUqE/7zVWvwQeAAAJ

[dorianps]

@oesteban thanks for the link. Looks like the discussion there was more conceptual than practical.

We have created a json structure to store our ratings. Looks like this for a manual rating on Freesurfer segmentations:

{
  "SchemeName": "FS_seg_SNAP",
  "RatedFolder": "derivatives/freesurfer/sub-A012345/sub-A012345_ses-04",
  "Files": [
    "mri/aparc.DKTatlas+aseg.nii.gz",
    "mri/norm.nii.gz"
  ],
  "CommandLine": "itksnap -g mri/norm.nii.gz -s mri/aparc.DKTatlas+aseg.nii.gz -l ../../snapdkt.txt",
  "Ratings": [
    {
      "RaterName": "DP",
      "RepeatedRatings": [
        {
          "RatingNumber": 1,
          "QCVal": 4,
          "Comments": "Claustrum overinclusion in putamen - ",
          "RatingDate": "2019-02-12",
          "RatingTime": "22:41:23-05:00",
          "RatingDuration": 13.4279119,
          "RatingDurationUnits": "seconds"
        }
      ]
    }
  ]
}

Here are a few important concepts we considered.

1. Ratings should go by scheme. I.e. Freesurfer data can be QC-ed for many things, and the protocol for rating something can be considered a scheme.
2. Each scheme should be at the root of the json data structure.
3. Each scheme can have both multiple raters, and multiple ratings from the same rater.
4. The files used for rating and the command call is stored in the json structure to know exactly what was rated and how.
5. The scheme can be extended to automated QC measures. For example, one could add a field "SchemeType: manual/automatic" to distinguish manual ratings from MRIQC score-based QC. And maybe we should add for each scheme allowable fields (i.e., 1-5 for a scheme, 0-1 for another scheme, or flexible text field for yet another one).
6. There was discussion above where should these QC files sit. In our opinion they should sit within the derivated data being QC-ed. We don't see yet the point of having QC as another derivated folder for two reasons:

• Because then you need to sync the QC derivated folder with the data it QCs in a dataset (e.g., you can't delete a Freesurfer folder leaving its QC results sitting elsewhere orphan). Same is true if you want to delete a single sessions or subject, there is no point in trying to find where else data related to that session are sitting.
• If we don't keep the QC file inside the result for each, then we need a full mirror of the derivated folder structure in the QC folder, which makes no sense.

Thus, in our current view, QC data should be flexible and should be considered a property of the data it refers to, not just another derivated data folder.

P.s. We are not experts of BIDS standards or heavy json users. The above solution is what we think can achieve long term usability/flexibility. If you see any incompatibility or mismatch with bids standards, please let us know.