Dcm2niix example : allowing BIDS validation of provenance records

examples/dcm2niix_5/README.md

@@ -0,0 +1,88 @@
+# BIDS Prov example #5 for `dcm2niix`
+
+This example aims at showing provenance traces from a DICOM to Nifti conversion, performed by `dcm2niix` on a Linux-based (Fedora) operating system.
+
+## Overview
+
+This time, we describe the provenance records using BIDS Prov, but inside several *JSON* files.
+
+Although this example generates only one file, it illustrates how to store provenance records in the case where the dcm2niix `Conversion` Activity is responsible for several conversions.
+
+After conversion, and adding provenance traces, the resulting directory tree looks like this:
+
+```
+prov/
+├── prov-dcm2niix_act.json
+├── prov-dcm2niix_ent.json
+├── prov-dcm2niix_env.json
+└── prov-dcm2niix_soft.json
+sourcedata/
+sub-02/
+└── anat
+    ├── sub-02_T1w.json
+    └── sub-02_T1w.nii
+```
+
+Note that the `sourcedata/` directory contains the source dataset described in the [section hereafter](#source-dataset).
+
+We use:
+
+* the `GeneratedBy` field of JSON sidecar, already existing in the BIDS specification
+* modality agnostic files inside the `prov/` directory
+
+Note that we assume that both `sub-02_T1w.nii` and its JSON sidecar `sub-02_T1w.json` were generated by the same Activity.
+
+## Provenance merge
+
+The python script `code/merge_prov.py` aims at merging all provenance records into one JSON-LD graph.
+
+```shell
+pip install bids-prov==0.1.0
+mkdir prov/merged/
+python code/merge_prov.py
+```
+
+## Provenance visualization
+
+We are then able to visualize these provenance files using the following commands (current directory is `examples/dcm2niix_3/`):
+
+```shell
+pip install bids-prov==0.1.0
+bids_prov_visualizer --input_file prov/merged/prov-dcm2niix.jsonld --output_file prov/merged/prov-dcm2niix.png
+```
+
+![](/examples/dcm2niix_3/prov/merged/prov-dcm2niix.png)
+
+## Source dataset
+
+Dataset is the same as the one for [example dcm2niix_1](/BEP028_BIDSprov/examples/dcm2niix_1/README.md#source-dataset).
+
+## Notes
+
+We introduce the following BIDS entity that is currently not existing:
+* `prov`
+    * Full name: Provenance traces
+    * Format: `prov-<label>`
+    * Definition: A grouping of provenance traces. Defining multiple provenance traces groups is appropriate when several processings have been performed on data.
+
+We introduce the following BIDS suffixes that are currently not existing:
+* `act`: the file describes BIDS Prov `Activities` for the group of provenance traces
+* `soft`: the file describes BIDS Prov `Software` for the group of provenance traces
+* `env`: the file describes BIDS Prov `Environments` for the group of provenance traces
+
+We use the `GeneratedBy` field of JSON sidecars to link to `Activities` that created the file the sidecars refers to.
+
+In this example, we rely on the fact that nodes defined in the `prov/*.jsonld` files have `bids::prov/` as base IRIs.
+
+The `code/merge_prov.py` code is responsible for:
+* merging the JSON provenance traces into the base JSON-LD graph;
+* create an `Entity` and linking it to the `Activity` described by the `GeneratedBy` field in the case of JSON sidecars.
+
+### Limitations
+
+1. The `Environments` term is not defined in the current BIDS Prov context, hence we define environments as `Entities`.
+
+2. Listing all the DICOM files used by the dcm2niix conversion steps would lower readability of the JSON-LD provenance files. Therefore we only listed the following directories as `Entities`:
+* `bids::sourcedata/hirni-demo/acq1/dicoms/example-dicom-structural-master/dicoms`
+
+although it is not allowed by the current version of the BIDS Prov specification to have directories as `Entities`.

examples/dcm2niix_5/code/merge_prov.py

@@ -0,0 +1,114 @@
+#!/usr/bin/python
+# coding: utf-8
+
+""" Merge available prov JSON files into one RDF graph """
+
+import json
+from pathlib import Path
+
+# List of available prov files
+prov_soft_files = [
+  'prov/prov-dcm2niix_soft.json'
+]
+prov_env_files = [
+  'prov/prov-dcm2niix_env.json'
+]
+prov_act_files = [
+  'prov/prov-dcm2niix_act.json'
+]
+prov_ent_files = [
+  'prov/prov-dcm2niix_ent.json'
+]
+sidecar_files = [
+  'sub-02/anat/sub-02_T1w.json'
+]
+
+# Base jsonld
+base_provenance = {
+  "Records": {
+    "Software": [],
+    "Activities": [],
+    "Entities": []
+  }
+}
+
+# Add context and version (in this example, we suppose that context and version are known by BIDS)
+base_provenance["BIDSProvVersion"] = "0.0.1"
+base_provenance["@context"] = "https://purl.org/nidash/bidsprov/context.json"
+
+# Parse Software
+for prov_file in prov_soft_files:
+  with open(prov_file, encoding = 'utf-8') as file:
+    data = json.load(file)
+    base_provenance['Records']['Software'] += data['Software']
+
+# Parse Environments
+for prov_file in prov_env_files:
+  with open(prov_file, encoding = 'utf-8') as file:
+    data = json.load(file)
+    # /!\ Workaround: environments are added in the Entities list because
+    # the Environments term is not defined in the BIDS Prov context yet
+    base_provenance['Records']['Entities'] += data['Environments']
+
+# Parse Entities
+for prov_file in prov_ent_files:
+  with open(prov_file, encoding = 'utf-8') as file:
+    data = json.load(file)
+    base_provenance['Records']['Entities'] += data['Entities']
+
+# Parse Activities
+for prov_file in prov_act_files:
+  with open(prov_file, encoding = 'utf-8') as file:
+    data = json.load(file)
+    base_provenance['Records']['Activities'] += data['Activities']
+
+# Parse Sidecar files
+for sidecar_file in sidecar_files:
+  # Identify data file(s) associated with the sidecar
+  sidecar_filename = Path(sidecar_file)
+  data_files = Path('').glob(f'{sidecar_filename.with_suffix("")}.*')
+  data_files = [str(f) for f in list(data_files) if str(sidecar_filename) not in str(f)]
+
+  # Write provenance
+  with open(sidecar_file, encoding = 'utf-8') as file:
+    data = json.load(file)
+    if 'GeneratedBy' in data:
+
+      # Get activity data and id
+      activity_data = data['GeneratedBy']
+      activity_id = ""
+      if "Id" in activity_data:
+        activity_id = activity_data['Id']
+      else:
+        activity_id = activity_data
+
+      # Provenance for the data file
+      for data_file in data_files:
+        base_provenance['Records']['Entities'].append(
+          {
+            "Id": f"bids::{data_file}",
+            "GeneratedBy": activity_data
+          }
+        )
+
+    if 'SidecarGeneratedBy' in data:
+
+      # Get activity data and id
+      activity_data = data['SidecarGeneratedBy']
+      activity_id = ""
+      if "Id" in activity_data:
+        activity_id = activity_data['Id']
+      else:
+        activity_id = activity_data
+
+      # Provenance for the sidecar
+      base_provenance['Records']['Entities'].append(
+        {
+          "Id": f"bids::{sidecar_filename}",
+          "GeneratedBy": activity_id
+        }
+      )
+
+# Write jsonld
+with open('prov/merged/prov-dcm2niix.jsonld', 'w', encoding = 'utf-8') as file:
+  file.write(json.dumps(base_provenance, indent = 2))

examples/dcm2niix_5/prov/merged/prov-dcm2niix.jsonld

@@ -0,0 +1,45 @@
+{
+  "Records": {
+    "Software": [
+      {
+        "Id": "bids::prov/#dcm2niix-khhkm7u1",
+        "Label": "dcm2niix",
+        "Version": "v1.0.20220720"
+      }
+    ],
+    "Activities": [
+      {
+        "Id": "bids::prov/#conversion-00f3a18f",
+        "Label": "Conversion",
+        "Command": "dcm2niix -o . -f sub-%i/anat/sub-%i_T1w sourcedata/hirni-demo/acq1/dicoms/example-dicom-structural-master/dicoms",
+        "AssociatedWith": "bids::prov/#dcm2niix-khhkm7u1",
+        "Used": [
+          "bids::prov/#fedora-uldfv058",
+          "bids::sourcedata/hirni-demo/acq1/dicoms/example-dicom-structural-master/dicoms"
+        ]
+      }
+    ],
+    "Entities": [
+      {
+        "Id": "bids::prov/#fedora-uldfv058",
+        "Label": "Fedora release 36 (Thirty Six)",
+        "OperatingSystem": "GNU/Linux 6.2.15-100.fc36.x86_64"
+      },
+      {
+        "Id": "bids::sourcedata/hirni-demo/acq1/dicoms/example-dicom-structural-master/dicoms",
+        "Type": "Entity",
+        "Label": "dicoms"
+      },
+      {
+        "Id": "bids::sub-02/anat/sub-02_T1w.nii",
+        "GeneratedBy": "bids::prov/#conversion-00f3a18f"
+      },
+      {
+        "Id": "bids::sub-02/anat/sub-02_T1w.json",
+        "GeneratedBy": "bids::prov/#conversion-00f3a18f"
+      }
+    ]
+  },
+  "BIDSProvVersion": "0.0.1",
+  "@context": "https://purl.org/nidash/bidsprov/context.json"
+}

examples/dcm2niix_5/prov/prov-dcm2niix_act.json

@@ -0,0 +1,14 @@
+{
+  "Activities": [
+    {
+      "Id": "bids::prov/#conversion-00f3a18f",
+      "Label": "Conversion",
+      "Command": "dcm2niix -o . -f sub-%i/anat/sub-%i_T1w sourcedata/hirni-demo/acq1/dicoms/example-dicom-structural-master/dicoms",
+      "AssociatedWith": "bids::prov/#dcm2niix-khhkm7u1",
+      "Used": [
+        "bids::prov/#fedora-uldfv058",
+        "bids::sourcedata/hirni-demo/acq1/dicoms/example-dicom-structural-master/dicoms"
+      ]
+    }
+  ]
+}

examples/dcm2niix_5/prov/prov-dcm2niix_ent.json

@@ -0,0 +1,9 @@
+{
+  "Entities": [
+    {
+      "Id": "bids::sourcedata/hirni-demo/acq1/dicoms/example-dicom-structural-master/dicoms",
+      "Type": "Entity",
+      "Label": "dicoms"
+    }
+  ]
+}

examples/dcm2niix_5/prov/prov-dcm2niix_env.json

@@ -0,0 +1,9 @@
+{
+  "Environments": [
+    {
+      "Id": "bids::prov/#fedora-uldfv058",
+      "Label": "Fedora release 36 (Thirty Six)",
+      "OperatingSystem": "GNU/Linux 6.2.15-100.fc36.x86_64"
+    }
+  ]
+}

examples/dcm2niix_5/prov/prov-dcm2niix_soft.json

@@ -0,0 +1,9 @@
+{
+  "Software": [
+    {
+      "Id": "bids::prov/#dcm2niix-khhkm7u1",
+      "Label": "dcm2niix",
+      "Version": "v1.0.20220720"
+    }
+  ]
+}

examples/dcm2niix_5/sub-02/anat/sub-02_T1w.json

@@ -0,0 +1,25 @@
+{
+    "Modality": "MR",
+    "ManufacturersModelName": "nifti2dicom",
+    "SoftwareVersions": "0.4.11",
+    "SeriesDescription": "anat-T1w",
+    "ProtocolName": "anat-T1w",
+    "ImageType": ["DERIVED", "SECONDARY"],
+    "RawImage": false,
+    "SeriesNumber": 401,
+    "AcquisitionTime": "13:25:18.000000",
+    "AcquisitionNumber": 1,
+    "SliceThickness": 0.666667,
+    "SpacingBetweenSlices": 0.666667,
+    "ImageOrientationPatientDICOM": [
+        0.999032,
+        -0.0217884,
+        0.0382096,
+        0.0265195,
+        0.991414,
+        -0.128044   ],
+    "ConversionSoftware": "dcm2niix",
+    "ConversionSoftwareVersion": "v1.0.20220720",
+    "GeneratedBy": "bids::prov/#conversion-00f3a18f",
+    "SidecarGeneratedBy": "bids::prov/#conversion-00f3a18f"
+}

examples/from_parsers/context_2025_04_09_13h50m02s.txt → examples/from_parsers/context_2025_05_21_07h43m45s.txt

@@ -1,4 +1,4 @@
-Date : 2025_04_09_13h50m02s
+Date : 2025_05_21_07h43m45s
 Processing files...
     file= nidmresults-examples/afni_alt_onesided_proc.sub_001
     file= nidmresults-examples/afni_alt_onesided_proc.sub_001
@@ -69,4 +69,4 @@ Processing files...
     file= nidmresults-examples/spm_thr_voxelfdrp05_batch.m
     file= nidmresults-examples/spm_thr_voxelfwep05_batch.m
     file= nidmresults-examples/spm_thr_voxelunct4_batch.m
-End of processed files. Results in dir : 'examples/from_parsers'. Time required: 0:00:02.120406
+End of processed files. Results in dir : 'examples/from_parsers'. Time required: 0:00:02.130006