Doc and cleaning the dcm2niix json example

Author: bclenet

examples/dcm2niix/README.md

@@ -3,6 +3,7 @@
 This example aims at showing the capture of provenance for a [`dcm2niix`](https://github.com/rordenlab/dcm2niix) usecase: converting DICOM data to Nifti files within a BIDS dataset.
 
 Source data for this example can be found here: https://github.com/psychoinformatics-de/hirni-demo. This is a datalad dataset containing anatomical and functional MRI acquisitions. The contents of this dataset can be downloaded using:
+
 ```shell
 mkdir sourcedata
 cd sourcedata
@@ -11,67 +12,95 @@ datalad install --recursive https://github.com/psychoinformatics-de/hirni-demo.g
 
 > [!NOTE] Note that the dataset must be added inside the `sourcedata/` directory.
 
-## Experiment #1
+## Purpose
+
+The aim of the example is to describe the provenance records using BIDS Prov, but inside several *JSON* files.
+We use:
+
+* the `GeneratedBy` field of JSON sidecars, already existing in the BIDS specification
+* modality agnostic files inside the `prov/` directory
+
+as follows:
 
-The aim of the experiment is to describe the provenance records inside several files.
-Here we use sidecars and modality agnostic files inside the `prov/` directory, as follows:
 ```
 .
 ├── prov
-│   ├── prov-dcm2niix_.prov.json
-│   ├── prov-dcm2niix_env.prov.json
-│   └── prov-dcm2niix_soft.prov.json
+│   ├── base.prov.json
+│   ├── prov-dcm2niix_ses-01_env.prov.json
+│   ├── prov-dcm2niix_ses-01_soft.prov.json
+│   ├── prov-dcm2niix_ses-02_env.prov.json
+│   └── prov-dcm2niix_ses-02_soft.prov.json
 └── sub_02
     ├── ses_20130717141500
     │   └── anat
-    │       └── sub-02_ses-20130717141500_T1w.prov.jsonld
+    │       └── sub-02_ses-20130717141500_T1w.json
     └── ses_20140425155335
         └── func
-            └── sub-02_ses-20140425155335_task-oneback_run-1_bold.prov.jsonld
+            └── sub-02_ses-20140425155335_task-oneback_run-1_bold.json
 ```
 
+## New features for BIDS / BIDS Prov
+
 We introduce the following BIDS suffixes that are currently not existing:
-* `soft`:
-* `env`:
+* `soft`: the file describes BIDS Prov `Software` for the provenance traces
+* `env`: the file describes BIDS Prov `Environments` for the provenance traces
 
 We introduce the following BIDS entity that is currently not existing:
 * `prov`
     * Full name: Provenance traces
     * Format: `prov-<label>`
     * Definition: A grouping of provenance traces. Defining multiple provenance traces groups is appropriate when several processings have been performed on data.
 
-* `sub-02_ses-20130717141500_T1w.prov.jsonld` and `sub-02_ses-20140425155335_task-oneback_run-1_bold.prov.jsonld` are sidecars defining provenance for the corresponding `.nii` files.
-* `environments.prov.jsonld` mutualises the declaration of software environments objects for lower level prov files
-* `software.prov.jsonld` mutualises the declaration of software pieces objects for lower level prov files
+We also use the `ses` entity for provenance files, in order to identify several groups of processes that happened at different times. In the example, `ses-01` and `ses-02` represent two different dcm2niix conversion sessions, with different versions of dcm2niix and environments.
+
+We use the `GeneratedBy` field of JSON sidecars to describe `Activities` that created the file the sidecars refers to.
+
+The `prov/base.prov.jsonld` contains an empty BIDS Prov JSON-LD graph as follows:
+```JSON-LD
+{
+  "@context": "https://purl.org/nidash/bidsprov/context.json",
+  "BIDSProvVersion": "0.0.1",
+  "Records": {
+    "Software": [],
+    "Activities": [],
+    "Entities": []
+  }
+}
+```
+
+## Merging JSON in a JSON-LD file and plotting graph
 
-The python script `code/exp1_merge_prov.py` aims at merging all these provenance records into one RFD graph.
+The python script `code/merge_prov.py` aims at merging all these provenance records into one RFD graph.
 
 ```shell
-pip install -r code/requirements.txt
-mkdir prov/experiment_1/
-python code/exp1_merge_prov.py
+mkdir prov/merged/
+python code/merge_prov.py
 ```
 
-From that, we generate the RDF graph `prov/experiment_1/merged_provenance.ttl`. Then we were able to plot the graph as a png file. We used this command:
+From that, we generate the JSON-LD graph `prov/merge/prov-dcm2niix.prov.jsonld`. Then we were able to plot the graph as a png file. We used this command:
 
 ```shell
-python code/plot_graph.py -i prov/experiment_1/merged_provenance_2.ttl -o prov/experiment_1/merged_provenance_2.png
+python ../../bids_prov/visualize.py --input_file prov/merged/prov-dcm2niix.prov.jsonld --output_file prov/merged/prov-dcm2niix.prov.png
 ```
 
-![](/examples/dcm2niix/prov/experiment_1/merged_provenance.png)
+![](/examples/dcm2niix/prov/merged/prov-dcm2niix.prov.png)
 
 ### Notes
 
-In this example, we rely on the fact that nodes defined in the `prov/*.prov.jsonld` files have `bids::prov/` as base IRIs. Here are the involved nodes:
-* `bids::prov/dcm2niix`
-* `bids::prov/fedora`
+In this example, we rely on the fact that nodes defined in the `prov/*.prov.jsonld` files have `bids::prov/` as base IRIs.
+
+The `code/merge_prov.py` code is responsible for:
+* merging the JSON provenance traces into the base JSON-LD graph;
+* create an `Entity` and linking it to the `Activity` described by the `GeneratedBy` field in the case of JSON sidecars.
 
 ### Limitations
 
-The `bids::prov/fedora` node defined in `prov/environments.prov.jsonld` (see grey node in the above graph plot) is not recognized as a `prov:Entity` as the current context (commit [ce0eb77](https://github.com/bids-standard/BEP028_BIDSprov/commit/ce0eb774abd9527e594bd69212a87d5047864678)) does not define the `Environments` term.
+1. The `Environments` term is not defined in the current BIDS Prov context, hence we define environments as `Entities`.
 
-Listing all the DICOM files used by the dcm2niix conversion steps would lower readability of the JSON-LD provenance files. Therefore we only listed the following directories as `Entities`:
+2. Listing all the DICOM files used by the dcm2niix conversion steps would lower readability of the JSON-LD provenance files. Therefore we only listed the following directories as `Entities`:
 * `bids::sourcedata/hirni-demo/acq1/dicoms/example-dicom-structural-master/dicoms/`
 * `bids::sourcedata/hirni-demo/acq2/dicoms/example-dicom-functional-master/dicoms/`
 
 although it is not allowed by the current version of the BIDS Prov specification to have directories as `Entities`.
+
+3. In this example, the provenance for the JSON sidecar files is not described.

examples/dcm2niix/code/exp1_merge_prov.py renamed to examples/dcm2niix/code/merge_prov.py

@@ -4,7 +4,6 @@
 """ Merge available prov JSON files into one RDF graph """
 
 import json
-from pyld import jsonld
 
 # List of available prov files
 prov_soft_files = [
@@ -55,5 +54,5 @@
 				})
 
 # Write jsonld
-with open('prov/experiment_1/merged_provenance.prov.jsonld', 'w', encoding = 'utf-8') as file:
+with open('prov/merged/prov-dcm2niix.prov.jsonld', 'w', encoding = 'utf-8') as file:
 	file.write(json.dumps(base_provenance, indent = 2))