Skip to content

Commit d2e49ea

Browse files
bclenetbclenet
authored
Merge pull request bids-standard#154 from bclenet/nipype_example
[Example] Simple nipype+FSL workflow
2 parents 9527c76 + 32740e0 commit d2e49ea

14 files changed

+360
-0
lines changed

‎examples/nipype/README.md‎

Lines changed: 96 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,96 @@
1+
# BIDS-Prov example for nipype
2+
3+
This example aims at showing provenance traces from a simple [nipype](https://nipype.readthedocs.io/en/latest/) workflow, performed on a container-based software environment.
4+
5+
## Workflow
6+
7+
The workflow code is inside `code/normalize.py` and performs:
8+
1. a brain extraction of a T1w anatomical file `sub-001/anat/sub-001_Tw.nii.gz`, using BET;
9+
2. a registration to MNI152 of the resulting file, using FLIRT;
10+
3. exporting relevant output files to a BIDS compliant name space.
11+
12+
See [hereafter](#running-the-workflow) for more details on how to run the workflow.
13+
14+
## Overview
15+
16+
In order to describe provenance records using BIDS Prov, we use:
17+
18+
* the `GeneratedBy` field of JSON sidecar files, already existing in the BIDS specification;
19+
* modality agnostic files inside the `derivatives/flirt/prov/` directory
20+
21+
After running the workflow and adding provenance traces, the resulting directory tree looks like this:
22+
23+
```
24+
.
25+
├── code
26+
│ └── normalize.py
27+
├── derivatives
28+
│ ├── bids_prov_workflow
29+
│ └── flirt
30+
│ ├── prov
31+
│ │ ├── prov-flirt_act.prov.json
32+
│ │ ├── prov-flirt_base.prov.json
33+
│ │ ├── prov-flirt_ent.prov.json
34+
│ │ ├── prov-flirt_env.prov.json
35+
│ │ └── prov-flirt_soft.prov.json
36+
│ └── sub-001
37+
│ └── anat
38+
│ ├── sub-001_space-mni152nlin2009casym_T1w_brain.json
39+
│ ├── sub-001_space-mni152nlin2009casym_T1w_brain.nii.gz
40+
│ ├── sub-001_T1w_brain.json
41+
│ └── sub-001_T1w_brain.nii.gz
42+
├── README.md
43+
└── sub-001
44+
└── anat
45+
└── sub-001_T1w.nii.gz
46+
47+
```
48+
49+
Note that the `derivatives/bids_prov_workflow/` directory is nipype's working directory for the workflow. Its contents are not exhaustively described by the provenance traces.
50+
51+
## Provenance merge
52+
53+
The python script `code/merge_prov.py` aims at merging all provenance records into one JSON-LD graph.
54+
55+
```shell
56+
pip install bids-prov==0.1.0
57+
mkdir derivatives/flirt/prov/merged/
58+
python code/merge_prov.py
59+
```
60+
61+
The `code/merge_prov.py` code is responsible for:
62+
* merging the JSON provenance traces into the base JSON-LD graph;
63+
* create an `Entity` and linking it to the `Activity` described by the `GeneratedBy` field in the case of JSON sidecars.
64+
65+
## Provenance visualization
66+
67+
We are then able to visualize these provenance files using the following commands (current directory is `examples/nipype/`):
68+
69+
```shell
70+
pip install bids-prov==0.1.0
71+
bids_prov_visualizer --input_file derivatives/flirt/prov/merged/prov-flirt.prov.jsonld --output_file derivatives/flirt/prov/merged/prov-flirt.prov.png
72+
```
73+
74+
![](/examples/nipype/derivatives/flirt/prov/merged/prov-flirt.prov.png)
75+
76+
## Running the workflow
77+
78+
We use of the `nipype/nipype:py38` docker image that contains both nipype and FSL.
79+
80+
Assuming we are inside the nipype example directory (`examples/nipype`)
81+
82+
```bash
83+
# Get the container and run the workflow
84+
docker pull nipype/nipype:py38
85+
docker run -u root -it --rm -v .:/work nipype/nipype:py38 python code/flirt.py
86+
```
87+
88+
## Limitations / open questions
89+
90+
1. We are not able yet to describe a file (Entity) that is not inside the bids dataset, and belongs to the software environment. Here the `MNI152_T1_1mm_brain.nii.gz` (MNI152 template) can be considered as a part of FSL ; we use the following IRI for it `"/usr/share/fsl/5.0/data/standard/MNI152_T1_1mm_brain.nii.gz"`. Ideally we would like to create a relation between this Entity and the Environment.
91+
92+
2. We use Nipype's `ExportFile` nodes to export the computed files to the location they belong to in the BIDS tree (if nothing is done, the computed files stay in the workflow working directory). These are Nipype nodes coded in python, and we are therefore not able to write a precise command line as attribute of the corresponding Activities.
93+
94+
3. We refer the software environment as `"bids::prov/#docker.io/nipype/nipype:py38-vavfao8v"` but this could be `docker.io/nipype/nipype:py38` ?
95+
96+
4. We may want to avoid describing temporary files from nipype's working directory (here `bids_prov_workflow/`). For example, we could use blank nodes instead of `"bids::derivatives/bids_prov_workflow/brain_extraction/sub-001_T1w_brain.nii.gz"`.

‎examples/nipype/code/flirt.py‎

Lines changed: 39 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,39 @@
1+
#!/usr/bin/python
2+
# coding: utf-8
3+
4+
"""
5+
A simple Nipype workflow performing brain extraction and normalisation
6+
of an anatomical file.
7+
"""
8+
from os.path import abspath
9+
10+
from nipype.pipeline.engine import Workflow, Node
11+
from nipype.interfaces.fsl import BET, FLIRT, Info
12+
from nipype.interfaces.io import ExportFile
13+
14+
# Create workflow
15+
workflow = Workflow(name='bids_prov_workflow')
16+
workflow.base_dir = abspath('derivatives/')
17+
18+
# Create nodes
19+
brain_extraction = Node(BET(), name = 'brain_extraction')
20+
brain_extraction.inputs.in_file = abspath('sub-001/anat/sub-001_T1w.nii.gz')
21+
22+
flirt = Node(FLIRT(), name = 'flirt')
23+
flirt.inputs.reference = Info.standard_image('MNI152_T1_1mm_brain.nii.gz')
24+
workflow.connect(brain_extraction, 'out_file', flirt, 'in_file')
25+
26+
export_brain = Node(ExportFile(), name = 'export_brain')
27+
export_brain.inputs.clobber = True
28+
export_brain.inputs.out_file = abspath(
29+
'derivatives/flirt/sub-001/anat/sub-001_T1w_brain.nii.gz')
30+
workflow.connect(brain_extraction, 'out_file', export_brain, 'in_file')
31+
32+
export_brain_MNI_space = Node(ExportFile(), name = 'export_brain_MNI_space')
33+
export_brain_MNI_space.inputs.clobber = True
34+
export_brain_MNI_space.inputs.out_file = abspath(
35+
'derivatives/flirt/sub-001/anat/sub-001_space-mni152nlin2009casym_T1w_brain.nii.gz')
36+
workflow.connect(brain_extraction, 'out_file', export_brain_MNI_space, 'in_file')
37+
38+
# Run workflow
39+
workflow.run()

‎examples/nipype/code/merge_prov.py‎

Lines changed: 94 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,94 @@
1+
#!/usr/bin/python
2+
# coding: utf-8
3+
4+
""" Merge available prov JSON files into one JSON-LD graph """
5+
6+
import json
7+
from pathlib import Path
8+
9+
# List of available prov files
10+
prov_soft_files = [
11+
'derivatives/flirt/prov/prov-flirt_soft.prov.json'
12+
]
13+
prov_env_files = [
14+
'derivatives/flirt/prov/prov-flirt_env.prov.json'
15+
]
16+
prov_act_files = [
17+
'derivatives/flirt/prov/prov-flirt_act.prov.json'
18+
]
19+
prov_ent_files = [
20+
'derivatives/flirt/prov/prov-flirt_ent.prov.json'
21+
]
22+
sidecar_files = [
23+
'derivatives/flirt/sub-001/anat/sub-001_T1w_brain.json',
24+
'derivatives/flirt/sub-001/anat/sub-001_space-mni152nlin2009casym_T1w_brain.json'
25+
]
26+
27+
# Base jsonld
28+
base_provenance = {
29+
"Records": {
30+
"Software": [],
31+
"Activities": [],
32+
"Entities": []
33+
}
34+
}
35+
36+
# Add context and version
37+
with open('derivatives/flirt/prov/prov-flirt_base.prov.json', encoding = 'utf-8') as file:
38+
base_provenance.update(json.load(file))
39+
40+
# Parse Software
41+
for prov_file in prov_soft_files:
42+
with open(prov_file, encoding = 'utf-8') as file:
43+
data = json.load(file)
44+
for key, value in data.items():
45+
value['Id'] = key
46+
base_provenance['Records']['Software'].append(value)
47+
48+
# Parse Environments
49+
for prov_file in prov_env_files:
50+
with open(prov_file, encoding = 'utf-8') as file:
51+
data = json.load(file)
52+
for key, value in data.items():
53+
value['Id'] = key
54+
# /!\ Workaround: environments are added in the Entities list because
55+
# the Environments term is not defined in the BIDS Prov context yet
56+
base_provenance['Records']['Entities'].append(value)
57+
58+
# Parse Activities
59+
for prov_file in prov_act_files:
60+
with open(prov_file, encoding = 'utf-8') as file:
61+
data = json.load(file)
62+
for key, value in data.items():
63+
value['Id'] = key
64+
base_provenance['Records']['Activities'].append(value)
65+
66+
# Parse Entities
67+
for prov_file in prov_ent_files:
68+
with open(prov_file, encoding = 'utf-8') as file:
69+
data = json.load(file)
70+
for key, value in data.items():
71+
value['Id'] = key
72+
base_provenance['Records']['Entities'].append(value)
73+
74+
# Parse Sidecar files
75+
for sidecar_file in sidecar_files:
76+
# Identify data file(s) associated with the sidecar
77+
sidecar_filename = Path(sidecar_file)
78+
data_files = Path('').glob(f'{sidecar_filename.with_suffix("")}.*')
79+
data_files = [str(f) for f in list(data_files) if str(sidecar_filename) not in str(f)]
80+
81+
# Write provenance
82+
with open(sidecar_file, encoding = 'utf-8') as file:
83+
data = json.load(file)
84+
if 'GeneratedBy' in data:
85+
activity_id = data['GeneratedBy']
86+
for data_file in data_files:
87+
base_provenance['Records']['Entities'].append({
88+
"Id": f"bids::{data_file}",
89+
"GeneratedBy": activity_id
90+
})
91+
92+
# Write jsonld
93+
with open('derivatives/flirt/prov/merged/prov-flirt.prov.jsonld', 'w', encoding = 'utf-8') as file:
94+
file.write(json.dumps(base_provenance, indent = 2))

‎examples/nipype/derivatives/flirt/prov/merged/prov-flirt.prov.jsonld‎

Lines changed: 97 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,97 @@
1+
{
2+
"Records": {
3+
"Software": [
4+
{
5+
"Label": "Nipype",
6+
"Version": "1.8.6.dev0",
7+
"AltIdentifier": "RRID:SCR_002502",
8+
"Id": "bids::prov/#nipype-s9w94f9u"
9+
},
10+
{
11+
"Label": "FSL",
12+
"Version": "5.0",
13+
"AltIdentifier": "RRID:SCR_002823",
14+
"prov:actedOnBehalfOf": "bids::prov/#nipype-s9w94f9u",
15+
"Id": "bids::prov/#fsl-e1oq534p"
16+
}
17+
],
18+
"Activities": [
19+
{
20+
"Label": "brain_extraction",
21+
"Command": "bet sub-001_T1w.nii.gz sub-001_T1w_brain.nii.gz",
22+
"AssociatedWith": "bids::prov/#fsl-e1oq534p",
23+
"Used": [
24+
{
25+
"Id": "bids::sub-001/anat/sub-001_T1w.nii.gz",
26+
"Type": "Entity",
27+
"Label": "sub-001_T1w.nii.gz"
28+
},
29+
"bids::prov/#docker.io/nipype/nipype:py38-vavfao8v"
30+
],
31+
"Id": "bids::prov/#bet-ys913vx4"
32+
},
33+
{
34+
"Label": "flirt",
35+
"Command": "flirt -in /work/derivatives/bids_prov_workflow/brain_extraction/sub-001_T1w_brain.nii.gz -ref /usr/share/fsl/5.0/data/standard/MNI152_T1_1mm_brain.nii.gz -out sub-001_T1w_brain_flirt.nii.gz -omat sub-001_T1w_brain_flirt.mat",
36+
"AssociatedWith": "bids::prov/#fsl-e1oq534p",
37+
"Used": [
38+
{
39+
"Id": "bids::derivatives/bids_prov_workflow/brain_extraction/sub-001_T1w_brain.nii.gz",
40+
"Type": "Entity",
41+
"Label": "sub-001_T1w_brain.nii.gz",
42+
"GeneratedBy": "bids::prov/#bet-ys913vx4"
43+
},
44+
"/usr/share/fsl/5.0/data/standard/MNI152_T1_1mm_brain.nii.gz",
45+
"bids::prov/#docker.io/nipype/nipype:py38-vavfao8v"
46+
],
47+
"Id": "bids::prov/#flirt-xzje9hjh"
48+
},
49+
{
50+
"Label": "export_brain",
51+
"Command": "shutil.copy",
52+
"AssociatedWith": "bids::prov/#nipype-s9w94f9u",
53+
"Used": [
54+
"bids::derivatives/bids_prov_workflow/brain_extraction/sub-001_T1w_brain.nii.gz",
55+
"bids::prov/#docker.io/nipype/nipype:py38-vavfao8v"
56+
],
57+
"Id": "bids::prov/#export_file-i1cblvll"
58+
},
59+
{
60+
"Label": "export_brain_MNI_space",
61+
"Command": "shutil.copy",
62+
"AssociatedWith": "bids::prov/#nipype-s9w94f9u",
63+
"Used": [
64+
{
65+
"Id": "bids::derivatives/bids_prov_workflow/flirt/sub-001_T1w_brain_flirt.nii.gz",
66+
"Type": "Entity",
67+
"Label": "sub-001_T1w_brain_flirt.nii.gz",
68+
"GeneratedBy": "bids::prov/#flirt-xzje9hjh"
69+
},
70+
"bids::prov/#docker.io/nipype/nipype:py38-vavfao8v"
71+
],
72+
"Id": "bids::prov/#export_file-fpw3jrwy"
73+
}
74+
],
75+
"Entities": [
76+
{
77+
"Label": "nipype/nipype:py38 docker image",
78+
"OperatingSystem": "GNU/Linux 6.2.15-100.fc36.x86_64",
79+
"Id": "bids::prov/#docker.io/nipype/nipype:py38-vavfao8v"
80+
},
81+
{
82+
"Label": "MNI152_T1_1mm_brain.nii.gz",
83+
"Id": "/usr/share/fsl/5.0/data/standard/MNI152_T1_1mm_brain.nii.gz"
84+
},
85+
{
86+
"Id": "bids::derivatives/flirt/sub-001/anat/sub-001_T1w_brain.nii.gz",
87+
"GeneratedBy": "bids::prov/#export_file-i1cblvll"
88+
},
89+
{
90+
"Id": "bids::derivatives/flirt/sub-001/anat/sub-001_space-mni152nlin2009casym_T1w_brain.nii.gz",
91+
"GeneratedBy": "bids::prov/#export_file-fpw3jrwy"
92+
}
93+
]
94+
},
95+
"@context": "https://purl.org/nidash/bidsprov/context.json",
96+
"BIDSProvVersion": "0.0.1"
97+
}

‎examples/nipype/derivatives/flirt/prov/merged/prov-flirt.prov.png‎

156 KB
Loading

‎examples/nipype/derivatives/flirt/prov/prov-flirt_base.prov.json‎

Lines changed: 4 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,4 @@
1+
{
2+
"@context": "https://purl.org/nidash/bidsprov/context.json",
3+
"BIDSProvVersion": "0.0.1"
4+
}

‎examples/nipype/derivatives/flirt/prov/prov-flirt_ent.prov.json‎

Lines changed: 5 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,5 @@
1+
{
2+
"/usr/share/fsl/5.0/data/standard/MNI152_T1_1mm_brain.nii.gz": {
3+
"Label": "MNI152_T1_1mm_brain.nii.gz"
4+
}
5+
}

‎examples/nipype/derivatives/flirt/prov/prov-flirt_env.prov.json‎

Lines changed: 6 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,6 @@
1+
{
2+
"bids::prov/#docker.io/nipype/nipype:py38-vavfao8v": {
3+
"Label": "nipype/nipype:py38 docker image",
4+
"OperatingSystem": "GNU/Linux 6.2.15-100.fc36.x86_64"
5+
}
6+
}

‎examples/nipype/derivatives/flirt/prov/prov-flirt_soft.prov.json‎

Lines changed: 13 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,13 @@
1+
{
2+
"bids::prov/#nipype-s9w94f9u": {
3+
"Label": "Nipype",
4+
"Version": "1.8.6.dev0",
5+
"AltIdentifier": "RRID:SCR_002502"
6+
},
7+
"bids::prov/#fsl-e1oq534p": {
8+
"Label": "FSL",
9+
"Version": "5.0",
10+
"AltIdentifier": "RRID:SCR_002823",
11+
"prov:actedOnBehalfOf": "bids::prov/#nipype-s9w94f9u"
12+
}
13+
}

‎examples/nipype/derivatives/flirt/sub-001/anat/sub-001_T1w_brain.json‎

Lines changed: 3 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,3 @@
1+
{
2+
"GeneratedBy": "bids::prov/#export_file-i1cblvll"
3+
}

0 commit comments

Comments
 (0)