Skip to content

Commit 98d46cd

Browse files
bclenetbclenet
committed
Cleaning example and updating README.md
1 parent f111723 commit 98d46cd

File tree

12 files changed

+51
-164
lines changed

12 files changed

+51
-164
lines changed

‎examples/dcm2niix/README.md‎

Lines changed: 10 additions & 41 deletions
Original file line numberDiff line numberDiff line change
@@ -11,14 +11,12 @@ datalad install --recursive https://github.com/psychoinformatics-de/hirni-demo.g
1111

1212
> [!NOTE] Note that the dataset must be added inside the `sourcedata/` directory.
1313
14-
1514
## Experiment #1
1615

1716
The aim of the experiment is to describe the provenance records inside several files. Here we use sidecars and modality agnostic files inside the `prov/` directory, as follows:
1817
```
1918
.
2019
├── prov
21-
│ ├── bidsprov_context.json
2220
│ ├── environments.prov.jsonld
2321
│ └── software.prov.jsonld
2422
└── sub_02
@@ -31,7 +29,6 @@ The aim of the experiment is to describe the provenance records inside several f
3129
```
3230

3331
* `sub-02_ses-20130717141500_T1w.prov.jsonld` and `sub-02_ses-20140425155335_task-oneback_run-1_bold.prov.jsonld` are sidecars defining provenance for the corresponding `.nii` files.
34-
* `bidsprov_context.json` is the BIDS Prov context from https://purl.org/nidash/bidsprov/context.json, with minor changes.
3532
* `environments.prov.jsonld` mutualises the declaration of software environments objects for lower level prov files
3633
* `software.prov.jsonld` mutualises the declaration of software pieces objects for lower level prov files
3734

@@ -43,51 +40,23 @@ mkdir prov/experiment_1/
4340
python code/exp1_merge_prov.py
4441
```
4542

46-
From that, we generate the RDF graph `prov/experiment_1/merged_provenance.ttl`.
47-
48-
### Limitation #1: type indexing
49-
50-
Notice that starting from line 50 of the file, the generated triplets are not correct.
51-
52-
```turtle
53-
[] prov:Activity <urn:conversion-5a66f5be> ;
54-
...
55-
```
56-
57-
This is probably due to the fact that rdflib 7.1.3 does not conform with JSON-LD 1.1 and is therefore not able to parse graphs using [type indexing](https://www.w3.org/TR/json-ld11/#example-106-indexing-data-in-json-ld-by-type) as we used it e.g. in the `Records_hirni-demo_software` term of the `prov/software.prov.jsonld` file.
58-
Furthermore, all the triplets describing the type of node objects are missing because of rdflib not understanding type indexing.
43+
From that, we generate the RDF graph `prov/experiment_1/merged_provenance.ttl`. Then we were able to plot the graph as a png file. We used this command:
5944

60-
As a workaround, we replaced the incorrect lines (50 to the end) by triplets describing the type of the objects in the merged document, as presented in `prov/experiment_1/merged_provenance_2.ttl`.
61-
62-
From that we were able to plot the following graph `prov/experiment_1/merged_provenance_2.ttl`. We used the
6345
```shell
6446
python code/plot_graph.py -i prov/experiment_1/merged_provenance_2.ttl -o prov/experiment_1/merged_provenance_2.png
6547
```
6648

67-
![](/examples/dcm2niix/prov/experiment_1/merged_provenance_2.png)
68-
69-
### Limitation #2: the `@graph` term
49+
![](/examples/dcm2niix/prov/experiment_1/merged_provenance.png)
7050

71-
In the original BIDSProv context, we define the term `Records` that refers to `@graph` while setting up type indexing.
51+
### Notes
7252

73-
Probably due to the fact that rdflib 7.1.3 does not conform with JSON-LD 1.1, parsing several JSON-LD files each one containing a `Records` would fail: it sets `Records` with the last parsed content, not a union of all containers as it should be.
53+
In this example, we rely on the fact that nodes defined in the `prov/*.prov.jsonld` files have `bids::prov/` as base IRIs. Here are the involved nodes:
54+
* `bids::prov/dcm2niix`
55+
* `bids::prov/fmriprep`
56+
* `bids::prov/fedora`
7457

75-
Therefore, we sticked to using the `@graph` term, in a local context `prov/bidsprov_context.json`.
58+
### Limitations
7659

77-
### Limitation #3: `bids` namespace for files
78-
79-
Defining the `@base` term in the context (see `sub_02/ses_20140425155335/func/sub-02_ses-20140425155335_task-oneback_run-1_bold.prov.json` line 5) allows give a default base IRI to relative IRIs.
80-
81-
In our case, the sidecar provenance files defined under the `sub_02/` (sub-)directory(ies) may want to refer to keywords inside the `prov/` directory. Setting `@base` to the `prov/` directory and passing relative IRIs inside sidecars files resolves this... But parsing the sidecar files with rdflib, we get the following RDF nodes:
82-
83-
```turtle
84-
<file:///data/dev/BEP028_BIDSprov/examples/dcm2niix/prov/fedora>
85-
```
86-
87-
although we would prefer:
88-
89-
```turtle
90-
<bids::prov/fedora>
91-
```
60+
The `bids::prov/fedora` node defined in `prov/environments.prov.jsonld` (see grey node in the above graph plot) is not recognized as a `prov:Entity` as the current context (commit [ce0eb77](https://github.com/bids-standard/BEP028_BIDSprov/commit/ce0eb774abd9527e594bd69212a87d5047864678)) does not define the `Environments` term.
9261

93-
> [!NOTE] Note that we had to alter the .ttl files to remove the root paths of the directories (e.g. replacing `file:///home/user/` with `file:///data/`)
62+
We cannot list all the

‎examples/dcm2niix/code/exp1_merge_prov.py‎

Lines changed: 9 additions & 22 deletions
Original file line numberDiff line numberDiff line change
@@ -1,41 +1,28 @@
1-
import json
1+
#!/usr/bin/python
2+
# coding: utf-8
3+
4+
""" Merge available prov JSON-LD files into one RDF graph """
25

3-
from rdflib import Graph
6+
import json
47
from pyld import jsonld
58

9+
# List of available prov files
610
prov_files = [
7-
#'prov/common.prov.jsonld',
811
'prov/software.prov.jsonld',
912
'prov/environments.prov.jsonld',
1013
'sub_02/ses_20130717141500/anat/sub-02_ses-20130717141500_T1w.prov.jsonld',
1114
'sub_02/ses_20140425155335/func/sub-02_ses-20140425155335_task-oneback_run-1_bold.prov.jsonld'
1215
]
1316

14-
with open('prov/bidsprov_context.json', encoding = 'utf-8') as file:
15-
context = json.load(file)
16-
17+
# Generate RDF graph of each file using pyld
1718
rdf_graph = ''
1819
for prov_file in prov_files:
1920
with open(prov_file, encoding = 'utf-8') as file:
20-
#expanded = json.dumps(jsonld.flatten(jsonld.expand(json.load(file)), ctx = context, options = {'processingMode': 'json-ld-1.1'}), indent = 2)
21-
#print(expanded)
2221
rdf_graph += f'# {prov_file}\n'
2322
rdf_graph += jsonld.normalize(
24-
json.load(file), {'algorithm': 'URDNA2015', 'format': 'application/n-quads'})
23+
jsonld.flatten(data), {'algorithm': 'URDNA2015', 'format': 'application/n-quads'})
2524
rdf_graph += '\n'
2625

27-
#graph.parse(data = expanded, format = 'json-ld')
28-
print(rdf_graph)
29-
30-
31-
"""
32-
graph.parse('prov/common.prov.jsonld', format = 'json-ld')
33-
graph.parse('prov/software.prov.jsonld', format = 'json-ld')
34-
graph.parse('prov/environments.prov.jsonld', format = 'json-ld')
35-
graph.parse('sub_02/ses_20130717141500/anat/sub-02_ses-20130717141500_T1w.prov.jsonld', format = 'json-ld')
36-
graph.parse('sub_02/ses_20140425155335/func/sub-02_ses-20140425155335_task-oneback_run-1_bold.prov.jsonld', format = 'json-ld')
37-
"""
38-
#print(len(graph))
39-
#graph.serialize(destination = 'prov/experiment_1/merged_provenance.ttl')
26+
# Write RDF graph
4027
with open('prov/experiment_1/merged_provenance.ttl', 'w' , encoding = 'utf-8') as file:
4128
file.write(rdf_graph)

‎examples/dcm2niix/code/requirements.txt‎

Lines changed: 1 addition & 2 deletions
Original file line numberDiff line numberDiff line change
@@ -1,2 +1 @@
1-
rdflib==7.1.3
2-
pyld==2.0.4
1+
pyld==2.0.4

‎examples/dcm2niix/prov/common.prov.jsonld‎

Lines changed: 0 additions & 4 deletions
This file was deleted.

‎examples/dcm2niix/prov/environments.prov.jsonld‎

Lines changed: 11 additions & 8 deletions
Original file line numberDiff line numberDiff line change
@@ -1,10 +1,13 @@
11
{
2-
"@context": "https://raw.githubusercontent.com/bclenet/BEP028_BIDSprov/refs/heads/context/context.json",
3-
"Environments": [
4-
{
5-
"Id": "bids::prov/fedora",
6-
"Label": "Fedora release 36 (Thirty Six)",
7-
"OperatingSystem": " GNU/Linux 6.2.15-100.fc36.x86_64"
8-
}
9-
]
2+
"@context": "https://purl.org/nidash/bidsprov/context.json",
3+
"BIDSProvVersion": "0.0.1",
4+
"Records": {
5+
"Environments": [
6+
{
7+
"Id": "bids::prov/fedora",
8+
"Label": "Fedora release 36 (Thirty Six)",
9+
"OperatingSystem": " GNU/Linux 6.2.15-100.fc36.x86_64"
10+
}
11+
]
12+
}
1013
}

‎examples/dcm2niix/prov/experiment_1/merged_provenance.png‎

-1.36 KB
Loading

‎examples/dcm2niix/prov/experiment_1/merged_provenance.ttl‎

Lines changed: 14 additions & 15 deletions
Original file line numberDiff line numberDiff line change
@@ -1,52 +1,51 @@
11
# prov/software.prov.jsonld
2+
<bids::prov/dcm2niix> <http://www.w3.org/1999/02/22-rdf-syntax-ns#type> <http://www.w3.org/ns/prov#Agent> .
23
<bids::prov/dcm2niix> <http://www.w3.org/2000/01/rdf-schema#label> "dcm2niix" .
4+
<bids::prov/fmriprep> <http://www.w3.org/1999/02/22-rdf-syntax-ns#type> <http://www.w3.org/ns/prov#Agent> .
35
<bids::prov/fmriprep> <http://www.w3.org/2000/01/rdf-schema#label> "fmriprep" .
4-
_:c14n0 <http://www.w3.org/ns/prov#Agent> <bids::prov/dcm2niix> .
5-
_:c14n0 <http://www.w3.org/ns/prov#Agent> <bids::prov/fmriprep> .
66

77
# prov/environments.prov.jsonld
88
<bids::prov/fedora> <http://www.w3.org/2000/01/rdf-schema#label> "Fedora release 36 (Thirty Six)" .
9-
_:c14n0 <http://www.w3.org/ns/prov#Entity> <bids::prov/fedora> .
109

1110
# sub_02/ses_20130717141500/anat/sub-02_ses-20130717141500_T1w.prov.jsonld
11+
<bids::sourcedata/hirni-demo/acq1/dicoms/example-dicom-structural-master/dicoms/N2D_0001.dcm> <http://www.w3.org/1999/02/22-rdf-syntax-ns#type> <http://www.w3.org/ns/prov#Entity> .
1212
<bids::sourcedata/hirni-demo/acq1/dicoms/example-dicom-structural-master/dicoms/N2D_0001.dcm> <http://www.w3.org/2000/01/rdf-schema#label> "N2D_0001.dcm" .
13+
<bids::sourcedata/hirni-demo/acq1/dicoms/example-dicom-structural-master/dicoms/N2D_0002.dcm> <http://www.w3.org/1999/02/22-rdf-syntax-ns#type> <http://www.w3.org/ns/prov#Entity> .
1314
<bids::sourcedata/hirni-demo/acq1/dicoms/example-dicom-structural-master/dicoms/N2D_0002.dcm> <http://www.w3.org/2000/01/rdf-schema#label> "N2D_0002.dcm" .
15+
<bids::sourcedata/hirni-demo/acq1/dicoms/example-dicom-structural-master/dicoms/N2D_0003.dcm> <http://www.w3.org/1999/02/22-rdf-syntax-ns#type> <http://www.w3.org/ns/prov#Entity> .
1416
<bids::sourcedata/hirni-demo/acq1/dicoms/example-dicom-structural-master/dicoms/N2D_0003.dcm> <http://www.w3.org/2000/01/rdf-schema#label> "N2D_0003.dcm" .
17+
<bids::sub-02/ses-20130717141500/anat/sub-02_ses-20130717141500_T1w.json> <http://www.w3.org/1999/02/22-rdf-syntax-ns#type> <http://www.w3.org/ns/prov#Entity> .
1518
<bids::sub-02/ses-20130717141500/anat/sub-02_ses-20130717141500_T1w.json> <http://www.w3.org/2000/01/rdf-schema#label> "sub-02_ses-20130717141500_T1w.json" .
1619
<bids::sub-02/ses-20130717141500/anat/sub-02_ses-20130717141500_T1w.json> <http://www.w3.org/ns/prov#wasGeneratedBy> <urn:conversion-00f3a18f> .
20+
<bids::sub-02/ses-20130717141500/anat/sub-02_ses-20130717141500_T1w.nii> <http://www.w3.org/1999/02/22-rdf-syntax-ns#type> <http://www.w3.org/ns/prov#Entity> .
1721
<bids::sub-02/ses-20130717141500/anat/sub-02_ses-20130717141500_T1w.nii> <http://www.w3.org/2000/01/rdf-schema#label> "sub-02_ses-20130717141500_T1w.nii" .
1822
<bids::sub-02/ses-20130717141500/anat/sub-02_ses-20130717141500_T1w.nii> <http://www.w3.org/ns/prov#wasGeneratedBy> <urn:conversion-00f3a18f> .
23+
<urn:conversion-00f3a18f> <http://www.w3.org/1999/02/22-rdf-syntax-ns#type> <http://www.w3.org/ns/prov#Activity> .
1924
<urn:conversion-00f3a18f> <http://www.w3.org/2000/01/rdf-schema#label> "Conversion" .
2025
<urn:conversion-00f3a18f> <http://www.w3.org/ns/prov#used> <bids::prov/fedora> .
2126
<urn:conversion-00f3a18f> <http://www.w3.org/ns/prov#used> <bids::sourcedata/hirni-demo/acq1/dicoms/example-dicom-structural-master/dicoms/N2D_0001.dcm> .
2227
<urn:conversion-00f3a18f> <http://www.w3.org/ns/prov#used> <bids::sourcedata/hirni-demo/acq1/dicoms/example-dicom-structural-master/dicoms/N2D_0002.dcm> .
2328
<urn:conversion-00f3a18f> <http://www.w3.org/ns/prov#used> <bids::sourcedata/hirni-demo/acq1/dicoms/example-dicom-structural-master/dicoms/N2D_0003.dcm> .
2429
<urn:conversion-00f3a18f> <http://www.w3.org/ns/prov#wasAssociatedWith> <bids::prov/dcm2niix> .
25-
_:c14n0 <http://www.w3.org/ns/prov#Activity> <urn:conversion-00f3a18f> .
26-
_:c14n0 <http://www.w3.org/ns/prov#Entity> <bids::sourcedata/hirni-demo/acq1/dicoms/example-dicom-structural-master/dicoms/N2D_0001.dcm> .
27-
_:c14n0 <http://www.w3.org/ns/prov#Entity> <bids::sourcedata/hirni-demo/acq1/dicoms/example-dicom-structural-master/dicoms/N2D_0002.dcm> .
28-
_:c14n0 <http://www.w3.org/ns/prov#Entity> <bids::sourcedata/hirni-demo/acq1/dicoms/example-dicom-structural-master/dicoms/N2D_0003.dcm> .
29-
_:c14n0 <http://www.w3.org/ns/prov#Entity> <bids::sub-02/ses-20130717141500/anat/sub-02_ses-20130717141500_T1w.json> .
30-
_:c14n0 <http://www.w3.org/ns/prov#Entity> <bids::sub-02/ses-20130717141500/anat/sub-02_ses-20130717141500_T1w.nii> .
3130

3231
# sub_02/ses_20140425155335/func/sub-02_ses-20140425155335_task-oneback_run-1_bold.prov.jsonld
32+
<bids::sourcedata/hirni-demo/acq2/dicoms/example-dicom-functional-master/dicoms/MR.1.3.46.670589.11.38317.5.0.4476.2014042516042547586> <http://www.w3.org/1999/02/22-rdf-syntax-ns#type> <http://www.w3.org/ns/prov#Entity> .
3333
<bids::sourcedata/hirni-demo/acq2/dicoms/example-dicom-functional-master/dicoms/MR.1.3.46.670589.11.38317.5.0.4476.2014042516042547586> <http://www.w3.org/2000/01/rdf-schema#label> "MR.1.3.46.670589.11.38317.5.0.4476.2014042516042547586" .
34+
<bids::sourcedata/hirni-demo/acq2/dicoms/example-dicom-functional-master/dicoms/MR.1.3.46.670589.11.38317.5.0.4476.2014042516042548588> <http://www.w3.org/1999/02/22-rdf-syntax-ns#type> <http://www.w3.org/ns/prov#Entity> .
3435
<bids::sourcedata/hirni-demo/acq2/dicoms/example-dicom-functional-master/dicoms/MR.1.3.46.670589.11.38317.5.0.4476.2014042516042548588> <http://www.w3.org/2000/01/rdf-schema#label> "MR.1.3.46.670589.11.38317.5.0.4476.2014042516042548588" .
36+
<bids::sourcedata/hirni-demo/acq2/dicoms/example-dicom-functional-master/dicoms/MR.1.3.46.670589.11.38317.5.0.4476.2014042516042551590> <http://www.w3.org/1999/02/22-rdf-syntax-ns#type> <http://www.w3.org/ns/prov#Entity> .
3537
<bids::sourcedata/hirni-demo/acq2/dicoms/example-dicom-functional-master/dicoms/MR.1.3.46.670589.11.38317.5.0.4476.2014042516042551590> <http://www.w3.org/2000/01/rdf-schema#label> "MR.1.3.46.670589.11.38317.5.0.4476.2014042516042551590" .
38+
<bids::sub-02/ses-20140425155335/func/sub-02_ses-20140425155335_task-oneback_run-1_bold.json> <http://www.w3.org/1999/02/22-rdf-syntax-ns#type> <http://www.w3.org/ns/prov#Entity> .
3639
<bids::sub-02/ses-20140425155335/func/sub-02_ses-20140425155335_task-oneback_run-1_bold.json> <http://www.w3.org/2000/01/rdf-schema#label> "sub-02_ses-20140425155335_task-oneback_run-1_bold.json" .
3740
<bids::sub-02/ses-20140425155335/func/sub-02_ses-20140425155335_task-oneback_run-1_bold.json> <http://www.w3.org/ns/prov#wasGeneratedBy> <urn:conversion-5a66f5be> .
41+
<bids::sub-02/ses-20140425155335/func/sub-02_ses-20140425155335_task-oneback_run-1_bold.nii> <http://www.w3.org/1999/02/22-rdf-syntax-ns#type> <http://www.w3.org/ns/prov#Entity> .
3842
<bids::sub-02/ses-20140425155335/func/sub-02_ses-20140425155335_task-oneback_run-1_bold.nii> <http://www.w3.org/2000/01/rdf-schema#label> "sub-02_ses-20140425155335_task-oneback_run-1_bold.nii" .
3943
<bids::sub-02/ses-20140425155335/func/sub-02_ses-20140425155335_task-oneback_run-1_bold.nii> <http://www.w3.org/ns/prov#wasGeneratedBy> <urn:conversion-5a66f5be> .
44+
<urn:conversion-5a66f5be> <http://www.w3.org/1999/02/22-rdf-syntax-ns#type> <http://www.w3.org/ns/prov#Activity> .
4045
<urn:conversion-5a66f5be> <http://www.w3.org/2000/01/rdf-schema#label> "Conversion" .
4146
<urn:conversion-5a66f5be> <http://www.w3.org/ns/prov#used> <bids::prov/fedora> .
4247
<urn:conversion-5a66f5be> <http://www.w3.org/ns/prov#used> <bids::sourcedata/hirni-demo/acq2/dicoms/example-dicom-functional-master/dicoms/MR.1.3.46.670589.11.38317.5.0.4476.2014042516042547586> .
4348
<urn:conversion-5a66f5be> <http://www.w3.org/ns/prov#used> <bids::sourcedata/hirni-demo/acq2/dicoms/example-dicom-functional-master/dicoms/MR.1.3.46.670589.11.38317.5.0.4476.2014042516042548588> .
4449
<urn:conversion-5a66f5be> <http://www.w3.org/ns/prov#used> <bids::sourcedata/hirni-demo/acq2/dicoms/example-dicom-functional-master/dicoms/MR.1.3.46.670589.11.38317.5.0.4476.2014042516042551590> .
4550
<urn:conversion-5a66f5be> <http://www.w3.org/ns/prov#wasAssociatedWith> <bids::prov/dcm2niix> .
46-
_:c14n0 <http://www.w3.org/ns/prov#Activity> <urn:conversion-5a66f5be> .
47-
_:c14n0 <http://www.w3.org/ns/prov#Entity> <bids::sourcedata/hirni-demo/acq2/dicoms/example-dicom-functional-master/dicoms/MR.1.3.46.670589.11.38317.5.0.4476.2014042516042547586> .
48-
_:c14n0 <http://www.w3.org/ns/prov#Entity> <bids::sourcedata/hirni-demo/acq2/dicoms/example-dicom-functional-master/dicoms/MR.1.3.46.670589.11.38317.5.0.4476.2014042516042548588> .
49-
_:c14n0 <http://www.w3.org/ns/prov#Entity> <bids::sourcedata/hirni-demo/acq2/dicoms/example-dicom-functional-master/dicoms/MR.1.3.46.670589.11.38317.5.0.4476.2014042516042551590> .
50-
_:c14n0 <http://www.w3.org/ns/prov#Entity> <bids::sub-02/ses-20140425155335/func/sub-02_ses-20140425155335_task-oneback_run-1_bold.json> .
51-
_:c14n0 <http://www.w3.org/ns/prov#Entity> <bids::sub-02/ses-20140425155335/func/sub-02_ses-20140425155335_task-oneback_run-1_bold.nii> .
5251

‎examples/dcm2niix/prov/experiment_1/merged_provenance_2.png‎

-184 KB
Binary file not shown.

0 commit comments

Comments
 (0)