ProvEntities for BIDS files and BIDS datasets

[bclenet]

Current status

The current version of the BEP028 PR for specification indicates :

Key name	Requirement Level	Data type	Description
Used	OPTIONAL	string or array of strings	Identifier(s) of the provEntity (one or more) or environment (one or more) used by this activity. Related provEntities MUST be described as specified in the ProvEntities section. Related environments MUST be described as specified in the Environments section.

And in the ProvEntities section:

Each file with a ent suffix is a JSON file describing provEntities.
These files SHOULD not contain provEntities describing data files that are available in the dataset. Use sidecar JSON files instead for this purpose (see Provenance of a BIDS file).
These files SHOULD not contain provEntities describing the current dataset. Use dataset_description.json files instead for this purpose (see Provenance of a BIDS dataset).

Issue

For simplification reasons we would like that it is not required to write a whole provEntity JSON object in a provenance file for BIDS files or BIDS datasets listed in the Used metadata field. (In other cases, we still want the data to be described by a provEntity because we cannot derive enough information from an identifier that is not a BIDS URI)

For now we cover the following cases:

• a BIDS file that is part of the dataset in which the provenance is described:
  • we use a BIDS URI to link to the file ;
  • we can even use the sidecar JSON of the file as a container for other metadata that would normally be stored in a provEntity.
• the BIDS dataset in which the provenance is described:
  • we use a BIDS URI to link to the dataset (bids::.) ;
  • we can even use the dataset_description.json as a container for other metadata that would normally be stored in a provEntity.

However, we do not cover the following cases:

• a BIDS file that is part of another dataset ;
• another BIDS dataset.

Partial solution

We do not require describing a provEntity for any BIDS file or any BIDS dataset, e.g.:

These files SHOULD not contain provEntities describing available BIDS files (see Provenance of a BIDS file for this purpose).
These files SHOULD not contain provEntities describing available BIDS datasets. (see Provenance of a BIDS dataset for this purpose).

This option works well unless users want to describe other metadata fields (e.g. Digest, Type, or other metadata) that would normally be inside a provEntity JSON object and that cannot be derived from a BIDS URI... In this case users have to go back to describing provEntities inside provenance files.

[cmaumet]

@bclenet Thanks for fleshing this problem out so clearly and +1 for your proposed solution.

About the limitation you highlight:

This option works well unless users want to describe other metadata fields (e.g. Digest, Type, or other metadata) that would normally be inside a provEntity JSON object and that cannot be derived from a BIDS URI...

I think those metadata fields (e.g. Digest, Type, or other metadata) could be in the dataset in which the BIDS file is located. Do you see any shortcomings in doing this?

[bclenet]

@cmaumet I see these problems in relying on other datasets to store provenance metadata:

• we don't want to modify source datasets to write parts of the provenance of a derivative dataset
• most of the time we do not have write access to these datasets
• maybe splitting the provenance graph between several datasets is something we want to avoid ?

That being said, it could work for study datasets but the above limitations are true in other cases.

[cmaumet]

Thanks @bclenet!

we don't want to modify source datasets to write parts of the provenance of a derivative dataset
most of the time we do not have write access to these datasets

In those cases, I think that it would be okay to write a ProvEntity in a _ent.json file but in general it seems more likely to me that the original dataset must be in charge of writing the metadata for its own files.

maybe splitting the provenance graph between several datasets is something we want to avoid ?

In principles splitting the graph in multiple datasets should work well with RDF but we need to test with as example (as you did for BIDS study example).

So I think we are good?

[bclenet]

@cmaumet

In those cases, I think that it would be okay to write a ProvEntity in a _ent.json file but in general it seems more likely to me that the original dataset must be in charge of writing the metadata for its own files.

If we consider that this is provenance metadata describing the creation of another dataset, I don't think the original dataset should store it (and also there might be conflicts e.g. with Type between what several derivatives datasets would like to write).

In principles splitting the graph in multiple datasets should work well with RDF but we need to test with as example (as you did for BIDS study example).

I agree that splitting the graph is ok in theory, but for BIDS Prov it makes the aggregation of provenance metadata harder.