-
Notifications
You must be signed in to change notification settings - Fork 129
Description
Relates to bids-standard/bids-specification#2155.
I have added a command to my package to test pybids
; see instructions here.
Currently the set of discordances between what pybids
yields and what I think would be the ideal outcomes is quite large (see below).
Most of these are attributed to scenarios where the dataset should be deemed non-conformant based on the specification (even if the current validators don't pick up all such instances). pybids
seems to me to not be making any attempt to identify violations of the Inheritance Principle in the input dataset, it is just yielding whatever metadata it deems suitable to associate based on its own internal logic. In my opinion, in many of these example datasets it is suitable to yield neither metadata nor an empty result; the metadata request should be met with an exception, as the determination of which files from which to source that metadata is ill-posed. If there is interest in making a dent in the list below, this would be the primary change to make.
Current output of tests as at 29a112cd4518bba9c05b762299aa45f74f14f6f6:
INFO:__main__:30 discrepancies in test outcomes
ERROR:__main__:Dataset: ip112e1bad, BIDS version: 1.1.2, expected violation; actual outcome match
ERROR:__main__:Dataset: ip112e1bad, BIDS version: 1.7.0, expected violation; actual outcome match
ERROR:__main__:Dataset: ip112badmetapathe1, BIDS version: 1.1.2, expected violation; actual outcome mismatch
ERROR:__main__:Dataset: ip112badmetapathe1, BIDS version: 1.7.0, expected match; actual outcome mismatch
ERROR:__main__:Dataset: ip112badmetapathe2v1, BIDS version: 1.1.2, expected violation; actual outcome mismatch
ERROR:__main__:Dataset: ip112badmetapathe2v1, BIDS version: 1.7.0, expected violation; actual outcome mismatch
ERROR:__main__:Dataset: ip112badmetapathe2v2, BIDS version: 1.1.2, expected violation; actual outcome mismatch
ERROR:__main__:Dataset: ip112badmetapathe2v2, BIDS version: 1.7.0, expected match; actual outcome mismatch
ERROR:__main__:Dataset: ip170e2, BIDS version: 1.1.2, expected violation; actual outcome match
ERROR:__main__:Dataset: ip170e2, BIDS version: 1.7.0, expected violation; actual outcome match
ERROR:__main__:Dataset: ip170badrelpath, BIDS version: 1.1.2, expected match; actual outcome mismatch
ERROR:__main__:Dataset: ip170badrelpath, BIDS version: 1.7.0, expected violation; actual outcome mismatch
ERROR:__main__:Dataset: ipexclnonsc, BIDS version: 1.1.2, expected warning; actual outcome match
ERROR:__main__:Dataset: ipexclnonsc, BIDS version: 1.7.0, expected warning; actual outcome match
ERROR:__main__:Dataset: ipi1195v1, BIDS version: 1.1.2, expected violation; actual outcome mismatch
ERROR:__main__:Dataset: ipi1195v1, BIDS version: 1.7.0, expected violation; actual outcome mismatch
ERROR:__main__:Dataset: ipi1195v2, BIDS version: 1.1.2, expected violation; actual outcome mismatch
ERROR:__main__:Dataset: ipi1195v2, BIDS version: 1.7.0, expected violation; actual outcome mismatch
ERROR:__main__:Dataset: ipdwi003, BIDS version: 1.1.2, expected violation; actual outcome match
ERROR:__main__:Dataset: ipdwi003, BIDS version: 1.7.0, expected violation; actual outcome match
ERROR:__main__:Dataset: iploosemeta, BIDS version: 1.1.2, expected warning; actual outcome match
ERROR:__main__:Dataset: iploosemeta, BIDS version: 1.7.0, expected warning; actual outcome match
ERROR:__main__:Dataset: ipmultielfce1v2, BIDS version: 1.1.2, expected violation; actual outcome mismatch
ERROR:__main__:Dataset: ipmultielfce1v2, BIDS version: 1.7.0, expected violation; actual outcome mismatch
ERROR:__main__:Dataset: ippr1003ae1, BIDS version: 1.1.2, expected violation; actual outcome mismatch
ERROR:__main__:Dataset: ippr1003ae1, BIDS version: 1.7.0, expected violation; actual outcome mismatch
ERROR:__main__:Dataset: ippr1003ae2, BIDS version: 1.1.2, expected violation; actual outcome match
ERROR:__main__:Dataset: ippr1003ae2, BIDS version: 1.7.0, expected violation; actual outcome match
ERROR:__main__:Dataset: ippr1003e1v2, BIDS version: 1.1.2, expected violation; actual outcome match
ERROR:__main__:Dataset: ippr1003e1v2, BIDS version: 1.7.0, expected violation; actual outcome match