Indexing is hardcoded

[nx10]

At the moment indexing is hardcoded and only follows subject/session directory structure.

BIDS spec / schema defines subfolders https://github.com/bids-standard/bids-specification/blob/master/src/schema/rules/directories.yaml

We should investigate how to let indexing follow the schema without degrading performance or bloating the codebase

[kaitj]

I am having a little bit of trouble my head around this one (more how much of this should be handled by validation vs indexing), though I do agree, generally, that if possible we should follow the schema.

Currently:

• b2t only indexes if a directory follows the subject/session directory structure (e.g. it doesn't index the Freesurfer output directories as they are outputted) and the identified files pass validation
• b2t doesn't make a distinction between "raw" and "derivative" datasets, just that if its a "bids" dataset. Unless I am missing something, I think this is still fine even with the schema

Initial thoughts:

• The schema entities that immediately relevant to b2t (and these might not span across all levels) are:

  subject:
   entity: subject
   level: optional
   opaque: false
   subdirs:
     - session
     - datatype
  session:
   entity: session
   level: optional
   opaque: false
   subdirs:
     - datatype
  template:
   entity: template
   level: optional
   opaque: false
   subdirs:
     - cohort
     - datatype
  datatype:
   value: datatype
   level: optional
   opaque: false

• we probably also want to consider all the directories where the entity is opaque: false with some exceptions (e.g. recursively searching through the derivatives directory)

Just some brief thoughts, but agree we should investigate this further.