LINST II - In-house TSV instrument descriptors

[ridz1208]

The existing LINST format is a text file where each row represents a field on a form (with some exceptions) with an ad-hoc number of attributes for each field depending on it's type. The attributes are separated by an inhouse {@} separator and are only indexed by location in the row with no header or description of what the attribute is meant to be. The format is hard to read and hard to edit by technical and non-technical users, one bad edit causes the instrument loading to fail in the most confusing ways.

Proposal:

A defined in-house TSV formated instrument descriptor where each attribute is clearly structured under a header (type, name, label, list options, min, max, ...) with validation when attributes are mis-associated with incorrect types. A TSV can easily be opened and edited by users reducing greatly potential errors stemming of mishandling of a text file. Delimiters are more standard. Explicit headers ensure the editor knows what each field serves. The current LINST format is also associated with a .meta, .rules and .score file here is the break down of changes proposed here:

1. .linst
   1. A described above, direct mapping of each field type to the new TSV format and organization of attributes under dedicated columns.
   2. For ENUM fields (selects, multiselects) the enum should probably follwo a JSON format as key:value pairs. JSON is easy to encode/decode and a popular enough structure to not confuse editors from different backgrounds
   3. _status fields historically are listed as their own row i nthe current standard. the new format would add the necessary headers such as include_status and status_options where the former defines if we need a status field and the latter enumerates the options as a JSON
2. .rules
   1. rules no longer need a separate file, they can be included in the main tsv
   2. the new format would explicitly define a required column to avoid implicitly making everything required and bypassing it with hacks like the "NEVER_REQUIRED" rules and the "norules" meta field
   3. a required_if column alternatively offers conditional requirement where if the cell is filled the rule in the cell applies. In the initial implementation the rules can be very simple and formed as reference_field_name==value operators can be limited to == and != and logic statements only need to support AND and OR.
3. .score
   No planned changes in the short term.
4. .meta
   1. Simplify the format to a TSV as well with no additional changes

In terms of implementation, a phased approach is recommended:

Phase 1:
1. Build basic template for the TSV with all the possible headers defined for instrument structure only (no rules)
2. Build NDB_BVL_Instrument extension class to support reading the TSV
3. Build LINST -> LINST2.0 conversion scripts

Phase 2:
1. Incorporate rules headers into the TSV template
2. Add rule parsing logic to the PHP class
3. create rule conversion script from .rule files to the main TSV

Phase 3:
1. Modify .meta files to become instrumentname_meta.tsv files

Phase 4:
1. Adapt instrument builder to use the new TSV format
2. Adapt instrument manager to support installing the TSV format

Phase 5:
1. Create LINST2.0->BIDS exporters

[driusan]

Phase 1,2, 4, and 5 make sense to me but I don't see the benefit from phase 3. The .meta files aren't really maintained by users and I don't see the benefit of changing them to tsv.

(And if we do need users to maintain them we might as well eliminate them entirely by putting any relevant data from them into a database table.)

[jeffersoncasimir]

The following may be entirely disregarded:

My proposal is to use the BIDS phenotype standard instead of creating a new one. Although the dedicated page and examples are lackluster, there are a many recognized terms, including one for data type (which I previously did not know about), which make it extremely versatile for our purposes.

A lot of my work on the BidsPhenotypeParser class (CBIG) can be reused and inspired from, which essentially converts BIDS phenotype "data dictionaries" (.json) to .linst, in order to work with LORIS. There is also a slight lift for using TSV instead of CSV and for converting the BIDS fields: participant_id and session_id to PSCID (or ExternalID) and Visit_label.

The main downside is the lack of a "builder" for JSON, which is the data dictionary that would not necessarily be built by someone with prior knowledge of the format. BIDS themselves recommend using a particular online JSON editor/validator amongst other tools.

The main upsides are the extensibility-by-design of the BIDS specification and non-uniformity of the JSON structure vs a comma-separated structure, which leads to empty fields in the data file, as well as the built-in validation of "Levels" (equivalent to "select" options).

The phases above can mostly be adapted for the BIDS recognized files (TSV data + JSON sidecar)