Advanced boutiques descriptors

Advanced boutiques descriptors

This page documents some advanced usage of boutiques descriptors in the context of VIP. The audience is assumed to already have successfully written and used basic boutiques descriptors as described in the packaging guide.

Custom properties

These optional VIP-specific properties in the "custom" section of the boutiques descriptor can be used to customize runtime behaviour.

• "vip:dot" (array of strings): a list of input IDs whose values are to be combined one to one (as in a "dot product") matching the Nth element of each list together, for a total N tuples. This implies that (i) there are at least two input IDs listed and that (ii) if a workflow is launched with N>1 values on one of those IDs, then all other IDs listed must have N values as well.
  Example: "vip:dot": ["seed","img", "mask"]
  In this example, if the three IDs listed have each two values corresponding to (s1, s2), (i1, i2) and (m1, m2), they will be combined into (s1, i1, i1) and (s2, i2, m2).
  The default behavior, for inputs not listed in a vip:dot property, is to do a cross combination of all values, combining each element of one set with every element of the other sets and, if a vip:dot is present, with every tuple of the "dot product".

• "vip:directoryInputs" (map of objects): a list of (one or multiple) input IDs that can be interpreted as directory in addition to file (which is the default type). The keys of the map are the input IDs, and the values are objects consisting of a single patterns key, containing an array of string. When VIP launches a workflow, if a directory is provided instead of a file, all files matching one of the patterns are passed as distinct inputs to the workflow.
  Example: "vip:directoryInputs":{"input1":{"patterns:["*.nii","*.nii.gz"]}}
  In this example, if the input named input1 is a directory, then all the files it contains that match *.nii or *.nii.gz will be used as inputs for the workflow.

• "vip:resultsDirectoryDefault" (string): default value for the results directory of a workflow. If not specified, it defaults to /vip/Home. This parameter is only relevant for executions launched from VIP web UI, executions launched from VIP API are responsible for setting the results directory explicitly.

• "vip:resultsDirectoryDescription (string): user-visible description for the results directory field in the execution launch form. This parameter is only relevant for executions launched from VIP web UI.

• "vip:resultsDirectorySuffix" (string): name of the subdirectory that is automatically appended to the results directory of a workflow. When this property is not set, the default behavior of VIP is to create a timestamp-based subdirectory where outputs will be stored. This parameter can be explicitly set to empty to completely disable output directory auto-append, and write outputs directly to the results directory.
  Example: "vip:resultsDirectorySuffix":"mysubdir"

• "vip:intIteratorInputs" (array of strings): a list of input IDs (often just one) that are expected to be runtime "iterators". When a workflow is submitted, an iterator input must have a single value, and it must be a strictly positive integer. This integer N>0 is then expanded into N values ranging from 0 to N-1, which are passed as inputs to N jobs. This is useful for tasks that can be split into N parallel subtasks.
  The recommended boutiques configuration for an iterator input is: "type":"String/Number" "integer":true, "minimum":1.

• "vip:overriddenInputs" (map of strings): a list of inputs IDs that should be invisible from the user, in VIP web UI launch form and API. The keys of the map must be the ID of another visible input, which will be used as the initial value for the hidden input. This can be combined with other custom properties to have several inputs available at runtime, all derived from a single user-visible input (for instance, an iterator count and a total value). Overridden inputs and their dependent inputs are assumed to be mandatory, there is no support for optional fields.
  Example: "vip:overriddenInputs":{"myHiddenInput":"myVisibleInput"}

• "vip:altContainerOpts" (array of strings): a list of app-specific runtime parameters to pass to the container system. These parameters are similar to those defined in the "container-image.container-opts" field of the boutiques descriptor, but they target the alternative runtime system: in practice, this field allows to define runtime options for singularity when the "container-image.type" field is docker, and vice-versa.

Outputs naming

When an app is suited to run multiple jobs within a single VIP execution workflow, for instance through a combination of input parameters with multiple values, special attention should be paid to the naming of output files. This is defined by the "path-template" field in the boutiques descriptors "outputs" section.

Assuming each job produces one output file, then the output file name must be constructed in "path-template" in a such way that:

• each job produces a unique output filename for a given combination of inputs. If two different jobs within a single execution workflow have the same output filename, then the second job that runs will overwrite the result of the first job, which is generally not the desired behaviour.
• the total number of jobs that get executed must exactly match the number of output filename combinations. It typically means that it's not possible to give full control of the output filename to end users : settings such as "path-template": "[OUTNAME]" should then be avoided, in favor of some more "mechanical" naming.

One way to achieve this is to build "path-template" as a combination of all input parameters that can take multiple independent values within a single execution workflow. The base name of an input file can be reused as a part of the output name, with some extra prefix and/or extensions stripping. For instance:

    "path-template": "output_[INPUT_FILE]_[PARAM1]_[PARAM2].out",
    "path-template-stripped-extensions": [".in"],