✅ Testing Desiderata

[shnizzedy]

Related problem

Much of our testing that could be automated is currently done manually

Proposed feature

Test Types (Outcome)

1. Build (Docker containers)
   a. Verify that Docker recipes complete and can be converted to singularity
2. Portability (List of checkmarks or X's)
   b. Verify consistent execution across equivalent Docker and Singularity runs
3. Smoke (List of checkmarks or X's)
   a. ✅ Verify successful build of Nipype graphs for each config #3 [ & 📝 "Matrix of data [×] pipeline configs coloured by status is automatically added to PRs." #7 ]
   b. Verify list of expected-outputs from prior/benchmark runs (smoke-regression)
4. Lightweight Regression (Heatmap of comparison)
   a. Verify similarity of pipeline outputs on downsampled data to prior/benchmark runs
5. Regression (Heatmap of comparison)
   a. Verify similarity of pipeline outputs on relevant data to prior/benchmark runs
6. Resource (Matrix of all nodes, cells coloured by CPU, RAM, and external consistency)
   a. Internally monitor performance and record it by saving callback.log files
   b. Externally monitor resource consumption/runtime as failsafe on the former
7. Unit-Node (Matrix of all nodes, cells coloured by: skipped, passed, failed)
   a. Starting from checkpoint data, verify the completion of new/modified nodes

Configurations

1. All packaged with C-PAC

Datasets (# and source)

1. Human (~10 random participants from the following)
   a. ADHD200
   • IDs?
     1. 1019436
     2. 2014113
     3. 3154996
     4. 3699991
     5. 3884955
   • "Standard"
     b. HBN
   • IDs?
     1. EPI field maps:
        a. NDARAA504CRN
        b. NDARAA947ZG5
        c. NDARAB348EWR
        d. NDARAB458VK9
     2. Gradient echo/“Phase-diff” field maps:
        a. NDARAD481FXF
        b. NDARAV894XWD
        c. NDARCA186WGH
   • Long scans
   • EPI Field maps for blip-up/blip-down
     c. NKI-RS
   • IDs?
   • Phase difference maps for FSL FUGUE
     d. CoRR HNU
   • IDs
     1. 0025453
     2. 0025428
     3. 0025429
     4. 0025452
     5. 0025448
2. NHP (5 random participants from PRIME)
   a. IDs?
3. Rodent (2–3 participants from what dataset?)
   a. IDs?
   […]

Workflow

1. Open PR, which triggers tests 1–4
2. Once GH actions tests pass:
   a. Request code review from development team
   b. Manually launch script that starts tests 2, 5–7
3. Once tests 5–7 complete:
   a. Run script to compile figures and draft GitHub comment for PR
   b. Post comments to PR
   c. Request results review from Steve
   d. Request complete review from Greg and/or Mike

― Testing Meeting Agenda#Testing Desiderata

Acceptance criteria

The described tests are automated (as possible)

Alternatives

No response

Additional context

No response