Skip to content

Fix session filter dropping sessionless datasets; add rbc-all integration test#290

Merged
nx10 merged 12 commits into
mainfrom
fix/session-null-filter-integration-test
Apr 10, 2026
Merged

Fix session filter dropping sessionless datasets; add rbc-all integration test#290
nx10 merged 12 commits into
mainfrom
fix/session-null-filter-integration-test

Conversation

@nx10
Copy link
Copy Markdown
Contributor

@nx10 nx10 commented Apr 9, 2026

Closes #289

Summary

Integration test that runs rbc all and the four individual subcommands (anatomical/functional/metrics/qc) on tests/data/ds000001, then verifies both produce the expected derivative files and match each other.

Writing this test exposed several pre-existing bugs:

  • Session filter drops sessionless datasets: pl.col("ses") != "longitudinal" evaluates to null (falsy) when ses is None. Fixed with ne_missing() in all three cross-sectional orchestration modules.
  • resolve_metrics / resolve_qc label mismatch: Resolve functions queried raw regressor names (e.g. "36-parameter") but exports use bids_safe_label() (e.g. "36parameter"). Fixed by applying bids_safe_label() in both resolve functions.
  • resolve_qc duplicate match on template_bold: Query matched both the untagged preproc bold and per-regressor preproc bolds. Fixed with extra={"reg": False} to exclude files with a reg entity.

Also pins uv version in the full test workflow to work around a raw.githubusercontent.com network block on the self-hosted runner.

Test plan

  • uv run pytest tests/integration/test_all.py -m slow --runner docker passes end-to-end
  • Existing unit tests still pass (uv run pytest -m unit)
  • Full test suite passes on self-hosted runner

@github-actions
Copy link
Copy Markdown

github-actions Bot commented Apr 9, 2026

Coverage

Coverage Report
FileStmtsMissCoverMissing
rbc
   __init__.py10100% 
   context.py25868%70, 75–77, 79–80, 93–94
   metadata.py560100% 
rbc/bids
   __init__.py90100% 
   _schema.py585499%776, 782, 790, 1101
   anatomical.py24387%44, 47–48
   builder.py72790%233–235, 362, 364–365, 386
   functional.py42588%46–49, 84
   longitudinal.py250100% 
   metrics.py23195%44
   qc.py170100% 
   query.py674237%103–107, 121–125, 127–128, 130–135, 137, 139, 153, 159–165, 198, 207, 209–216, 225, 257, 266–267
   session.py470100% 
rbc/cli
   __init__.py10100% 
   all.py49295%84, 109
   anatomical.py25292%47, 63
   base.py71987%56, 62, 123–125, 131–133, 156
   functional.py33293%67, 87
   longitudinal.py27292%46, 63
   main.py420100% 
   metrics.py42295%77, 95
   qc.py25292%45, 61
rbc/core
   __init__.py30100% 
   common.py261253%43–45, 62–70
   fileops.py27485%69–72
   fsl2itk.py420100% 
   nifti.py192597%236–237, 244–245, 524
   niwrap.py56198%58
rbc/core/anatomical
   __init__.py40100% 
   registration.py15473%59, 151, 166, 183
   segmentation.py24866%64, 74–76, 92, 114, 125, 141
rbc/core/functional
   __init__.py130100% 
   coregistration.py7271%44, 55
   despiking.py7357%32, 36–37
   distortion.py1304069%269–271, 321, 324, 332–335, 341–346, 349, 352–353, 356, 365–369, 375–376, 387–388, 391, 397, 443–444, 447, 455, 461, 470–471, 474, 484, 491
   erosion.py32196%50
   initialization.py9455%35, 42–43, 63
   masking.py342526%53, 55–56, 58–59, 62–65, 69, 91, 134, 183, 197, 208, 223, 233, 249, 258, 271, 285, 296, 306, 319, 328
   motion.py573735%62, 64–67, 69, 71, 73, 76–77, 83–84, 86–87, 95–97, 99, 102, 105, 107, 124–125, 135–138, 159, 169, 171–172, 175, 177–179, 181, 183
   nuisance.py816025%78, 80–85, 87, 89–90, 93, 96–98, 100–102, 104–110, 112–116, 118, 163, 165, 167, 170–171, 173–175, 178, 181–182, 185–187, 190–193, 197, 203–205, 207, 235, 243, 269, 278, 307, 316, 322
   regressors.py89693%163, 193, 325–328
   resampling.py544320%37–42, 74–76, 78, 80–81, 85–87, 90, 93, 105, 107–108, 112, 114, 150–152, 154–155, 159, 161–162, 167–169, 173–174, 177, 180, 187, 199, 201–202, 207, 209
   timing.py161131%46–47, 49–53, 58, 60, 66–67
rbc/core/longitudinal
   __init__.py10100% 
   transform.py46784%106–107, 165–168, 170
rbc/core/metrics
   __init__.py30100% 
   alff.py90198%265
   reho.py660100% 
   smoothing.py7357%36, 42–43
   standardization.py271159%64, 66–68, 70, 72–75, 77–78
   timeseries.py57198%120
rbc/core/qc
   __init__.py60100% 
   dvars.py260100% 
   motion.py310100% 
   registration.py410100% 
   xcp.py410100% 
rbc/orchestration
   __init__.py32293%90, 93
   all.py41978%131–132, 137, 145, 153–154, 171, 173–174
   anatomical.py372240%49–52, 57–58, 60–63, 85–87, 89–90, 94, 101, 104, 109–110, 116, 118
   functional.py411270%124–126, 128–129, 133, 139, 142, 147–148, 157, 159
   longitudinal.py56198%150
   metrics.py463034%32–35, 38–40, 67, 75–76, 100–102, 104–106, 110, 118–121, 123–124, 130–132, 137, 145, 152, 154
   qc.py340100% 
rbc/workflows
   __init__.py100100% 
   anatomical.py471959%90–93, 97–102, 107, 177–180, 182–184, 188
   functional.py985642%113–114, 122, 127–128, 136, 207–208, 211–212, 215–216, 219, 222–225, 235–237, 247–248, 251, 254, 257–258, 266–267, 274–275, 282–283, 290–291, 294–296, 299–302, 314–315, 327, 329–332, 335–336, 345–346, 354, 361, 430, 436
   metrics.py411856%80, 83–84, 93–94, 99–102, 105–108, 112–115, 119
   qc.py553438%88–89, 92–93, 96, 99, 102–107, 110, 119–121, 125–128, 130–134, 136–137, 139–141, 144, 161, 167, 169
rbc_resources
   __init__.py380100% 
TOTAL324458382% 

Tests Skipped Failures Errors Time
730 0 💤 0 ❌ 0 🔥 9.822s ⏱️

@nx10 nx10 force-pushed the fix/session-null-filter-integration-test branch from 9dce206 to c94b0aa Compare April 9, 2026 18:22
nx10 added 8 commits April 9, 2026 20:55
…gration test (#289)

`pl.col("ses") != "longitudinal"` evaluates to null (falsy) when ses is
None, silently skipping every row in datasets without session labels.
Replace with `ne_missing()` in all three cross-sectional orchestration
modules so null sessions are preserved.

Add an integration test that runs `rbc all` on tests/data/ds000001 with
Docker and verifies key derivative files are produced.
…al stages

Tests both invocation styles (rbc all vs anatomical/functional/metrics/qc
in sequence) and verifies they produce the same set of derivative files.
Uses session-scoped fixtures so each pipeline variant runs only once.
The functional/metrics/qc stages need to discover derivatives from
previous stages via load_table, so the output dir must be included
in the input dirs alongside the raw BIDS dataset.
resolve_metrics and resolve_qc queried extra entities with raw regressor
names (e.g. "36-parameter") but export_functional saves them through
bids_safe_label (e.g. "36parameter"). This mismatch caused
FileNotFoundError when running rbc metrics or rbc qc standalone.
Copies ds000001 and truncates the BOLD timeseries before running the
pipeline, cutting functional processing time significantly while still
exercising the full code path (motion correction, nuisance regression,
metrics, QC).
25 volumes was too few: after discarding 2 TRs, only 23 timepoints
remained, which is fewer than the ~36 columns in the 36-parameter
nuisance regressor model, causing 3dTproject to fail.
Remove the volume truncation optimization so we can shake out all bugs
with the real data first. Can revisit truncation once the pipeline
passes end-to-end.
The sequential test now copies anat derivatives from the rbc-all run
instead of re-running brain extraction and registration, saving ~30-40
min of CI time. The disk round-trip handoff between functional, metrics,
and QC stages is still fully tested.
@nx10 nx10 force-pushed the fix/session-null-filter-integration-test branch from f793cc7 to 4404b9a Compare April 10, 2026 00:57
nx10 added 4 commits April 9, 2026 21:01
The BIDS path builder formats integer run entities as "run-1" (not
"run-01"), so update the test assertion accordingly.

resolve_qc's template_bold query matched both the untagged preproc bold
and per-regressor preproc bolds. Add without=["reg"] to disambiguate.
Use extra={"reg": False} instead of without=["reg"] since reg is an
extra entity (not a standard BIDS column). Also relax the QC glob
pattern since entity ordering puts reg before desc in the filename.
compute_alff, compute_reho, and compute_timeseries default to writing
outputs next to their input files. When the standalone metrics/QC
orchestration resolves inputs from the derivatives dir, these
intermediates leak into the output alongside the BIDS-named exports.

Use generate_exec_folder() to create a dedicated work directory under
the niwrap runner's data_dir, and pass explicit out_file/out_dir paths
so intermediates stay out of the derivatives tree.
@nx10 nx10 merged commit 8837200 into main Apr 10, 2026
8 checks passed
@nx10 nx10 deleted the fix/session-null-filter-integration-test branch April 10, 2026 06:48
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Add integration test for rbc all pipeline stage handoff

1 participant