ENH: Allow data and output directories of tutorials to be specified (#125)

* ENH: Allow data and output directories of tutorials to be specified

These directories now consume tens of gigabytes.   These changes
allow the source and data directories to be separate.

Also allows testing to wipe the source directory without having to
re-download data.

* STYLE: Fix line lengths etc

Author: aylward

.gitattributes

@@ -2,3 +2,4 @@
 *.mha filter=lfs diff=lfs merge=lfs -text
 docs/assets/*.gif filter=lfs diff=lfs merge=lfs -text
 docs/assets/*.png filter=lfs diff=lfs merge=lfs -text
+tests/baselines/**/*.png filter=lfs diff=lfs merge=lfs -text

.github/workflows/README.md

@@ -163,7 +163,7 @@ GPU tests require self-hosted runners with:
 **Option 3: Run Locally**
 ```bash
 # Install with CUDA + PhysicsNeMo (matches the self-hosted GPU runner).
-# Requires Python >= 3.11 (nvidia-physicsnemo does not support 3.10).
+# Requires Python 3.11-3.13 (the range nvidia-physicsnemo supports).
 uv pip install -e ".[test,cuda13,physicsnemo]"
 
 # Run GPU tests
@@ -237,7 +237,7 @@ pytest tests/ -m "unit and not requires_gpu" --cov=physiotwin4d
 ### GPU Tests
 ```bash
 # Install with CUDA + PhysicsNeMo (matches the self-hosted GPU runner).
-# Requires Python >= 3.11 (nvidia-physicsnemo does not support 3.10).
+# Requires Python 3.11-3.13 (the range nvidia-physicsnemo supports).
 uv pip install -e ".[test,cuda13,physicsnemo]"
 
 # Run GPU tests

.github/workflows/ci.yml

@@ -35,7 +35,7 @@ jobs:
       fail-fast: false
       matrix:
         os: [ubuntu-latest, windows-latest]
-        python-version: ['3.11', '3.12']
+        python-version: ['3.11', '3.12', '3.13']
 
     steps:
     - name: Checkout code
@@ -272,7 +272,6 @@ jobs:
     timeout-minutes: 30
     # Only run GPU tests on manual trigger or if 'run-gpu-tests' label is present
     if: github.event_name == 'workflow_dispatch' || contains(github.event.pull_request.labels.*.name, 'run-gpu-tests')
-    continue-on-error: true
 
     steps:
     - name: Checkout code
@@ -281,15 +280,31 @@ jobs:
         lfs: true
 
     - name: Create venv in RUNNER_TEMP
-      # Python 3.11 is required because the [physicsnemo] extra pulls in
-      # nvidia-physicsnemo, which requires Python >= 3.11.
+      # The oldest interpreter the project supports, so this job is the one
+      # that catches a regression at the floor. nightly-health.yml runs the
+      # newest (3.13) for the opposite reason. Both are on the runner.
       run: |
         & "C:\Program Files\Python311\python.exe" -m venv "$env:RUNNER_TEMP\physiotwin4d-venv"
         echo "$env:RUNNER_TEMP\physiotwin4d-venv\Scripts" >> $env:GITHUB_PATH
 
     - name: Check GPU availability
       run: nvidia-smi
 
+    - name: Check nvcc availability
+      # torch-scatter compiles CUDA kernels from source when no matching
+      # pre-built wheel exists. Without nvcc on PATH the build silently falls
+      # back to a CPU-only extension that only fails later, at test time, with
+      # an opaque scatter error. Fail here instead.
+      run: |
+        $nvcc = Get-Command nvcc -ErrorAction SilentlyContinue
+        if (-not $nvcc) {
+            Write-Error "nvcc not found on PATH; torch-scatter would build CPU-only"
+            exit 1
+        }
+        Write-Output "nvcc found at $($nvcc.Source)"
+        nvcc --version
+        if ($LASTEXITCODE -ne 0) { exit $LASTEXITCODE }
+
     - name: Cache uv packages
       uses: actions/cache@v4
       with:
@@ -308,7 +323,9 @@ jobs:
       # requires_physicsnemo-marked tests can run on this GPU runner.
       # Invoke via python -m uv so uv targets the active venv interpreter.
       run: |
-        python -m uv pip install -e ".[test,cuda13,physicsnemo]"
+        python -m uv pip install -e ".[cuda13]"
+        if ($LASTEXITCODE -ne 0) { exit $LASTEXITCODE }
+        python -m uv pip install -e ".[test,cuda13,physicsnemo]" --no-build-isolation-package torch-scatter
 
     - name: Assert CUDA is accessible
       run: |
@@ -334,14 +351,26 @@ jobs:
         pip list
 
     - name: Run GPU tests
-      # External self-hosted GPU runner: enable every opt-in bucket via --run-all.
-      # Tests whose host requirements (e.g. a licensed Simpleware install)
-      # aren't met on the runner will runtime-skip cleanly via their
-      # internal availability guards.
+      # A pre-merge gate, so it runs the GPU buckets only and finishes inside
+      # timeout-minutes. The exhaustive --run-all sweep, tutorials included,
+      # belongs to nightly-health.yml: running it here too would duplicate six
+      # hours of work on this same runner and could never fit the timeout.
+      #
+      # --require-tutorial-data turns missing data into a failure rather than a
+      # skip, so this job cannot report green having run nothing.
       run: |
-        pytest tests/ -v --run-all --cov=physiotwin4d --cov-report=xml --cov-report=term --cov-report=html
+        pytest tests/ -v --run-gpu --run-physicsnemo --require-tutorial-data --max-test-seconds=600 --timeout=1800 --cov=physiotwin4d --cov-report=xml --cov-report=term --cov-report=html
       env:
         CUDA_VISIBLE_DEVICES: 0
+        # The datasets, the results and the trained networks live outside the
+        # checkout, which actions/checkout wipes on every run. Each root has a
+        # "test" subtree that this suite reads and writes, so a CI run never
+        # touches a full run's files. Unset, each falls back to its in-repo
+        # default; see data/README.md. These are the same paths
+        # nightly-health.yml uses, because both jobs run on this same runner.
+        PHYSIOTWIN_INPUT_DATA_DIR: D:\PhysioTwin4D\nightly-runner\data
+        PHYSIOTWIN_OUTPUT_DATA_DIR: D:\PhysioTwin4D\nightly-runner\output
+        PHYSIOTWIN_WEIGHTS_DIR: D:\PhysioTwin4D\nightly-runner\network_weights
 
     - name: Upload coverage to Codecov
       uses: codecov/codecov-action@v4

.github/workflows/nightly-health.yml

@@ -42,7 +42,8 @@ jobs:
 
     outputs:
       # Captures the pytest step's actual outcome (success / failure / skipped)
-      # even though the job itself is not failed by a test failure.
+      # for the dashboard. The job itself is failed by the gate step at the end,
+      # after the artifacts have been uploaded.
       test-outcome: ${{ steps.run-tests.outcome }}
 
     steps:
@@ -70,8 +71,9 @@ jobs:
           if ($LASTEXITCODE -ne 0) { exit $LASTEXITCODE }
 
       - name: Create venv in RUNNER_TEMP
-        # Python 3.13 is required because the [physicsnemo] extra pulls in
-        # nvidia-physicsnemo, which requires Python >= 3.11.
+        # The newest interpreter the project supports, so this job is the one
+        # that catches a dependency shipping no wheel for it. ci.yml runs the
+        # floor (3.11) for the opposite reason. Both are on the runner.
         run: |
           & "C:\actions-runner\python\WPy64-3.13.12.0\python\python.exe" -m venv "$env:RUNNER_TEMP\physiotwin4d-venv"
           echo "$env:RUNNER_TEMP\physiotwin4d-venv\Scripts" >> $env:GITHUB_PATH
@@ -157,14 +159,33 @@ jobs:
         # continue-on-error keeps the job running so artifacts are always uploaded.
         # The step outcome (success/failure) is still captured and passed downstream.
         continue-on-error: true
+        # --max-test-seconds fails a test that finishes but took too long, so the
+        # overrun is reported with its duration and the rest of the suite still
+        # runs. --timeout raises the pytest-timeout backstop above it: on
+        # Windows that backstop can only kill the whole process, which would
+        # take the JUnit XML and every other result with it, so it is left to
+        # catch a genuine hang and nothing else. Both numbers are provisional
+        # until the per-tutorial timings are measured.
         run: |
-          pytest tests/ -v --run-all `
+          pytest tests/ -v --run-all --require-tutorial-data `
+            --max-test-seconds=900 `
+            --timeout=3600 `
             --cov=physiotwin4d `
             --cov-report=xml `
             --cov-report=json `
             --junitxml=test-results.xml
         env:
           CUDA_VISIBLE_DEVICES: 0
+          # The datasets, the results and the trained networks live on the
+          # runner's own disk rather than in the checkout, which
+          # actions/checkout wipes every run. Each root has a "test" subtree
+          # that this suite reads and writes, so a nightly run never touches a
+          # full run's files, and the downsampled subsets the fixtures build
+          # under <input>/test survive between runs instead of being rebuilt.
+          # Unset, each falls back to its in-repo default; see data/README.md.
+          PHYSIOTWIN_INPUT_DATA_DIR: D:\PhysioTwin4D\nightly-runner\data
+          PHYSIOTWIN_OUTPUT_DATA_DIR: D:\PhysioTwin4D\nightly-runner\output
+          PHYSIOTWIN_WEIGHTS_DIR: D:\PhysioTwin4D\nightly-runner\network_weights
 
       - name: Upload test results
         uses: actions/upload-artifact@v6
@@ -177,6 +198,18 @@ jobs:
             coverage.json
           retention-days: 90
 
+      - name: Fail the job if the test suite did not pass
+        # The pytest step is continue-on-error so that the artifacts and the
+        # dashboard are produced whatever happened. Without this gate the job
+        # would report success even when tests failed or the step timed out,
+        # and the workflow status badge would say green while nothing passed.
+        # build-dashboard runs on if: always(), so it still gets its inputs.
+        if: steps.run-tests.outcome != 'success'
+        run: |
+          Write-Output "Health test suite outcome: ${{ steps.run-tests.outcome }}"
+          Write-Output "See the health-test-results artifact for the JUnit XML."
+          exit 1
+
   # ──────────────────────────────────────────────────────────────────────────
   # 2. Build the HTML dashboard from test results (runs even if tests failed)
   # ──────────────────────────────────────────────────────────────────────────

data/README.md

@@ -26,6 +26,39 @@ downloader — DIR-Lab distributes each case individually and may require
 registration, so it must be obtained manually; see
 [DirLab-4DCT/README.md](DirLab-4DCT/README.md).
 
+## Keeping the Data Outside the Clone
+
+The tutorials resolve three roots through `ParametersBase` in
+[`tutorials/parameters_base.py`](../tutorials/parameters_base.py), each
+defaulting to its location in the clone and each overridable by environment
+variable. Point them elsewhere when the data should outlive the checkout — on a
+CI runner, for instance, where every run starts from a fresh working tree:
+
+| Variable | Default | Holds |
+| --- | --- | --- |
+| `PHYSIOTWIN_INPUT_DATA_DIR` | `<repo>/data` | The datasets in this directory |
+| `PHYSIOTWIN_OUTPUT_DATA_DIR` | `<repo>/tutorials/output` | What the tutorials write |
+| `PHYSIOTWIN_WEIGHTS_DIR` | `<repo>/tutorials/network_weights` | Networks the tutorials train |
+
+Each root has a `test` subdirectory used when a tutorial runs under
+`PHYSIOTWIN_RUNNING_AS_TEST`, so a test run reads the small downsampled subsets
+and writes beside them rather than touching a full run's datasets, results, or
+checkpoints. The subsets under `<input root>/test` are built on demand by the
+fixtures in `tests/conftest.py`; putting that root outside the clone means they
+survive a checkout and are built once rather than every run.
+
+The layout under an overridden input root is the same as here:
+
+```text
+<PHYSIOTWIN_INPUT_DATA_DIR>/
+  DirLab-4DCT/             Case1Pack_T00.mha, ...
+  Duke-Heart-4DLabelmaps/  pm0027/*_labelmap.nii.gz, *_landmark.mrk.json
+  Chest-CT/                Chest-CT.mha
+  KCL-Heart-Model/         average_mesh.vtk, input_meshes/
+  Slicer-Heart-CT/         slice_000.mha, ...
+  test/                    built by the pytest fixtures
+```
+
 ## Notes
 
 - Always cite the original data source in publications — see each dataset's

data/test/.gitignore

@@ -1,2 +1,5 @@
-slicer_heart
-slicer_heart_small
+# Everything here is a pytest-built cache; only this file and the README are
+# tracked. See README.md for which fixture builds which directory.
+*
+!.gitignore
+!README.md

data/test/README.md

@@ -4,9 +4,14 @@ This directory is **automatically managed by the pytest infrastructure**
 (`tests/conftest.py`) — it is a cache, not a dataset you download or
 maintain by hand. It holds data used to run the unit test suite.
 
-This data is **not** used by the workflows, tutorials, or CLIs of the
-PhysioTwin4D library; those consume the datasets documented in
-[`data/README.md`](../README.md) instead.
+It is the `test` subdirectory of the input data root, so setting
+`PHYSIOTWIN_INPUT_DATA_DIR` moves it too; see
+[`data/README.md`](../README.md#keeping-the-data-outside-the-clone). The paths
+below are written relative to that root, which defaults to `data/`.
+
+The workflows, tutorials, and CLIs of the PhysioTwin4D library consume the
+full datasets documented in [`data/README.md`](../README.md). They read the
+subsets here only when run as tests, under `PHYSIOTWIN_RUNNING_AS_TEST`.
 
 ## What Lives Here
 
@@ -16,10 +21,30 @@ PhysioTwin4D library; those consume the datasets documented in
 - `slicer_heart_small/` — the same phases downsampled to 1.5x1.5x1.5 mm,
   used by tests that need a smaller/faster image (labelmaps and
   transforms computed from this data are cached here too).
-
-Both subdirectories are created on demand by `tests/conftest.py` fixtures
-the first time a test needs them, and are `.gitignore`d — do not commit
-their contents.
+- `KCL-Heart-Model/` — downloaded by the `download_kcl_heart_model` fixture.
+- `DirLab-4DCT/` — a few cases from `<input root>/DirLab-4DCT`, downsampled to
+  3 mm by the `dirlab_test_data` fixture.
+- `Duke-Heart-4DLabelmaps/` — a few cases from
+  `<input root>/Duke-Heart-4DLabelmaps`, their labelmaps downsampled to 2 mm
+  nearest-neighbour by the `duke_heart_test_data` fixture.
+- `Chest-CT/` — `<input root>/Chest-CT` downsampled to 3 mm by the
+  `chest_ct_test_data` fixture.
+
+Here `<input root>` is whatever `PHYSIOTWIN_INPUT_DATA_DIR` names, defaulting to
+the `data/` directory of the clone — so each subset is built from the full
+dataset alongside it, wherever that root has been pointed.
+
+Every subdirectory is created on demand by `tests/conftest.py` fixtures
+the first time a test needs them, and is `.gitignore`d — do not commit
+their contents. The subsets derived from another dataset are only built
+when that source dataset is present under the input root; otherwise the tests
+that need them skip, or fail if `--require-tutorial-data` was passed.
+
+The tutorials read these directories rather than the full datasets whenever
+`PHYSIOTWIN_RUNNING_AS_TEST` is set, and write to the matching `test` subtree of
+the output and weights roots — `tutorials/output/test/` and
+`tutorials/network_weights/test/` by default. A test run therefore never reads
+or overwrites the datasets, results, or trained checkpoints of a full run.
 
 ## Regenerating
 

docs/faq.rst

@@ -68,10 +68,12 @@ torchvision, and torchaudio are sourced from
 What Python version is required?
 ---------------------------------
 
-Python 3.10, 3.11 and 3.12 are supported.
+Python 3.11, 3.12 and 3.13 are supported.
 
-The one exception is the optional ``[physicsnemo]`` extra: ``nvidia-physicsnemo``
-requires Python >= 3.11, so the AI-surrogate tutorials need 3.11 or 3.12.
+The floor is set by the optional ``[physicsnemo]`` extra: ``nvidia-physicsnemo``
+supports >= 3.11, < 3.14, and the AI-surrogate tutorials need it. The rest of
+the library would run on 3.10, but declaring 3.10 would promise an install
+that cannot resolve that extra.
 
 Usage Questions
 ===============

docs/installation.rst

@@ -10,7 +10,7 @@ Prerequisites
 System Requirements
 -------------------
 
-* **Python**: 3.10, 3.11, or 3.12
+* **Python**: 3.11, 3.12, or 3.13
 * **GPU**: NVIDIA GPU with CUDA 13 — required for full capability and best performance; a CPU-only PyPI installation is a supported fallback, but it is slow, emits a runtime warning, and cannot run the AI-surrogate workflows
 * **RAM**: 16GB minimum (32GB+ recommended for large datasets)
 * **Storage**: 10GB+ for package and model weights