Merge pull request #34 from JosephBrunet/improve-slurm-launcher

Improve slurm launcher

Author: JosephBrunet

docs/advanced/index.md

@@ -6,6 +6,7 @@ Pages:
 
 - [Transmural Analysis](./cardio-analysis.md): Extract transmural profiles from angle and FA maps with `cardio-analysis`.
 - [Streamlines](./streamlines.md): Generate and visualize 3D streamlines with `cardio-generate-streamlines` and `cardio-visualize-streamlines`.
+- [SLURM Launcher](./slurm-launcher.md): Submit chunked `cardio-tensor` jobs on HPC clusters.
 
 See also:
 

docs/advanced/slurm-launcher.md

@@ -0,0 +1,63 @@
+# SLURM Launcher
+
+Use `cardio-tensor-slurm` to submit chunked orientation jobs as SLURM array tasks.
+
+## What It Does
+
+- Splits a dataset into chunks.
+- Generates a `.slurm` script.
+- Submits one or more SLURM arrays with `sbatch`.
+- Monitors output progress.
+
+The launcher calls `cardio-tensor` on each task with:
+
+```bash
+cardio-tensor <conf_file> --start_index <start> --end_index <end>
+```
+
+## Basic Usage
+
+```bash
+cardio-tensor-slurm path/to/parameters.conf
+```
+
+## Useful Options
+
+```bash
+cardio-tensor-slurm path/to/parameters.conf \
+  --start_index 0 \                     # First slice index (inclusive)
+  --end_index 5000 \                    # Last slice index (exclusive)
+  --chunk_size 100 \                    # Slices processed per SLURM task
+  --time_limit 4:00:00 \                # Wall-time limit for each task
+  --cpus_per_task 8 \                   # CPU cores requested per task
+  --mem_gb 64 \                         # Memory requested per task (GB)
+  --array_parallel 50 \                 # Max concurrent tasks in the SLURM array
+  --partition nice \                    # Optional partition/queue name
+  --log_dir /path/to/logs \             # Where SLURM stdout/stderr logs are written
+  --submit_dir /path/to/slurm_scripts   # Where generated .slurm scripts are saved
+```
+
+Other useful flags:
+
+- `--no_monitor`: submit jobs and return immediately (do not watch output progress).
+- `--dry_run`: generate the `.slurm` script and print `sbatch` command without submitting.
+
+Run help for the full list:
+
+```bash
+cardio-tensor-slurm -h
+```
+
+## Troubleshooting
+
+- **`sbatch` fails**  
+  Verify partition/account/time/memory policy on your cluster.
+
+- **Job fails before running `cardio-tensor`**  
+  Check logs in `OUTPUT_PATH/slurm/log/`.
+
+- **No progress in monitor**  
+  Confirm output folder permissions and that tasks can write outputs.
+
+- **Want to debug without submitting jobs**  
+  Use `--dry_run`.

docs/reference/cli.md

@@ -9,6 +9,10 @@ Compute myocyte orientation from a 3D volume using a configuration file.
 - `cardio-tensor`
   Computes structure tensor, helix/transverse angle, FA, and eigenvectors.
 
+- `cardio-tensor-slurm`
+  Submits chunked `cardio-tensor` runs as SLURM array jobs.
+  See [SLURM Launcher](../advanced/slurm-launcher.md).
+
 See the [example](../getting-started/examples.md) to get started.
 
 ---

mkdocs.yml

@@ -34,6 +34,7 @@ nav:
     - advanced/index.md
     - Transmural Analysis: advanced/cardio-analysis.md
     - Streamlines: advanced/streamlines.md
+    - SLURM Launcher: advanced/slurm-launcher.md
   - Reference:
     - reference/index.md
     - CLI Commands: reference/cli.md

paper/paper.md

@@ -59,7 +59,7 @@ For non-diffusion imaging modalities, such as micro-CT [@reichardt_fiber_2020],
 
 Cardiotensor addresses this gap by providing an open-source Python package specifically tailored to structure tensor analysis of large cardiac volumes. Rather than relying on diffusion modeling, cardiotensor infers tissue orientation directly from image intensity gradients, making it applicable across a wide range of modalities and scales. Previous studies have demonstrated strong agreement between structure tensor–based orientation and DT-MRI–derived metrics when applied to the same human hearts [@teh_validation_2016]. The package supports full pipelines from raw image stacks to myocyte orientation maps and tractography. Its architecture is optimized for large datasets, using chunked and parallel processing suitable for high-performance computing environments. The overall processing workflow, from input volumes to orientation maps and tractography, is summarized in \autoref{fig:pipeline}.
 
-Cardiotensor has already been successfully applied in published work to characterize 3D cardiomyocyte architecture in healthy and diseased human hearts using synchrotron tomography [@brunet_multidimensional_2024] to datasets over a terabyte in size. While cardiotensor was conceived for cardiac imaging, the package is modality‑ and tissue‑agnostic. Any volumetric dataset exhibiting coherent fibrous microstructure can be analyzed, including brain white matter, skeletal muscle, and tendon. This generality makes the library useful for both cardiovascular and broader anatomical or histological studies.
+Cardiotensor has already been successfully applied in published work to characterize 3D cardiomyocyte architecture in healthy and diseased human hearts using synchrotron tomography [@brunet_multidimensional_2024]. In practice, cardiotensor has been applied to whole-heart HiP-CT datasets exceeding 1–5 TB (teravoxel scale). While cardiotensor was conceived for cardiac imaging, the package is modality‑ and tissue‑agnostic. Any volumetric dataset exhibiting coherent fibrous microstructure can be analyzed, including brain white matter, skeletal muscle, and tendon. This generality makes the library useful for both cardiovascular and broader anatomical or histological studies.
 
 ![Cardiotensor pipeline for 3D cardiac orientation analysis and tractography.
 (a) Input whole‑ or partial‑heart volume with optional myocardial mask.
@@ -70,7 +70,7 @@ The third eigenvector field (smallest eigenvalue) is visualized as arrows color
 
 ## Implementation
 
-Cardiotensor is implemented in Python and designed to efficiently process very large 3D cardiac imaging datasets. It relies primarily on NumPy [@van_der_walt_numpy_2011] for numerical computation, with I/O accelerated by tifffile [@gohlke_cgohlketifffile_2025], Glymur [@evans_quintusdiasglymur_2025], and OpenCV [@bradski_opencv_2000]. Dask [@rocklin_dask_2015] is used exclusively to parallelize file reading, while the core computations rely on Python’s multiprocessing module for local parallelism. The package builds on the structure-tensor library [@jeppesen_quantifying_2021] to calculate the 3D structure tensor and eigenvector decomposition.
+Cardiotensor is implemented in Python and designed to efficiently process terabyte-scale 3D cardiac imaging datasets. It relies primarily on NumPy [@van_der_walt_numpy_2011] for numerical computation, with I/O accelerated by tifffile [@gohlke_cgohlketifffile_2025], Glymur [@evans_quintusdiasglymur_2025], and OpenCV [@bradski_opencv_2000]. Dask [@rocklin_dask_2015] is used exclusively to parallelize file reading, while the core computations rely on Python’s multiprocessing module for local parallelism. The package builds on the structure-tensor library [@jeppesen_quantifying_2021] to calculate the 3D structure tensor and eigenvector decomposition.
 
 The package supports multiple use cases: