Ablation study

[Luugaaa]

In this issue, we report the results of our ablation study. This ablation is run on open-ms FLAIR modality, as segmenting MS lesion is a difficult task we believe this will highlight the importance of each part of our method. The training configuration is the same as for other open-ms training.

[Luugaaa]

Everything is trained with 3 folds.

We have 2 ablation experiments.

1. we use our method without auglab. This way we can directly compare with baseline and synthseg. For this method the base is the baseline. The different exps are:

• • baseline
• • baseline + k-means + label remapping with noise (no texture preserved) ===> very similar to synthseg EM
• • baseline + k means + label remapping with noise (no texture preserved) + voronoi subparcellation ===> a bit like an improved version of synthseg EM
• • baseline + k means + label remapping with noise (no texture preserved) + voronoi subparcellation + histogram remapping

2. we use our method with auglab. This way we can compare with auglab and confirm our ablation is still valid in when our method is mixed with auglab. For this method, the base is auglab. The different exps are :

• • auglab
• • auglab + k-means + label remapping with noise (no texture preserved)
• • auglab + k means + label remapping with noise (no texture preserved) + voronoi subparcellation
• • auglab + k means + label remapping with noise (no texture preserved) + voronoi subparcellation + histogram remapping

Baseline-based ablation:

Dice results:

HD95 results:

Auglab based -ablation

Dice results:

HD95 results:

Conclusions:

The histogram remapping is responsible for only a few percent increase. It's that texture preservation is important and likely explain why we surpass other methods.
We can also conclude that the main driver of contrast agnosticism is the remapping of regions of the volumes (k-means + labels remapping), it seems voronoi help too.

[Luugaaa]

To confirm that histogram remapping does in fact perserve texture, we ran a texture analysis study, also on open-ms flair modality.

1. Why "noblur"

Two things are compared: 5 augmentation methods (palette, auglab_default, synthseg_em, synthseg_noem, v26_6_2_noisefill_v2), each applied to real open-ms source volumes (FLAIR, T1w). Each method's config can optionally apply resolution-degrading blur as part of its augmentation. Early on we found SynthSeg's generator always applies a σ=0.5-voxel blur even when its data_res/blur_range knobs are set to "no blur" (functional.py:551 hard-codes this when data_res==atlas_res) — the actual switch is "apply_resolution": false. We use the noblur configs (data/configs_noblur_nospatial/, confirmed: apply_resolution=False for both SynthSeg variants, p_rotation=p_scaling=0 for all 5 methods) so that what we measure is the per-region intensity fill, not blur or spatial jitter contributing a shared artifact to every method. This was necessary because blur was found to manufacture a spurious "shared boundary" correlation between source and generated images independent of any real texture preservation.

2. Preprocessing (exactly, in order)

1. Generation: source volume → z-scored ((x - mean)/std, per-volume) → fed through the method's transform (e.g. PALETTE's K-means/Voronoi + signed affine remap, or SynthSeg's GMM paint) → 10 random variant volumes saved per (method, subject, contrast).
2. ROI: foreground = intensity > 10% of that source volume's 99th percentile (FOREGROUND_FRAC=0.10) — a crude whole-brain mask, no anatomical knowledge. (We verified: 100% of MS lesion voxels fall inside it, since lesions are FLAIR-hyperintense — it is not a lesion-excluding mask.)
3. Erosion: the foreground mask is eroded by 3 voxels (--erode-iters 3, a 3×3×3 min-pool applied 3 times) to drop a thin outer shell — this is the fix for the "boundary given for free" artifact (the shell near any conditioning-label boundary trivially matches across methods; interior voxels must genuinely derive from the source). Foreground survives this at full power (30/30 subjects, ≥200 interior voxels each); the lesion ROI does not (kills most lesions) — which is one of the two reasons lesion is no longer a reported ROI.
4. This interior mask is then used to compute the metric on each (source, generated-variant) pair.

3. Metric — Normalized Gradient Field (NGF)

Reference: Haber, E. & Modersitzki, J. (2006), "Intensity gradient based registration and fusion of multi-modal images," MICCAI, pp. 726–733 — proposed for comparing images of genuinely different contrast in multimodal registration; confirmed in continued current use via its citation in the 2020 Learn2Reg challenge (Häger et al.). Our formula was verified line-by-line against a maintained implementation (github.com/BailiangJ/normalized_gradient_field_pytorch), not just the paper's description.

Formula, per voxel: gradients computed via a standard 3×3×3 Sobel kernel (∇src, ∇gen), then

ngf(voxel) = (∇src · ∇gen)² / ((|∇src|² + ε²)(|∇gen|² + ε²)) [ε = 1e-5, numerical stability only]

— the squared cosine of the angle between the two local gradient vectors. Squaring makes it inversion-invariant (a sign flip in either gradient cancels) and, by the chain rule, gradient direction survives any locally-monotonic intensity remap regardless of how nonlinear it is — no block-size hyperparameter needed (unlike the retired rank-correlation family, which needed one to avoid PALETTE's region-to-region sign cancellation). We report ngf_all = mean over all ROI voxels.

4. The chance floor — 1/3, not 0

Reference: standard directional-statistics fact — Fisher, Lewis & Embleton, Statistical Analysis of Spherical Data (Cambridge, 1987); Mardia & Jupp, Directional Statistics (Wiley, 2000). For an isotropic random 3-D direction (what i.i.d. noise-fill gradients look like) compared against any other direction, E[cos²θ] = 1/3 by symmetry (3-line proof: unit vector ⇒ v_x²+v_y²+v_z²=1; symmetry ⇒ the three expectations are equal; so each is 1/3). Verified independently three ways: the --sanity synthetic phantom (0.333), a fresh 20-million-sample Monte Carlo (0.333429 ± 0.000067), and real data (synthseg_noem landing at 0.319–0.332). The math is textbook; applying it to characterize this metric's floor is our own reasoning, not a cited precedent — stated plainly in LITERATURE_REVIEW.md. Not drawn on the plots (your call), kept as text/documentation only.

5. Final results

foreground-interior NGF (ngf_all), n=30 subjects × 10 variants, noblur:

method	FLAIR	T1w
palette	0.808	0.894
auglab_default	0.772	0.777
synthseg_em	0.461	0.422
v26_6_2_noisefill_v2	0.453	0.401
synthseg_noem	0.332	0.319

Paired Wilcoxon, palette vs auglab (per-subject, n=30): FLAIR p=7.6e-3 (21/30 subjects favor palette), T1w p=1.2e-6 (29/30 subjects) — palette significantly preserves more structure than conventional augmentation. synthseg_noem sits at the derived floor; synthseg_em/noisefill_v2 sit modestly above it because their spatial partition (EM sub-clustering / K-means, respectively) is informed by real image intensities even though the fill itself is random noise — a real, defensible residual, not an artifact. Causal chain: the intensity fill carries almost all of the preserved structure; a real-intensity-informed partition alone contributes a modest residual; an uninformed partition contributes nothing above chance.

[Luugaaa]

with blur (real conditions):

method	FLAIR	T1w
palette	0.764	0.859
auglab_default	0.703	0.727
v26_6_2_noisefill_v2	0.466	0.416
synthseg_em	0.405	0.407
synthseg_noem	0.332	0.324

[Luugaaa]

flair plots:

noblur:

blur (real conditions):

[Luugaaa]

Noise fill (basically = our palette without the histogram reampping, just a basic flat coloring, just like synthseg) preserves ~as little texture as synthsegEM. Therefore, histogram is the main driver of texture preservation.

[Luugaaa]

Clearer ablation ladder, missing the final val100:

[Luugaaa]

val100 is in — closing the gap from the last update. Full ladder with both v000/v100 rungs + per-step significance is in supplementary:

rung	Open-MS FLAIR	Brats-GLI T1n	CHAOS T1in	CHAOS T2spir	ON-Harm. T1w
+AL(v000)	30.6	46.9	87.6	86.3	64.2*
+AL(v100)	31.3	47.6*	87.1	87.1*	64.2

Dedicated val000-vs-val100 section confirms they're statistically indistinguishable at task level (Dice p=1.00, HD95 p=0.31) — val100 reported as a free bonus, not a separate contribution.

[Luugaaa]

Idea: maybe we should look into more details about what test contrasts show an improvement with texture and which do not. however this can prove heavy to report.

[Luugaaa]

For open ms flair, it's clear and easy that adding texture (real fill) benefits to all eval contrasts:

However, for open ms t1w, it's beneficial only for t2w, not flair, which could mean flair has a different texture pattern:

For brats t1n it's also clear that texture (real fill) help for all contrasts (while for openms t1 it's not the case!!):

While on t2w training it's harmful for t1n and helpful for the others:

[Luugaaa]

BraTS:

• training on T1n, texture transfers well to t2w (+7.13)
• training on T2w, texture transfers badly to t1n (−4.60).
  Same two contrasts, opposite direction depending on which one we train on.

Open-MS shows the same asymmetry:

• FLAIR→T1w transfers well (+6.17)
• T1w→FLAIR trends the other way (−3.77, though not significant at this n).

[Luugaaa]

We need to figure out:

• how to present this in the paper without making it too heavy
• how to explain why texture doesn't transfer well sometimes

[Luugaaa]

Proposed figures:

Answer to the open question above -- broke the fill-swap step down PER eval contrast instead of pooling (new: compute_fillswap_per_contrast.py).

Per-contrast Delta Dice (Holm-corrected within each ladder's own contrast family):

• Open-MS FLAIR: t1w +6.17*, t2w +9.22* -- pooled +7.69, p=6.1e-05
• Open-MS T1w: flair -3.77 (ns), t2w +5.34* -- pooled +0.79, p=0.43 (ns)
• Brats-GLI T1n: t1c +2.01*, t2w +7.13*, t2f +12.52* -- pooled +7.22, p=2.7e-26
• Brats-GLI T2w: t1n -4.60*, t1c +2.59*, t2f +3.80* -- pooled +0.60, p=0.12 (ns)
• CHAOS T1in: t1out +1.16 (ns), t2spir +0.83 (ns), ct +1.22* -- pooled +1.15
• CHAOS T2spir: t1in -2.22 (ns), t1out -2.55 (ns), ct +1.36 (ns) -- pooled +0.29 (ns)
• ON-Harmony T1w: T2w -2.72, bold -4.86, dwi_ap -2.62, epi_ap -2.09, gre_echo1_mag -10.09 -- none individually significant -- pooled -4.59 (macro -4.48), p=6.6e-07
• ON-Harmony T2w: T1w -0.65* (only sig. one), bold -0.95, dwi_ap -0.79, epi_ap -2.30, gre_echo1_mag -2.46 -- pooled -1.30 (macro -1.43), p=1.8e-08

* p<0.05, Holm-corrected within that row's own contrast family.

BraTS-GLI T2w's "flat" pooled result isn't flat -- it's t1n reversing sign (-4.60) against t1c/t2f holding positive. Broke T2w->T1n further by label: the reversal is almost entirely SNFH (oedema): +14.7 Dice when trained T1n->T2w vs. -9.8 when trained T2w->T1n (n=70, p=4.3e-11 / p=4.5e-05) -- the only label whose sign flips between directions. RC shows the same reversal, smaller (+2.9 / -6.5). NCR is flat both ways; ET only ever helps. Wrote this up in supplementary with a before/after zoom figure. Candidate mechanism (flagged, not tested): shortcut learning (Geirhos et al. 2020, Nat Mach Intell) / simplicity bias (Shah et al. 2020, NeurIPS) -- a texture cue learned for a diffuse label from one contrast may actively mislead on another rather than just fail to transfer; Ouyang et al. 2023 (IEEE TMI) makes the same contrast-specific-cue argument for cross-modality/cross-sequence medical segmentation directly.

Separately, this also resolved the ON-Harmony question (see #20): its pooled fill-swap effect is the largest, most significant negative in the whole table, which reads like a mystery on its own. Per-contrast it isn't one -- 9 of its 10 (train, eval) curves are individually non-significant (n=4-8/contrast, all pointing negative), only 1/10 reaches significance alone (T2w->T1w). Checked its labels: all 31 regions are bounded by a genuine tissue-type interface (ventricles/CSF, cortex-nuclei/white matter) -- same category as CHAOS, not a third pattern. Larger magnitude than CHAOS is plausibly a labelling-density effect (near-total brain volume vs. a handful of organs) -- open question for a second densely-labelled task, not resolved here.