Try ensembling/bagging for prediction

[jcohenadad]

Concept

Training several models in production (split 80/20/0) with different seeds, the idea is to average all predictions to generate a more reliable soft prediction (ie: bagging ensemble).

Prototyping script

# Loop across multiple subjects
list_sub=(001 002 003 004 005)
for sub in ${list_sub[@]}; do
  # Crop input image
  sct_deepseg_sc -i ~/data.neuro/basel-mp2rage/sub-P${sub}/anat/sub-P${sub}_UNIT1.nii.gz -c t1 
  sct_maths -i sub-P${sub}_UNIT1_seg.nii.gz -dilate 5 -shape ball -o sub-P${sub}_UNIT1_seg_dilate.nii.gz
  sct_maths -i sub-P${sub}_UNIT1_seg_dilate.nii.gz -dilate 32 -dim 1 -shape disk -o sub-P${sub}_UNIT1_seg_dilate.nii.gz
  sct_crop_image -i /Users/julien/data.neuro/basel-mp2rage/sub-P${sub}/anat/sub-P${sub}_UNIT1.nii.gz -m sub-P${sub}_UNIT1_seg_dilate.nii.gz -o sub-P${sub}_UNIT1_crop.nii.gz
  # Loop across models and run prediction
  list_seed=(7 8 9 10 11)
  for seed in ${list_seed[@]}; do ivadomed_segment_image -i sub-P${sub}_UNIT1_crop.nii.gz -m models/seed${seed}/model_seg_lesion_mp2rage -s _pred${seed}; done
  # Average
  sct_image -i sub-P${sub}_UNIT1_crop_pred*.nii.gz -concat t -o sub-P${sub}_UNIT1_crop_predMean.nii.gz
  sct_maths -i sub-P${sub}_UNIT1_crop_predMean.nii.gz -mean t -o sub-P${sub}_UNIT1_crop_predMean.nii.gz
  # Display
  fsleyes -S ~/data.neuro/basel-mp2rage/sub-P${sub}/anat/sub-P${sub}_UNIT1.nii.gz sub-P${sub}_UNIT1_crop_predMean.nii.gz -cm red-yellow
done

[jcohenadad]

Results

The prediction is averaged across 5 models (yellow: high confidence, red: low confidence).

P001

P002

P003

P004

[Nilser3]

Hi @jcohenadad
Following this pipeline,
Test on a UNI and T1q contrasts from CRMBM, Marseille dataset,

Results UNI image:



DSC (lesion_GT VS lesion_seg_thr_0.5) = 0.456
cDSC (lesion_GT VS lesion_seg) = 0.39

Results T1q image:

Fail to segment T1q contrast, models trained in UNI contrast only, perhaps.

[jcohenadad]

Very cool @Nilser3 ! Could you please clarify which version of the model you used? Which release (if you used a release). I find that results are pretty decent 😊

[Nilser3]

yes! @jcohenadad
I used this release : r20230210

[Nilser3]

Hi @jcohenadad

Continuing the ensembling test on the Marseille data (40 subjects with 3T UNI contrast)
I calculated the Continuous Dice Coefficient for "soft masks" based on this implementation and the Classic Dice between the GT and binarized masks (thresholding at 1/5 = 0.2) because we have 5 segmentation models.

There are 2 empy masks from GT (subject 3 and 4)