How to tweak method/parameters for training a new model?

[MrMarkusJ]

I wonder how to decide on a good strategy to train a new model from scratch for MULTI-LABEL purpose.

Currently I am training a model for segmentation of rodent skull (3 labels: skull, left mandible, right mandible).
The image size is ~ 500 x 500 x 700, and only 8 GB GPU RAM are available. Further, ~ 15 manuall annotated
volumes are used for the training. The training is done with the 'segmentation' app.

Unfortunately the accuracy is ~ 60%, which is far too bad (even after 500 epochs).

Now I would like to improve this performance but I am not sure where to start and which parameters/ settings
to adapt.
Therefore it would be great if the MONAILabel team could point out on the options one could try to
improve the accuracy based on the segmentation task and available data.

Starting point: image data

• dimension of the data
• data format
• image intensity range
• image modality
• orientation/ alignment of the images
• what scrutures should be segmented? (ambiguous?)

In case labels are available

• consisten numbering
• data format

How to choose the most appropriate application based on this initial information?

• deepedit
• segmentation

In case of deepedit

• GPU RAM requirement, since the entire volume is loaded

in case of segmentation

• works on batches (defined by self.spatial_size)

Which one of these two methods (deepedit, segmentation) results in higher accuracy?

• is there a general rule?
• depending always on the segmentation task?

In case of large image dimension:

• would it improve accuracy if the data were downsampled to make it suitable for the Deepedit app?

Where to find information about setting parameters like "self.spatial_size" and about the options available in the MonAI plugin of Slicer3D like "lambda".

Such information would help the user to "tweak" the training process according to the actual segmentation task.

Thanks for your valuable input.

Best,
Markus

[SachidanandAlle]

Simple question.. let's say you are not using monailabel and with the data you have.. using the same, are you able train to any CNN model (say some UNET) for segmentation? what is the number of epochs and accuracy you are getting?

[diazandr3s]

Thanks for the summary, @MrMarkusJ
We're trying to condense this and other tips on AutoML algorithm to automatically select the best hyperparameters. Future versions will benefit from this.

For now, considering the available GPU memory my suggestion is to use the segmentation app with the following changes:

1/ Use a better intensity normalization transform along with gaussian smooth. Here is an example of vertebra segmentation: https://github.com/Project-MONAI/MONAILabel/blob/main/sample-apps/radiology/lib/trainers/localization_spine.py#L80-L82
The numbers for this transform can be adjusted to the Hounsfield units you think enclosed the regions you want to segment.

2/ Start with a patch size of 96x96x96

3/ Use these sliding window arguments: https://github.com/Project-MONAI/MONAILabel/blob/main/sample-apps/radiology/lib/trainers/localization_spine.py#L117-L119 for both inference and training.

In general, you could follow the localization_spine segmentation model. Just update the spacing here to 1.0, label names and give it a try.

Hope this helps

[diazandr3s]

Hi @MrMarkusJ,

Thanks for the update.

Didn´t applied the "localization_spine" segmentation app;

You don't need to. I just wanted to make you aware of another set of pre-transforms/inferer and network set up that might work for this task :)

Anyway, the accuracy improved to 65%, the validation of the training reached levels of ~ 90%.

Validation accuracy is a bit low. However, the accuracy of the validation set highly depends on the images used for validation. How is the segmentation quality?
You could use this model as a "seed" for an active learning approach in which you get predictions from the model, modify them and then update the model.

Now running monailabel 0.4.2.
But the bad thing now ---> can not start the training anymore. Looks like a "genearl" issue with

There is an issue with the scribbles we're solving right now #958
Can you please use the 0.4.1 version (pip install monailabel==0.4.1) while we solve this? We don't want to block your development

[MrMarkusJ]

Hi @diazandr3s,

thanks for you support!

Your suggestion regarding downgrading to monailabel 0.4.1 solved the issues.
"Scribbles" works, and also the "training" can be started.

You are not blocking my development ;-)

Currently I am comparing the app you have suggested:

• _You don't need to. I just wanted to make you aware of another set of pre-transforms/inferer and
• network set up that might work for this task :)

I already spotted several differences..... which give rise to more questions :-) !

Will take some time to "study" the example you suggested.
E.g. (\lib\trainer) def train_pre_transforms --> some RandFlip transforms and also the CropForeGroundd commands are not applied in the localization_spine_model.

For sure there is a reason behind it...
Understanding the strategy behind it helps to tweak the model.

Best,
Makus

[diazandr3s]

This is a very question. My guide is the test set nature. I mean, how the test set looks like? does it vary a lot from the training set? Some test sets don't have flipped organs, some times test images are cropped around the organ you're segmenting, etc

Regarding intensity normalization, it may help to chat with a clinical expert to know more about the Haunsfiled unit range or the intensity range of an image on which the organ can be better highlighted.

Another way of sharpening this skill is by performing more experiments and seeing successful models on tasks similar to ours :)

I hope this helps

[MrMarkusJ]

Dear @diazandr3s ,

I have taken a look at the "spine_localization" example and made following changes:

\lib\configs\segmentation.py:

• channels=[16, 32, 64, 128], // 256 omitted
• strides=[2, 2, 2], // only 3 instead of previously 4

\lib\infers\segmentation.py:

• ScaleIntensityRanged(keys="image", a_min=-20000, a_max=32767, b_min=0.0, b_max=1.0, clip=True), // pre_transforms
• roi_size=self.roi_size, sw_batch_size=2, overlap=0.4, padding_mode="replicate", mode="gaussian" // inferer

The intensity range is taken from the images; not given in Hounsfield! but all 3 bone structures of interest are within this range; the
3 labels to segment do not differ in terms of the intensity range; the difference is e.g. shape & location in the image

\lib\tainers\segmentation.py:

• ScaleIntensityRanged(keys="image", a_min=-20000, a_max=32767, b_min=0.0, b_max=1.0, clip=True),
• GaussianSmoothd(keys="image", sigma=0.4),
• ScaleIntensityd(keys="image", minv=-1.0, maxv=1.0), // in val_pre_transforms and in train_pre_transforms

And this definition is not used in the spine_localization app:

def norm_labels(self):
# This should be applied along with NormalizeLabelsInDatasetd transform
new_label_nums = {}
for idx, (key_label, val_label) in enumerate(self._labels.items(), start=1):
if key_label != "background":
new_label_nums[key_label] = idx
if key_label == "background":
new_label_nums["background"] = 0
return new_label_nums

Okay. 13 volumes used for training; 3 volumes used for validation. 3 labels: jaw_l, jaw_r, skull.

Performance after 500 epochs:
train_loss: ~ 0.0193 (looks like it would still decrease with more epochs)
training looks real great: rain_dice: 0.9686 train_jaw_l_dice: 0.9680 train_jaw_r_dice: 0.9752 train_skull_dice: 0.9627
but the validation is off: val_jaw_l_dice: 0.5741 val_jaw_r_dice: 0.5832 val_mean_dice: 0.5368 val_skull_dice: 0.4532

Why is the validation "suboptimal" ? Might it has something to do with the function norm_labels ?

Example of a labelled data set:

Any good idea on how to improve validation is very welcome :-)

Best,
Markus

[diazandr3s]

Hi @MrMarkusJ,

It seems the model is overfitting the training set. I'd recommend you add random intensity and spatial transformations during training i.e. RandShiftIntensity, random affine. These tutorial are great as they show how to use some of the transforms: https://github.com/Project-MONAI/tutorials/tree/main/3d_segmentation

Hope this helps. Please let us know.