Differences between MONAI and MONAILabel Segmentation Prediction

[egrabke]

I've been using MONAI to train a segmentation model before importing to MONAILabel. Evaluating the same model on the same image using both MONAI and using MONAILabel's generic segmentation app inferrer (ensuring the same pre- and post-transforms) yield different results ranging from minute pixel-level differences to segmentations with zero overlap.

These are the transforms used within MONAILabel's inferrer
Pre-transforms

LoadImaged(keys=["image"]),
EnsureChannelFirstd(keys=["image"]),                                   
ScaleIntensityd(keys=["image"]), 
EnsureTyped(keys=["image"]),

Post-transforms

Activationsd(keys="pred", sigmoid=True), # softmax
AsDiscreted(keys="pred", threshold_values=True), # argmax
ToNumpyd(keys="pred"),
Restored(keys="pred", ref_image="image"),
BoundingBoxd(keys="pred", result="result", bbox="bbox"),

Vs the MONAI inferrer
Pre-transforms

LoadImaged(keys=["img", "seg"]),
EnsureChannelFirstd(keys=["img", "seg"]),                                   
ScaleIntensityd(keys=["img","seg"]), 
SpatialPadd(keys=["img", "seg"], spatial_size = spatial_size, method= 'symmetric'),
EnsureTyped(keys=["img", "seg"]),

Post-transforms
[EnsureType(), Activations(sigmoid=True), AsDiscrete(threshold_values=True)])]

A possible source of difference:

• Restored() seems to be required for MONAILabel (segmentation won't output anything without it), but it will erroneously scale down segmentations if a spatial pad is used as a pre-transform

Despite this however, I also found though that significant differences were noticed even on images that would not have been padded.

I was wondering what further steps can be taken to reduce the model output differences?

[SachidanandAlle]

What is provided in the example app is not going to work for all kind of dataset/images/use-cases. This is only reference implementation over one kind of images/dataset. As the data changes, you might need something more in your pre/post transforms..

For example, you don't have to use Restored..
If you have everything taken care as part of post transform (pred is ready to be written as nii file) you can skip this...

In the example app, using Restored may not enough for all kind of images... how you restore the pred based on the original image actually depends on the pre-transforms.

In some cases, inverse transform can be a simple trick to rollback all the restore... But for some this might not be enough..
https://github.com/Project-MONAI/MONAILabel/blob/main/sample-apps/deepedit/lib/infer.py#L80

You can also override writer in your infer task.. and at the end produce binary mask and any json result which would like to send it back to client.. as a result of infer action
https://github.com/Project-MONAI/MONAILabel/blob/main/monailabel/interfaces/tasks/infer.py#L417

[diazandr3s]

Thanks for reporting this, @egrabke. This is a very good question.

In addition to what @SachidanandAlle says, I also wanted to ask you whether the training you're running in MONAI is also patch-based as it is done in the Segmentation App in MONAI Label (https://github.com/Project-MONAI/MONAILabel/blob/main/sample-apps/segmentation/lib/train.py#L102).

In the infer.py file of the Segmentation App in MONAI Label you could see that the inferrer used there is a sliding window one (https://github.com/Project-MONAI/MONAILabel/blob/main/sample-apps/segmentation/lib/infer.py#L65). This is used when the training is patch-based.

Please let us know if both configurations (MONAI vs MONAI Label) are the same in those aspects: type of training (patch-based or whole image) and type of inferrer

[egrabke]

Thank you both for the comments.

@SachidanandAlle : Thank you for bringing the invertible transforms to my attention, I wasn't sure how to invert padding and can make note of that. Despite this though, even when I don't have spatial padding in the MONAILabel pre_transforms I still need the restored() transform to produce an output, I'm not entirely sure why. Would there be any other resizing done during inference beyond what's listed in apps/segmentation/lib/infer.py?

@diazandr3s : The training for MONAI is patch-based. Patch size for MONAI training as well as sliding window size for both the MONAI and MONAILabel inferences were all set to the same values (full image x and y with cropped z), and the sliding window parameters (roi_size, sw_batch_size, overlap) were set identically between the two with the rest kept to default. The only difference was for MONAI I used sliding_window_inference from monai.inferers, vs for MONAILabel I used SlidingWindowInferer from monai.inferers. I haven't done any training with MONAILabel though and have made sure to keep the 3D Slicer Auto-Update Model box unchecked.

[diazandr3s]

Thanks for your response, @egrabke.

Out of curiosity, why don't you train the model directly with MONAI Label? It already has everything to do training and inference (data augmentation, backbone definition, trainers, inferrers, etc).

Despite this though, even when I don't have spatial padding in the MONAILabel pre_transforms I still need the restored() transform to produce an output, I'm not entirely sure why. Would there be any other resizing done during inference beyond what's listed in apps/segmentation/lib/infer.py?

You're not using Spacingd transformation or any other transform that changes the spatial size, right?

I'd suggest using exactly the same pre-transforms in both setups. SpatialPadd might be the one creating that difference during inference. Other than that, I don't see other differences.

Please let us know how that goes.

[egrabke]

So I copied the MONAI transforms directly to MONAILabel pre-transforms and then overwrote the inverse_transforms method as:

def inverse_transforms(self):
    return []

I still need restored() but model now outputs exactly as expected! Thank you both for your help!