Original labels not loading in 3D Slicer after batch inference

[Nachimak28]

Hi there,

As per another [discussion](#856) which proved really helpful. I am trying to build a Proof of Concept workflow.
Context: Training a custom 3D segmentation model and hosting the monailabel server in a CPU since we can't afford a GPU letting the monailabel server act as a backend to supply the labels without any training happening then and there.

What all I have done till this point:

• Trained a custom model - just the [COVID CT lung lesion segmentation model](https://github.com/Project-MONAI/tutorials/tree/main/3d_segmentation/challenge_baseline)
• Added the new model as per the [model addition guide](https://github.com/Project-MONAI/MONAILabel/tree/main/sample-apps/radiology#how-to-add-new-model)
• 3D slicer successfully shows the label when the model inference is run.
• Ran the batch inference api to generate all the original labels

Issue faced:
Even if the original labels are present in the images/labels/original directory, I dont see the label being loaded in 3D slicer and still have to run the model inference everytime.
I also checked the plugin config in 3D slicer's application settings and the Original Labels option is enabled.

I presume I might be doing something wrong with the lib/config and lib/infer app codebase that the original label isn't being transmitted when I press the next sample button.

Here are my config and infer files
Config: https://github.com/Nachimak28/monailabel_custom_model_inference/tree/main/config

Infer: https://github.com/Nachimak28/monailabel_custom_model_inference/blob/main/infers/lung_lesion_seg.py

Note: Both these are simply inspired from the segmentation_spleen.py files as mentioned in the model addition guide with only minor necessary changes.

I also tried a similar experiment with the spleen segmentation example given in the documentation, it still does not load the already present label.

monailabel version: 0.4.0
Slider 3D version: 5.0.2 stable
(would the original label load in a preview release ?)

[diazandr3s]

Hi @Nachimak28, thanks for your detailed post. This is great!

It's strange why this is happening. Could you please try with the Slicer preview release? If labels are in the images/labels/original, they should also be loaded in Slicer when fetching the next sample.

Please make sure the option is activated:

[Nachimak28]

Hi @diazandr3s sorry to bother you again.
To make things simpler, I simply tried out the spleen segmentation example given in the docs with a small subset of images while still maintaining the directory structure as given.

I tried out with the latest preview release of slicer 3D - 5.1.0 build 2022-07-09 as well as with an older preview release 5.1.0 build 2022-06-22.

unfortunately for both of these slicer versions, the original labels do not load. The slicer logs show the followng message

Also my application setting has the Original label option enabled

I'm not sure what I'm doing wrong here. If you want more details from my end or if you wish to provide any specific instructions to follow, I am happy to do so.

My folder structure inside the images directory is as follows:

.
├── ./datastore_v2.json
├── ./labels
│   ├── ./labels/final
│   └── ./labels/original
│       ├── ./labels/original/spleen_6.nii.gz
│       ├── ./labels/original/spleen_60.nii.gz
│       ├── ./labels/original/spleen_61.nii.gz
│       ├── ./labels/original/spleen_62.nii.gz
│       └── ./labels/original/spleen_63.nii.gz
├── ./spleen_6.nii.gz
├── ./spleen_60.nii.gz
├── ./spleen_61.nii.gz
├── ./spleen_62.nii.gz
└── ./spleen_63.nii.gz

My datastore_v2.json (truncated) is as follows:
Note: although the json here is truncated, it still consists the full nested dict for a single image.

{
  "name": "new-dataset",
  "description": "New Dataset",
  "images_dir": ".",
  "labels_dir": "labels",
  "objects": {
    "spleen_60": {
      "image": {
        "ext": ".nii.gz",
        "info": {
          "ts": 1657775965,
          "name": "spleen_60.nii.gz",
          "strategy": {
            "random": {
              "ts": 1657869417,
              "client_id": "user-xyz"
            }
          }
        }
      },
      "labels": {
        "original": {
          "ext": ".nii.gz",
          "info": {
            "ts": 1657775965,
            "checksum": "SHA256:315f7d834e93e3a24b65835139a6769dff00bfb235774e554b000f9ec92172e3",
            "name": "spleen_60.nii.gz"
          }
        }
      }
    },

I'm running this on a standard CPU VM in google cloud with 1 port open - 8888. The VM config is v2 CPUs and 7.5 GB RAM.

I tried to read the slicer plugin code in the monailabel repo and think that I might have located the issue:
https://github.com/Project-MONAI/MONAILabel/blob/main/plugins/slicer/MONAILabel/MONAILabel.py

Line number: 1224 to 1234 - the try except block
Especially the line number: 1226

labels = datastore["objects"][image_id]["labels"]["original"]["info"]["params"]["label_names"]

Here if the datastore dictionary has the exact same schema as the datastore_v2.json, it seems to be asking for a non-existent key-value of
params-> label_names which as seen in the above mentioned json file is not present as a key value pair.

I might be wrong here but could this be the issue ?
In case you want me to try out some logic change in some part of the lib to help in the debugging, I shall be happy to do so.

[diazandr3s]

Hi @Nachimak28,

Thanks for the follow-up.
Sorry, I didn't see the issue before. Please convert this into a dictionary: https://github.com/Nachimak28/monailabel_custom_model_inference/blob/main/config/lung_lesion_seg.py#L40

I've just created a PR changing this bit for the segmentation spleen model. You can see here the change: https://github.com/Project-MONAI/MONAILabel/blob/fixSpleenSeg/sample-apps/radiology/lib/configs/segmentation_spleen.py#L49-L51

Please let me know how that goes :)

[Nachimak28]

Hi @diazandr3s unfortunately the changes that you suggested didnt work out and the situation was the same. However a small change in the datastore json solved the issue.
As I had pointed out about the slicer plugin code issue:
https://github.com/Project-MONAI/MONAILabel/blob/main/plugins/slicer/MONAILabel/MONAILabel.py

Line number: 1224 to 1234 - the try except block
Especially the line number: 1226

labels = datastore["objects"][image_id]["labels"]["original"]["info"]["params"]["label_names"]

Here the nested dict path params.label_names is missing in the datastore_v2.json and even after the code changes and the batch inference.

I manually added it in the json file and Slicer 3D started loading the original labels.

Below is the before/after comparison of the json file
Before

      "labels": {
        "original": {
          "ext": ".nii.gz",
          "info": {
            "ts": 1657971125,
            "checksum": "SHA256:315f7d834e93e3a24b65835139a6769dff00bfb235774e554b000f9ec92172e3",
            "name": "spleen_60.nii.gz"
          }
        }
      }
    },

After

      "labels": {
        "original": {
          "ext": ".nii.gz",
          "info": {
            "ts": 1657971125,
            "checksum": "SHA256:315f7d834e93e3a24b65835139a6769dff00bfb235774e554b000f9ec92172e3",
            "name": "spleen_60.nii.gz",
            "params": {
              "label_names": {
                "spleen": 1
              }
            }
          }
        }
      }
    },

The difference is adding the following part in the labels.original.info nested dict path

"params": {
              "label_names": {
                "spleen": 1
              }
            }

Output with the json change:

I have all I need for now and can make the whole workflow work with this minor update in the datastore_v2.json

The steps I followed are:
1.) Create a folder with the images
2.) Start the monailabel server and run the batch_inference API
3.) Once done, edit the datastore_v2.json carefully and add the params key value in each labels.original.info section for all image_ids, could easily write a small python script for this in case there are too many images in the datastore.
4.) Connect to the server in Slicer and click on "Next Label"
DONE

[diazandr3s]

Hi @Nachimak28,

Thanks for your response.
Can you please try the following steps and let us know how that goes?

1/ Delete the datastore_v2.json file
2/ Delete labels/original folder
3/ Start the server
4/ Run batch inference

The latest PR should automatically add params argument to the JSON file. I've tried it locally on my PC and it works.

Please let us know.

[Nachimak28]

Hi @diazandr3s
Apologies for the late response.
I tried out the workflow with the latest monailabel codebase (installation from the git repo instead of pypi) and the original labels got loaded in slicer without having to modify the datastore_v2.json

Thank you so much for your help and prompt responses.

Steps I followed:

1. Create a fresh conda env and install pytorch etc
2. Installed monailabel from github
3. Create a new datasets/spleen_experiment and copy 3-5 files into it
4. Start the monailabel server with the necessary arguments
5. Run the batch inference API
6. Load the image in slicer 3d (the label loads too)
7. Verified that the datastore_v2.json successfully storee the params key value for every original label

I've tried this out with deepedit, segmentation_spleen models and also one custom model (lung lesion segmentation) and it all worked as expected.

Thank you so so much and I'm extremely sorry to have bothered you with this issue or taken too much of your time.

Also, if you don't mind, I plan to make some videos demonstrating this workflow from scratch since a couple of other discussions (including mine) are asking for a similar kind of workflow. I can send you the video(s) once I make them and you can upload them to the monai youtube channel if you want.

[SachidanandAlle]

That's helpful.. please go ahead and make the videos.. we can share across..

[diazandr3s]

Thanks for reporting back, @Nachimak28. This is great!