Automatic Evaluation with Simulated Annotator

[Zrrr1997]

Hi,

thanks for this wonderful project! I was wondering whether it is possible to automatically evaluate trained models by simulating the user during the testing stage. Most works in interactive segmentation start from a random initialization of clicks and then iteratively add clicks in the center of largest connected error component, similar to the training handlers in MONAI Label.

Is there a tutorial or a piece of code that I missed out regarding doing this during the evaluation? I would like to see how a trained model, e.g., DeepEdit, performs on the whole dataset, for which it would be impractical to conduct a user study with real annotators. A simulated annotator would also give a good indication of how the performance increases with the number of interactions (i.e. IoU@NoC).

Thanks very much in advance and keep up the great work!

Best,
Zdravko

[tangy5]

As far as I know, MONAI Label performs evaluation on validation set or auto partitioned set when conducting training. And active learning or scoring process such as epistemic scoring can score each unlabeled data with an uncertainty measure. Except these, there is no further "simulated annotator", maybe @diazandr3s can help more here about DeepEdit.
But I think this is a good point, evaluate how good a current trained model is on entire dataset might be very useful when the interactive annotation performs. If currently no perfect solution, you are very welcome to contribute the "simulated annotator", the open-sourced project is looking for ideas/discussion/feedback/contribution.

Thank you for the discussion.

[Zrrr1997]

Interesting, I think that a simulated user would be a very nice contribution since it would enable us to automatically verify and compare methods on an equal footing. Most works I have read go for using simply the center of the object as an initialization foreground click and then just sample from the center of the largest connected error component, which could be either the background or foreground (ITIS: BMVC 2018, RITM: ICIP 2022).

I dug around the tutorials and actually found the way DeepEdit does this: It uses the Chamfer Distance Transform (CDT) over the discrepancy regions (over- and undersegmentations) and the click is sampled from the pseudo-probability of the logits of the network (in this case DeepEdit based on DynUNet or Swin UNetr). So the clicks are more randomized than just using the center of the largest error. I have only seen this method used in Sakinis et al. (2019), which seems to be the base idea behind DeepEdit.

The code for this is here, but you really need to dive into the source code to understand what exactly these transformations are doing.

I think this will do for now, but I can gladly add a contribution regarding further annotator simulations, e.g., the center of largest error or extreme points or internal margin points (and many others).

[SachidanandAlle]

FYI... we do that for Deepgrow models..