Split image-level processing from cross-image processing

[piercus]

Hello @lartpang, thanks for this great lib.

I would like to be able, on a big dataset, to store image-level metrics.

Context

For example

Image1 :

• WeightedFmeasure : 92%
• Emeasure : 94%
• tags: foo, bar

Image2 :

• WeightedFmeasure : 93%
• Emeasure : 90%
• tags: bar

Image3 :

• WeightedFmeasure : 88%
• Emeasure : 97%
• tags : foo

And then to be able to run the evaluation on different tags (foo/bar) (without re-running the image-level metrics computation)

Actual

In the current lib it is not direct to do this, because the image-level processing, and the cross-images processing are made together, and there is no cross-metric convention.

Suggestion

In metric.step(pred, gt)

• return the image-level value of the metric (could be an array for dynamic results)
• Run in 2 different steps
  • Compute the metric metric.compute(pred, gt) -> value
  • Store the value internally metric.load(pred, gt)

As a result

• We can store image-level metrics when running metric.step(pred, gt)
• We can reuse pre-stored metrics using metric.load(value) afterward, before running metric.get_results()

Would you be interested to change the API for this ? Would you like some help ?

[lartpang]

@piercus

Thanks for the idea, maybe the following extension code is what you want?

This code below is used to evaluate the algorithms for the video tasks.

https://github.com/lartpang/PySODEvalToolkit/blob/f12dcf5925750c9ea73c535ee8d99ed40cf1cb4c/utils/recorders/metric_recorder.py#L201-L279