Multi-label support

[rth]

Thanks for sharing this package and writing the paper.

It would be nice if this package also supported binary multi-label classification problems.

What would be the best way to aggregate for instance "Smallest Margin" computed for each label into a per-sample score in your opinion?

[dsleo]

Thanks for the feedback.

We haven't experimented much with specific query samplers for multi-label classification tasks.

If you are using a "one-vs-the-rest" classifier, you can aggregate the individual binary classifiers uncertainty measure directly. Active Learning Strategies for Multi-Label Text Classification explores various aggregations over confidence-based sampling. Another option of course would be to use clustering-based samplings which are task-independent.

What type of multi-label classifier are you currently using ?

We'll try to benchmark a bit the aggregation approach and see if we can propose something. Let us know if you come across an effective strategy !

[AlexandreAbraham]

Hello Roman,

In fact, we have not yet explored this setting. As you may have seen, we offer for now few methods that have been tried and tested on balanced, or slightly imbalanced classification tasks. In this setting, we do not correct for label distribution (yet) as we do not expect it to be paramount. But the distribution of label is crucial in multi-label classification.

Considering all of this, I would first try the method described in "Active Learning with Multi-Label SVM Classification". This method proposes to measure uncertainty using Max-Margin uncertainty, and they also propose a label cardinality consistency metric to ensure that the distribution of target labels is homogeneous to train. I guess that the adaptive part can be left out in a first time.

This method is implemented in alipy (among others) if you like: https://github.com/NUAA-AL/ALiPy/blob/86960cd02fc5a366e470231387c1cf63a9274ec4/alipy/query_strategy/multi_label.py#L973

The two other major python AL package modal and libact propose their own methods:
https://libact.readthedocs.io/en/latest/examples/multilabel_active_learning.html
https://modal-python.readthedocs.io/en/latest/content/apireference/multilabel.html

If you want to explore any of these options, do not hesitate to reach out to me with your usecase and maybe we could find a good algo to integrate in cardinal.