Classification of imblanaced data

[pawarbi]

How does FLAML handle imbalanced data (unequal distribution of target classes in classification task)? RandomForest has class_weight argument, xgboost has sample_weight and LGBM has class_weight as well? Should we scan for class imbalance first and if it's detected, set those arguments to 'balanced'. It would be better to throw a warning and let the user know about class imbalance before training. It would be nice if imbalanced-learn is used automatically in preprocessing the dataset if imblance is detected

Thoughts?

[sonichi]

Currently FLAML does several things for imbalanced data.

1. When a class contains fewer than 20 examples, we repeatedly add these examples to the training data until the count is at least 20.
2. We use stratified sampling when doing holdout and kf.
3. We make sure no class is empty in both training and holdout data.
4. We allow users to pass sample_weight to AutoML.fit().

I'm supportive of making the warning about imbalance.
I'm not clear how to use imbalanced-learn automatically as there are multiple choices offered by that library. Feel free to elaborate more or propose other ideas.

[sonichi]

Updated answer: https://microsoft.github.io/FLAML/docs/FAQ#how-does-flaml-handle-imbalanced-data-unequal-distribution-of-target-classes-in-classification-task

[DSgUY]

Hi @sonichi, I was able to use sample_weight doing some refactor of weights from class_weight.compute_class_weight from skelearn as I was getting a dimension error.

sample code:

sklearn_weights = class_weight.compute_class_weight(class_weight = 'balanced',
                                                    classes = classes,
                                                    y = y_train)

y_weight = y_train.copy()
y_weight = np.where (y_weight, sklearn_weights[1], sklearn_weights[0])

then:

automl = AutoML()
settings = {
    "time_budget": 10 * 60,      
    "metric": 'f1',           
    "task": 'classification',   
    "seed": 43,                 
    "verbose": 0,
    'sample_weight': y_weight
    }

automl.fit(X_train=X_train, y_train=y_train,  **settings)

is this ok?

[sonichi]

Looks good to me. Does it give you better performance than not using sample_weight?

[DSgUY]

I have an extreme imbalance dataset [1784989 5947] so I was carring out a quick tests. With sample_weight I have more FN and much more FP so I think it 'improves' the classification task.

[sonichi]

Could you let me know if it gives better results in the end? If so, it'll be good to add this method to the FAQ.