Ensemble

[pawarbi]

I ran the demo example from here: https://github.com/microsoft/FLAML/blob/main/notebook/flaml_demo.ipynb
Couple of things:

• I had to install openml. It would be good if flaml installed it. Should openml be added to https://github.com/microsoft/FLAML/blob/main/setup.py considering that's the first thing required to run the demo datasets using from flaml.data import load_openml_dataset
• Does setting ensemble=True in settings force flaml to create and output an ensemble model? Expected behavior should be if ensemble=True, train individual models and ensembles but return the best model based for the chosen metric. If you run the demo notebook from above, an ensemble model is returned which has lower accuracy than the single best model (lgbm). Expected output in this case would be best LGBMClassifier model not the StackingClassifer which has lower accuracy.

[sonichi]

1. For openml, we made it optional in the default package. In the notebook option, openml is included. So one can use pip install flaml[notebook] to make sure the notebook is runnable. I'll add !pip install flaml[notebook] in the beginning of the notebook so that it ensures the dependencies are installed.
2. Without access to test data we won't know whether the best single model is better or the ensemble is better. What we can do is to keep both models available by adding a property best_individual_model when ensemble=True. Would you like to work on that?

[wjlgatech]

Hi @sonichi, is the emsemble option available now? Is that a demo for its use? Thanks

[sonichi]

ensemble – boolean or dict | default=False. Whether to perform ensemble after search. Can be a dict with keys ‘passthrough’ and ‘final_estimator’ to specify the passthrough and final_estimator in the stacker.

FLAML/test/test_automl.py

Line 253 in ddc1a63

"ensemble": True,

FLAML/test/test_automl.py

Line 164 in ddc1a63

def test_ensemble(self):

[wjlgatech]

@sonichi Thanks for the references. When you set ensemble=True, after automl.fit(X_train=X_train, y_train=y_train, **settings), is there a place you can extract the ensemble model? and do something like this

#pseudo code here:
y_pred = automl.ensemble_model.predict(X_test)
y_prob = automl.ensemble_model.predict_proba(X_test)

[wjlgatech]

Moreover, what are the ensemble methods FLAML use?

Is it a simple average or more sophisticate one like linear regression/MLP on the top of all trained models?

# simple average ensemble
y_prob_average = (y_prob1 + y_prob2 + y_prob3)/3

[sonichi]

We use StackingClassifier or StackingRegressor from sklearn.ensemble. When ensemble=True, you can directly use automl.predict or automl.predict_proba to apply the ensemble model. To extract the ensemble model, use automl.model.