Add class weights to the EnsembleVotingClassifier

[rp1908]

I used gradient boosting classifier to build a classification model. I am trying to improve the model by using a stack up model. I want to ensemble 3 different models, let's say, gbm, randomforests, logistic regression (except for gbm, other models subject to change). In my GBM model, I used weights in the fit function by giving higher weights to positive target variable. I want too try the same thing in ensemble, but I am unable to figure out how to implement weights in the source code of the ensemblevotingclassifier. I am new to this, so would like to receive suggestions regarding implementation of weights

Thanks

[rasbt]

Hi,
via the weights parameter of the classifier only controls the weights of the models (e.g,. the GBM, random forest, and logistic regression). To clarify ... when you say "positive target variable", do you mean a high value of your performance metric, or the "class label 1" in a binary classification setting?

While you have to tune the weights in the EnsembleVoteClassifier manually, the i.e., via hyperparameter tuning such as grid search or random search, there's also an alternative approach called Stacking. Since you mentioned that you want to

using a stack up model

you are maybe interesting in the Stacking alternative? I've implemented the simple version in mlxtend, and the documentation can be found here: http://rasbt.github.io/mlxtend/user_guide/classifier/StackingClassifier/

However, note that this is the original StackingClassifier version, not the cross-validation variant (for more info: #29). I was planning to add a StackingCVClassifier one day, though.

PS: I am closing this "issue" because it doesn't seem to be a bug in mlxtend, but please feel free to comment further in this thread.

[rp1908]

Hi Sebastian,

Thanks for your response. I wasn't sure how to ask the question, hence I
used "issue" to ask it.

By weights (enriching the target variable in a biased sample), I mean
assigning higher weight to class variable 1 in the target variable. For
example, if there are 10 class 1 target variables and 90 class 0 target
variables in a dataset, I would be assigning a weight of 9 to class 1 and a
weight of 1 to class 0. This is possible in GBM or logistic by using
sample_weights argument in the fit function. But, this is not possible in
the ensemblevoteclassifier as I see in the source code.
In the source code of gradient boosting classifier, this weights is present
in the source code of the fit function. Do you think it would be possible
to incorporate such weights in ensemblevoteclassifier too? I'd really
appreciate any help I can receive.

Regards,
Ruthvik

On Tue, Aug 2, 2016 at 3:39 AM, Sebastian Raschka notifications@github.com
wrote:

Hi,
via the weights parameter of the classifier only controls the weights of
the models (e.g,. the GBM, random forest, and logistic regression). To
clarify ... when you say "positive target variable", do you mean a high
value of your performance metric, or the "class label 1" in a binary
classification setting?

While you have to tune the weights in the EnsembleVoteClassifier
manually, the i.e., via hyperparameter tuning such as grid search or random
search, there's also an alternative approach called Stacking. Since you
mentioned that you want to

using a stack up model

you are maybe interesting in the Stacking alternative? I've implemented
the simple version in mlxtend, and the documentation can be found here:
http://rasbt.github.io/mlxtend/user_guide/classifier/StackingClassifier/

However, note that this is the original StackingClassifier version, not
the cross-validation variant (for more info: #29
#29). I was planning to add a
StackingCVClassifier one day, though.

PS: I am closing this "issue" because it doesn't seem to be a bug in
mlxtend, but please feel free to comment further in this thread.

—
You are receiving this because you authored the thread.
Reply to this email directly, view it on GitHub
#80 (comment), or mute
the thread
https://github.com/notifications/unsubscribe-auth/ATz2TIHHlZkF8vJScXAqENc608sw69NPks5qbm6KgaJpZM4JZiF-
.

[rasbt]

Oh I see, based on the terms you used, I would have assumed that you meant the class labels :)
One way to go about it would be to use the class_weights of the respective classifier implementations, e.g., the LogisticRegression and RandomForestClassifier in scikit-learn have one, too.
But adding it to the EnsembleVoteClassifier would also be a nice enhancement (and the results may be a bit different from setting the class weights in the initial classifiers). I think it would not be too complicated to add. It currently uses the weights as an optional weighting of the individual classifiers, i.e.,

            maj = np.apply_along_axis(lambda x:
                                      np.argmax(np.bincount(x,
                                                weights=self.weights)),
                                      axis=1,
                                      arr=predictions)

As an alternative to weight on class labels, one could just multiply the predictions I guess. E.g.,

if we have 3 classifiers that predict the three samples as [0, 1, 1, 0, 1] and you say that class 0 should be weighted twice as much as class 1, then it would become [0, 0, 1, 1, 0, 0, 1] from which you then can select the majority class label as a prediction.

I think that may be a useful feature, so please feel free to submit a pull request if you like!