dfs() builtin filter function

[grayskripko]

What if suggest a user an opportunity to filter features in the process of building a feature matrix?

As I noticed, the output of dfs() can be millions of highly correlated useless features. The vast amount of time was wasted on them. Would be great to give a user an opportunity to filter out highly correlated features. As well as features with a big ratio of missing values that will be dropped anyway. I assume it can increase the ratio of useful features, allow use a bigger value of max_depth and other expensive settings of dfs().

I expect this function can be inserted after transformation or the next step.

[kmax12]

That is an interesting suggestion. Would this make sense as a part of DFS or as a separate function compatible with the output of DFS? We would have to calculate all of the features in order to begin filtering, right? That would mean adding it to DFS might not save any time.

If it was separate it could be like this, which is similar to how ft.encode_features works.

fm, fl = ft.dfs(...)

fm_filtered, fl_filtered = ft.drop_correlated(fm, fl, threshold=.9)

[grayskripko]

the idea is exactly opposite. Probably, it requires some research. I assume that highly correlated features in any step of a recursive call will mostly produce the same useless highly correlated garbage. Yes, there should be a lot of exceptions but, in general, I believe that it's worth to increase max_depth (and other important settings of dfs()) using the opportunity to shrink the feature space inside of _run_dfs() presumably after the "step 3", the transformation step, or later.

At least, this is not a big deal to run something like drop_correlated() after dfs() by user means applying the function to the dfs() output. It's much harder for the user to inject the logic inside _run_dfs()

[windowshopr]

This would be a lovely feature! Right now, after building a new dataset of a huge number of features, I use FeatureSelector (https://github.com/WillKoehrsen/feature-selector) to filter out any highly correlated, low important, etc. features, but having this be a built in function would be ideal during the DFS() phase. Just my two cents.

[grayskripko]

as it was told in the issue, there are some callback examples like xgboost. Another I found is a pretty popular fastai package. It contains dozens of them allowing a user to inject a callback in any moment of forwarding and backpropagation: https://docs.fast.ai/callback.core.html.
Regarding the design, I think it can be like:

class Callback:
    def __init__(self, callback_fun, location):
        assert location.split(' ')[0] is in ['before', 'after']
        assert location.split(' ')[1] is in ['aggregate', 'transform']  # or like that
...

then you call dfs(..., callback=Callback(...)
and run some validation in dfs for callback function that it's compatible with its location

I forgot terminology for 2 years but wanted to stress the whole idea with an example. Despite the great opportunities pandas gives its users I still can't remember what should I pass as an input and an output when I create my functions for to apply(), transform(), applymap(), agg(), pipe() in pd.Series, pd.GroupedDataframe, pd.DataFrame. This is a nightmare for me. It would be great to avoid such a problem somehow.