how do I use FLAML to tune my fastai tabular model?

[wjlgatech]

Hi FLAML team,
I want to use FLAML tools to fine tune my fastai v1 tabular model. The model does NOT has sklearn API.

# load data
from fastai.tabular import *

path = untar_data(URLs.ADULT_SAMPLE)

df = pd.read_csv(path/'adult.csv')
valid_idx = range(len(df)-2000, len(df))
dep_var = 'salary'
cat_names = ['workclass', 'education', 'marital-status', 'occupation', 'relationship', 'race', 'sex', 'native-country']

# data preprocessing
procs = [FillMissing, Categorify, Normalize]

# prep data for tabular_learner()
data = TabularDataBunch.from_df(path, df, dep_var=dep_var, valid_idx=valid_idx, procs=procs, cat_names=cat_names)

#metrics
f1=FBeta()
precision = Precision()
recall = Recall()

# train a classifier
learner = tabular_learner(data, layers=[200,100], ps=[0.1, 0.01], emb_drop=0.1, emb_szs={'native-country': 10}, metrics=[accuracy, precision, recall, f1])

learner.fit_one_cycle(1, 1e-2)

Here I want to fine tune 2 hyper-parameters in tabular_learner() : layers, ps

• the number of layers: n_layer choose from [2, 3, 4, 5]
• the number of units each layer has n_units chooses from [100, 200, 300, 400, 500, 600, 700, 800, 900, 1000, 1100, 1200]`
• the dropout ratio of each layer. ps choose from [0.001,0.01,0.05,0.1,0.2,0.3,0.4,0.5]

Since the customized models in your notebooks are fr. skearn, it is not very clear to me how to construct a class that can be consume by automl.add_learner().

Here is my experiment but it ran into an error:

from flaml import tune

class MyFastai1TabularModel(TabularModel):


    def __init__(self, n_jobs=-1, **params):
        '''Constructor
        
        Args:
            n_jobs: An integer of the number of parallel threads
            params: A dictionary of the hyperparameter names and values
        '''
        super().__init__(**params)

        # convert to int for integer hyperparameters
        self.params = {
            "n_jobs": n_jobs,
            'n_layers': int(params['n_layers']),
            'n_units': int(params['n_units']),
            'ps': params['ps'],
            'emb_drop': params['emb_drop']
        }    

    @classmethod
    def search_space(cls,  **params):
        '''[required method] search space

        Returns:
            A dictionary of the search space. 
            Each key is the name of a hyperparameter, and value is a dict with
                its domain and init_value (optional), cat_hp_cost (optional) 
                e.g., 
                {'domain': tune.randint(lower=1, upper=10), 'init_value': 1}
        '''
        space = {        
            'n_layers': {'domain': tune.randint(lower=2, upper=6), 'init_value': 2, 'low_cost_init_value': 2},
            'n_units': {'domain': tune.randint(lower=200, upper=1200), 'init_value': 200, 'low_cost_init_value': 200},
            'ps': {'domain': tune.loguniform(lower=1e-4, upper=1e-1), 'init_value': 0.01, 'low_cost_init_value': 0.01},
        }
        return space

automl = AutoML()
automl.add_learner(learner_name='fastai1tabular', learner_class=MyFastai1TabularModel)

settings = {
    "time_budget": 600,  # total running time in seconds
    "metric": 'f1', 
    "estimator_list": ['fastai1tabular', ],  # list of ML learners ['fastai1tabular', 'lgbm', 'rf', 'xgboost']
    "task": 'classification',  # task type    
    "log_file_name": 'airlines_experiment_custom_learner.log',  # flaml log file 
    "log_training_metric": True,  # whether to log training metric
}

automl.fit(X_train = X_train, y_train = y_train, **settings)

```python
The error was:
```python
AttributeError                            Traceback (most recent call last)
<ipython-input-36-4ba32f7e0310> in <module>()
      9 }
     10 
---> 11 automl.fit(X_train = X_train, y_train = y_train, **settings)

/usr/local/lib/python3.7/dist-packages/flaml/automl.py in fit(self, X_train, y_train, dataframe, label, metric, task, n_jobs, log_file_name, estimator_list, time_budget, max_iter, sample, ensemble, eval_method, log_type, model_history, split_ratio, n_splits, log_training_metric, mem_thres, pred_time_limit, train_time_limit, X_val, y_val, sample_weight_val, groups_val, groups, verbose, retrain_full, split_type, learner_selector, hpo_method, starting_points, seed, n_concurrent_trials, keep_search_state, append_log, early_stop, **fit_kwargs)
   1506         for estimator_name in estimator_list:
   1507             estimator_class = self._state.learner_classes[estimator_name]
-> 1508             estimator_class.init()
   1509             self._search_states[estimator_name] = SearchState(
   1510                 learner_class=estimator_class,

AttributeError: type object 'MyFastai1TabularModel' has no attribute 'init'

Is there any thing I can fix class MyFastai1TabularModel(TabularModel)? or I should go for the route like what it shows here: https://github.com/microsoft/FLAML/blob/main/notebook/flaml_pytorch_cifar10.ipynb ?

[sonichi]

The second route would definitely work. To decide whether the first route could work, the main question is whether you can inherit flaml.model.BaseEstimator and implement the required functions: fit(), predict() and predict_proba(). Feel free to chat on gitter if you like.

[wjlgatech]

Feel free to make changes and write comments on that colab notebook.

[sonichi]

There are some bugs in train_clf_config. I've fixed them. Also, there seems to be a bug with ray when using return instead of tune.report. So I replaced return with tune.report, and it works now.

[wjlgatech]

@sonichi Thank you so much! It works!

I have another question: I want to create a search space for a neural network with n layers (randomly chosen from [2,3,4,5]) and each layer has certain units (randomly chosen from [100, 200, 300])
For example:

if the network has 2 layers, then the units could be [100, 300], [300, 200], [200, 200] , etc.

if the network has 3 layers, then the units could be [100, 200, 100], [300, 100, 200], [200,300,300], etc.

The code could look like

def train_nn(config):
      ...

# define search space dictionary
space = {"n_layers": tune.choice([2,3,4,5])}

# add an additional item to the search space dictionary
space["n_units"= [tune.choice([100, 200,300]) for i in range(space['n_layers'])]

analysis = tune.run(
    train_nn,
    config = space,
)

Here I try to create space['n_units'] by the randomly selected value of space['n_layers']. But there is syntax error:

TypeError                                 Traceback (most recent call last)
<ipython-input-13-e7959554a989> in <module>()
      1 # search space
      2 space = {"n_layers": tune.choice([2,3,4,5]), 
----> 3          "n_units": [tune.choice([100, 200,300]) for i in range(space['n_layers'])]
TypeError: 'Categorical' object cannot be interpreted as an integer

It's because tune.choice([2,3,4,5]) is of type ray.tune.sample.Categorical which I do not know how to turn it into int.
Your advise is much appreciated!

[wjlgatech]

The implement in optuna is as following; what is the corresponding implementation in FLAML/ Ray?

def objective(trial: optuna.Trial):
    num_layers = trial.suggest_int('n_layers', 2, 5)  # `num_layers` is  2, 3, 4, or 5.
    layers= []  # define the number of unit of each layer / the ratio of dropout of each layer
    for i in range(n_layers - 1):  # `TabularModel` automatically adds the last layer.
        num_units = trial.suggest_categorical(f'num_units_layer_{i}', [100, 200, 300])
        layers.append(num_units)
        
    emb_drop = trial.suggest_discrete_uniform('emb_drop', 0, 1, 0.05)
    learn = tabular_learner(data, layers=layers, emb_drop=emb_drop, y_range=y_range, metrics=exp_rmspe)
    
    learn.fit_one_cycle(5, 1e-3, wd=0.2)
    return learn.validate()[-1].item()  # Of course you can use the last record of `learn.recorder`.
    
study = optuna.create_study()
study.optimize(objective)
best_trial = study.best_trial

[sonichi]

I see that you opened another discussion #241 . Please see my answer there.