Does FLAML work for self-built neural network?

[hanhanwu]

Hello FLAML team,

I was trying to compare the performance between FLAML and Optuna in deep learning params tuning.

Starting with a basic CNN.

This is my Optuna code, which works. Params need to tune can be put in the objective function:

Then I was trying to figure out how to use FLAML to tune the same CNN model. I have checked the notebook here: https://github.com/microsoft/FLAML/blob/main/notebook/flaml_finetune_transformer.ipynb

And mainly defined train_cnn() that's similar to your train_distilbert() in the notebook:

I guess the **config is the search space? And here's the code tried to tune the model:

But this doesn't work.

If FLAML also works for self-built neural networks, is that possible for you guys to add a simple notebook example for that, so that users can follow? (If you are able to tell what's wrong in my code here, that's even better :) )
I feel pre-trained model from transformer is harder to understand since I'm not familiar with transformer and guess many other users can be similar :)

Thanks a bunch!!

[sonichi]

num_train_epochs is not in the search space. So it needs to be removed from low_cost_partial_config. Could you comment the line with low_cost_partial_config and try again?

[sonichi]

We have a working example under test/test_pytorch_cifar10.py. But from the error message it seems you are using windows, and I haven't been able to make ray work in windows. Ray's support for windows is experimental. Could you try commenting ray.shutdown(), ray.init() and use_ray=True to see if it works without ray?

[sonichi]

Added a notebook example in this PR: #189

[hanhanwu]

Great! Thank you very much!

[sonichi]

The PR has been merged. Is there any follow-up question?

[hanhanwu]

Yeah, I saw the notebook last week. No more question so far. Thank you very much!