[python-package] `Booster.refit()` raises a misleading warning when using categorical features

[jameslamb]

Description

Seeing this warning in Python unit tests:

tests/python_package_test/test_engine.py::test_linear_trees
/Users/runner/miniforge/envs/test-env/lib/python3.12/site-packages/lightgbm/basic.py:2137: UserWarning: categorical_feature keyword has been found in params and will be ignored.
Please use categorical_feature argument of the Dataset constructor to pass this parameter.
_log_warning(

That test case is not directly passing categorical_feature through params... that might be happening internal to lightgbm, and therefore something that users cannot avoid.

Reproducible example

I haven't not narrowed it down further yet, but can reliably reproduce it via running the tests.

pytest 'tests/python_package_test/test_engine.py::test_linear_trees'

Environment info

LightGBM version or commit hash: 226e7f7

Command(s) you used to install LightGBM

cmake -B build -S . -DUSE_OPENMP=OFF
cmake --build build --target _lightgbm -j4
sh build-python.sh install --precompile

Additional Comments

I suspect that maybe this is a result of categorical_feature being treated as both a "param" here:

LightGBM/python-package/lightgbm/basic.py

Lines 2021 to 2025 in 3654eca

	if self.params is not None:
	# no min_data, nthreads and verbose in this function
	dataset_params = _ConfigAliases.get(
	"bin_construct_sample_cnt",
	"categorical_feature",

But also being a keyword argument to Dataset.__init__():

LightGBM/python-package/lightgbm/basic.py

Line 1784 in 3654eca

categorical_feature: _LGBM_CategoricalFeatureConfiguration = "auto",

[zacharyftw]

@jameslamb i would like to work on this issue

[jameslamb]

Sure, thanks!

[zodchi94]

@jameslamb is it still in progress? I'd like to solve this one

[jameslamb]

Thanks @zodchi94 , no one is actively working on this so you're welcome to pick it up. I've assigned it to you, please submit a pull request when you have time / interest. You're welcome to open a draft PR and use comments there to ask questions if you need help.

[arjunprakash027]

Hey @jameslamb - I'd love to take a stab at this if its not in progress

[jameslamb]

There isn't any change in progress as far as I know, I've reassigned it to you. We'd be happy for the help, thanks @arjunprakash027

[arjunprakash027]

Hey @jameslamb , I’d really appreciate your input on this (I can include code if needed).

My current understanding is:

There is a warning that categorical_feature cannot be specified both in the params passed to train() and in the Dataset object, and the recommended/default way is to set it on the Dataset side rather than via train params

In this case, the warning appears when loading a model from a local text file and initializing a Booster from it. From digging into the code, it looks like parameters are loaded from the C++ side when not explicitly passed from Python, and those parameters include categorical_feature.

I also observed that if categorical_feature is not passed to the Dataset constructor (left as the default "auto"), it is not written into the text model file, and therefore is not read back from C++ on load. I assume from this that we are also saving Dataset parameters into the same string as training parameters.

Given that, I see a few possible solutions:

1. Change the C++ side to avoid persisting categorical_feature as a string when saving the model, and instead store it separately for the Dataset (or rely on the default "auto" behavior).
2. After converting parameters from C++ to a Python dict, strip out categorical_feature on the Python side (or redirect it into the Dataset constructor in a way that does not trigger the warning).
3. If categorical_feature is present in the params and the Dataset has categorical_feature="auto" (default), move the value from params into the Dataset and clear it from params.
4. Separate model parameters and Dataset‑specific parameters in the C++ save/load path so they are not mixed and do not conflict later. (same as step 2)

What do you think? I could really use your suggestion here

Thank you!

[arjunprakash027]

LightGBM/python-package/lightgbm/basic.py

Lines 3613 to 3623 in 33e93d6

	train_set.construct()
	# copy the parameters from train_set
	params.update(train_set.get_params())
	params_str = _param_dict_to_str(params)
	_safe_call(
	_LIB.LGBM_BoosterCreate(
	train_set._handle,
	_c_str(params_str),
	ctypes.byref(self._handle),
	)
	)

if you look at the above code in basic.py, parameters of Dataset object is combined with that of train parameters before handing everything off to c++ to create a booster and train.

And in c++ we dump all the config (including that of Dataset) while exporting the model as text file and therefore we have categorical_feature as a model parameter while loading the model from disk.

I suppose this behaviour is expected, should we change something here?