[python-package] How do I reproduce LightGBM's L2 loss with a custom objective?

[btngcm]

I am currently using a custom L2 loss function and would like to try to achieve the same effect as the built-in L2 loss function in LGBM. However, based on my testing, when I set the parameter 'feature_fraction', according to eval_result, the result will have a deviation of the order of 1e-4. Here is my code

import numpy as np
import pandas as pd
import lightgbm as lgb
from sklearn.model_selection import train_test_split
import matplotlib.pyplot as plt
from sklearn.datasets import load_breast_cancer

def l2_loss(y, data):
    t = data.get_label()
    grad = (y - t)
    hess = np.ones_like(y)
    return grad, hess

def l2_eval(y, data):
    t = data.get_label()
    loss = (y - t) ** 2 
    return 'l2_custom', loss.mean(), False

cancer = load_breast_cancer()
x = cancer.data
y = cancer.target
X_train, X_test, y_train, y_test = train_test_split(x, y, random_state=0)


lgb_train = lgb.Dataset(X_train, y_train)
lgb_test = lgb.Dataset(X_test, y_test)
lgbm_params = {
    'boosting_type': 'gbdt', 
    'objective': 'regression', 
    'metric': 'l2', 
    'feature_fraction': 0.95,
    'verbose': -1, 
    "seed": 3, 
    "bagging_seed": 3, 
    "feature_fraction_seed": 3,
    'num_threads': 1,
    }
eval_result = {}
model = lgb.train(lgbm_params, 
                  lgb_train,
                  num_boost_round=100,
                  valid_sets=[lgb_train, lgb_test],
                  callbacks=[lgb.record_evaluation(eval_result=eval_result)])

lgb_train = lgb.Dataset(X_train, y_train, init_score=np.ones_like(y_train) * np.mean(y_train))
lgb_test = lgb.Dataset(X_test, y_test, init_score=np.ones_like(y_test) * np.mean(y_train))
lgbm_params = {
    'boosting_type': 'gbdt', 
    'objective': 'regression', 
    'metric': 'None', 
    'feature_fraction': 0.95,
    'verbose': -1, 
    "seed": 3, 
    "bagging_seed": 3, 
    "feature_fraction_seed": 3,
    'num_threads': 1,
    }
eval_result_custom = {}
model = lgb.train(lgbm_params, 
                  lgb_train,
                  fobj=l2_loss,
                  feval=l2_eval,
                  num_boost_round=100,
                  valid_sets=[lgb_train, lgb_test],
                  callbacks=[lgb.record_evaluation(eval_result=eval_result_custom)])


np.array(eval_result["valid_1"]["l2"]) - np.array(eval_result_custom["valid_1"]["l2_custom"])

# array([ 1.09155228e-04,  7.29931531e-05,  1.29470861e-03, -7.57841913e-04,
       -5.25649308e-04,  1.61685018e-03,  3.72543557e-04, -6.27124544e-04,
       -1.11300630e-03, -1.71263685e-03, -7.36518101e-04, -1.66719504e-05,
       -2.65152156e-05, -7.72906106e-05, -1.32754737e-04, -8.42054739e-04,
       -1.10504478e-03, -1.27733287e-03, -1.52907734e-03, -1.32965297e-03,
       -1.66672198e-03, -1.75084555e-03, -1.85820361e-03, -2.20488819e-03,
       -1.86205781e-03, -1.67845150e-03, -1.46834841e-03, -1.75160841e-03,
       -2.12522323e-03, -2.44374316e-03, -2.67094134e-03, -2.25995018e-03,
       -2.27950347e-03, -2.05786825e-03, -2.48343245e-03, -2.12190731e-03,
       -2.24762728e-03, -2.43148097e-03, -2.09623719e-03, -1.76362730e-03,
       -1.49269336e-03, -1.55342882e-03, -1.66717828e-03, -1.72470297e-03,
       -1.40780989e-03, -1.31962252e-03, -1.49693263e-03, -1.58855790e-03,
       -1.26178383e-03, -1.50026131e-03, -1.55791090e-03, -1.61025457e-03,
       -1.52606887e-03, -1.49666063e-03, -1.66413700e-03, -1.72256752e-03,
       -1.88702369e-03, -1.85907193e-03, -1.82764922e-03, -1.72238464e-03,
       -1.91252787e-03, -1.74053387e-03, -1.88331845e-03, -1.73742180e-03,
       -1.78331144e-03, -1.82270777e-03, -1.84575238e-03, -1.71488980e-03,
       -1.61404212e-03, -1.67033804e-03, -1.72049221e-03, -1.76970137e-03,
       -1.79598923e-03, -1.79350876e-03, -1.80647167e-03, -1.73345429e-03,
       -1.63948193e-03, -1.59036185e-03, -1.42593379e-03, -1.45313660e-03,
       -1.42657691e-03, -1.49853759e-03, -1.56282731e-03, -1.64057159e-03,
       -1.67995755e-03, -1.74710340e-03, -1.68834089e-03, -1.72641421e-03,
       -1.70694513e-03, -1.62733158e-03, -1.61523343e-03, -1.76219151e-03,
       -1.66909772e-03, -1.63843771e-03, -1.72622580e-03, -1.67351035e-03,
       -1.67849786e-03, -1.60731855e-03, -1.75988751e-03, -1.79778481e-03])

But when I annotate this parameter "feature_fraction", the deviation will decrease to 1e-18. What is the reason for this?

lgb_train = lgb.Dataset(X_train, y_train)
lgb_test = lgb.Dataset(X_test, y_test)
lgbm_params = {
    'boosting_type': 'gbdt', 
    'objective': 'regression', 
    'metric': 'l2', 
    'verbose': -1, 
    "seed": 3, 
    "bagging_seed": 3, 
    "feature_fraction_seed": 3,
    'num_threads': 1,
    }
eval_result = {}
model = lgb.train(lgbm_params, 
                  lgb_train,
                  num_boost_round=100,
                  valid_sets=[lgb_train, lgb_test],
                  callbacks=[lgb.record_evaluation(eval_result=eval_result)])

lgb_train = lgb.Dataset(X_train, y_train, init_score=np.ones_like(y_train) * np.mean(y_train))
lgb_test = lgb.Dataset(X_test, y_test, init_score=np.ones_like(y_test) * np.mean(y_train))
lgbm_params = {
    'boosting_type': 'gbdt', 
    'objective': 'regression', 
    'metric': 'None', 
    'verbose': -1, 
    "seed": 3, 
    "bagging_seed": 3, 
    "feature_fraction_seed": 3,
    'num_threads': 1,
    }
eval_result_custom = {}
model = lgb.train(lgbm_params, 
                  lgb_train,
                  fobj=l2_loss,
                  feval=l2_eval,
                  num_boost_round=100,
                  valid_sets=[lgb_train, lgb_test],
                  callbacks=[lgb.record_evaluation(eval_result=eval_result_custom)])


np.array(eval_result["valid_1"]["l2"]) - np.array(eval_result_custom["valid_1"]["l2_custom"])



# array([-1.11022302e-16, -1.94289029e-16, -1.94289029e-16,  1.11022302e-16,
        1.38777878e-17,  1.38777878e-17,  1.38777878e-17,  1.38777878e-17,
       -1.38777878e-17,  1.38777878e-17,  6.93889390e-18,  0.00000000e+00,
        6.93889390e-18, -2.77555756e-17,  1.38777878e-17,  0.00000000e+00,
       -6.93889390e-18,  0.00000000e+00, -6.93889390e-18,  0.00000000e+00,
        0.00000000e+00,  6.93889390e-18,  6.93889390e-18,  6.93889390e-18,
       -6.93889390e-18,  0.00000000e+00,  6.93889390e-18,  6.93889390e-18,
        1.38777878e-17,  0.00000000e+00, -3.46944695e-18, -3.46944695e-18,
       -6.93889390e-18,  6.93889390e-18,  1.38777878e-17, -3.46944695e-18,
        1.04083409e-17,  3.46944695e-18,  6.93889390e-18,  1.73472348e-17,
       -6.93889390e-18, -6.93889390e-18,  0.00000000e+00, -1.04083409e-17,
        0.00000000e+00, -3.46944695e-18,  3.46944695e-18,  0.00000000e+00,
       -6.93889390e-18, -3.46944695e-18, -6.93889390e-18,  0.00000000e+00,
        6.93889390e-18,  3.46944695e-18,  0.00000000e+00,  3.46944695e-18,
       -1.04083409e-17,  1.04083409e-17,  1.04083409e-17, -6.93889390e-18,
       -3.46944695e-18,  0.00000000e+00,  0.00000000e+00,  6.93889390e-18,
        6.93889390e-18,  1.38777878e-17,  3.46944695e-18, -1.38777878e-17,
        0.00000000e+00, -3.46944695e-18, -3.46944695e-18, -3.46944695e-18,
        0.00000000e+00,  0.00000000e+00,  6.93889390e-18, -6.93889390e-18,
       -1.04083409e-17,  1.38777878e-17,  0.00000000e+00,  3.46944695e-18,
        0.00000000e+00, -3.46944695e-18,  3.46944695e-18,  3.46944695e-18,
        3.46944695e-18,  3.46944695e-18,  3.46944695e-18, -6.93889390e-18,
        6.93889390e-18, -1.38777878e-17,  3.46944695e-18,  0.00000000e+00,
        6.93889390e-18, -3.46944695e-18,  1.04083409e-17, -6.93889390e-18,
        6.93889390e-18, -3.46944695e-18, -1.04083409e-17,  6.93889390e-18])

[jameslamb]

I see that you double-posted this here and on Stack Overflow (link). Please do not do that.

Maintainers here also monitor the [lightgbm] tag on Stack Overflow. I could have been spending time preparing an answer here while another maintainer was spending time answering your Stack Overflow post, which would have been a waste of maintainers' limited attention that could otherwise have been spent improving this project. Double-posting also makes it less likely that others with a similar question will find the relevant discussion and answer.

[btngcm]

Hi @jameslamb Thanks for reply, I will delete my post on Stack Overflow.

[AhmedThahir]

So what's the answer to the question?

[Shanxce]

I tried a lot of parameters and read the source code and other answers, there were two questions:

1. When using Custom Objective, the feature_fraction is different from the Default case
2. When using custom objective, the init_score is 0 and the default case is mean(labels)
   problem 2 can be solved using init_score, and problem 1 is mentioned in Feature fraction gives different column selections when using a custom objective #6053.

init_score = np.full(y.shape, fill_value=np.mean(y))
train_data = lgb.Dataset(X, label=y, init_score=init_score)