[CUDA] [python-package] Regarding the time consumption of training LightGBM with CUDA

[Jessee-Jiao]

Hi, here's the context regarding the three potential issues you mentioned earlier:

First, my dataset size is in the millions (referring to the number of rows, sometimes close to ten million), and the number of features is about 250. Is this number of columns considered small, as you mentioned? I'm not sure.

Second, I did use a custom metric function: R-squared. The specific code is as follows:

def r2_metric(pred, dtrain):
    label = dtrain.get_label()
    
    loss = (pred - label) ** 2
    var = (label - np.mean(label)) ** 2
    
    r2 = 1 - sum(loss) / sum(var)
    return 'r2', r2, True

Lastly, the training time I measured, which showed almost no improvement, refers solely to the time taken by lgb.train(), excluding other times such as data loading, plotting, etc.

I will provide some details about the parameter settings below, excluding classic parameters like learning rate, number of leaves, depth, etc. (I don’t think these are relevant to this issue).

'device': 'cuda' if use_cuda else 'cpu',
'gpu_platform_id': 0,
'gpu_device_id': 1,
'tree_type': 'data_parallel',
'gpu_use_dp': True,


'verbosity': -10,
'n_jobs': 32,

evals_result = {}

callbacks = [log_evaluation(period=40), early_stopping(stopping_rounds), record_evaluation(evals_result)]

gbm = lgb.train(
    params,
    lgb_train,
    num_boost_round=400,
    feval=r2_metric,
    valid_sets=[lgb_train, lgb_valid],
    valid_names=['train', 'validation'],
    callbacks=callbacks,
)

Additionally, I’d like to mention that I used Optuna for hyperparameter tuning. Could this have any impact? (However, the training time I mentioned above still refers to the time taken for a single training run with one set of parameters.) I’m sorry, but I also want to point out that when I switched to GPU with CatBoost and used Optuna, the training speed improved significantly. However, with LightGBM, there was no such improvement. Moreover, CatBoost doesn’t support GPU computation for R-squared, but that doesn’t affect its speed. I really need to use LightGBM, so I’m quite confused. Sorry again.
Thank you！

[jameslamb]

Thanks for opening this.

here's the context regarding the three potential issues you mentioned earlier

Please, use links. This helps everyone reading to understand how things relate to each other.

In this case, I know what you're referring to and will do it for you: #6969 (comment)

my dataset size is in the millions (referring to the number of rows, sometimes close to ten million), and the number of features is about 250. Is this number of columns considered small, as you mentioned

No, that's not what I meant by "small". It should be possible to achieve some speedups using the CUDA version on a dataset of that size.

GPU

Please share the output of running this command:

nvidia-smi

excluding classic parameters

Please, provide the exact parameters you used. You are asking us to investigate and give you ideas on what is happening... we need you to tell us enough information to replicate what you are seeing in our own development environments.

It would be helpful if you could provide a minimal, reproducible example, as I requested in #6969 (comment).

Start from something like this:

from sklearn.datasets import make_regression
import lightgbm as lgb

X, y = make_regression(n_samples=10_000, n_features=10, random_state=708)
params = {
    "num_leaves": 50
}

dtrain = lgb.Dataset(X, label=y, params=params)
bst = lgb.train(
    train_set=dtrain,
    params=params,
    num_boost_round=100,
)

Then modify the code to the MOST MINIMAL one that replicates the issue you are seeing.

Could [Optuna] have any impact? (However, the training time I mentioned above still refers to the time taken for a single training run with one set of parameters.)

Please, try to reduce this to a minimal example.

If you see this slowdown switching from CPU training to GPU training...

• ... without optuna? exclude optuna from your example.
• ... using a built-in objective instead of a custom one? use the built-in one.
• ... without the use of other callbacks? exclude the other callbacks

The less external libraries and customization in your code, the easier it will be for someone to help you.

I’m sorry, but I also want to point out that when I switched to GPU with CatBoost and used Optuna, the training speed improved significantly. However, with LightGBM, there was no such improvement.

• are you sure you used similar parameters (like max_depth)?
• is the CatBoost model actually equivalent / better on evaluation metrics? If not, that's not a fair comparison.
• did you use early stopping with CatBoost as well? did training stop at exactly the same iteration?

CatBoost very well may be faster for your dataset and problem, but if you want to use claims like this to help identify performance problems in LightGBM, you need to be very careful and precise about what you are comparing.

[Jessee-Jiao]

Thank you very much for your detailed response. I’m sorry that I didn’t have access to a device earlier to provide the details you requested.Also, I’m very sorry, but I’m not very familiar with the operations for opening an issue on GitHub. Please forgive me. And thank you very much for your patience in helping me resolve the issue.

nvidia-smi

First, here is the output of nvidia-smi:

... using a built-in objective instead of a custom one? use the built-in one.

Second, I conducted experiments regarding the custom metric:
(1) W/o R2 (using L2) - CPU: 113s
(2) W/o R2 (using L2) - CUDA: 43s
(3) W/ R2 - CPU: 500s
(4) W/ R2 - CUDA: 472s

It’s almost certain that the custom R2 evaluation metric is the cause. Is there a way to resolve this? If I want to benefit from faster training, do I have to abandon using R2?

are you sure you used similar parameters

did you use early stopping with CatBoost as well? did training stop at exactly the same iteration?

Regarding the CatBoost issue you mentioned last, both models have similar parameters (e.g., max depth, number of leaves, learning rate, regularization coefficients, etc.), but since their algorithms are different, it’s not guaranteed that early stopping would occur at the same number of iterations on the same data. However, they do achieve nearly comparable performance. I’m not actually trying to compare their performance, as that’s quite complex. I’m just puzzled because CatBoost can achieve fast training on GPU even with R2, while I couldn’t achieve the same with LightGBM.

[jameslamb]

It’s almost certain that the custom R2 evaluation metric is the cause

Thanks for measuring those timings! Yes, I agree. It looks like you got a good speedup with the CUDA variant relative to the CPU variant, but that the evaluation of your custom R2 metric dominates the training time.

Is there a way to resolve this? If I want to benefit from faster training, do I have to abandon using R2?

You could try speeding up your custom metric. For example, pre-compute things that are not changing and store them on the class, so you don't spend time re-computing them on every iteration. For example, np.mean(label) is probably not changing over the course of training, so re-computing it every time is wasteful.

def CustomR2:

    def __init__(self):
        self.initialized = False
        self.label_mean = -1.0

    def __call__(self, pred, dtrain):
        label = dtrain.get_label()
        # compute self.label_mean if it hasn't been computed before
        if not self.initialized:
            self.label_mean = np.mean(label)
            self.initialized = True
         residuals = (pred - label) ** 2
         var = (label - self.label_mean) ** 2
         r2 = 1 - sum(loss) / sum(var)
        return 'r2', r2, True

You could also try using something like scikit-learn's R^2 metric (https://scikit-learn.org/stable/modules/generated/sklearn.metrics.r2_score.html), maybe it will be faster than your custom one.

I'm sorry but until you provide us a minimal, reproducible example, as I've asked for several times now, there is not much more we can do to help.

_ I’m just puzzled because CatBoost can achieve fast training on GPU even with R2, while I couldn’t achieve the same with LightGBM_

I’m not actually trying to compare their performance, as that’s quite complex.

Then why mention CatBoost? If your goal is to help identify performance issues in LightGBM or your own code, then I could see how mentioning the performance of a similar project would be helpful... the issues may lie in the differences between LightGBM and CatBoost.

But otherwise, mentioning it but not providing enough detail (reproducible example(s)) for us to investigate does not really help us make any progress on your report.

[Jessee-Jiao]

Thank you very much for your response. I apologize for not yet providing a minimal, reproducible example. This is partly because my work environment was inconvenient yesterday (weekend), and partly because, after testing, I’m almost certain the issue is related to the custom metric. In fact, most of the details in my code have already been shared with you (nothing particularly special), and it should suffice to add parameters like num_leaves: 128, learning_rate: 0.08, etc., to the parameters I initially provided.

Of course, your response regarding the custom R² metric has already been very helpful, and I’ll conduct some tests and experiments on this issue first.

Lastly, as a side note, could LightGBM consider including R² as a built-in metric to make it more convenient and faster for users to utilize? In any case, thank you so much for your efforts.

[jameslamb]

most of the details in my code have already been shared with you (nothing particularly special)

Ok, until you can provide a minimal, reproducible example, I personally am not going to spend any more time investigating this.

You might also want to look at the open issues tagged gpu (CUDA) and feature request to see things that do not yet have a CUDA implementation here: https://github.com/microsoft/LightGBM/issues?q=is%3Aissue%20state%3Aopen%20label%3A%22gpu%20(CUDA)%22%20label%3A%22feature%20request%22

For example, LightGBM does not yet have multi-GPU support (#6138), so only 1 of your GPUs will be used.

could LightGBM consider including R² as a built-in metric to make it more convenient and faster for users to utilize?

Sure, we can take that as a feature request. That's a good idea! I've opened #6983 to document that.

[jameslamb]

using the sklearn method, the performance improved from 470s to 110s

Hey great!!

the r2 value turns negative, even though I used gpu_use_dp=True

It is absolutely possible for R^2 to be negative if there is overfitting. As https://scikit-learn.org/stable/modules/generated/sklearn.metrics.r2_score.html says "... it can be negative (because the model can be arbitrarily worse)".

Using gpu_use_dp=True may actually make that more likely, because using lower numerical precision could act as a form of regularization (defense against overfitting).

once the program in terminal1 finishes running, running the program again in a new terminal results in a normal iteration process

It isn't clear what "normal iteration process" means. But my interpretation of what you're saying is "different training runs are producing different results".

In general, the CUDA implementation of LightGBM is not deterministic. CUDA is fundamentally about parallel programming, and some of the forms of parallelization used by LightGBM result in non-deterministic results because the time it takes different parallel computations to complete is non-deterministic.

minimal example code.

Thanks but this is not exactly reproducible. Try copying that code and running it in a new session... it won't work.

For example:

• train_X, train_Y, valid_X, and valid_Y are not defined
• log_evaluation and early_stopping are not defined
• r2_metric is not defined

[Jessee-Jiao]

Hi, I originally wanted to refine the minimal code I provided, but after conducting some experiments, I found that the issue I discovered cannot be reproduced with the minimal code (i.e., randomly generated dummy data) - it works normally!
Therefore, I narrowed down the problem to my data. After some comparative experiments, I confirmed that certain features in my X are causing the issue, leading to mutual interference when the same data and code run simultaneously on CUDA, which results in a very strange training process.
Unfortunately, I cannot present the X data to you to help me, so even providing a directly runnable minimal code would be of no help. I'm not sure if this is a CUDA bug or something else - perhaps it happens with sparse data? I haven't thought of any other potential issues with the data.

However, it's at least unrelated to the custom metric, so I can temporarily use sklearn's R2. But I still need to wait for the "CUDA ignoring max depth" issue to be successfully fixed before I can use CUDA with confidence. Thank you very much for all your responses.