[Bug]: Training Server fails to start with LATENCY_MODEL_TYPE=xgboost due to unconditional LightGBM import

[modassarrana89-new]

Contact Details

modassar.rana@ibm.com

What happened?

Problem

The Training Server fails to start when LATENCY_MODEL_TYPE=xgboost is configured because training_server.py imports the LightGBM module during application startup, before the configured model type is evaluated.

Although the ConfigMap specifies:

LATENCY_MODEL_TYPE=xgboost

the application still executes:

import lightgbm as lgb

during module import.

On IBM Z (s390x), the installed LightGBM shared library fails to load:

OSError:
/usr/local/lib/python3.14/site-packages/lightgbm/lib/lib_lightgbm.so:
undefined symbol: _ZTVN10__cxxabiv117__class_type_infoE

This exception occurs before the application reads the configuration or constructs the predictor, causing the Training Server to terminate and enter CrashLoopBackOff.

Expected Behavior

When LATENCY_MODEL_TYPE=xgboost is configured, the Training Server should initialize only the XGBoost backend. Optional ML frameworks (e.g. LightGBM) should not prevent the service from starting.

If LightGBM is unavailable, it should not impact XGBoost deployments.

Actual Behavior

Application startup sequence:

1. Python starts
2. Imports XGBoost
3. Imports LightGBM
4. LightGBM shared library fails to load (OSError)
5. Process exits
6. Pod enters CrashLoopBackOff

The configured model type is never evaluated.

Environment

• Platform: OpenShift
• Architecture: IBM Z (s390x)
• Python: 3.14
• Config:
  LATENCY_MODEL_TYPE=xgboost

Suggested Fix

One of the following approaches:

1. Perform lazy imports based on the configured model type:
   • Import XGBoost only when MODEL_TYPE == xgboost
   • Import LightGBM only when MODEL_TYPE == lightgbm

OR

2. Treat LightGBM as an optional dependency by catching both ImportError and OSError during import so that unsupported native libraries do not prevent the Training Server from starting.

This issue may also affect the Prediction Server because it follows the same startup/import pattern.

Version

main

Steps to Reproduce

Build or use the Training Server image containing both XGBoost and LightGBM dependencies.
Deploy the Training Server on an OpenShift cluster (IBM Z / s390x).

Configure the Training Server with:

LATENCY_MODEL_TYPE=xgboost
Start the Training Server.
Observe the Training Server pod entering CrashLoopBackOff.
Observe the application failing during startup while importing LightGBM.

Environment

Relevant log output

[modassarrana89-new]

/assign @modassarrana89-new