LdSvmTrainer slow behaviour when loading from a remote database

[superichmann]

System Information (please complete the following information):

• OS & Version: Windows 10
• ML.NET Version: 4.0.0
• .NET Version: 9

Describe the bug
When using DatabaseLoader from a remote database, the loading time and overall running time is much longer then other trainers.
As well the memory does not fills up quickly in comparison to different trainers

To Reproduce
Run LdSvmTrainer on a large dataset with hundreds of columns and millions of rows, should leave memory footprint. but it just consumes more and more data from the network sql and not fills up the memory, also takes about 100 times longer then lgbm / ff.

Run a different trainer (lgbm/ff) on the same dataset and see the memory footprint.

Expected behavior
LdSvmTrainer shoud load the entire data onto memory, similar to other ml.net trainers and then work on them.

[michaelgsharp]

ML.NET by default usually does not cache data. If you want to cache the data, in general you can do mlContext.Data.Cache(data);. In the case of the LdSvmTrainer, there is an option you can set in the trainer options themselves to directly cache the data. That does default to true, but if you aren't seeing it cached its probable you are overwriting the default somehow. That should resolve your issue.

I'm going to leave this open but mark it as needing action from you. If it works, you don't need to do anything and the issue will auto close. If it doesn't work let us know and it won't be auto closed.

[superichmann]

Hi @michaelgsharp, thanks for the reply.

I am adding my code example so you could accurately assess the cache situation.

All data is cached with command
mlContext.Data.Cache(data); as you mentioned.

options are:

      var options = new LdSvmTrainer.Options
      {
        LabelColumnName = targetCol,
        FeatureColumnName = "Features",
        Cache=true,
        NumberOfIterations=1000
      };

Exact pipeline is (deduce the pipelines by its names):

var pipeline = MinMax.Append(DoubleToSinglePipeline).Append(mlc.Transforms.Categorical.OneHotEncoding(CategoricalColumns.ToArray())).Append(mlc.Transforms.Concatenate("Features", featuresArray.ToArray()).Append((mlc.BinaryClassification.Trainers.LdSvm(options))));
model = pipeline.Fit(data);

My original issue as posted compares LightGbm and LdSvm with these exact code examples.

[michaelgsharp]

dumb question, but when you do the mlContext.Data.Cache(data);, are you also re-assigning it? data = mlContext.Data.Cache(data);?

How much data are you using? Like in terms of MB? Is the amount of data you have small enough to fit into memory?

[michaelgsharp]

And your pipeline is helpful, but a lot of the options aren't included in your code samples. Could you by chance share everything for a complete repro?

[superichmann]

data is couple of gigas, fits into memory.
full pipeline:

DatabaseLoader loader = mlContextontext.Data.CreateDatabaseLoader(columns); // DatabaseLoader.Column[] columns was populated beforehand
DatabaseSource dbSource = new DatabaseSource(NpgsqlFactory.Instance, connectionString, sqlSelect, TIMEOUT_SECONDS); // sqlSelect contains all columns explicitly, not *
var train = loader.Load(dbSource);
train = mlContext.Data.Cache(train);
var doubleColumns = train.Schema.Where(col => col.Type == NumberDataViewType.Double).Select(col => new InputOutputColumnPair($"{col.Name}", col.Name));
var preDoubleToSingle = mlContext.Transforms.Conversion.ConvertType(doubleColumns.ToArray(), DataKind.Single);

var options = new LdSvmTrainer.Options
{
	LabelColumnName = targetCol,
	FeatureColumnName = "Features",
	Cache=true,
	NumberOfIterations=1000
};
List<InputOutputColumnPair> CATsArray = CI.CategoricalColumnNames.Select(col => new InputOutputColumnPair(col)).ToList(); // CI ColumnInformation was created beforehand with all string/bool based cols.
EstimatorChain<NormalizingTransformer> MinMax = new EstimatorChain<NormalizingTransformer>();
foreach (var col in CI.NumericColumnNames)
{
	MinMax = MinMax.Append(this.mlContext.Transforms.NormalizeMinMax(col));
}
TransformerChain<TransformerChain<BinaryPredictionTransformer<LdSvmModelParameters>>> model;
var pipeline = MinMax.Append(preDoubleToSingle).Append(mlContext.Transforms.Categorical.OneHotEncoding(CATsArray.ToArray())).Append(mlContext.Transforms.Concatenate("Features", featuresArray.ToArray()).Append((mlContext.BinaryClassification.Trainers.LdSvm(options))));
model = pipeline.Fit(train);

in the last few lines related to the LdSvm itself, if you use the same pipeline but replace to lgbm, you will see a different memory footprint (much more memory in use) and way less network traffic, also finishes much faster.