GlassNet predictions are inconsistent after updating from 0.5.3 to 0.6.0

[Lexachoc]

Hi,

I tested the GlassNet prediction results for the new version glasspy v0.6.0 with scikit-learn 1.7.1 as required for the new VITRIFY model if framework="sklearn".

I've noticed that GlassNet's prediction results differ noticeably:

GlassNet (GlassPy) v0.5.3 with scikit-learn 1.6.1 (in SciGlass Next)

GlassNet (GlassPy) v0.6.0 with scikit-learn 1.7.1 (in SciGlass Next)

I didn't realize that the version of scikit-learn could make such noticeable difference of about 5 to 20 °C until I did this update.
The dependencies list only specifies "scikit-learn>=1.5.0".

What do you sugguest?

Maybe we should freeze the version of scikit-learn that you used to train the GlassNet originally? Or should there be new scaler files (joblib files) created for the newer version of scikit-learn?

Or I'll just keep using scikit-learn 1.6.1, since I'm actually only using the default framework with onnxruntime.

But I'm still wondering if GlassNet's prediction results with scikit-learn 1.6.1 also differ from the original prediction (Which version did you use?)

UPDATE: The real cause is not the version of scikit-learn, but the file https://github.com/drcassar/glasspy/blob/v0.6.0/glasspy/data/load.py !

[drcassar]

Thank you for bringing this issue to my attention, @Lexachoc.

The original scikit-learn version used to create GlassNet was 1.2.0 and since 1.0.0 there are no major changes in the way the scalers used in GlassNet work. That is why the scalers are not offered as joblib files and instead are generated during first run.

However, I recently changed the glasspy.data submodule. This may be the root of the problem. I will look into it.

Thanks again and please use 0.5.3 until I fix this issue.

[Lexachoc]

Thanks for the quick reply! @drcassar

I actually also tested to locate the problem by modifying the version.json (changing the scikit-learn version back to 1.6.1 and then pip install scikit-learn==1.6.1) to see how it worked.

It turned out that once I rollback scikit-learn to 1.6.1, the GlassNet results are the same as the previous version.

Hope this info helps.

[drcassar]

This is an interesting info! Thank you for letting me know!

[Lexachoc]

@drcassar

Oh! Wait a minute! You're right. Somehow, my local version was still using the old dataset (old joblib files) during testing, which led me to the wrong conclusion that the scikit-learn version was the cause of the inconsistent results...

Now I can confirm that I am using the new joblib files generated from the new dataset. The file sizes of these two scaler files differ from those of the previous ones.

The scikit-learn is not the cause!! This file https://github.com/drcassar/glasspy/blob/v0.6.0/glasspy/data/load.py is!

I replaced the load.py file (0.6.0) with the older version (0.5.3), and the results are consistent regardless of the scikit-learn version used (1.6.1 or 1.7.1).

[drcassar]

Great, thank you! I will look into this issue and try to fix it this week. As you run the SciGlass Next website, it is better to stay on the previous version for now. Thanks again for all your help!

[Lexachoc]

BTW, I was concerned about the initialization overhead of the onnx framework, so I ran a few benchmarks in Jupyter Notebook.

Tested with Intel(R) Core(TM) i7-10850H CPU @ 2.70GHz (6 Cores) with Windows 10.

from glasspy.predict import VITRIFY
import pandas as pd
import numpy as np
import time

start = time.perf_counter()
model_onnx = VITRIFY(framework="onnx")
end = time.perf_counter()
print(f"init time: {end - start:.6f} s")

init time: 114.079646 s

Almost 2 mins! Do you think the init time of around 2 mins with onnx is normal?

====================================================

start = time.perf_counter()
model_sklearn = VITRIFY(framework="sklearn")
end = time.perf_counter()
print(f"init time: {end - start:.6f} s")

init time: 0.782374 s

Very fast!

====================================================

start = time.perf_counter()
model_onnx_unzipped = VITRIFY(framework="onnx") # but I unzip the CHEM.onnx.gz and then use rt.InferenceSession(path/to/CHEM.onnx)
end = time.perf_counter()
print(f"init time: {end - start:.6f} s")

init time: 109.743247 s

So skipping the unzipping step only saves about ~ 5s

====================================================

And the prediction performance test (run 10 times for average):

base = pd.DataFrame(
    [[1, 0, 2], [0, 1, 2], [1, 1, 2]],
    columns=["Li2O", "Na2O", "SiO2"]
)

def make_df(n):
    reps = int(np.ceil(n / len(base)))
    return pd.concat([base] * reps, ignore_index=True).iloc[:n]

def bench(model, df, repeats=10):
    # warm-up
    model.predict_proba_glass(df)

    times = []
    for _ in range(repeats):
        start = time.perf_counter()
        model.predict_proba_glass(df)
        end = time.perf_counter()
        times.append((end - start)*1000) # in ms

    return {
        "mean_ms": np.mean(times),
        "median_ms": np.median(times)
    }

sizes = [1, 100, 1000, 5000]

results = []

for n in sizes:
    df = make_df(n)

    r_onnx = bench(model_onnx, df)
    r_sklearn = bench(model_sklearn, df)

    results.append({
        "n": n,
        "onnx_mean_ms": round(r_onnx["mean_ms"], 3),
        "onnx_median_ms": round(r_onnx["median_ms"], 3),
        "sklearn_mean_ms": round(r_sklearn["mean_ms"], 3),
        "sklearn_median_ms": round(r_sklearn["median_ms"], 3),
        "speedup_onnx_over_sklearn": round(
            r_sklearn["median_ms"] / r_onnx["median_ms"],
            3
        ),
    })

pd.DataFrame(results)

n	onnx_mean_ms	onnx_median_ms	sklearn_mean_ms	sklearn_median_ms	speedup_onnx_over_sklearn
1	1.693	1.590	64.180	62.165	39.087
100	3.706	3.660	86.189	86.564	23.650
1000	15.849	16.222	113.031	108.945	6.716
5000	71.164	70.834	272.438	268.124	3.785

n is the number of compositions as input for prediction.

It is obvious that using onnx framework is much faster than using sklearn. This is especially true for a small number of compositions (which is usually the case).

Since the model will run on a server in the end, the init overhead is not very important to me. I will therefore stick with the default onnx framework due to its much faster prediction speed.

[drcassar]

That is a very interesting analysis, @Lexachoc! Thank you!

I just added the onnx framework because I wanted something to resist the test of time, as sklearn keeps changing, but I also wanted to have the original sklearn files available for those who are interested in the original model. Keep in mind that onnx framework gives slightly different predictions than sklearn due to the 64-to-32 bits conversion that was necessary to convert the sklearn model to onnx. I am not that familiar with onnx to say if a 2 min init time is reasonable or not, sorry, but I guess your usecase justify the use of onnx framework as you are in a server, as you said.

[drcassar]

Hi @Lexachoc, could you please try to install the dev branch (https://github.com/drcassar/glasspy/tree/dev) and see if the GlassNet predictions are what you were expecting from the previous version of GlassNet? If yes I will move forward to rebuild the documentation and upload to pypi.

[Lexachoc]

Hi @Lexachoc, could you please try to install the dev branch (https://github.com/drcassar/glasspy/tree/dev) and see if the GlassNet predictions are what you were expecting from the previous version of GlassNet? If yes I will move forward to rebuild the documentation and upload to pypi.

I've tested the dev branch, and the GlassNet prediction results now agree with those of 0.5.3.

P.S. BTW I have some doubts about the steps (like the order of drop_duplicates, set_index, and convert, etc) of processing for cache, not sure if this can make a difference for the source data using SciGlass() with cache or without. As I only tested for GlassNet model predictions. But the results for GlassNet are the same.

[drcassar]

Great! Thanks a lot, @Lexachoc. I will also review the data processing steps because the code changed a bit when I introduced the cache process.