Saving and reloading boosters using IOBuffer

[gzarpapis]

Hello everyone. I'm trying to write my own keep_best_model_instance function, by saving my Booster when the validation loss decreases. I understand there is a specific function to predict on the best instance, but I need the model instance itself.

All my tries with save and load, serialize and deserialize with an IOBuffer have been unsuccessful and I could not find any documentation or examples on the topic.

If anyone has done this before, would you kindly guide how to achieve this? Also, maybe include this on the documentation?

Thank you for your time.

[ExpandingMan]

All my tries with save and load, serialize and deserialize with an IOBuffer have been unsuccessful and I could not find any documentation or examples on the topic.

Can you provide an MWE or, at the very least, an explicit error dump? I don't know what to make of this description. There are no model finalization steps, so as far as I know you should be able to call XGBoost.save(booster, io) at any time between calls to update! or updateone!.

[gzarpapis]

Here's my MWE. All attempts except 2, 3 and 4 throw error. My attempts 2, 3, 4 however give wrong result. Next comment will be my output.

    d_train = DMatrix(x_train, label=y_train);
    d_test = DMatrix(x_test, label=y_test);


    bst = Booster(d_train; max_depth = 20, XGBoost.regression(eval_metric = "mape")..., XGBoost.randomforest()...);
    best_loss = typemax(Float32);
    best_i = 0;
    buffer_save = IOBuffer(read=true, write=true);
    buffer_serial = IOBuffer(read=true, write=true);

    for i in 1:10
        update!(bst, d_train, watchlist = watch);
        pred = XGBoost.predict(bst, x_test);
        curr_loss = error(pred, y_test);
        println(curr_loss);
        if curr_loss < best_loss
            
            best_i = i;
            best_loss = curr_loss

            XGBoost.save(bst, buffer_save);
            buffer_serial = IOBuffer(XGBoost.serialize(bst));
            println(best_i);        
        end
    end

    assert_it_works = final -> (begin; pred = XGBoost.predict(final, x_test); curr_loss = error(pred, y_test); return (curr_loss, "Works: " * string(curr_loss == best_loss)); end);

    final_bst = Booster(DMatrix[]);
    try
        seekstart(buffer_save);
        final_bst = XGBoost.load(typeof(XGBoost.Booster));

        println(assert_it_works(final_bst));
    catch e;
        write(stdout, string(e));
        println("\nAttempt 1 Failed\n")
    end
    
    try
        seekstart(buffer_save);
        final_bst = XGBoost.Booster(DMatrix[], model_buffer = buffer_save);

        println(assert_it_works(final_bst));
    catch e;
        write(stdout, string(e));
        println("\nAttempt 2 Failed\n")
    end

    try
        seekstart(buffer_save);
        XGBoost.load!(final_bst, buffer_save);

        println(assert_it_works(final_bst));
    catch e;
        write(stdout, string(e));
        println("\nAttempt 3 Failed\n")
    end

    try
        seekstart(buffer_save);
        XGBoost.load!(final_bst, take!(buffer_save));

        println(assert_it_works(final_bst));
    catch e;
        write(stdout, string(e));
        println("\nAttempt 4 Failed\n")
    end
    
    try
        seekstart(buffer_save);
        XGBoost.load!(final_bst, buffer_save);

        println(assert_it_works(final_bst));
    catch e;
        write(stdout, string(e));
        println("\nAttempt 5 Failed\n")
    end
    
    try
        seekstart(buffer_serial);
        final_bst = XGBoost.deserialize!(final_bst, buffer_serial);

        println(assert_it_works(final_bst));
    catch e;
        write(stdout, string(e));
        println("\nAttempt 6 Failed\n")
    end

    try
        seekstart(buffer_serial);
        final_bst = XGBoost.deserialize!(final_bst, take!(buffer_serial));

        println(assert_it_works(final_bst));
    catch e;
        write(stdout, string(e));
        println("\nAttempt 7 Failed\n")
    end

    try
        seekstart(buffer_serial);
        final_bst = XGBoost.deserialize!(final_bst, read(buffer_serial));

        println(assert_it_works(final_bst));
    catch e;
        write(stdout, string(e));
        println("\nAttempt 8 Failed\n")
    end

    try
        final_bst = XGBoost.deserialize(buffer_serial);

        println(assert_it_works(final_bst));
    catch e;
        write(stdout, string(e));
        println("\nAttempt 9 Failed\n")
    end

    try
        final_bst = XGBoost.deserialize(Type{XGBoost.Booster}, take!(buffer_serial));

        println(assert_it_works(final_bst));
    catch e;
        #write(stdout, string(e));
        println("\nAttempt 10 Failed\n")
    end

    try
        final_bst = XGBoost.deserialize(typeof(final_bst), take!(buffer_serial));

        println(assert_it_works(final_bst));
    catch e;
        write(stdout, string(e));
        println("\nAttempt 11 Failed\n")
    end

[gzarpapis]

[ Info: [1]     train-mape:1.51966291478350146  valid-mape:1.54929523610775366
1.5492952
1
[ Info: [2]     train-mape:0.40160081844888185  valid-mape:0.46602336089583418
0.46602336
2
[ Info: [3]     train-mape:0.17652566519803181  valid-mape:0.26984500867009859
0.269845
3
[ Info: [4]     train-mape:0.12303823920340601  valid-mape:0.23781751415377347
0.23781751
4
[ Info: [5]     train-mape:0.10996602844796843  valid-mape:0.23461653884055558
0.23461653
5
[ Info: [6]     train-mape:0.10870173813734055  valid-mape:0.23464444234406565
0.23464446
[ Info: [7]     train-mape:0.10713973679701468  valid-mape:0.23546314605696411
0.23546316
[ Info: [8]     train-mape:0.10624772975655812  valid-mape:0.23561708124796898
0.23561707
[ Info: [9]     train-mape:0.10658877726512184  valid-mape:0.23602501327988257
0.236025
[ Info: [10]    train-mape:0.10552233684595115  valid-mape:0.23619752531929100
0.23619755
MethodError(XGBoost.load, (DataType,), 0x0000000000007be8)
Attempt 1 Failed

(1.5492952f0, "Works: false")
(1.5492952f0, "Works: false")
(1.5492952f0, "Works: false")
XGBoost.Lib.XGBoostError(XGBoost.Lib.XGBoosterLoadModelFromBuffer, "[10:17:59] /workspace/srcdir/xgboost/src/learner.cc:1003: Check failed: fi->Read(&mparam_, sizeof(mparam_)) == sizeof(mparam_) (0 vs. 136) : BoostLearner: wrong model format\nStack trace:\n  [bt] (0) /home/zarpapis/.julia/artifacts/271facf086d4d0b748a1835be4e1208876f382f9/lib/libxgboost.so(+0x47fd34) [0x7fac29e07d34]\n  [bt] (1) /home/zarpapis/.julia/artifacts/271facf086d4d0b748a1835be4e1208876f382f9/lib/libxgboost.so(xgboost::LearnerIO::LoadModel(dmlc::Stream*)+0x2b1) [0x7fac29e2a5a1]\n  [bt] (2) /home/zarpapis/.julia/artifacts/271facf086d4d0b748a1835be4e1208876f382f9/lib/libxgboost.so(XGBoosterLoadModelFromBuffer+0x40) [0x7fac29b0e150]\n  [bt] (3) [0x7fac8f8c8fc7]\n  [bt] (4) [0x7fac8f8c9037]\n  [bt] (5) /home/zarpapis/julia-1.10.0/bin/../lib/julia/libjulia-internal.so.1.10(ijl_apply_generic+0x2ae) [0x7faca59f799e]\n  [bt] (6) [0x7fac8f86388e]\n  [bt] (7) [0x7fac8f892e77]\n  [bt] (8) [0x7fac8f897703]\n\n")
Attempt 5 Failed

MethodError(XGBoost.deserialize!, (Booster(), IOBuffer(data=UInt8[...], readable=true, writable=false, seekable=true, append=false, size=10387011, maxsize=Inf, ptr=1, mark=-1)), 0x0000000000007be8)
Attempt 6 Failed

XGBoost.Lib.XGBoostError(XGBoost.Lib.XGBoosterUnserializeFromBuffer, "[10:17:59] /workspace/srcdir/xgboost/src/learner.cc:1182: Check failed: header == serialisation_header_: If you are loading a serialized model (like pickle in Python, RDS in R) or\nconfiguration generated by an older version of XGBoost, please export the model by calling\n`Booster.save_model` from that version first, then load it back in current version. See:\n\n    https://xgboost.readthedocs.io/en/stable/tutorials/saving_model.html\n\nfor more details about differences between saving model and serializing.\n\nStack trace:\n  [bt] (0) /home/zarpapis/.julia/artifacts/271facf086d4d0b748a1835be4e1208876f382f9/lib/libxgboost.so(+0x47fd34) [0x7fac29e07d34]\n  [bt] (1) /home/zarpapis/.julia/artifacts/271facf086d4d0b748a1835be4e1208876f382f9/lib/libxgboost.so(xgboost::LearnerIO::Load(dmlc::Stream*)+0x287) [0x7fac29e2d177]\n  [bt] (2) /home/zarpapis/.julia/artifacts/271facf086d4d0b748a1835be4e1208876f382f9/lib/libxgboost.so(XGBoosterUnserializeFromBuffer+0x48) [0x7fac29b0dbf8]\n  [bt] (3) [0x7fac8f8cb587]\n  [bt] (4) [0x7fac8f8cb5f7]\n  [bt] (5) /home/zarpapis/julia-1.10.0/bin/../lib/julia/libjulia-internal.so.1.10(ijl_apply_generic+0x2ae) [0x7faca59f799e]\n  [bt] (6) [0x7fac8f86388e]\n  [bt] (7) [0x7fac8f893852]\n  [bt] (8) [0x7fac8f897703]\n\n")
Attempt 7 Failed

XGBoost.Lib.XGBoostError(XGBoost.Lib.XGBoosterUnserializeFromBuffer, "[10:17:59] /workspace/srcdir/xgboost/src/learner.cc:1182: Check failed: header == serialisation_header_: If you are loading a serialized model (like pickle in Python, RDS in R) or\nconfiguration generated by an older version of XGBoost, please export the model by calling\n`Booster.save_model` from that version first, then load it back in current version. See:\n\n    https://xgboost.readthedocs.io/en/stable/tutorials/saving_model.html\n\nfor more details about differences between saving model and serializing.\n\nStack trace:\n  [bt] (0) /home/zarpapis/.julia/artifacts/271facf086d4d0b748a1835be4e1208876f382f9/lib/libxgboost.so(+0x47fd34) [0x7fac29e07d34]\n  [bt] (1) /home/zarpapis/.julia/artifacts/271facf086d4d0b748a1835be4e1208876f382f9/lib/libxgboost.so(xgboost::LearnerIO::Load(dmlc::Stream*)+0x287) [0x7fac29e2d177]\n  [bt] (2) /home/zarpapis/.julia/artifacts/271facf086d4d0b748a1835be4e1208876f382f9/lib/libxgboost.so(XGBoosterUnserializeFromBuffer+0x48) [0x7fac29b0dbf8]\n  [bt] (3) [0x7fac8f8cb587]\n  [bt] (4) [0x7fac8f8cb5f7]\n  [bt] (5) /home/zarpapis/julia-1.10.0/bin/../lib/julia/libjulia-internal.so.1.10(ijl_apply_generic+0x2ae) [0x7faca59f799e]\n  [bt] (6) [0x7fac8f86388e]\n  [bt] (7) [0x7fac8f893ff2]\n  [bt] (8) [0x7fac8f897703]\n\n")
Attempt 8 Failed

MethodError(XGBoost.deserialize, (IOBuffer(data=UInt8[...], readable=true, writable=false, seekable=true, append=false, size=10387011, maxsize=Inf, ptr=10387012, mark=-1),), 0x0000000000007be8)
Attempt 9 Failed


Attempt 10 Failed

MethodError(Booster, (), 0x0000000000007be8)
Attempt 11 Failed

[ExpandingMan]

So, when you call load it is providing the buffer as a parameter to the model constructor, however, when you call load! it's calling a different method called XGBoosterLoadModelFromBuffer. Confusingly, it seems that this is NOT the inverse of XGBoosterSaveModelToBuffer which is what we use to save. Confusion over the various serialization methods in libxgboost has popped up a number of times, and it seems it's still not fully resolved.

I have opened this issue in libxgboost to try to get some clarification. I agree that, regardless of what the maintainers say, the current state of load, load! and save functions in the Julia wrapper is extremely confusing (safe to say me and probably any other maintainers are themselves confused about the intended use of these functions) so I think we should either get those functions to work the way you are trying to use them, or we should deprecate load!.

In the meantime, I suggest you simply use save and load and not load. It's not clear to me there's any real disadvantage to this, as the overhead of creating the booster object in the first place is pretty low.

[gzarpapis]

After looking into it some more, I've realized a few things.

1. I was using serialize wrong. It should look like this:

# Make a new IOBuffer or ...
# ... empty the buffer if reusing it in a loop
b = IOBuffer(read = true, write = true);

# Write to buffer
write(b, XGBoost.serialize(bst));

# - - - - - Do other things - - - - - #

# Create a new Booster
final_bst = Booster(DMatrix[]);

# Go to position 0 of buffer and deserialize
seekstart(b);
deserialize!(final_bst, read(b)); # Or take!(b)

The final_bst object should now contain the proper instance of the model and not its latest. In my above example I was using the IOBuffer incorrectly, so don't pay attention to that.

2. save and load work with their string implementation for creating a file and reading from it, but I still don't know if it works with buffers.

3. Former functions are not even necessary if you don't care about a human-readable json format. What I mean is, the IOBuffer object can be written to and read by a file, no problem. So serializing - deserializing can be used for all purposes instead.