XGBoost different inference results on AMD64, ARM and PPC

[smorovic]

As discussed in PR #44473
we noticed discrepancy in XGBoost inference result with the new unit test RecoEgamma/PhotonIdentification/test/test_PhotonMvaXgb.cc.
Unit test passes on x86_64, but fails in identical fashion and with identical discrepancies on both PPC64 LE and ARM 64, happening in 4 out of 10 tests:

src/RecoEgamma/PhotonIdentification/test/test_PhotonMvaXgb.cc:44: FAILED:
  CHECK_THAT( xgbScore, Catch::Matchers::WithinAbs(mva_score_v1[i], 0.0001) )
with expansion:
  0.91074f is within 0.0001 of 0.9863399863

src/RecoEgamma/PhotonIdentification/test/test_PhotonMvaXgb.cc:44: FAILED:
  CHECK_THAT( xgbScore, Catch::Matchers::WithinAbs(mva_score_v1[i], 0.0001) )
with expansion:
  0.82656f is within 0.0001 of 0.9750099778

src/RecoEgamma/PhotonIdentification/test/test_PhotonMvaXgb.cc:44: FAILED:
  CHECK_THAT( xgbScore, Catch::Matchers::WithinAbs(mva_score_v1[i], 0.0001) )
with expansion:
  0.0f is within 0.0001 of 0.00179

src/RecoEgamma/PhotonIdentification/test/test_PhotonMvaXgb.cc:44: FAILED:
  CHECK_THAT( xgbScore, Catch::Matchers::WithinAbs(mva_score_v1[i], 0.0001) )
with expansion:
  0.93808f is within 0.0001 of 0.9837399721

PR was submitted to disable the check on non-x86_64 for now:
#44531

[cmsbuild]

cms-bot internal usage

[cmsbuild]

A new Issue was created by @smorovic.

@makortel, @Dr15Jones, @smuzaffar, @antoniovilela, @sextonkennedy, @rappoccio can you please review it and eventually sign/assign? Thanks.

cms-bot commands are listed here

[makortel]

assign reconstruction, ml

[cmsbuild]

New categories assigned: reconstruction,ml

@jfernan2,@mandrenguyen,@valsdav,@wpmccormack you have been requested to review this Pull request/Issue and eventually sign? Thanks

[smorovic]

I fetched xgboost v1.7.5 (and for the subpackage dmlc-core master branch) from github and compiled with default options (cmake . ; make).
On x86_64 RHEL8 (gcc8) as well as lxplus9-arm (gcc11) it gives the same result for the test, which is consistent with the first check in the unit test.

#include <stdio.h>
#include <string>
#include <sstream>
#include <iostream>
extern "C" {
#include "xgboost/c_api.h"
}

int main() {

  int best_ntree_limit_ = 158;
  BoosterHandle booster;
  XGBoosterCreate(NULL, 0, &booster);
  XGBoosterLoadModel(booster, "/afs/cern.ch/user/s/smorovic/public/Photon_NTL_158_Endcap_v1.bin");

  std::stringstream config;
  config << "{\"training\": false, \"type\": 0, \"iteration_begin\": 0, \"iteration_end\": " << best_ntree_limit_
         << ", \"strict_shape\": false}";
  std::string config_ = config.str();

  float var[9];

  var[0] = 134.303;
  var[1] = 0.945981;
  var[2] = 0.0264346;
  var[3] = 0.012448;
  var[4] = 0.0208734;
  var[5] = 113.405;
  var[6] = 1.7446;
  var[7] = 0.00437808;
  var[8] = 0.303464;

  DMatrixHandle dmat;
  XGDMatrixCreateFromMat(var, 1, 9, -999.9f, &dmat);
  uint64_t const* out_shape;
  uint64_t out_dim;
  const float* out_result = NULL;
  XGBoosterPredictFromDMatrix(booster, dmat, config_.c_str(), &out_shape, &out_dim, &out_result);
  float ret = out_result[0];
  XGDMatrixFree(dmat);

  float exp = 0.98634;
  printf(" ===TEST===     VAL: %f\n", ret);
  printf(" ===EXPECTED=== VAL: %f\n", exp);
}

 ===TEST===     VAL: 0.986344
 ===EXPECTED=== VAL: 0.986340

If applying this cmsdist patch:
https://github.com/cms-sw/cmsdist/blob/IB/CMSSW_14_1_X/master/xgboost-arm-and-ppc.patch
There is also no difference.

Finally,compiling with gcc12 (14_0_2 cmsenv) also give the same result.

Making above into unit test and running it on ARM actually passes:

Fail    2s ... RecoEgamma/PhotonIdentification/RecoEgammaPhotonIdentificationTest
Pass    1s ... RecoEgamma/PhotonIdentification/RecoEgammaPhotonIdentificationTest2

so I'm starting to be suspicious that there is something wrong with the unit test itself (regarding portability).

[smorovic]

Found it.
if (abs(etaSC) < 1.5) --> (std::abs(etaSC) < 1.5) fixes the unit test on ARM:

Pass    1s ... RecoEgamma/PhotonIdentification/RecoEgammaPhotonIdentificationTest
Pass    1s ... RecoEgamma/PhotonIdentification/RecoEgammaPhotonIdentificationTest2

I noticed earlier that on x86_64 abs is a floating point version, but apparently not on other architectures?

I will push this fix to the unit test (reusing the existing PR and backport).

[makortel]

I noticed earlier that on x86_64 abs is a floating point version, but apparently not on other architectures?

The C abs() function is actually for int, whereas fabs() is for double, i.e. the behavior on ARM was the correct one.

The only (or "easiest") way I could imagine the compiler's behavior on x86-64 would be that somehow it picked the C++ std::abs() instead of C abs(). But even that would sound strange.