fixed expected model output files to match current code, and added README.md to explain details

[matthewcornell]

From @trobacker :

For future reference:

We unplugged the underlying models (lgb.LGBMRegressor and sarix.SARIX -> numpyro, jax, etc.) - which we decided were nondeterministic with respect to OS - with fixed results. We felt that this solution that we came up with (mocking out the underlying models) has value because it's still testing the code in idmodels, and assuming the underlying models are tested within their own packages.

We essentially retested/rediscovered that the reproducibility of model_output files was not working across different hardware. We found that it didn't seem to be anything about setting rng at a high level in the code but somewhere in the modeling pieces/source code, (i.e. something within jax or lightgbm somewhere) it seemed to have differences across hardware.

Given that these models have performed well in the past, I don't think this is a reason to not use them, so we used the patch approach to get the tests to pass for now keeping in mind that there is this odd irreproducibility across hardware.

We might want to consider keeping track of what hardware we run models on (docker?) due to this issue.

---

[matthewcornell]

@trobacker and I are debugging the failing tests.

[trobacker]

Yes!

[trobacker]

For future reference:

We unplugged the underlying models (lgb.LGBMRegressor and sarix.SARIX -> numpyro, jax, etc.) - which we decided were nondeterministic with respect to OS - with fixed results. We felt that this solution that we came up with (mocking out the underlying models) has value because it's still testing the code in idmodels, and assuming the underlying models are tested within their own packages.

We essentially retested/rediscovered that the reproducibility of model_output files was not working across different hardware. We found that it didn't seem to be anything about setting rng at a high level in the code but somewhere in the modeling pieces/source code, (i.e. something within jax or lightgbm somewhere) it seemed to have differences across hardware.

Given that these models have performed well in the past, I don't think this is a reason to not use them, so we used the patch approach to get the tests to pass for now keeping in mind that there is this odd irreproducibility across hardware.

We might want to consider keeping track of what hardware we run models on (docker?) due to this issue.