Non-deterministic model parameters across training runs

[leostimpfle]

What happens?

Training the same model on the same data (using the DuckDB backend) results in slightly different model parameters (m/u probabilities) in each run. The differences are very small (<1e-10) so likely just some floating point issue.

Although the differences are small, it makes development of linkage models a bit more cumbersome because the parameters will always change slightly and it's not immediately clear where differences between runs come from.

I'm currently often resorting to rounding the model parameters in splink.Linker._settings_obj after training (to 10 decimals which seems to work reliably) but this feels like a hack.

To Reproduce

reproduce_em_nondeterminism.py

OS:

MacOS

Splink version:

4.0.12

Have you tried this on the latest master branch?

• I agree

Have you tried the steps to reproduce? Do they include all relevant data and configuration? Does the issue you report still appear there?

• I agree

[ADBond]

You're right that it's a floating-point issue - specifically it is the fact that floating point addition is not associative, combined with the indeterminism stemming from SQL engine query freedom. There are some details in the discussion of issue #2808.

If this is solvable, then it would certainly impact training performance, as it would require the use of ORDER BY to ensure we have a repeatable order. But I am not certain if it even would be solvable, as I am not sure we have enough control over duckdb to ensure it would execute the operations in a deterministic order. I'll see if I can find time to have a little look into this to see if it is possible or feasible - if it was, it would likely be an opt-in because of the performance aspect.

[leostimpfle]

Thanks for your prompt response @ADBond and sorry for not spotting the related issue #2808. In my case, I would likely not want to sacrifice performance to ensure consistent estimates beyond the 10th decimal place and would therefore likely stick to rounding model parameters to sensible precision.

Maybe just two questions:

• Would it be helpful to add a (very brief) note somewhere in the documentation for users looking for reproducible setups (for example in the training section)? I'd be happy to look into this and create a PR.
• Would it make sense to let the user control optional rounding of parameter estimates for reproducibility? Are there any cases where differences in the 13th decimal place materially matter at all or would determinism be more valuable for the "average" user?

Thanks again!

[ADBond]

Would it be helpful to add a (very brief) note somewhere in the documentation for users looking for reproducible setups (for example in the training section)? I'd be happy to look into this and create a PR.

Yes I think that would be a good idea - a PR would be most welcome.

Would it make sense to let the user control optional rounding of parameter estimates for reproducibility? Are there any cases where differences in the 13th decimal place materially matter at all or would determinism be more valuable for the "average" user?

That certainly seems like it could be an option - differences of that magnitude will have negligible effect, and it would not be reasonable to claim that the parameter estimates are anything like that precise (or even that that level of precision is meaningful).

[gringer]

Rounding model parameters to something that leads to reproducible results would be excellent.

[gringer]

Here's something I wrote to round the comparison levels that are output to a saved JSON file to 4 s.f.. Because our workflow includes loading from this JSON file as a component (in the next step), doing this rounding on the intercepted dictionary is good enough:

# [code unwrapped from linker.misc.save_model_to_json so that rounding can be added]
filename = "settings_model.json"
out_path = os.path.join(outputs_folder, filename)
model_dict = linker._settings_obj.as_dict()
model_dict["probability_two_random_records_match"] = float(format(model_dict["probability_two_random_records_match"], ".4g"))
# Round model m/u probabilities to 4 s.f, rounding the complement for very high probabilities
for comparison in model_dict["comparisons"]:
    for comparison_level in comparison["comparison_levels"]:
        if("u_probability" in comparison_level):
            if(comparison_level["u_probability"] > 0.9):
                comparison_level["u_probability"] = 1 - float(format(1 - comparison_level["u_probability"], ".4g"))
            else:
                comparison_level["u_probability"] = float(format(comparison_level["u_probability"], ".4g"))
        if("m_probability" in comparison_level):
            if(comparison_level["m_probability"] > 0.9):
                comparison_level["m_probability"] = 1 - float(format(1 - comparison_level["m_probability"], ".4g"))
            else:
                comparison_level["m_probability"] = float(format(comparison_level["m_probability"], ".4g"))
with open(out_path, "w", encoding="utf-8") as f:
    json.dump(model_dict, f, indent=4)
print(f"Model saved to: {out_path}")

The results are still not identical, but it's close enough that I can evaluate different blocking rule sets without having to worry about interference from model error.

Edit: after inspection of the results, I noticed a few high probabilities that got rounded to 1, so I've updated the code to round (1-p) to 4 s.f. for those probabilities.

[gringer]

Curiously, even after rounding I'm getting differences in u probabilities (some noticeably large), despite using a random number seed. I was under the impression from previous comments that the u probabilities, at least, should be reproducible.

This is using 1e8 u samples with a seed of 1:

$ diff -y settings_model_run1.json settings_model_run2.json; echo
{                                                               {
    "link_type": "link_only",                                       "link_type": "link_only",
    "probability_two_random_records_match": 1.868e-09,              "probability_two_random_records_match": 1.868e-09,
    "retain_matching_columns": true,                                "retain_matching_columns": true,
    "retain_intermediate_calculation_columns": true,                "retain_intermediate_calculation_columns": true,
    "additional_columns_to_retain": [                               "additional_columns_to_retain": [
        "snz_uid",                                                      "snz_uid",
        "snz_unique_nbr"                                                "snz_unique_nbr"
    ],                                                              ],
    "sql_dialect": "duckdb",                                        "sql_dialect": "duckdb",
    "linker_uid": "65knnba7",                                 |     "linker_uid": "x4iye18f",
    "em_convergence": 0.0001,                                       "em_convergence": 0.0001,
    "max_iterations": 25,                                           "max_iterations": 25,
    "bayes_factor_column_prefix": "bf_",                            "bayes_factor_column_prefix": "bf_",
    "term_frequency_adjustment_column_prefix": "tf_",               "term_frequency_adjustment_column_prefix": "tf_",
    "comparison_vector_value_column_prefix": "gamma_",              "comparison_vector_value_column_prefix": "gamma_",
    "unique_id_column_name": "unique_id",                           "unique_id_column_name": "unique_id",
    "source_dataset_column_name": "source_dataset",                 "source_dataset_column_name": "source_dataset",
    "blocking_rules_to_generate_predictions": [                     "blocking_rules_to_generate_predictions": [
        {                                                               {
            "blocking_rule": "(l.\"splink_birth_date\" = r.\"               "blocking_rule": "(l.\"splink_birth_date\" = r.\"
            "sql_dialect": "duckdb"                                         "sql_dialect": "duckdb"
        },                                                              },
        {                                                               {
            "blocking_rule": "(l.\"splink_first_names_array\"               "blocking_rule": "(l.\"splink_first_names_array\"
            "sql_dialect": "duckdb",                                        "sql_dialect": "duckdb",
            "arrays_to_explode": [                                          "arrays_to_explode": [
                "splink_first_names_array",                                     "splink_first_names_array",
                "splink_last_names_array"                                       "splink_last_names_array"
            ]                                                               ]
        },                                                              },
        {                                                               {
            "blocking_rule": "(l.\"splink_birth_date\" = r.\"               "blocking_rule": "(l.\"splink_birth_date\" = r.\"
            "sql_dialect": "duckdb",                                        "sql_dialect": "duckdb",
            "arrays_to_explode": [                                          "arrays_to_explode": [
                "NYSIIS_first_names_array"                                      "NYSIIS_first_names_array"
            ]                                                               ]
        },                                                              },
        {                                                               {
            "blocking_rule": "(l.\"splink_birth_date\" = r.\"               "blocking_rule": "(l.\"splink_birth_date\" = r.\"
            "sql_dialect": "duckdb",                                        "sql_dialect": "duckdb",
            "arrays_to_explode": [                                          "arrays_to_explode": [
                "NYSIIS_last_names_array"                                       "NYSIIS_last_names_array"
            ]                                                               ]
        },                                                              },
        {                                                               {
            "blocking_rule": "(l.\"NYSIIS_first_names_array\"               "blocking_rule": "(l.\"NYSIIS_first_names_array\"
            "sql_dialect": "duckdb",                                        "sql_dialect": "duckdb",
            "arrays_to_explode": [                                          "arrays_to_explode": [
                "NYSIIS_first_names_array",                                     "NYSIIS_first_names_array",
                "NYSIIS_last_names_array"                                       "NYSIIS_last_names_array"
            ]                                                               ]
        }                                                               }
    ],                                                              ],
    "comparisons": [                                                "comparisons": [
        {                                                               {
            "output_column_name": "splink_first_names",                     "output_column_name": "splink_first_names",
            "comparison_levels": [                                          "comparison_levels": [
                {                                                               {
                    "sql_condition": "\"splink_first_names_l\                       "sql_condition": "\"splink_first_names_l\
                    "label_for_charts": "splink_first_names i                       "label_for_charts": "splink_first_names i
                    "fix_m_probability": false,                                     "fix_m_probability": false,
                    "fix_u_probability": false,                                     "fix_u_probability": false,
                    "is_null_level": true                                           "is_null_level": true
                },                                                              },
                {                                                               {
                    "sql_condition": "\"splink_first_names_l\                       "sql_condition": "\"splink_first_names_l\
                    "label_for_charts": "[E] splink_first_nam                       "label_for_charts": "[E] splink_first_nam
                    "m_probability": 0.977,                                         "m_probability": 0.977,
                    "u_probability": 2.084e-05,               |                     "u_probability": 2.064e-05,
                    "fix_m_probability": true,                                      "fix_m_probability": true,
                    "fix_u_probability": false,                                     "fix_u_probability": false,
                    "tf_adjustment_column": "splink_first_nam                       "tf_adjustment_column": "splink_first_nam
                    "tf_adjustment_weight": 1.0                                     "tf_adjustment_weight": 1.0
                },                                                              },
                {                                                               {
                    "sql_condition": "\n                LEAST                       "sql_condition": "\n                LEAST
                    "label_for_charts": "[PJW0.95] Strong sub                       "label_for_charts": "[PJW0.95] Strong sub
                    "m_probability": 0.1201,                  |                     "m_probability": 0.12,
                    "u_probability": 0.0004006,               |                     "u_probability": 0.0004196,
                    "fix_m_probability": false,                                     "fix_m_probability": false,
                    "fix_u_probability": false,                                     "fix_u_probability": false,
                    "tf_adjustment_column": "splink_first_nam                       "tf_adjustment_column": "splink_first_nam
                    "tf_adjustment_weight": 1.0                                     "tf_adjustment_weight": 1.0
                },                                                              },
                {                                                               {
                    "sql_condition": "list_max(\n                                   "sql_condition": "list_max(\n            
                    "label_for_charts": "[JW0.95] splink_firs                       "label_for_charts": "[JW0.95] splink_firs
                    "m_probability": 0.03685,                 |                     "m_probability": 0.03696,
                    "u_probability": 0.004214,                |                     "u_probability": 0.004115,
                    "fix_m_probability": false,                                     "fix_m_probability": false,
                    "fix_u_probability": false,                                     "fix_u_probability": false,
                    "tf_adjustment_column": "splink_first_nam                       "tf_adjustment_column": "splink_first_nam
                    "tf_adjustment_weight": 1.0                                     "tf_adjustment_weight": 1.0
                },                                                              },
                {                                                               {
                    "sql_condition": "ELSE",                                        "sql_condition": "ELSE",
                    "label_for_charts": "All other comparison                       "label_for_charts": "All other comparison
                    "m_probability": 0.023,                                         "m_probability": 0.023,
                    "u_probability": 0.995364,                |                     "u_probability": 0.995445,
                    "fix_m_probability": true,                                      "fix_m_probability": true,
                    "fix_u_probability": false                                      "fix_u_probability": false
                }                                                               }
            ],                                                              ],
            "comparison_description": "CustomComparison"                    "comparison_description": "CustomComparison"
        },                                                              },
        {                                                               {
            "output_column_name": "splink_last_names",                      "output_column_name": "splink_last_names",
            "comparison_levels": [                                          "comparison_levels": [
                {                                                               {
                    "sql_condition": "\"splink_last_names_l\"                       "sql_condition": "\"splink_last_names_l\"
                    "label_for_charts": "splink_last_names is                       "label_for_charts": "splink_last_names is
                    "fix_m_probability": false,                                     "fix_m_probability": false,
                    "fix_u_probability": false,                                     "fix_u_probability": false,
                    "is_null_level": true                                           "is_null_level": true
                },                                                              },
                {                                                               {
                    "sql_condition": "\"splink_last_names_l\"                       "sql_condition": "\"splink_last_names_l\"
                    "label_for_charts": "[E] splink_last_name                       "label_for_charts": "[E] splink_last_name
                    "m_probability": 0.998,                                         "m_probability": 0.998,
                    "u_probability": 0.0001719,               |                     "u_probability": 0.0001789,
                    "fix_m_probability": true,                                      "fix_m_probability": true,
                    "fix_u_probability": false,                                     "fix_u_probability": false,
                    "tf_adjustment_column": "splink_last_name                       "tf_adjustment_column": "splink_last_name
                    "tf_adjustment_weight": 1.0                                     "tf_adjustment_weight": 1.0
                },                                                              },
                {                                                               {
                    "sql_condition": "\n                LEAST                       "sql_condition": "\n                LEAST
                    "label_for_charts": "[PJW0.95] Strong sub                       "label_for_charts": "[PJW0.95] Strong sub
                    "m_probability": 0.09444,                 |                     "m_probability": 0.09443,
                    "u_probability": 7.949e-05,               |                     "u_probability": 8.682e-05,
                    "fix_m_probability": false,                                     "fix_m_probability": false,
                    "fix_u_probability": false,                                     "fix_u_probability": false,
                    "tf_adjustment_column": "splink_last_name                       "tf_adjustment_column": "splink_last_name
                    "tf_adjustment_weight": 1.0                                     "tf_adjustment_weight": 1.0
                },                                                              },
                {                                                               {
                    "sql_condition": "list_max(\n                                   "sql_condition": "list_max(\n            
                    "label_for_charts": "[JW0.95] splink_last                       "label_for_charts": "[JW0.95] splink_last
                    "m_probability": 0.001822,                |                     "m_probability": 0.001823,
                    "u_probability": 8.18e-05,                |                     "u_probability": 7.761e-05,
                    "fix_m_probability": false,                                     "fix_m_probability": false,
                    "fix_u_probability": false,                                     "fix_u_probability": false,
                    "tf_adjustment_column": "splink_last_name                       "tf_adjustment_column": "splink_last_name
                    "tf_adjustment_weight": 1.0                                     "tf_adjustment_weight": 1.0
                },                                                              },
                {                                                               {
                    "sql_condition": "ELSE",                                        "sql_condition": "ELSE",
                    "label_for_charts": "All other comparison                       "label_for_charts": "All other comparison
                    "m_probability": 0.002,                                         "m_probability": 0.002,
                    "u_probability": 0.9996668,               |                     "u_probability": 0.9996567,
                    "fix_m_probability": true,                                      "fix_m_probability": true,
                    "fix_u_probability": false                                      "fix_u_probability": false
                }                                                               }
            ],                                                              ],
            "comparison_description": "CustomComparison"                    "comparison_description": "CustomComparison"
        },                                                              },
        {                                                               {
            "output_column_name": "splink_sex",                             "output_column_name": "splink_sex",
            "comparison_levels": [                                          "comparison_levels": [
                {                                                               {
                    "sql_condition": "\"splink_sex_l\" IS NUL                       "sql_condition": "\"splink_sex_l\" IS NUL
                    "label_for_charts": "splink_sex is NULL",                       "label_for_charts": "splink_sex is NULL",
                    "fix_m_probability": false,                                     "fix_m_probability": false,
                    "fix_u_probability": false,                                     "fix_u_probability": false,
                    "is_null_level": true                                           "is_null_level": true
                },                                                              },
                {                                                               {
                    "sql_condition": "\"splink_sex_l\" = \"sp                       "sql_condition": "\"splink_sex_l\" = \"sp
                    "label_for_charts": "[E] splink_sex",                           "label_for_charts": "[E] splink_sex",
                    "m_probability": 0.995129,                                      "m_probability": 0.995129,
                    "u_probability": 0.501,                   |                     "u_probability": 0.5009,
                    "fix_m_probability": false,                                     "fix_m_probability": false,
                    "fix_u_probability": false                                      "fix_u_probability": false
                },                                                              },
                {                                                               {
                    "sql_condition": "ELSE",                                        "sql_condition": "ELSE",
                    "label_for_charts": "All other comparison                       "label_for_charts": "All other comparison
                    "m_probability": 0.004871,                                      "m_probability": 0.004871,
                    "u_probability": 0.499,                   |                     "u_probability": 0.4991,
                    "fix_m_probability": false,                                     "fix_m_probability": false,
                    "fix_u_probability": false                                      "fix_u_probability": false
                }                                                               }
            ],                                                              ],
            "comparison_description": "CustomComparison"                    "comparison_description": "CustomComparison"
        },                                                              },
        {                                                               {
            "output_column_name": "splink_birth_date",                      "output_column_name": "splink_birth_date",
            "comparison_levels": [                                          "comparison_levels": [
                {                                                               {
                    "sql_condition": "\"splink_birth_date_l\"                       "sql_condition": "\"splink_birth_date_l\"
                    "label_for_charts": "splink_birth_date is                       "label_for_charts": "splink_birth_date is
                    "fix_m_probability": false,                                     "fix_m_probability": false,
                    "fix_u_probability": false,                                     "fix_u_probability": false,
                    "is_null_level": true                                           "is_null_level": true
                },                                                              },
                {                                                               {
                    "sql_condition": "\"splink_birth_date_l\"                       "sql_condition": "\"splink_birth_date_l\"
                    "label_for_charts": "[E] splink_birth_dat                       "label_for_charts": "[E] splink_birth_dat
                    "m_probability": 0.995,                                         "m_probability": 0.995,
                    "u_probability": 2.911e-05,               |                     "u_probability": 2.915e-05,
                    "fix_m_probability": true,                                      "fix_m_probability": true,
                    "fix_u_probability": false,                                     "fix_u_probability": false,
                    "tf_adjustment_column": "splink_birth_dat                       "tf_adjustment_column": "splink_birth_dat
                    "tf_adjustment_weight": 1.0                                     "tf_adjustment_weight": 1.0
                },                                                              },
                {                                                               {
                    "sql_condition": "ELSE",                                        "sql_condition": "ELSE",
                    "label_for_charts": "All other comparison                       "label_for_charts": "All other comparison
                    "m_probability": 0.005,                                         "m_probability": 0.005,
                    "u_probability": 0.99997089,              |                     "u_probability": 0.99997085,
                    "fix_m_probability": true,                                      "fix_m_probability": true,
                    "fix_u_probability": false                                      "fix_u_probability": false
                }                                                               }
            ],                                                              ],
            "comparison_description": "CustomComparison"                    "comparison_description": "CustomComparison"
        }                                                               }
    ]                                                               ]
}                                                               }