MCMC non-convergence and catastrophic predictive_accuracy on 1.6.0 with official tfp-nightly dependency

[Delong-Li-Martech]

Summary

Upgrading from google-meridian==1.4.0 to google-meridian==1.6.0 with the official dependency stack from pyproject.toml (tfp-nightly==0.26.0.dev20260130) causes complete MCMC non-convergence and nonsensical goodness-of-fit metrics, while the same training data, config, and priors work on 1.4.0.

Environment

• Platform: linux aarch64 (Docker on Apple Silicon)
• Python: 3.11
• google-meridian: 1.6.0
• tensorflow: 2.20.0
• tfp-nightly: 0.26.0.dev20260130 (as required by pyproject.toml)
• numpy: 2.3.5 (< 2.4)

Working baseline (1.4.0):

• google-meridian: 1.4.0
• tensorflow: 2.20.0
• tensorflow-probability: 0.25.0 (stable; declared in v1.4.0 pyproject.toml)

Reproduction

National/geo MMM, kpi_type=non_revenue, no revenue_per_kpi, ROI priors via PriorDistribution(roi_m=LogNormal(...)), holdout_id mask, MCMC: 8 chains / 2000 adapt / 1000 burnin / 2000 keep.

Install official 1.6.0 stack:

pip install 'numpy<2.4' 'google-meridian==1.6.0'
pip freeze | grep -iE 'meridian|tensorflow|tfp'

Train model, then:

from meridian.analysis import analyzer
from meridian.analysis.review import reviewer

analysis = analyzer.Analyzer(mmm)
print(analysis.predictive_accuracy(use_kpi=True).to_dataframe())

health = reviewer.ModelReviewer(mmm).run()
print(health.overall_status, health.health_score, health.summary_message)

Actual results (1.6.0 + official tfp-nightly)

Predictive accuracy (ALL geo model):

metric	geo_granularity	evaluation_set	value
R_Squared	national	All Data	-68817.18
MAPE	national	All Data	62.83
wMAPE	national	All Data	64.02

Health check:

• overall_status: FAIL
• health_score: 0.0
• max_r_hat for roi_m: 338660163584.00
• Message: "Failed: Model did not converge. Other checks were skipped."

Expected results (1.4.0 + tensorflow-probability 0.25.0)

Same data and config on 1.4.0:

metric	geo_granularity	evaluation_set	value
R_Squared	national	All Data	0.908
MAPE	national	All Data	0.051
wMAPE	national	All Data	0.053

Notes

1. tfp-nightly is an explicit dependency in 1.5.3+ pyproject.toml, not an accidental resolver artifact.
2. 1.5.3 also declares the same tfp-nightly pin; we have not fully validated whether 1.5.3 official stack reproduces the issue.
3. Replacing tfp-nightly with tensorflow-probability==0.25.0 after installing 1.6.0 is a possible workaround but violates declared dependencies and may break if 1.6.0 calls 0.26-only APIs.

[beanscg]

I checked current main, and the dependency surface matches the version-delta report:

• pyproject.toml:37-39 pins TensorFlow and tfp-nightly == 0.26.0.dev20260130 in the base dependencies.
• pyproject.toml:74-80 also pins the JAX extra to jax==0.7.2, jaxlib==0.7.2, and the same tfp-nightly[substrates-jax] build.
• I did not see an open PR that appears to cover this 1.6.0 convergence / predictive-accuracy regression.

A useful next packet would be a clean-env version matrix for 1.4.0, 1.5.3, and 1.6.0 with the official dependency set, recording whether the same Meridian input reaches comparable R-hat / predictive accuracy. If it reproduces only on 1.6.0, the smallest guard may be a dependency smoke test around the pinned tfp-nightly build before changing model code.

[cpulavarthi]

Hello @Delong-Li-Martech and @beanscg,

Thank you for your feedback.

Unfortunately, it's hard to tell the exact reason for the convergence issues you experience. We don't expect worse convergence in Meridian 1.6.0 vs. 1.4.0 though. We are also not aware of any convergence issues related to pinning to version tfp-nightly == 0.26.0.dev20260130.

The reason for pinning to this particular version of tfp is that the tensorflow probability hasn't released a version for a while and the latest released version (0.25.0) was not compatible with jax==0.7.2 which we support since meridian==1.5.3.

The convergence issues you observed may not be directly caused by a different tfp version but rather a side effect of different seeds in each setup. We recommend following the Model debugging guide to improve the convergence.

Feel free to reach out if you have any further questions!

Google Meridian Support Team