System crash for geo level MMM

[rohiitthakur]

Hi team, i am working upon geo level model with 6 geos and weekly 3 years of data making total 942 total observation and 15 media channels, 4 controlled, 2 non media and 1 organic channels, knots 7, i am facing a continues issue where when ever i am using multichain model the sample posterior code crashes and restarts the runtime. i have tried using 1 chain and it works with n_keeps of 500, i need to use all media channels as combining them is not an option. i am using n1-highmem-8 , i am seeing while running the model code it utilizes only 10GB of system ram and then it crashes and restarts the instance. please help me with what changes i can make to run it smoothly with multichains,

[cpulavarthi]

Hello @rohiitthakur,

Thank you for contacting us!

For starters you may try passing n_chains as a list of integers that will add up to your chosen number of chains as suggested in our documentation on Model Debugging. For example, if n_chains=10 is making the model crash, you may instead use n_chains=[5,5] which will take more time for the execution but will overcome the issue you are facing.

You may also enable the JAX backend instead of TensorFlow as it allows efficient compilation of the computational graph, thus reducing the memory requirements for MCMC sampling. The code to enable JAX and detailed considerations regarding numerical differences and performance are noted in our documentation linked above.

I would also suggest checking if the instance is crashing before using the available RAM, i.e., is the RAM of your instance higher than 10 GB. If so, check if there are any memory allocation caps you may need to disable.

Feel free to reach out if you have any further questions or suggestions for us.

Google Meridian Support Team