How do I calibrate the model after positive results?

[eseanwork]

Hi everyone,

I have successfully fitted a Meridian model and tested its recommended media mix using two geo-tests. The results have been very positive, better than expected. Here is a summary of the results:

• Actual Lift (Geo-tests): Estimated average lift of 10%, with a confidence interval of [6%, 17%].
• Predicted Lift (Meridian): Expected lift of 5%, with an interval of [2%, 7%].

We have ruled out any seasonality factors affecting these results, so we are confident that the increase is directly driven by the implementation of the recommended media mix.

My question is: How can I use this information to recalibrate the model? Given that the actual lift was higher than expected, I understand that my actual media contribution percentage is higher than what the model initially determined. Would it make sense to scale it and retrain the model using a higher prior for media contribution?

I would appreciate your thoughts on the best approach to feed these findings back into the model.

Thanks in advance!

[cpulavarthi]

Hello @eseanwork,

Thank you for contacting us!

It would make perfect sense to recalibrate your model using the geo-test results. In Meridian, this is done by adjusting your prior distributions to reflect your new, data-backed belief about media effectiveness.

Depending on how your geo-test results are structured, you can approach this in a couple of ways:

• Channel-level calibration: If your tests isolated specific channels, you can adjust the specific channel-wise ROI priors (e.g., your roi_m parameters).
• Total contribution calibration: If it was a broader mix test, you can adjust the total paid media contribution prior. You can achieve this by setting a common ROI prior on all channels such that the total media contribution implies a prior mean and standard deviation that aligns with your 10% lift finding. This approach is advantageous as it doesn't require you to formulate an intuition for each individual channel's performance.

Please note that if your geo-tests only evaluated a subset of your media mix, using the total paid media contribution approach will apply the same common ROI prior to all channels in your model. This may not be appropriate if the untested channels are expected to perform differently than the tested ones. If you only have test data for certain channels, you should set custom channel-level priors (such as specific ROI or contribution_m priors) specifically for the tested channels. For the remaining untested channels, you will still need to establish and set a reasonable default prior.

We recommend reviewing the following documentation to guide your specific implementation:

• ROI priors and calibration
• Setting custom ROI priors using past experiments
• Total paid media contribution priors
• Custom total paid media contribution prior - Implementation example

Feel free to reach out if you have any further questions or need help with the implementation.

Google Meridian Support Team