Credible Interval spikes in a certain time period

[superli3]

Reference link from Discord thread
https://discord.com/channels/1397974595952185365/1443276393520828588

I have a Meridian model at weekly level. When looking at the Summary Output HTML, I notice that there is a spike of credible interval at the beginning of the model fit chart. However, looking at the performance metrics table, the overall performance of the model seems to be good. What could be the possible cause to this scenario? Is there any recommendation on how to surpass this spike of credible interval?

The reason why the model is built with weekly national data is just to align with the project requirements. So basically, my hands are tied to whatever the granularity as it is. I can definitely give a try on the priors

Debugging info:
1). The data runs over 3 years at weekly level
2). The model is built at national level
3). There are 11 paid media channels plus 1 organic search channel
4). The model measures revenue
5). The data was pulled from different marketing platforms (e.g. SA360, etc.)
6). I use the default priors (i.e. lognormal with mu=0.2 and sigma=0.9)
7). Currently page visits and number of promotions are used as controls
8). Please check the attached screenshot. Seems the model passes the quality check.
10). Not explicitly specified in the ModelSpec, use default settings as of now

[gousesb]

Hello @superli3 ,

Thank you for contacting us!

To address this, first, I recommend investigating the data for that period to see if there was anything unusual happening in that week, such as a large, sudden increase in the media spend for any specific media channel.

Second, I recommend reviewing the knot settings within the model configuration. The spike could be caused by an assigned knot located in that week. If a knot is found to be the cause, remove that specific knot or turn off the automatic knot selection feature to resolve the issue.

Feel free to reach out for any further queries or feedback regarding Meridian.

Google Meridian Support Team

[sam-li-stacktics]

Hi @gousesb,

Sorry for a late reply. I just did some tests on my model, and I found these:

• I used AKS to automatically place knots for my model
• The AKS method would assign knots on some special dates. For example, it would place knots on, before, and after the week of Black Friday in 2022 and 2023 (e.g. It assigned knots at 15, 16, 17, 18).
• When I copied the list of knots generated by AKS and removed the knot at 17 and trained the model again, the spike of credible intervals disappeared.

However, I have ran into another issue. I was supposed to run 10 iterations of random holdout test with random seed set from 1 to 10 (80% to train and 20% to test). I found that with some random seeds, the testing performance of the model (i.e. test R^2 score) will drop significantly from somewhere around 0.92 to somewhere around 0.75 or even lower (0.55 as the worst). I did find that in some of the held out test data, special days are included, for example Black Friday, etc. and in my model, knots are also placed on those particular dates. So my follow up question is, in general, how should I place my knots? Is it a bad practice to place knots adjacently (e.g. knots on 15, 16, 17, 18, 19) and should I avoid placing knots on special events (e.g. Black Friday, Back To School Season, etc.) I did create a column in my DataFrame called 'ctrl_special_events' where it indicated 1 for Black Friday and for some other special events, and 0 for the rest of the dates. Based on this scenario, what would you recommend for knots placement for my model?

Thanks in advance!

[gousesb]

Hello @sam-li-stacktics ,

Thank you for contacting us!

For a national weekly model, the time-varying intercept should be kept parsimonious. It is recommended to start with a minimal number of knots and only increase them if clear underfitting is observed, rather than using many closely spaced knots. Although removing the knot at week 17 fixed the spike in the credible interval, the existing dense knot cluster around Black Friday still results in an over-parameterized structure. This dense clustering captures localized noise instead of a generalizable trend, making the model's testing performance highly sensitive to the specific holdout split. Knots should be used to define the broader piecewise linear trend over time, they should be dense only where low bias is necessary and sparse where low variance is preferred.

For discrete events like Black Friday or back-to-school periods, binary indicators are recommended. They model time effects as consistent across affected time periods and use degrees of freedom efficiently, while having no impact on the weeks outside the indicated window. This fits your setup where you already have a ctrl_special_events column, it is better to let that indicator represent the event and avoid stacking several adjacent knots (for example, weeks 15–19) on top of it unless there is strong evidence that each week has a structurally different baseline. Automatic Knot Selection (AKS) is still useful as a starting point, but after retrieving the selected knot locations, review them, thin out lightly packed knots around special events, and if needed, rerun AKS with a higher regularization penalty and a smaller range for the number of internal knots so that you end up with only a few well-placed knots plus your special-event indicator.

Feel free to reach out for any further queries or feedback regarding Meridian.

Google Meridian Support Team