incremental_output() assigning extremely low outputs for certain channels with new_data despite high output from input data.

[mikeang101]

Good afternoon! Hoping to get some assistance with an issue.

We're attempting to use incremental_output() module for scenario planning. In order to test it, we've run a model using weekly data for 2021 - 2024 (156 weeks) with 6 channels. When running the function without passing new data, each channel has historical incremental attributed in a split that makes sense to their share of spend and what we expect.

However, when passing a new_data tensor containing 50 weeks of 2025 spends, incremental_output() only gives outputs to 3 of 6 channels, with the other three being near zero.

The tensor only contains spends by week. No tensors for controls or impressions data (simulating future scenario where these are unknown, aside from holidays.).

For example, historical incremental might show three weeks as:

Where as passing new data looks like this:

For debugging purposes:

1. Model trained on 156 weeks of data.
2. National level model (a single market).
3. Six channels. TV, Radio, Out-Of-Home, Brand Google Search, Nonbrand Google Search, Local Service Ads.
4. We are measuring conversions, no revenue as it is not standardized between conversions.
5. Traditional channel data from invoices, digital channel data directly from Google Analytics.
6. We are using contribution priors. 25% TV, 15% Brand, 10% everything else.
7. Using 3 controls. Estimated competitor spends, binary is there a holiday that week, and car crashes per week.
8. Model Quality Report Card

9. Adstock and Saturation are set to meridian default for this model, 6 weeks max lag. 10. None of the data contains zeros.

Thanks for any assistance!

[cpulavarthi]

Hello @mikeang101,

Thank you for sharing detailed information regarding your issue!

The new_data argument accepts optional tensors for media, reach, frequency, organic_media, organic_reach, organic_frequency, non_media_treatments and revenue_per_kpi. If new_data is provided, the incremental outcome (from the incremental_outcome() method) is calculated using the new tensors in new_data and the original values of the remaining tensors. Therefore, simulated controls aren't required as the Analyzer would use the controls values from the original dataset and they are held constant during these computations.

However, the following two points are more likely to be causing the differences you observed and you may want to dig deeper in these areas:

1. You haven’t mentioned if you are using media spend as the exposure as well as the spend metric. The new_data argument expects the media exposure metric (media, and not media_spend) and if this is different from media spend while training the model, this can explain the differences observed in the results while using the new data. Meridian is trained to predict outcome based on the values of the exposure metric (media) and not spend unless you explicitly set media spend as your exposure metric as well.

   In this case you should ideally obtain the new data in the units of the exposure metric used during training for the post-modeling window of 50 weeks in 2025 that you mentioned. You can use historical average impression cost (historical CPM) to translate your spend into impression values. Alternatively, you may consider using media spend itself as the exposure metric while training (this isn’t recommended practice but a reasonable alternative).

2. If you are using media spend as both the spend and exposure metrics, you may still observe changes in the incremental outcome values if there is significant change in the budget allocation to the media channels. You may want to manually verify this is not the case.

Based on the debugging information you provided, your data seems to be of high quality and sufficient for the national model, and the model quality checks also don’t raise any concerns. The differences you observed are likely not because of improper model training, but instead because of differences in the exposure metric used during model training vs. what is used in the new data.

Hope this helps you better debug your issue. Let us know if adjusting the exposure metric input to the new data resolves these discrepancies. Do reach out if you have any further questions regarding this.

Google Meridian Support Team

[mikeang101]

Thank you for your reply! Sorry for the delay, I was away for the holidays.

The exposure metric was exactly the issue. When we initially ran forecasting, we were still collecting impressions data, so we used spend again in place of impressions. Since we passed a new budget tensor only, we had useable results. However once we switched to including actual impressions data, your explanation of the exposure metric explains why we no longer had useable results when using a new budget tensor.

I have confirmed that it works again by re-training a model with spend and will attempt to forecast with the old models after I create and clean CPM to predicted impressions data for the training budget.

Two follow up questions:

1. In our case, our business model is basically converting clients after car crashes, so more something that people rarely need rather than something more people would want like a product or service. Because of this, we never focused too much on impressions and CPM in our own analytics, relying more on conversions and CPA. I know that you mentioned that double spend is not ideal, but In our case, would you recommend using spend again as the exposure metric?

2. Is there a way to shift the time frame that the model uses to forecast the incremental or is it an average of the entire duration provided to train the model? I ask this because after training the model and running the historical incremental, the first two weeks are very low in 2020 as I assume the model ramped up from having a low point as the first data point (January / Winter historically see dips in conversions). While this dip isn't as significant in later years in the historical, the same ramping up shows up in the new budget forecast.

Part of the above question too is what if we wanted to forecast incremental for the back half of the year after reviewing performance from the first half and making changes? Is there a way to shift the model to account for this (ie adding an ISO Week control variable) when running incremental forecast? We usually see the best returns in summer and the worst in winter (like a sine wave) so in this example I wouldn't want to run an incremental forecast for a budget starting in summer and ending in winter and have it represented by the model as a time period starting in winter and ending in summer.

Thanks again for all your help!

[cpulavarthi]

Hello @mikeang101,

Thank you for confirming that the mismatch of media spend and media exposure metrics in the training data vs. the new data was causing your earlier issues.

To answer your follow up questions:

1. I mentioned that using the media spend as the exposure metric isn't ideal. To clarify my reasoning, I meant that changing the exposure metric only because the incremental outcome results aren't as expected isn't ideal and the alternative of converting media spend into the initial exposure metric using CPM to scale it accordingly would be better. Media spend can be used as an exposure metric as long as it makes business sense, which in your case it does. You will only need to ensure to read your results accordingly and adjust your interpretations considering the fact that you are now using media spend as your exposure metric.

2. Please note incremental outcome does not model temporal seasonality, which appears in the model's baseline. Incremental outcome provides (as the docstring says) the incremental outcome from media (and users can toggle include_non_paid_channels). By definition (as a mental model), seasonal KPI spikes unrelated to media shouldn't appear in incremental_outcome. And any seasonal spikes driven by media should be captured in the temporally ordered media values passed to new_data. You may check our documentation linked here for further reading.

Feel free to reach out if you have any further questions regarding this.

Google Meridian Support Team