new updates resulting in priors having little impact on results

[taymoursiddiqui]

Previously when using meridian, I would be able to see changes and adjustments being made in the estimates as a result of the priors chosen. My results also are very different than what I saw just last week and have not been able to replicate the results from last week.

However, now even choosing priors that are almost entirely centered on 0 still do not do anything to rein in crazy ROI estimates like 400. Has anyone else experienced this, just the priors having no impact ?

[gousesb]

Hello @taymoursiddiqui ,

Thank you for contacting us!

We're currently unable to reproduce the specific issues you're facing. To help us better understand and diagnose the problem, we suggest two main approaches. First, please try running your model (with exact same specifications) using an older version of Meridian and compare the results with those from the latest release. This will help us determine if the observed behavior is indeed a result of the recent Meridian update. Second, to assist us in reproducing your exact scenario, could you provide more information about both the model you are using and share minimal code to replicate the issue with demo data if possible? Understanding the specifics of your model configuration and the characteristics of your dataset will be incredibly helpful in our investigation.

Feel free to reach out for any further queries or feedback regarding Meridian.

Thank you

Google Meridian Support Team

[taymoursiddiqui]

Hello @gousesb Thank you for your suggestions! I will try testing out the older versions of meridian. I think I figured out where the issue is coming from, it seems like the priors I'm setting are being put on the wrong channel/feature.

When I do the posterior and prior plots, I see the wrong channels priors being changed from the rest of the channels, instead of the one I intended. This is why the crazy ROI of the channel that I was trying to rein in was not moving because the prior i was changing was being applied to the wrong channel.

This may be because I'm not properly inputting the priors into the model, but I don't think so because this is how we've been inputting the priors from the beginning and I didn't have this issue before. I also checked the order of coord_to_columns media, media_to_channel, media_spend_to_channel, the objects being input into the prior object (roi_m_sigma). The orders of all of these match and are consistent, but the order changes when results are output.

The order of meridian.input_data.media_channel does not match the order of everything that was input, and also again as I was mentioning, the prior/posterior plots indicate that the wrong channels prior is being changed.

Please let me know if you have any other questions or need any more information, and I'm happy to share more details.

I'm not sure if this code will be helpful but this is where I'm defining and setting up the priors:

priors_config[['roi_m_mu', 'roi_m_sigma']] = [0.1, 0.6]

priors_config.loc[
    priors_config['model_columns'] == 'paid_social__meta__brand__none__spend',
    ['roi_m_mu', 'roi_m_sigma']
] = [0.1, 0.3]

decay_low = priors_config[['decay_low']].values.flatten().astype(np.float64).tolist()
decay_high = priors_config[['decay_high']].values.flatten().astype(np.float64).tolist()
roi_m_mu = priors_config[['roi_m_mu']].values.flatten().astype(np.float64).tolist()
roi_m_sigma = priors_config[['roi_m_sigma']].values.flatten().astype(np.float64).tolist()
knot_values = 200.0

# pass priors into a object
prior = prior_distribution.PriorDistribution(
    roi_m=tfp.distributions.LogNormal(roi_m_mu, roi_m_sigma, name=constants.ROI_M),
    alpha_m=tfp.distributions.Uniform(decay_low, decay_high, name=constants.ALPHA_M)
  , knot_values = tfp.distributions.HalfNormal(knot_values, name=constants.KNOT_VALUES)
  ,    # -- Beta prior for each OM channel --
    contribution_om = tfp.distributions.Beta(
        concentration1 = [60.0, 1.0],   # α: mean 20% and 1%
        concentration0 = [240.0, 99.0],  # β: (α+β) controls variance
        name           = constants.CONTRIBUTION_OM
    )
)

model_spec = spec.ModelSpec(
    prior=prior,
    media_effects_dist='log_normal',
    hill_before_adstock=False,
    max_lag=8,
    unique_sigma_for_each_geo=False,
    roi_calibration_period=None,
    rf_roi_calibration_period=None,
    organic_media_prior_type='contribution',
    knots=29, # quarterly
    baseline_geo=None,
    holdout_id=holdout_id,
    control_population_scaling_id=control_scaling,
)

[taymoursiddiqui]

@gousesb So I just tested one run with an older version of merdian and it seems like the prior is being applied to the right channel.

[gousesb]

Hello @taymoursiddiqui ,

Thank you for providing details about the issue you are encountering.

We believe this may be related to a known channel prior mismatch bug that was introduced in a previous version and was tracked in Discussion #805 . This bug has been addressed and fixed in a more recent release.

To resolve this, we recommend ensuring you are using the most updated version of Meridian, which is 1.1.4. Could you please update to this version and compare the results with version 1.1.1 or earlier to see if they are consistent? This will help confirm if the bug you are experiencing is the same one that has since been fixed. Please let us know if the issue persists after updating.

Feel free to reach out for any further queries or feedback regarding Meridian.

Thank you

Google Meridian Support Team