How can I extract the statistical significance of each variable's coefficient?

[alexanderhe88]

Hi Meridian experts, so far the model diagnostics have been focusing on the entire model (R2, MAPE, training vs test....). But how can I check each variable coefficient's statistical significance? What is the threshold of p-value for that?

Thanks,
Alex

[cpulavarthi]

Hello Alex,

Thank you for contacting us!

I’d like to point out that p-values are a concept from frequentist hypothesis testing. In the context of regression coefficients, they represent the probability of the null hypothesis being true, which is typically that the coefficient is zero (i.e., the null hypothesis is that there is no relationship between the predictor and the outcome).

Meridian uses Bayesian Causal Inference, and hence doesn’t produce p-values as it operates within a different statistical framework. Meridian focuses on posterior distributions for model parameters. Instead of providing a single point estimate and p-value for the coefficients, it gives a posterior distribution for each coefficient that represents the uncertainty of the coefficient’s value. You can access the entire posterior distributions of each coefficient using the inference_data attribute of the model which returns an arviz.InferenceData object.

model_name.inference_data.posterior

Validating the quality of causal inference directly is challenging and necessitates well-designed experiments. For insights into assessing model fit, refer to the documentation on Assess the model fit and access the results.

You can calculate custom metrics using the posterior data available in model_name.inference_data.posterior.

Evaluating coefficient significance against zero

You can do so by computing the mean and credible interval of a parameter and check whether the credible interval includes 0. The following code snippet uses gamma_c as an example.

analyzer.get_central_tendency_and_ci(model_name.inference_data.posterior.gamma_c, confidence_level=0.9)

array([[ 0.02231027, -0.09809182,  0.1441703 ],
       [-0.14423704, -0.34139094,  0.02163551],
       [ 0.06480705, -0.6765277 ,  0.7822834 ],
       [ 1.1798875 ,  0.2182529 ,  2.1138403 ]], dtype=float32)

Columns are mean, ci_lo, ci_hi. Rows are control variables.
Note, even if a control variable's credible interval covers zero, it is often still useful for debiasing the causal inference of treatments.

Note that the ROI posterior will always be positive due to the positive prior. Therefore, checking its significance against zero is not meaningful.

Bayesian models make probabilistic statements about unknown parameters, combining prior belief and data observed. For example, with a Bayesian approach you can answer questions like, "What is the probability that a channel's ROI is greater than 1?". Direct probability statements on unknown parameters are interpretable and often closely aligned with how people think.

If you want to identify “good” performing channels, you can obtain model results summaries through the MediaSummary class (Generate the model results summary). This calculates the mean and credible intervals for each channel’s outcome metrics (incremental outcome, contribution, ROI, mROI, effectiveness) and media summary metrics (impressions, spend).

Also consider these custom metrics for evaluating ROIs.

Probability of ROI being at least 1

(model_name.inference_data.posterior.roi_m >= 1).mean(dim=('chain', 'draw'))

<xarray.DataArray 'roi_m' (media_channel: 3)> Size: 24B
array([0.5926, 0.5544, 0.5905])
Coordinates:
  * media_channel  (media_channel) <U7 84B 'tv' 'youtube' 'search'

This metric indicates the likelihood of a channel reaching an ROI breakeven point (generating sales equal to or greater than the investment).

Percentile of ROI

model_name.inference_data.posterior.roi_m.quantile([0.25, 0.75], dim=('chain', 'draw'))

<xarray.DataArray 'roi_m' (quantile: 2, media_channel: 3)> Size: 48B
array([[0.69005117, 0.63706601, 0.66975348],
       [2.19474739, 1.92793575, 2.21480644]])
Coordinates:
  * media_channel  (media_channel) <U7 84B 'tv' 'youtube' 'search'
  * quantile       (quantile) float64 16B 0.25 0.75

Feel free to reach out if you have any further questions regarding this.

Thank you

Google Meridian Support Team