_sec-overdispersion.qmd

---

::: notes

The Poisson distribution model **forces** the conditional variance
to equal the conditional mean
($\Var{Y \mid \vX=\vx, T=t} = \Expp[Y \mid \vX=\vx, T=t]$).
In practice, observational count data frequently exhibit variance
substantially larger than the mean
(or occasionally smaller, termed underdispersion).

:::

{{< slidebreak >}}

:::: {#def-overdispersion}
#### Overdispersion

Write $m_{\P}(\vx, t)$ and $v_{\P}(\vx, t)$
for the conditional mean and variance
that a model $\P(Y=y \mid \vX=\vx, T=t)$ specifies;
for a Poisson model, $v_{\P}(\vx, t) = m_{\P}(\vx, t) = \mu(\vx, t)$.
Let $\P$ specify $Y$'s conditional mean correctly,
so that $m_{\P}(\vx, t) = \Expp[Y \mid \vX=\vx, T=t]$
for every $\vx$ and $t$.
Then $Y$ is **overdispersed** relative to $\P$ if
its conditional variance exceeds the one $\P$ specifies
at some covariate pattern and exposure:

$$\Var{Y \mid \vX=\vx, T=t} > v_{\P}(\vx, t)$$

The same-mean requirement is what makes this a statement about dispersion:
without it, any model that simply understates the mean
would look overdispersed.
In practice we detect overdispersion by comparing
the conditional empirical variance in a dataset
against $v_{\est{\P}}(\vx, t)$, the variance a *fitted* model predicts.

::::

In Poisson regression, unmodeled heterogeneity, clustering,
and omitted predictors all inflate the conditional variance.
Where the mean model remains correct,
that inflation is overdispersion in the sense of @def-overdispersion.
An omitted predictor may instead --- or additionally --- misspecify the mean,
and to whatever extent it does,
the model is failing the correct-mean requirement @def-overdispersion imposes,
which is a different problem requiring a different remedy.
When overdispersion is present but ignored,
the point estimates $\evb$ remain consistent,
but the standard errors produced by standard maximum likelihood estimation
are severely underestimated.
This underestimation leads to overly narrow confidence intervals
and inflated false-positive (type I error) rates during hypothesis testing.

When overdispersion is detected via residual diagnostics
(such as the deviance or Pearson $\chi^2$ statistic,
divided by its residual degrees of freedom,
substantially exceeding 1),
practitioners can address it by incorporating missing predictors,
using quasipoisson estimation, or fitting a negative binomial regression model.

::: notes

c.f.
@dobson4e §3.2.1, 7.7, 9.8;
@vittinghoff2e §8.1.5;
and <https://en.wikipedia.org/wiki/Overdispersion>.

:::