docs(count-regression): improve narrative flow, standardize math notation, and fill gaps

_subfiles/count-regression/_note_glm-naming.qmd

@@ -2,11 +2,12 @@
 
 Logistic regression is named after the (inverse) link function.
 Poisson regression is named after the outcome distribution.
-I think this naming convention reflects the strongest 
-(most questionable assumption) in the model.
+This naming convention reflects the strongest
+(and often most questionable) assumption in each model.
 In binary data regression, 
 the outcome distribution essentially *must* be Bernoulli (or Binomial),
-but the link function could be logit, log, identity, probit, or something more unusual.
+but the link function could be logit, log, identity, probit,
+or something more unusual.
 In count data regression,
 the outcome distribution could have many different shapes,
 but the link function will probably end up being log,

_subfiles/count-regression/_sec-overdispersion.qmd

@@ -1,23 +1,42 @@
-
 ---
 
 ::: notes
 
-The Poisson distribution model **forces** the variance to equal the mean.
-In practice, many count distributions will have a variance substantially larger than the mean (or occasionally, smaller).
+The Poisson distribution model **forces** the conditional variance
+to equal the conditional mean ($\Var{Y \mid \vX} = \Expp[Y \mid \vX]$).
+In practice, observational count data frequently exhibit variance
+substantially larger than the mean
+(or occasionally, smaller, termed underdispersion).
 
 :::
 
+{{< slidebreak >}}
+
 :::: {#def-overdispersion}
 #### Overdispersion
 
-A random variable $X$ is **overdispersed**
-relative to a model $\p(X=x)$ if
-its empirical variance in a dataset is larger than
-the value predicted by the fitted model $\hat{\p}(X=x)$.
+A random variable $Y$ is **overdispersed**
+relative to a parametric model $\P(Y=y \mid \vX)$ if
+its conditional empirical variance in a dataset exceeds
+the theoretical variance imposed by the fitted model $\est{\P}(Y=y \mid \vX)$.
 
 ::::
 
+In Poisson regression, unmodeled heterogeneity, clustering,
+or omitted predictors cause overdispersion.
+When overdispersion is present but ignored,
+the point estimates $\evb$ remain unbiased,
+but the standard errors produced by standard maximum likelihood estimation
+are severely underestimated.
+This underestimation leads to overly narrow confidence intervals
+and inflated false-positive (type I error) rates during hypothesis testing.
+
+When overdispersion is detected via residual diagnostics
+(such as a mean deviance or Pearson $\chi^2$ statistic
+significantly exceeding 1),
+practitioners can address it by incorporating missing predictors,
+using quasipoisson estimation, or fitting a negative binomial regression model.
+
 ::: notes
 
 c.f.

_subfiles/count-regression/_sec_poisson_dx.qmd

@@ -7,17 +7,18 @@ $$e \eqdef y - \hat y$$
 
 #### Pearson residuals
 
-$$r = \frac{e}{\hse{e}} \approx \frac{e}{\sqrt{\hat y}}$$
+$$r \eqdef \frac{e}{\hse{e}} \approx \frac{e}{\sqrt{\hat y}}$$
 
 #### Standardized Pearson residuals
 
-$$r_p = \frac{r}{\sqrt{1-h}}$$
+$$r_p \eqdef \frac{r}{\sqrt{1-h}}$$
 where $h$ is the "leverage" (which we will continue to leave undefined).
 
 #### Deviance residuals
 
 $$
-d_k = \signt(y - \hat y)\left\{\sqrt{2[\ell_{\text{full}}(y) - \ell(\hat\beta; y)]}\right\}
+d_k \eqdef \signt(y - \hat y)
+  \left\{\sqrt{2[\ell_{\text{full}}(y) - \ell(\hat\beta; y)]}\right\}
 $$
 
 :::{.callout-note}

_subfiles/count-regression/_sec_zero-inflation.qmd

@@ -1,66 +1,84 @@
 
 ### Models for zero-inflated counts
 
-We assume a latent (unobserved) binary variable, $Z$, which we model using logistic regression:
+We assume a latent (unobserved) binary variable, $Z$,
+which we model using logistic regression:
 
-$$P(Z=1|X=x) = \pi(x) = \expit(\gamma_0 + \gamma_1 x_1 +...)$$
+$$
+\ba
+\P(Z=1 \mid \vX=\vx)
+&\eqdef \pi(\vx)\\
+&= \expit(\gamma_0 + \gamma_1 x_1 + \dots)
+\ea
+$$
 
-According to this model, if $Z=1$, then $Y$ will always be zero, regardless of $X$ and $T$:
+According to this model, if $Z=1$,
+then $Y$ will always be zero,
+regardless of $\vX$ and $T$:
 
-$$P(Y=0|Z=1,X=x,T=t) = 1$$
+$$\P(Y=0 \mid Z=1, \vX=\vx, T=t) = 1$$
 
-Otherwise (if $Z=0$), $Y$ will have a Poisson distribution, conditional on $X$ and $T$, as in a standard Poisson regression model.
+Otherwise (if $Z=0$), $Y$ will have a Poisson distribution,
+conditional on $\vX$ and $T$,
+as in a standard Poisson regression model.
 
-Even though we never observe $Z$, we can estimate the parameters $\gamma_0$-$\gamma_p$, via maximum likelihood:
+Even though we never observe $Z$,
+we can estimate the parameters $\gamma_0, \dots, \gamma_p$
+via maximum likelihood:
 
 $$
 \begin{aligned}
-\P(Y=y|X=x,T=t) &= \P(Y=y,Z=1|...) + \P(Y=y,Z=0|...)
+\P(Y=y \mid \vX=\vx, T=t) &= \P(Y=y, Z=1 \mid \vX=\vx, T=t) \\
+&\quad + \P(Y=y, Z=0 \mid \vX=\vx, T=t)
 \end{aligned}
 $$
 (by the Law of Total Probability)
 
 where
 $$
 \begin{aligned}
-P(Y=y,Z=z|...)
-&= P(Y=y|Z=z,...)P(Z=z|...)
+\P(Y=y, Z=z \mid \vX=\vx, T=t)
+&= \P(Y=y \mid Z=z, \vX=\vx, T=t)\,\P(Z=z \mid \vX=\vx)
 \end{aligned}
 $$
 
 ---
 
 ::: {#exr-zinf-pmf}
-Expand $P(Y=0|X=x,T=t)$, $P(Y=1|X=x,T=t)$ and $P(Y=y|X=x,T=t)$ into expressions involving $P(Z=1|X=x)$ and $P(Y=y|Z=0,X=x,T=t)$.
+Expand $\P(Y=0 \mid \vX=\vx, T=t)$, $\P(Y=1 \mid \vX=\vx, T=t)$,
+and $\P(Y=y \mid \vX=\vx, T=t)$
+into expressions involving $\P(Z=1 \mid \vX=\vx)$
+and $\P(Y=y \mid Z=0, \vX=\vx, T=t)$.
 :::
 
 ::: {.solution}
 
-Let $\pi = \P(Z=1|X=x)$ and $\mu_0 = \Expp[Y|Z=0,X=x,T=t]$.
+Let $\pi \eqdef \P(Z=1 \mid \vX=\vx)$
+and $\mu_0 \eqdef \Expp[Y \mid Z=0, \vX=\vx, T=t]$.
 
-**$P(Y=0)$:** $Y=0$ occurs either because $Z=1$ (always zero)
+**$\P(Y=0)$:** $Y=0$ occurs either because $Z=1$ (always zero)
 or because $Z=0$ and the Poisson draw equals 0:
 
 $$
 \ba
-\P(Y=0|X=x,T=t)
-&= \P(Z=1|X=x) + \P(Z=0|X=x)\,\P(Y=0|Z=0,X=x,T=t)\\
+\P(Y=0 \mid \vX=\vx, T=t)
+&= \P(Z=1 \mid \vX=\vx) + \P(Z=0 \mid \vX=\vx)\,\P(Y=0 \mid Z=0, \vX=\vx, T=t)\\
 &= \pi + (1-\pi)\,e^{-\mu_0}
 \ea
 $$
 
-**$P(Y=1)$:** $Z=1$ can never produce $Y=1$, so:
+**$\P(Y=1)$:** $Z=1$ can never produce $Y=1$, so:
 
 $$
-\P(Y=1|X=x,T=t)
-= (1-\pi)\,\P(Y=1|Z=0,X=x,T=t)
+\P(Y=1 \mid \vX=\vx, T=t)
+= (1-\pi)\,\P(Y=1 \mid Z=0, \vX=\vx, T=t)
 = (1-\pi)\,\mu_0 e^{-\mu_0}
 $$
 
-**$P(Y=y)$ for $y \geq 1$:** Identical reasoning gives
+**$\P(Y=y)$ for $y \geq 1$:** Identical reasoning gives
 
 $$
-\P(Y=y|X=x,T=t)
+\P(Y=y \mid \vX=\vx, T=t)
 = (1-\pi)\,\frac{\mu_0^y e^{-\mu_0}}{y!}
 $$
 
@@ -70,50 +88,55 @@ $$
 
 ::: {#exr-zinf-moments}
 
-Derive the expected value and variance of $Y$, conditional on $X$ and $T$, as functions of $\pi = P(Z=1|X=x)$ and $\mu_0 = \Expp[Y|Z=0,X=x,T=t]$.
+Derive the expected value and variance of $Y$, conditional on $\vX$ and $T$,
+as functions of $\pi \eqdef \P(Z=1 \mid \vX=\vx)$
+and $\mu_0 \eqdef \Expp[Y \mid Z=0, \vX=\vx, T=t]$.
 :::
 
 ::: {.solution}
 
-Let $\pi = \P(Z=1|X=x)$ and $\mu_0 = \Expp[Y|Z=0,X=x,T=t]$.
+Let $\pi \eqdef \P(Z=1 \mid \vX=\vx)$
+and $\mu_0 \eqdef \Expp[Y \mid Z=0, \vX=\vx, T=t]$.
 
 **Expected value.** By the Law of Total Expectation
-(conditioning on $Z$, within the subpopulation $\{X=x, T=t\}$):
+(conditioning on $Z$, within the subpopulation $\{\vX=\vx, T=t\}$):
 
 $$
 \ba
-\Expp[Y|X=x,T=t]
-&= \Expp[Y|Z=1,X=x,T=t]\,\pi + \Expp[Y|Z=0,X=x,T=t]\,(1-\pi)\\
+\Expp[Y \mid \vX=\vx, T=t]
+&= \Expp[Y \mid Z=1, \vX=\vx, T=t]\,\pi \\
+&\quad + \Expp[Y \mid Z=0, \vX=\vx, T=t]\,(1-\pi)\\
 &= 0 \cdot \pi + \mu_0(1-\pi)\\
 &= (1-\pi)\,\mu_0
 \ea
 $$
 
-The substitution $\Expp[Y|Z=0,X=x,T=t] = \mu_0$ follows immediately
+The substitution $\Expp[Y \mid Z=0, \vX=\vx, T=t] = \mu_0$ follows immediately
 from the definition of $\mu_0$.
 
 **Variance.** By the Law of Total Variance.
-To reduce clutter, we suppress the $(X=x, T=t)$ conditioning in the
+To reduce clutter, we suppress the $(\vX=\vx, T=t)$ conditioning in the
 intermediate steps: every expectation and variance is taken within
-the subpopulation $\{X=x, T=t\}$, and we restore the explicit conditioning
+the subpopulation $\{\vX=\vx, T=t\}$, and we restore the explicit conditioning
 in the final line.
 
 $$
-\Var{Y} = \Expp[\Var{Y|Z}] + \Var{\Expp[Y|Z]}
+\Var{Y} = \Expp[\Var{Y \mid Z}] + \Var{\Expp[Y \mid Z]}
 $$
 
-For the first term, since $\Var{Y|Z=1}=0$ and $\Var{Y|Z=0}=\mu_0$ (Poisson):
+For the first term,
+since $\Var{Y \mid Z=1}=0$ and $\Var{Y \mid Z=0}=\mu_0$ (Poisson):
 
 $$
-\Expp[\Var{Y|Z}] = 0 \cdot \pi + \mu_0(1-\pi) = (1-\pi)\mu_0
+\Expp[\Var{Y \mid Z}] = 0 \cdot \pi + \mu_0(1-\pi) = (1-\pi)\mu_0
 $$
 
-For the second term, $\Expp[Y|Z]$ takes the value 0 (with prob $\pi$)
+For the second term, $\Expp[Y \mid Z]$ takes the value 0 (with prob $\pi$)
 or $\mu_0$ (with prob $1-\pi$), so:
 
 $$
 \ba
-\Var{\Expp[Y|Z]}
+\Var{\Expp[Y \mid Z]}
 &= \pi(0 - (1-\pi)\mu_0)^2 + (1-\pi)(\mu_0 - (1-\pi)\mu_0)^2\\
 &= \pi(1-\pi)^2\mu_0^2 + (1-\pi)\pi^2\mu_0^2\\
 &= \pi(1-\pi)\mu_0^2[\,(1-\pi)+\pi\,]\\
@@ -124,12 +147,16 @@ $$
 Combining:
 
 $$
-\Var{Y|X=x,T=t} = (1-\pi)\mu_0 + \pi(1-\pi)\mu_0^2
+\Var{Y \mid \vX=\vx, T=t} = (1-\pi)\mu_0 + \pi(1-\pi)\mu_0^2
 = (1-\pi)\mu_0\bigl(1 + \pi\mu_0\bigr)
 $$
 
-Since $(1-\pi)\mu_0\bigl(1+\pi\mu_0\bigr) \geq (1-\pi)\mu_0 = \Expp[Y|X=x,T=t]$ for any $\pi > 0$,
-zero-inflated count models always exhibit overdispersion relative to a Poisson model
-with the same mean.
+Since
+$$
+(1-\pi)\mu_0(1+\pi\mu_0) \geq (1-\pi)\mu_0 = \Expp[Y \mid \vX=\vx, T=t]
+$$
+for any $\pi > 0$,
+zero-inflated count models always exhibit overdispersion
+relative to a Poisson model with the same mean.
 
 :::

chapters/count-regression.qmd

@@ -71,38 +71,56 @@ This content is adapted from:
 ## Negative binomial models
 
 ::: notes
-There are alternatives to the Poisson model.
+There are alternatives to the Poisson model when overdispersion is present.
 Most notably,
 the [negative binomial model](probability.qmd#sec-nb-dist).
 :::
 
+When diagnostics reveal overdispersion,
+the standard Poisson assumption $\Var{Y \mid \vX} = \Expp[Y \mid \vX]$
+is violated,
+leading to artificially narrow standard errors
+and inflated type I error rates.
 The [negative binomial distribution](probability.qmd#sec-nb-dist)
-is a common alternative to the Poisson distribution for count outcomes.
-It adds a dispersion parameter that allows the variance to exceed the mean,
-making it more flexible when overdispersion is present.
-We can still model $\mu$ as a function of $X$ and $T$ as before,
-and we can combine this model with zero-inflation
-(as the conditional distribution for the non-zero component).
+serves as a natural generalization of the Poisson distribution
+for count outcomes.
+It introduces an overdispersion parameter $\rho$
+that allows the conditional variance to exceed the mean
+($\Var{Y \mid \vX} = \mu + \mu^2 / \rho$).
+We still model $\mu(\vx, t) = t \exp{\eta(\vx)}$ as before,
+preserving the rate-ratio interpretation for regression coefficients.
+Furthermore, negative binomial models can be combined with zero-inflation
+to account for both structural zeros
+and variance expansion in count data.
 
 ---
 
-### Example: needle-sharing
+### Example: needle-sharing model extensions
 
 {{< include exr-needle-sharing-extensions.qmd >}}
 
-## Quasipoisson
+{{< slidebreak >}}
 
-Another way to handle overdispersion —
-rather than switching to the negative binomial distributional family —
-is the "quasipoisson" approach.
-It is a method-of-moments-type *inference method*:
-rather than specifying a full probability distribution and fitting by maximum likelihood,
-it specifies only the mean-variance relationship $\Var{Y} = \mu\theta$,
-and estimates $\theta$ accordingly.
-This approach is simpler to implement,
-but provides less information than the full negative binomial likelihood.
+## Quasipoisson regression
 
-See `?quasipoisson` in R for more.
+Another flexible approach to address overdispersion
+— without changing the underlying Poisson point estimates —
+is quasi-likelihood estimation via the quasipoisson model family.
+Rather than specifying a complete likelihood function
+and fitting by maximum likelihood,
+quasipoisson models specify only the mean-variance relationship
+$\Var{Y \mid \vX} = \theta \mu(\vx, t)$,
+where $\theta$ is a dispersion parameter estimated from Pearson residuals.
+
+While point estimates for regression coefficients $\evb$
+remain identical to standard Poisson regression,
+their estimated standard errors are scaled by $\sqrt{\eth}$.
+This method-of-moments approach provides robust standard error estimates
+and valid $p$-values when overdispersion is multiplicative,
+though it does not specify a full parametric distribution
+for prediction intervals or model likelihood comparisons.
+
+See `?quasipoisson` in R for implementation details.
 
 # More on count regression