feat: new article

Author: DiogoRibeiro7

_posts/2024-10-28-understanding normality tests a deep dive into their power and limitations.md

@@ -0,0 +1,159 @@
+---
+author_profile: false
+categories:
+- Data Analysis
+classes: wide
+date: '2024-10-28'
+excerpt: An in-depth look at normality tests, their limitations, and the necessity of data visualization.
+header:
+  image: /assets/images/data_science_5.jpg
+  og_image: /assets/images/data_science_5.jpg
+  overlay_image: /assets/images/data_science_5.jpg
+  show_overlay_excerpt: false
+  teaser: /assets/images/data_science_5.jpg
+  twitter_image: /assets/images/data_science_5.jpg
+keywords:
+- Normality Tests
+- Statistics
+- Data Analysis
+- QQ Plots
+seo_description: An in-depth exploration of normality tests, their limitations, and the importance of visual inspection for assessing whether data follow a normal distribution.
+seo_title: 'Understanding Normality Tests: A Deep Dive'
+seo_type: article
+summary: This article delves into the intricacies of normality testing, revealing the limitations of common tests and emphasizing the importance of visual tools like QQ plots and CDF plots.
+tags:
+- Normality Tests
+- Statistical Methods
+- Data Visualization
+title: 'Understanding Normality Tests: A Deep Dive into Their Power and Limitations'
+---
+
+**Abstract:**  
+In statistical analysis, assessing whether data follow a normal distribution is a critical step that influences subsequent tests and interpretations. However, the concept of "normality tests" is often misunderstood. This article explores the intricacies of normality testing, highlighting the limitations of these tests, the variety of methods available, and the importance of understanding what each test measures. We will delve into a unique distribution that challenges several normality tests, demonstrating why visual inspection and a comprehensive understanding of the data are indispensable.
+
+## The Role of Normality in Statistical Analysis
+
+The normal distribution holds a place of central importance in statistics. Many statistical models and tests assume that data follow a normal (Gaussian) distribution, characterized by a bell-shaped curve and symmetry around the mean. This assumption underpins inferential statistics, making normality testing a key first step in any analysis.
+
+Despite their common usage, normality tests are often misunderstood. These tests do not confirm that data are normal; rather, they test whether the data provide enough evidence to reject the null hypothesis ($$H_0$$) that the data come from a normal distribution. A failure to reject $$H_0$$ merely suggests that the data are not significantly different from a normal distribution, within the power and constraints of the test used.
+
+## The Special Distribution: A Case Study for Normality Testing
+
+Consider a specially constructed distribution with features that challenge normality tests. This distribution is symmetric with near-zero skewness and kurtosis approximating that of a normal distribution. Despite these properties, it is bimodal, featuring two distinct peaks—clearly deviating from the single-peaked nature of a normal distribution.
+
+### Key Characteristics of the Distribution:
+
+- **Perfect Symmetry (Skewness ≈ 0):** The left and right tails mirror each other.
+- **Normal Kurtosis (Kurtosis ≈ 3):** The distribution's tails are similar in weight to those of a normal distribution.
+- **Bimodality:** Unlike a normal distribution, the distribution has two peaks with a noticeable dip in the middle.
+
+Despite this bimodality, tests that focus only on skewness and kurtosis may fail to detect the deviation from normality, as these metrics align closely with those of a true normal distribution.
+
+## Understanding Normality Tests
+
+Normality tests formulate a null hypothesis ($$H_0$$) that the data come from a normal distribution and calculate a test statistic to determine whether to reject $$H_0$$. These tests have limitations that affect their reliability, depending on sample size, test type, and the specific features of the data.
+
+### The Null Hypothesis and Type II Errors
+
+It is crucial to understand that failing to reject $$H_0$$ does not confirm normality. It indicates that there is insufficient evidence to conclude the data are not normal, which may reflect a Type II error—failing to detect a deviation when one exists.
+
+### Limitations of Normality Tests:
+
+1. **Sample Size Sensitivity:** Smaller samples may not provide enough power to detect deviations.
+2. **Test Specificity:** Different tests respond to different features (e.g., skewness, kurtosis, overall shape).
+3. **Multiple Testing Issues:** Running multiple normality tests can lead to conflicting results.
+
+## Moment-Based Tests: Skewness, Kurtosis, and Jarque-Bera
+
+### Skewness and Kurtosis
+
+- **Skewness** measures the asymmetry of a distribution. A skewness of zero indicates symmetry.
+- **Kurtosis** measures the heaviness of the tails. A kurtosis of 3 indicates a normal distribution.
+
+### The Jarque-Bera Test
+
+The **Jarque-Bera test** combines skewness and kurtosis into a test statistic:
+
+$$
+JB = \frac{n}{6} \left( S^2 + \frac{(K - 3)^2}{4} \right)
+$$
+
+Where $$S$$ is skewness, $$K$$ is kurtosis, and $$n$$ is the sample size. In our special distribution, since skewness and kurtosis are close to zero, the test may fail to reject $$H_0$$—even though the distribution is bimodal.
+
+### Geary's Kurtosis: A Robust Alternative
+
+**Geary's kurtosis** (or Geary's ratio) compares the **Median Absolute Deviation (MAD)** to the **Standard Deviation (SD)**:
+
+$$
+\text{Geary's Ratio} = \frac{\text{MAD}}{\text{SD}}
+$$
+
+This ratio is more robust to outliers and better captures the shape of the distribution. In our bimodal distribution, Geary's test would be more likely to detect the deviation from normality.
+
+## Tests Based on CDF Comparison
+
+These tests compare the empirical cumulative distribution function (ECDF) of the sample to the theoretical CDF of a normal distribution.
+
+### Kolmogorov-Smirnov (K-S) Test
+
+The **K-S test** calculates the maximum difference between the ECDF and the theoretical CDF:
+
+$$
+D = \sup_x |F_n(x) - F(x)|
+$$
+
+### Cramér-von Mises Test
+
+The **Cramér-von Mises test** evaluates the squared difference between the ECDF and the theoretical CDF:
+
+$$
+W^2 = n \int_{-\infty}^{\infty} [F_n(x) - F(x)]^2 dF(x)
+$$
+
+### Anderson-Darling Test
+
+The **Anderson-Darling test** improves upon the Cramér-von Mises test by giving more weight to deviations in the tails.
+
+These CDF-based tests are sensitive to differences across the entire distribution. In our special case, the Anderson-Darling test would likely detect the non-normality due to the distribution’s bimodal structure.
+
+## Shapiro-Wilk and Shapiro-Francia Tests
+
+### Shapiro-Wilk Test
+
+The **Shapiro-Wilk test** measures the correlation between the data and the expected normal scores:
+
+$$
+W = \frac{\left( \sum_{i=1}^n a_i X_{(i)} \right)^2}{\sum_{i=1}^n (X_i - \bar{X})^2}
+$$
+
+Where $$X_{(i)}$$ are the ordered data and $$a_i$$ are constants based on expected normal values.
+
+### Shapiro-Francia Test
+
+The **Shapiro-Francia test** is a variant of the Shapiro-Wilk test, adjusted for larger sample sizes.
+
+### Performance on the Special Distribution
+
+For our bimodal distribution, these tests may not effectively detect the non-normality. Although useful for detecting skewness or tail deviations, they might overlook the bimodality.
+
+## Contradictory Test Results: Understanding the Discrepancies
+
+Different tests often yield conflicting results. For instance, while moment-based tests like **Jarque-Bera** may suggest normality, CDF-based tests like **Anderson-Darling** could strongly reject it. These contradictions arise because each test measures different aspects of the distribution.
+
+## The Importance of Data Visualization
+
+### Quantile-Quantile (QQ) Plots
+
+A **QQ plot** compares the sample quantiles to the theoretical normal quantiles. Deviations from a straight line indicate non-normality, which may be due to skewness, kurtosis, or modality.
+
+### Empirical CDF (ECDF) Plots
+
+An **ECDF plot** shows the cumulative probabilities of the sample data. Overlaying the theoretical CDF provides a visual comparison, highlighting differences across the entire distribution.
+
+### Application to the Special Distribution
+
+Visualizing our bimodal distribution with QQ or ECDF plots would immediately reveal the deviations from normality—particularly the dip in the middle of the distribution.
+
+## Conclusion
+
+Normality tests are essential tools, but they must be used with a deep understanding of their limitations. While formal tests can provide statistical evidence, they should be complemented by visual inspection tools like QQ plots and ECDF plots. Moreover, different tests have different sensitivities, so choosing the right test for the data’s characteristics is crucial. In practice, a comprehensive approach, combining statistical tests with visualization, will yield more robust and accurate conclusions.