Backward Elimination using Multiple Linear Regression

Overview

This project demonstrates the implementation of the Backward Elimination feature selection technique using Multiple Linear Regression in Python.

The program starts by fitting a regression model using all available independent variables and then iteratively removes variables that are not statistically significant based on their p-values. The process continues until all remaining variables satisfy the chosen significance level.

---

Requirements

Install the required libraries before running the program:

pip install numpy pandas statsmodels

---

Dataset

The program expects a data file named:

AS3Data.txt

Data Format

• First column: Dependent variable (Y)
• Remaining columns: Independent variables (X1, X2, X3, X4, X5, ...)

---

How It Works

1. Load the dataset using Pandas.
2. Separate the dependent variable (Y) and independent variables (X).
3. Add an intercept term to the model.
4. Fit a full Ordinary Least Squares (OLS) regression model.
5. Examine the p-values of all predictors.
6. Remove the predictor with the highest p-value if it exceeds the significance level (α = 0.05).
7. Refit the model and repeat the process.
8. Stop when all remaining predictors have p-values less than or equal to 0.05.
9. Display the final reduced regression model.

---

Significance Level

The backward elimination procedure uses:

alpha = 0.05

Any variable with a p-value greater than 0.05 is removed from the model.

---

Output

The program prints:

• Full model regression summary
• Full model equation
• Variables removed at each elimination step
• Final reduced model summary
• Reduced model equation

Example:

Step 1: removing x3 (p = 0.6841)
Step 2: removing x1 (p = 0.1297)

Final model:

Y = (6.4242) + (0.5826)*x2 + (1.2059)*x4 + (-0.6032)*x5

---

Running the Program

Place AS3Data.txt in the same directory as the Python file and run:

python backward_elimination.py

---

Concepts Used

• Multiple Linear Regression
• Ordinary Least Squares (OLS)
• Hypothesis Testing
• p-values
• Feature Selection
• Backward Elimination