LendingClub Credit Default Risk Analysis

An end-to-end data science project predicting loan defaults using an automated AWS cloud pipeline, Python machine learning models, and an interactive Tableau dashboard.

---

Project Summary

Dataset	LendingClub Accepted Loans 2007–2018
Rows analyzed	1,175,915
Best model	XGBoost
Model AUC	0.7210
Defaults caught	27,018 (test set)
Estimated losses prevented	$258,133,451
Net benefit	$35,131,266

---

Table of Contents

• Project Overview
• AWS Architecture
• Tech Stack
• Project Structure
• Phase 1 — AWS Data Pipeline
• Phase 2 — EDA & Modeling
• Phase 3 — Tableau Dashboard
• Model Results
• Business Recommendations
• How to Run

---

Project Overview

LendingClub is a peer-to-peer lending platform where investors fund personal loans. When borrowers stop making payments and their debt is written off, this is called a charge-off — or default. This project builds a production-grade pipeline to predict which loans will default based on borrower characteristics at the time of loan origination.

The project covers the full data science lifecycle:

• Raw data ingestion and cleaning on AWS
• Exploratory data analysis with Python
• Feature engineering and model training
• Business impact quantification
• Interactive Tableau dashboard for stakeholder communication

---

AWS Architecture

The pipeline is fully automated and event-driven:

1. Raw CSV lands in S3 raw/ folder
2. Lambda detects the upload and triggers Glue ETL automatically
3. Glue PySpark job cleans, transforms, and writes partitioned Parquet files
4. DataBrew profiles the cleaned data for quality assurance
5. Glue Crawler registers the schema in the Data Catalog
6. Athena enables serverless SQL queries directly on S3
7. Python/Jupyter connects to Athena for modeling
8. Tableau connects to Athena for dashboarding

---

Tech Stack

Layer	Tools
Cloud storage	AWS S3
ETL	AWS Glue (PySpark)
Data profiling	AWS Glue DataBrew
Query engine	AWS Athena
Orchestration	AWS Lambda + Step Functions
Analysis	Python, Jupyter Notebooks
Modeling	scikit-learn, XGBoost
Visualization	Tableau Public

---

Project Structure

Loan/
├── Clean_data/
│   └── lending_club_clean.csv          # cleaned dataset (1.17M rows)
├── Model_data/
│   ├── X_train.csv                     # training features
│   ├── X_test.csv                      # test features
│   ├── y_train.csv                     # training labels
│   ├── y_test.csv                      # test labels
│   └── lending_club_processed.csv      # fully encoded dataset
├── notebooks/
│   ├── 01_eda.ipynb                    # exploratory data analysis
│   ├── 02_feature_engineering.ipynb    # feature prep and encoding
│   ├── 03_modeling.ipynb               # model training and comparison
│   └── 04_model_evaluation.ipynb       # evaluation and business impact
├── Outputs/
│   ├── plots/                          # all saved charts
│   ├── scored_loans.csv                # model predictions on test set
│   ├── tableau_data.csv                # data for Tableau dashboard
│   ├── model_comparison.csv            # model metrics summary
│   └── model_recommendation.txt        # final recommendation write-up
└── README.md

---

Phase 1 — AWS Data Pipeline

Data source

The LendingClub dataset contains 1.5M+ accepted loan records from 2007 to 2018 with 74 columns covering borrower demographics, credit history, loan details, and repayment outcomes.

Glue ETL job

The PySpark cleaning script performs the following transformations:

• Selected 22 relevant columns from 74
• Filtered to closed loans only (Fully Paid and Charged Off)
• Stripped % from int_rate and revol_util, cast to float
• Removed months from term, cast to integer
• Standardised emp_length to numeric years
• Created binary target variable default_flag (1 = Charged Off)
• Extracted issue_year from date strings
• Dropped rows with nulls in critical columns
• Wrote output as Parquet partitioned by issue_year

DataBrew profile results

Key findings from the data profile:

• 99% valid cells across all 23 columns
• 0 duplicate rows
• Less than 1% missing values
• Strong positive correlation between int_rate and default_flag (r = 0.30)
• Strong negative correlation between fico_avg and int_rate (r = -0.40)

---

Phase 2 — EDA & Modeling

Notebook 01 — Exploratory Data Analysis

Screenshot: Outputs/plots/default_rate_by_grade.png

The relationship between loan grade and default rate is almost perfectly linear. Grade A borrowers default at 5.7% while Grade G borrowers default at 50.5% — nearly a coin flip. This confirmed that grade would be the dominant predictive feature.

Screenshot: outputs/plots/int_rate_distribution.png

Defaulted loans cluster at higher interest rates. The blue (paid) curve peaks around 10–13% while the red (default) distribution is right-shifted with a longer tail beyond 20%.

Screenshot: outputs/plots/default_rate_by_purpose.png

Small business loans default at 30.7% — the highest of any category. Car loans default at just 15%. Loan purpose adds meaningful signal beyond grade alone.

Screenshot: outputs/plots/correlation_matrix.png

Key insight: open_acc and total_acc are highly correlated (r ≈ 0.7), indicating multicollinearity. Only one was retained for modeling.

Notebook 02 — Feature Engineering

Key steps:

• Dropped open_acc (correlated with total_acc), sub_grade (redundant with grade), issue_year (not available at origination)
• Capped outliers at 99th percentile for annual_inc, revol_bal, loan_amnt, dti
• Ordinal encoded grade (A=1 through G=7)
• One-hot encoded home_ownership, verification_status, purpose
• Stratified 80/20 train/test split preserving 19.86% default rate

Final feature set: 32 features, 940,732 training rows, 235,183 test rows

Notebook 03 — Modeling

Three models were trained and compared:

Model	AUC	Precision	Recall	F1
Logistic Regression	0.7098	0.321	0.638	0.427
Random Forest	0.7129	0.310	0.684	0.426
XGBoost	0.7210	0.320	0.677	0.435

Screenshot: outputs/plots/roc_curve_comparison.png

XGBoost achieved the best AUC of 0.7210 — a strong result on real-world financial data consistent with published academic benchmarks on this dataset.

Screenshot: outputs/plots/feature_importance.png

grade accounts for 71% of XGBoost feature importance — confirming that LendingClub's proprietary grading system already encodes most of the credit risk signal. The marginal contribution of additional features is limited, which suggests alpha in peer-to-peer lending comes from identifying mispriced grades rather than building better default models.

Notebook 04 — Model Evaluation

Screenshot: outputs/plots/confusion_matrices.png

XGBoost confusion matrix on 235,183 test loans:

• True negatives (correctly approved): 121,156
• True positives (defaults correctly caught): 31,644
• False negatives (missed defaults): 15,071
• False positives (good loans rejected): 67,312

Screenshot: outputs/plots/threshold_analysis.png

Business Impact Analysis — XGBoost at optimal threshold:

Defaults correctly caught:    27,018
Estimated losses prevented:   $258,133,451
Estimated revenue lost:       $223,002,185
Net benefit:                  $35,131,266

On the 235K test set alone, the model generates $35M net benefit. Scaled to the full loan portfolio this represents approximately $175M in net value.

---

Phase 3 — Tableau Dashboard

Screenshot your full Tableau dashboard here

Live dashboard: View on Tableau Public

The dashboard contains four interactive views:

• Default rate by grade — A=6% through G=50.9% with red-green color coding
• Risk tier distribution — loan volume by model-assigned risk tier
• Business KPIs — total loans, AUC, defaults caught, losses prevented, net benefit
• Loan volume by year — 2013–2018 trend with default rate as color

All views respond to a Grade filter allowing stakeholders to drill into specific borrower segments.

---

What the feature importance tells us

The dominance of grade (71% importance) is the most interesting finding of the project. It means LendingClub's internal credit scoring already captures the vast majority of default risk. Additional borrower-level features add surprisingly little signal once grade is known. This has two implications:

1. For investors: diversifying across grades matters far more than trying to cherry-pick within a grade
2. For LendingClub: the current grading model is highly effective but could potentially incorporate more behavioral data (payment history patterns, income verification) to improve sub-grade differentiation

Business Recommendations

1. Deploy XGBoost at the 0.35 probability threshold

The default 0.5 threshold is too conservative for this use case. At 0.35 the model achieves the best F1 score by catching more defaults at an acceptable false positive rate. This translates directly to $35M+ net benefit on the test set.

2. Flag borderline loans for manual review

Loans with predicted default probability between 0.35 and 0.55 should be flagged for human underwriter review rather than auto-rejected. This reduces false positives while maintaining strong default detection — protecting both revenue and borrower fairness.

3. Re-train quarterly

Default rates are sensitive to macroeconomic conditions (interest rate environment, unemployment). The model was trained on 2013–2018 data. Quarterly re-training on recent loans ensures the model adapts to current credit conditions rather than overfitting to historical patterns.

4. Investigate Grade F and G lending

Grade G borrowers default at 50.9% — essentially a coin flip. The interest rate premium charged on these loans (30%+) may not fully compensate for the default losses when operating costs are factored in. A cost-benefit analysis of discontinuing Grade G lending is recommended.

5. Focus on small business loan underwriting

Small business loans have the highest default rate of any purpose category (30.7%) — nearly 10 percentage points above average. Enhanced underwriting criteria specifically for small business applicants (cash flow verification, business age, industry risk) could meaningfully reduce losses in this segment.

6. Use FICO score for second-level screening only

Despite strong correlation with default in EDA, FICO adds minimal predictive power once grade is known (feature importance ~1%). This suggests FICO is already embedded in the grade calculation. Underwriters should not over-weight FICO in manual reviews of borderline loans.

How to Run

Prerequisites

• Python 3.10+
• AWS account with appropriate IAM permissions
• Tableau Public (free)

AWS setup

# Configure AWS credentials
aws configure

# Upload raw data
python upload_data_s3.py

Python environment

pip install pyathena pandas numpy matplotlib seaborn scikit-learn xgboost imbalanced-learn boto3 jupyterlab sqlalchemy

Run notebooks in order

jupyter lab
# Open and run:
# notebooks/01_eda.ipynb
# notebooks/02_feature_engineering.ipynb
# notebooks/03_modeling.ipynb
# notebooks/04_model_evaluation.ipynb

Tableau

Open Tableau Public and connect to Outputs/tableau_data.csv

---

Dataset

LendingClub Loan Data (2007–2018) — available on Kaggle: https://www.kaggle.com/datasets/wordsforthewise/lending-club

---

Built by Rohan