🔬 ClusterLens

A production-ready clustering interpretation platform for business decision-making.

ClusterLens interprets the output of any clustering algorithm (KMeans, HDBSCAN, GMM, Spectral Clustering, etc.) and provides powerful tools to understand, diagnose, and operationalize clusters. Built for business users who need to understand segments, take decisions, and generate insights.

🏗 Architecture Overview

┌─────────────────────────────────────────────────────────────────┐
│                    React Frontend (Vite)                         │
│  ┌──────────┐ ┌──────────┐ ┌──────────┐ ┌───────────────────┐  │
│  │ Dashboard │ │ Explorer │ │ VisMap   │ │ Instance Explorer │  │
│  └──────────┘ └──────────┘ └──────────┘ └───────────────────┘  │
│  ┌──────────────┐ ┌──────────┐                                  │
│  │ Diagnostics  │ │ Insights │                                  │
│  └──────────────┘ └──────────┘                                  │
│          ▼ Fetch JSON via /api/*  ▼                              │
└─────────────────────────────────────────────────────────────────┘
                          │
┌─────────────────────────────────────────────────────────────────┐
│                     FastAPI Backend                              │
│  ┌────────────────────────────────────────────────────────────┐ │
│  │                 API Router Layer (10 routes)                │ │
│  │  /upload  /overview  /profiles  /diagnostics               │ │
│  │  /projections  /instances  /similarity  /insights /naming  │ │
│  └────────────────────────────────────────────────────────────┘ │
│  ┌────────────────────────────────────────────────────────────┐ │
│  │               Core Analytical Engine                       │ │
│  │  data_loader │ cluster_diagnostics │ cluster_profiles      │ │
│  │  cluster_explainability │ projection_engine                │ │
│  │  cluster_similarity │ cluster_naming │ business_insights   │ │
│  └────────────────────────────────────────────────────────────┘ │
│  ┌────────────────────────────────────────────────────────────┐ │
│  │          Data Layer (Polars + DuckDB + Cache)              │ │
│  └────────────────────────────────────────────────────────────┘ │
└─────────────────────────────────────────────────────────────────┘
                          │
┌─────────────────────────────────────────────────────────────────┐
│               Storage (Parquet + JSON Metadata)                  │
└─────────────────────────────────────────────────────────────────┘

🧰 Technology Stack

Layer	Technology	Why
Backend API	FastAPI	Async, high-perf, auto-docs via OpenAPI, Pydantic schemas
Data Processing	Polars + DuckDB	Polars for vectorized transforms; DuckDB for SQL analytics
Storage	Parquet + DuckDB	Columnar storage for speed; DuckDB for ad-hoc aggregation
Visualization	Plotly.js	Interactive, rich charts with React integration
Frontend	React 18 + Vite	Modern SPA with fast HMR, component-based UI
Caching	File-based (pickle)	Disk cache for expensive projections/diagnostics
Icons	Lucide React	Beautiful, consistent iconography

📐 Input Data Schema

ClusterLens accepts a universal input format that works with any clustering algorithm.

Primary Data: `observations.parquet`

Column	Type	Required	Description
`id`	string/int	✅	Unique observation identifier
`cluster_id`	int	✅	Assigned cluster label (-1 for noise)
`feature_*`	float	✅	Clustering features (any number)
`biz_*`	any	❌	Business variables for insights
`prob_*`	float	❌	Soft clustering probabilities
`dist_to_center`	float	❌	Distance to assigned cluster center
`umap_x`, `umap_y`	float	❌	Pre-computed UMAP projections

Metadata: `clusters_metadata.json`

{
  "algorithm": "KMeans",
  "n_clusters": 5,
  "cluster_centers": [[0.1, 0.2, ...], ...],
  "cluster_labels": { "0": "High Value", "1": "At Risk" },
  "parameters": { "n_init": 10 }
}

Metadata: `features_metadata.json`

{
  "features": {
    "feature_recency": {
      "display_name": "Recency (days)",
      "type": "numeric",
      "role": "clustering"
    }
  }
}

Column auto-detection: The system automatically categorizes columns by prefix (feature_* → clustering, biz_* → business, prob_* → probability, umap_*/tsne_* → projection).

🧠 Analytical Engine

8 Core Modules

Module	Responsibility
`data_loader`	Loads Parquet + JSON, auto-detects feature roles, registers in DuckDB
`cluster_diagnostics`	Silhouette, Davies-Bouldin, Calinski-Harabasz, variance, separation matrix
`cluster_profiles`	Per-cluster stats, lift vs population, effect sizes (Cohen's d), rankings
`cluster_explainability`	Instance-level: "Why is X in cluster C?" — distances, contributions, KNN
`projection_engine`	UMAP, t-SNE, PCA with caching and downsampling for large datasets
`cluster_similarity`	Centroid + Wasserstein distance, composite similarity, merge suggestions
`cluster_naming`	Auto-generates names like "High Value · Low Recency" from top features
`business_insights`	Generates insights like "Cluster 2 = 8% of users but 31% of revenue"

Diagnostics Engine

All metrics are translated to business-readable labels:

Metric	Range	Labels
Silhouette Score	−1 to 1	High (>0.5), Medium, Low, Poor
Davies-Bouldin	0 to ∞	Good (<1), Acceptable, Poor
Calinski-Harabasz	0 to ∞	High (>500), Medium, Low

Cluster Profiling

For each cluster, the engine computes:

Feature lift: cluster_mean / global_mean — how much a feature deviates
Effect size (Cohen's d): standardized difference vs population
Top discriminative features: ranked by absolute effect size

Instance Explainability

For any observation, answers "Why this cluster?" via:

Distance to own cluster center vs all other centers
Feature contribution scores (deviation normalized by cluster std)
K-nearest neighbors (same cluster + cross-cluster)

Business Insight Generation

Automatically surfaces surprising patterns using templates:

Size anomalies: "Cluster X is a niche segment with only 3% of population"
Value concentration: "Cluster Y = 8% of customers but 31% of revenue"
Feature extremes: "Cluster Z has significantly higher Recency (+2.1σ)"

Insights are ranked by a surprise score (deviation from uniform distribution).

🔌 API Endpoints

Method	Path	Description
POST	`/api/upload`	Upload Parquet + metadata files
GET	`/api/overview`	Cluster sizes, summary stats
GET	`/api/profiles`	All cluster profiles
GET	`/api/profiles/{cluster_id}`	Single cluster profile
GET	`/api/diagnostics`	Quality metrics + separation matrix
GET	`/api/projections?method=umap`	2D/3D projection data
GET	`/api/instances/{id}`	Instance-level explainability
GET	`/api/similarity`	Similarity matrix + merge suggestions
GET	`/api/insights`	Auto-generated business insights
GET	`/api/naming`	Auto-generated cluster names
GET	`/api/features`	Feature metadata + roles
GET	`/api/health`	Health check

All endpoints return:

{
  "status": "ok",
  "data": { ... },
  "meta": { "computation_time_ms": 123 }
}

Interactive API docs: http://localhost:8000/docs

🖥 UI Pages

1. Dashboard

4 metric cards (observations, clusters, features, algorithm)
Cluster size bar chart + composition donut chart
Top 3 business insights with surprise scores

2. Cluster Explorer

Clickable cluster cards with size/percentage
Radar chart comparing feature lifts across all clusters
Effect size heatmap (cluster × feature)
Detailed feature breakdown table for selected cluster

3. Visual Map

UMAP / t-SNE / PCA toggle (recomputes on switch)
Interactive WebGL scatter plot (handles 50K+ points)
Color by cluster with hover tooltips
Zoom, pan, and select

4. Instance Explorer

Search by observation ID
Distance to cluster centers bar chart
Feature contribution horizontal bar chart
Nearest neighbors table (clickable to chain-explore)
Assigned cluster with auto-generated name

5. Diagnostics

Global quality metrics with business-readable labels
Inter-cluster distance heatmap
Per-cluster silhouette bar chart
Quality metrics table

6. Insights

Ranked business insight cards with surprise scores
Cluster similarity heatmap
Merge suggestions with confidence levels and rationale

🚀 Getting Started

Prerequisites

Python 3.10+
Node.js 18+ and npm

1. Set up the backend

We recommend using uv as your package manager for much faster installations:

cd backend
uv venv
# Windows: .venv\Scripts\activate
# macOS/Linux: source .venv/bin/activate
uv pip install -r requirements.txt

(Alternatively, you can use standard python -m venv venv and pip install -r requirements.txt)

2. Generate sample data

python ../scripts/generate_sample_data.py

This creates a 10K-row RFM-style customer segmentation dataset in data/sample/.

3. Start the backend

uvicorn main:app --reload --port 8000

Visit http://localhost:8000/docs to see the API documentation.

4. Set up the frontend

cd frontend
npm install
npm run dev

Visit http://localhost:5173 to use the application.

5. Bring your own data

Upload via the API or place files in data/uploads/:

curl -X POST http://localhost:8000/api/upload \
  -F "observations=@my_clusters.parquet" \
  -F "clusters_metadata=@my_metadata.json" \
  -F "features_metadata=@my_features.json"

📁 Project Structure

ClusterLens/
├── backend/
│   ├── main.py                     # FastAPI app entry point
│   ├── config.py                   # Settings and paths
│   ├── api/                        # Route handlers
│   │   ├── routes_upload.py
│   │   ├── routes_overview.py
│   │   ├── routes_profiles.py
│   │   ├── routes_diagnostics.py
│   │   ├── routes_projections.py
│   │   ├── routes_instances.py
│   │   ├── routes_similarity.py
│   │   ├── routes_insights.py
│   │   ├── routes_naming.py
│   │   └── routes_features.py
│   ├── analytics/                  # Core analytical engine
│   │   ├── data_loader.py
│   │   ├── cluster_diagnostics.py
│   │   ├── cluster_profiles.py
│   │   ├── cluster_explainability.py
│   │   ├── projection_engine.py
│   │   ├── cluster_similarity.py
│   │   ├── cluster_naming.py
│   │   ├── business_insights.py
│   │   └── cluster_transitions.py
│   ├── models/
│   │   └── schemas.py              # Pydantic response models
│   ├── cache/
│   │   └── __init__.py             # Disk cache utilities
│   └── requirements.txt
├── frontend/
│   ├── package.json
│   ├── vite.config.js
│   ├── index.html
│   └── src/
│       ├── main.jsx
│       ├── App.jsx                 # Router setup
│       ├── index.css               # Design system
│       ├── api/client.js           # API wrapper
│       ├── utils/
│       │   ├── colors.js           # Cluster palette
│       │   └── formatters.js       # Number formatters
│       ├── components/
│       │   └── Layout.jsx          # Sidebar + Outlet
│       └── pages/
│           ├── Dashboard.jsx
│           ├── ClusterExplorer.jsx
│           ├── VisualMap.jsx
│           ├── InstanceExplorer.jsx
│           ├── Diagnostics.jsx
│           └── Insights.jsx
├── scripts/
│   └── generate_sample_data.py     # Sample data generator
├── data/
│   └── sample/                     # Generated sample dataset
└── README.md

⚡ Performance & Scalability

Strategy	Implementation
Columnar storage	Parquet for disk, DuckDB for ad-hoc SQL
Vectorized ops	Polars for all DataFrame transforms (10-100× vs Pandas)
Disk caching	Pickle-based cache with TTL for projections/diagnostics
Downsampling	Scatter plots capped at 50K points for rendering performance
Async I/O	FastAPI async endpoints for non-blocking reads
DuckDB aggregations	SQL GROUP BY for cluster stats on 1M rows in <1s
WebGL rendering	Plotly `scattergl` for interactive scatter with 50K+ points

Tested Scale

10K rows: All operations instant (<100ms)
100K rows: Diagnostics/profiles <2s, projections <10s (cached)
1M rows: Projections ~60s first load (cached after), all other ops <5s

🔮 Stretch Features

Cluster evolution over time: Compare snapshots, detect drift
Transition matrices: Sankey diagram showing observation movement
Scenario simulation: "What if we merge clusters 3 & 7?"
Cluster stability: Bootstrap resampling + Jaccard stability index
Cluster drift detection: KL divergence between time periods
Export to PowerPoint: Auto-generate slide deck with cluster profiles

📄 License

MIT License — see LICENSE for details.

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

🔬 ClusterLens

📋 Table of Contents

🏗 Architecture Overview

🧰 Technology Stack

📐 Input Data Schema

Primary Data: `observations.parquet`

Metadata: `clusters_metadata.json`

Metadata: `features_metadata.json`

🧠 Analytical Engine

8 Core Modules

Diagnostics Engine

Cluster Profiling

Instance Explainability

Business Insight Generation

🔌 API Endpoints

🖥 UI Pages

1. Dashboard

2. Cluster Explorer

3. Visual Map

4. Instance Explorer

5. Diagnostics

6. Insights

🚀 Getting Started

Prerequisites

1. Set up the backend

2. Generate sample data

3. Start the backend

4. Set up the frontend

5. Bring your own data

📁 Project Structure

⚡ Performance & Scalability

Tested Scale

🔮 Stretch Features

📄 License

FilesExpand file tree

README.md

Latest commit

History

README.md

File metadata and controls

🔬 ClusterLens

📋 Table of Contents

🏗 Architecture Overview

🧰 Technology Stack

📐 Input Data Schema

Primary Data: observations.parquet

Metadata: clusters_metadata.json

Metadata: features_metadata.json

🧠 Analytical Engine

8 Core Modules

Diagnostics Engine

Cluster Profiling

Instance Explainability

Business Insight Generation

🔌 API Endpoints

🖥 UI Pages

1. Dashboard

2. Cluster Explorer

3. Visual Map

4. Instance Explorer

5. Diagnostics

6. Insights

🚀 Getting Started

Prerequisites

1. Set up the backend

2. Generate sample data

3. Start the backend

4. Set up the frontend

5. Bring your own data

📁 Project Structure

⚡ Performance & Scalability

Tested Scale

🔮 Stretch Features

📄 License

Primary Data: `observations.parquet`

Metadata: `clusters_metadata.json`

Metadata: `features_metadata.json`