dfxpy is a high-performance Python library designed to reduce the entire data preparation and analysis workflow into a few powerful, deterministic operations. It helps data scientists and engineers go from raw, messy datasets to model-ready features in seconds, not hours.
While pandas is the industry standard for data manipulation, cleaning a dataset for machine learning often requires writing repetitive boilerplate code. dfxpy acts as an accelerator, automating the "grunt work" while providing intelligent diagnostics and insights.
- Fast & Deterministic: Optimized operations without the unpredictability of AI.
- Production-Ready: Type-hinted, modular, and follows PEP8 standards.
- Lightweight: Minimal dependencies—built on top of
pandas,numpy, andscikit-learn. - Self-Contained: Load and analyze data without needing to import multiple libraries.
One-line pipeline for column normalization (snake_case), duplicate removal, smart type inference (Object → Numeric/Datetime), and missing value imputation.
Intelligent dataset diagnostics that detect:
- ID-like columns (uniqueness checks)
- High cardinality categoricals
- Multicollinearity (correlated features)
- Data skewness and low-variance columns
Transform your data into X and y instantly. Automatically handles categorical target encoding (LabelEncoding), feature encoding (One-Hot), and optional scaling.
Generates structured, human-readable summaries of shapes, nulls, unique counts, and correlation matrices.
Detect and handle outliers using the IQR (Interquartile Range) method with options to remove or cap.
Generate beautiful, standalone HTML EDA reports with distributions and missing value summaries—zero dependencies required.
Instantly track changes between two versions of a dataset. Detects shape shifts, column changes, and cell-level value modifications.
Handle imbalanced datasets with ease using Oversampling, Undersampling, or synthetic interpolation.
Enforce rigorous data contracts with schema validation. Ensure your pipeline only processes data that meets your quality standards.
Build, save, and reuse complex cleaning workflows with the new chainable pipeline engine.
Get instant recommendations on which ML algorithms (XGBoost, LightGBM, etc.) to use based on your dataset's specific characteristics.
Understand exactly what happened to your data. Logs every transformation and repair for full transparency.
Install the latest version via pip:
pip install dfxpyFor detailed usage and parameter definitions, see the Full API Reference.
import dfxpy as dfx
# 1. Load data directly
df = dfx.load("raw_data.csv")
# 2. Run auto-clean (names, types, nulls, encoding)
df_clean = dfx.auto(df)
# 3. Get intelligent insights
dfx.audit(df_clean)
# 4. Prepare for Machine Learning
X, y = dfx.prepare(df_clean, target="outcome")# Analyze a dataset instantly
dfxpy analyze data.csv
# Prepare data for ML via CLI
dfxpy prepare data.csv --target price --output cleaned_features.csv| Task | Standard Pandas | dfxpy |
|---|---|---|
| Load Data | pd.read_csv("data.csv") |
dfx.load("data.csv") |
| Clean Names | df.columns = [c.lower().replace(' ', '_') for c in df.columns] |
dfx.auto(df) |
| Handle Nulls | df['val'].fillna(df['val'].median()) |
dfx.auto(df) |
| ML Prep | 10+ lines (Split, Encode, Scale) | dfx.prepare(df, target='y') |
| Audit | Manual inspection | dfx.audit(df) |
| HTML Report | 50+ lines (Matplotlib/Jinja) | dfx.report(df) |
| Dataset Diff | df1.compare(df2) (limited) |
dfx.compare(df1, df2) |
| Explainability | Manual logging | dfx.analyze_cleaning(df) |
We welcome contributions! Please see our CONTRIBUTING.md for guidelines on how to submit issues, feature requests, and pull requests.
dfxpy is licensed under the MIT License.
Built with ❤️ for the Data Science Community.