README.md

Awesome DuckLake

A curated list of awesome DuckLake tools and resources.

Contents

• Interfaces
• Libraries
• Transformation
• Implementations
• Resources
  • Documentation
  • Examples
  • Videos
  • Presentations

Interfaces

• DBeaver - Universal database tool and SQL client
• DuckDB CLI - DuckDB command-line interface
• DuckDB UI Extension - A user interface for your local DuckDB instance
• Harlequin - The SQL IDE for your terminal
• marimo - Reactive Python notebook
• Positron - A next-generation data science IDE
• Sling - Efficient data transfer between various sources and destinations

Libraries

• dlt - Python library that loads data from various data sources into well-structured, live datasets
• Ibis - Portable Python dataframe library
• Narwhals - Lightweight and extensible compatibility layer between dataframe libraries
• Pointblank - Powerful, yet elegant data validation framework for Python
• PySpark - Python API for Apache Spark

Transformation

• dbt Core - Open-source tool that enables transformation using analytics engineering best practices
• lea - Minimalist SQL orchestrator
• SQLMesh - Scalable and efficient data transformation framework

Implementations

• DuckDB - DuckDB core extension for attaching to databases stored in the DuckLake format
• MotherDuck - Cloud data warehouse with support for managed DuckLake
• DataFusion-DuckLake - DuckLake query engine for Rust, built with DataFusion.

Resources

Documentation

• DuckLake DuckDB extension - Documentation for the ducklake DuckDB extension
• Manifesto - The DuckLake manifesto from Mark Raasveldt and Hannes Mühleisen
• Specification - The specification for the DuckLake format

Examples

• DuckDB Python API - Python notebook demonstrating usage of the DuckDB Python API
• Ibis - Python notebook demonstrating usage of the Ibis dataframe library
• Pointblank - Python notebook demonstrating usage of the Pointblank data validation framework
• PySpark - Python notebook demonstrating usage of PySpark

Videos

• DuckLake & The Future of Open Table Formats - An in-depth discussion about DuckLake with Jordan Tigani and Hannes Mühleisen
• DuckLake - The SQL-Powered Lakehouse Format for the Rest of Us - A talk on the design rationale and principles of DuckLake by Hannes Mühleisen
• DuckLake: Learning from Cloud Data Warehouses to Build a Robust “Lakehouse” - A seminar by Jordan Tigani
• Introducing DuckLake - A podcast on the DuckLake project by Hannes Mühleisen and Mark Raasveldt

Presentations

• DuckLake: The SQL-Powered Lakehouse Format - Presentation by Hannes Mühleisen