The recent Axiom composable query engines blog highlights several of Axiom’s key advantages. DuckLake is a strong fit for Axiom because it cleanly separates table metadata from table data in a way that aligns naturally with Axiom’s connector architecture.
DuckLake stores schemas, snapshots, data file lists, partition values, and statistics in a relational catalog, while the actual data resides in Parquet files. In Axiom, the connector layer is responsible for planning-time metadata resolution, and Velox already provides a mature Parquet scan path. As a result, a DuckLake connector can focus on translating DuckLake metadata into Velox scan splits, while reusing the existing Hive/DWIO Parquet reader for execution.
Today, Axiom’s local Hive connector is primarily used for development and testing. Its metadata model is intentionally simple and largely derived from directory structure—table locations, partition paths, and files on disk. While this works well for Hive-style layouts, it also limits metadata expressiveness to what can be inferred from file paths.
DuckLake introduces a richer local lakehouse metadata layer. Instead of encoding table and partition information in directory structures, it stores metadata explicitly in a relational catalog database such as DuckDB, SQLite, or PostgreSQL, while keeping table data in Parquet files.
Conceptually, DuckLake is close to Apache Iceberg. In both systems, the engine does not simply scan directories of files. Instead, the connector must first resolve table metadata—schema, current snapshot, live data files, partition information, and statistics—before planning execution. The key difference lies in how metadata is represented: Iceberg uses metadata and manifest files, whereas DuckLake stores this information in relational catalog tables.
This model offers several advantages for Axiom:
• Richer metadata than directory-based Hive
Hive-style metadata is largely inferred from filesystem layout. DuckLake metadata is explicit and queryable, allowing Axiom to read schemas, snapshots, data files, partition values, and statistics directly from catalog tables.
• Support for hidden partitioning
Hive layouts expose partitions through directory names. DuckLake supports partitioning as metadata, removing the need to encode partition columns in file paths. This aligns better with modern lakehouse designs and provides a more realistic environment for metadata-driven pruning.
An initial version of the connector can remain intentionally scoped and read-only:
• DuckDB-backed DuckLake catalog
• Current snapshot only
• Parquet data files only
• No delete file support initially
• No data inlining initially
• Explicit errors for unsupported features
@mbasmanova @Yuhta would love your thoughts on whether this would be valuable for Axiom.
The recent Axiom composable query engines blog highlights several of Axiom’s key advantages. DuckLake is a strong fit for Axiom because it cleanly separates table metadata from table data in a way that aligns naturally with Axiom’s connector architecture.
DuckLake stores schemas, snapshots, data file lists, partition values, and statistics in a relational catalog, while the actual data resides in Parquet files. In Axiom, the connector layer is responsible for planning-time metadata resolution, and Velox already provides a mature Parquet scan path. As a result, a DuckLake connector can focus on translating DuckLake metadata into Velox scan splits, while reusing the existing Hive/DWIO Parquet reader for execution.
Today, Axiom’s local Hive connector is primarily used for development and testing. Its metadata model is intentionally simple and largely derived from directory structure—table locations, partition paths, and files on disk. While this works well for Hive-style layouts, it also limits metadata expressiveness to what can be inferred from file paths.
DuckLake introduces a richer local lakehouse metadata layer. Instead of encoding table and partition information in directory structures, it stores metadata explicitly in a relational catalog database such as DuckDB, SQLite, or PostgreSQL, while keeping table data in Parquet files.
Conceptually, DuckLake is close to Apache Iceberg. In both systems, the engine does not simply scan directories of files. Instead, the connector must first resolve table metadata—schema, current snapshot, live data files, partition information, and statistics—before planning execution. The key difference lies in how metadata is represented: Iceberg uses metadata and manifest files, whereas DuckLake stores this information in relational catalog tables.
This model offers several advantages for Axiom:
• Richer metadata than directory-based Hive
Hive-style metadata is largely inferred from filesystem layout. DuckLake metadata is explicit and queryable, allowing Axiom to read schemas, snapshots, data files, partition values, and statistics directly from catalog tables.
• Support for hidden partitioning
Hive layouts expose partitions through directory names. DuckLake supports partitioning as metadata, removing the need to encode partition columns in file paths. This aligns better with modern lakehouse designs and provides a more realistic environment for metadata-driven pruning.
An initial version of the connector can remain intentionally scoped and read-only:
• DuckDB-backed DuckLake catalog
• Current snapshot only
• Parquet data files only
• No delete file support initially
• No data inlining initially
• Explicit errors for unsupported features
@mbasmanova @Yuhta would love your thoughts on whether this would be valuable for Axiom.