Skip to content

Add metadata-level partition pruning to reduce file metadata fetch time for large catalogs #21

@shefeek-jinnah

Description

@shefeek-jinnah

When querying tables with millions of files, datafusion-ducklake fetches all file metadata from the catalog database before any filtering occurs. This causes performance degradation for large catalogs.

This issue was reported in the upstream DuckLake project: duckdb/ducklake#640

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Type

    No type
    No fields configured for issues without a type.

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions