Skip to content

Writer ignores the table's "delta.dataSkippingStatsColumns" and "delta.dataSkippingNumIndexedCols" #189

@kpapadatos

Description

@kpapadatos

Creating a table with "delta.dataSkippingStatsColumns" or "delta.dataSkippingNumIndexedCols" using delta-rs and then writing to it, results in having min/max values only for the specified columns in the delta log files.

When kafka-delta-ingest writes to that table, it always adds min/max values for all columns, making it unusable because the delta log becomes huge.

Metadata

Metadata

Assignees

No one assigned

    Labels

    bugSomething isn't working

    Type

    No type

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions