Skip to content

MetadataCleanUp may make the present versions not available #606

Open
@YannByron

Description

MetadataCleanup will delete the expired delta logs(json and checkpoint.parquet). But if the present version depends on these which will be cleaned up, it cannot replay to the whole commits. For example: we have delta logs from version 0 to version 10 as following: 000.json ~ 009.json, 010.json, 010.checkpoint.parquet.

When Commit 10 is operated, MetadataCleanup works. If we assume the logs before 9 (not contained) should be cleaned up, then the rest files are: 009.json, 010.json, 010.checkpoint.parquet.
In fact, Version 9 is not available, and only Version 10 is shown by desc history.

Metadata

Assignees

No one assigned

    Labels

    acknowledgedThis issue has been read and acknowledged by Delta admins

    Type

    No type

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions