Skip to content

Delete dataset URIs through object storage and propagate deletion failures #250

Description

@beinan

The rollout delete handler removes the registry entry and calls tokio::fs::remove_dir_all with the dataset URI. For a file://, s3://, or gs:// URI this is not a valid local filesystem deletion. Errors are only logged and the endpoint still returns HTTP 204.

Local reproduction using file:// (no cloud resources involved):

  1. Create a rollout store, write a row, and merge it.
  2. Delete it: the route returns 204.
  3. Create another store with the same name.
  4. The supposedly deleted row reappears.

Expected: successful deletion removes the dataset; failed deletion is reported and remains retryable.
Actual: the registry hides a dataset whose data still exists, and same-name recreation reopens it.

Relevant code: crates/lance-context-server/src/routes/rollouts.rs::delete_rollout_store. Other store-kind delete handlers use the same local-filesystem pattern and should be checked.

Please use the configured object-store backend, close resident writers before removal, preserve a retry path on failure, and add URI-backed deletion/recreation tests. Cloud behavior follows from the code path but was not tested against production storage.

Found while reviewing main at e292518; fixes will be based on refreshed main (currently 8fe27b2).

Activity

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Type

    No type

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions