Vacuum support ?

[rgrunbla]

Hello,

I'm using dagster-delta with the merge strategy to create a scd type 2 version of some external database, in s3. It works well, refreshing my data and only updating what needs to be updated, but i was wondering : is there a way to manage vacuuming for the assets ?

E.g. I have the following merge stats:

{
  "execution_time_ms": 16926,
  "num_output_rows": 5901596,
  "num_source_rows": 5970635,
  "num_target_files_added": 12,
  "num_target_files_removed": 7,
  "num_target_files_scanned": 8,
  "num_target_files_skipped_during_scan": 0,
  "num_target_rows_copied": 5899543,
  "num_target_rows_deleted": 0,
  "num_target_rows_inserted": 1912,
  "num_target_rows_updated": 141,
  "rewrite_time_ms": 39548,
  "scan_time_ms": 340752
}

According to the stats, 7 file are to be removed, but I can still see them in s3 : will this plugin take care of removing them after e.g. 1 week ?

Thanks

[ion-elgreco]

@rgrunbla this plugin won't handle that but what you can do easily nowadays is use hooks on assets, so you have a hook that does the vacuuming after writing