Rewriting history in ducklake

[c-herrewijn]

Background

Consider the following scenarios:

• you realize that your ducklake contains sensitive data that never should have been there (e.g. unencrypted creditcard data or passwords)
• someone demands their data to be removed based on right to be forgotten (which might also apply to backups if technically possible / reasonable)

True deletion can be obtained by expiring past snapshots (in combination with ducklake_cleanup_old_files()). The drawback of that method is that you then loose all time travel options in the affected time window.

Recreating the affected parquet files via a user created script that replaces some data with anonymized values might be an option, but that is cumbersome and requires a deeper understanding of the ducklake format. (At the risk of corrupting the datalake)

Suggestion

Would logic to 'rewrite history' be a useful addition to the ducklake specification / implementation?

This would likely mean alternative versions of UPDATE, DELETE, and DROP statements to achieve "delete as if it never existed", or "update as if it always was like this".