-
Notifications
You must be signed in to change notification settings - Fork 26
Open
Labels
Description
Describe the feature
Introduce a migration tool to retroactively deduplicate existing vaults. Currently, when deduplication is enabled on an existing vault, previously stored chunks are not deduplicated.
Why is this feature needed?
- Existing vaults cannot benefit from deduplication without re-adding files.
- Users waste storage space since duplicate chunks remain untouched.
- Improves consistency with new vaults that already have dedup enabled.
Proposed solution
- Add a
sietch dedup migratecommand. - This command should scan existing chunks in the vault, compute their hashes, and update the deduplication index.
- Duplicate chunks should be consolidated and properly tracked in the index.
Alternatives considered
- Manually re-adding files after enabling dedup, which is time-consuming and inefficient.
Additional context
- The limitation is already noted in the code at
cmd/dedup.go:124-126. - Providing a migration command would ensure backward compatibility and help existing users save space without manual intervention.