Skip to content

Add Migration Tool to Retroactively Deduplicate Existing Vaults #94

@S4tvara

Description

@S4tvara

Describe the feature
Introduce a migration tool to retroactively deduplicate existing vaults. Currently, when deduplication is enabled on an existing vault, previously stored chunks are not deduplicated.

Why is this feature needed?

  • Existing vaults cannot benefit from deduplication without re-adding files.
  • Users waste storage space since duplicate chunks remain untouched.
  • Improves consistency with new vaults that already have dedup enabled.

Proposed solution

  • Add a sietch dedup migrate command.
  • This command should scan existing chunks in the vault, compute their hashes, and update the deduplication index.
  • Duplicate chunks should be consolidated and properly tracked in the index.

Alternatives considered

  • Manually re-adding files after enabling dedup, which is time-consuming and inefficient.

Additional context

  • The limitation is already noted in the code at cmd/dedup.go:124-126.
  • Providing a migration command would ensure backward compatibility and help existing users save space without manual intervention.

Metadata

Metadata

Assignees

No one assigned

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions