Skip to content

0x90d/videoduplicatefinder

BranchesTags

Repository files navigation

Video Duplicate Finder

Video Duplicate Finder is a cross-platform software to find duplicated video (and image) files on hard disk based on similarity. Unlike other duplicate finders this one also finds duplicates which have a different resolution, frame rate and even watermarked.

Features

  • Cross-platform
  • Fast scanning speed
  • Ultra fast rescan
  • Optional calling ffmpeg functions natively for even more speed
  • Finds duplicate videos / images based on similarity (optional scan against pHash at zero cost)
  • Partial clip detection — finds when a shorter video is a partial clip of a longer one (audio fingerprinting)
  • Optional AI matching — neural image embeddings find cropped, mirrored, zoomed and heavily edited copies the classic methods miss, and locate trimmed clips inside longer recordings without needing audio. Runs 100% locally.
  • Desktop GUI (Windows, Linux, macOS)
  • Headless CLI for scripting and automation
  • Web UI for remote/headless/NAS use
  • Docker image for easy self-hosting

Partial Clip Detection

VDF can detect when a shorter video is a partial clip of a longer one — for example, a scene ripped from a movie, or a clip saved from a longer recording. Candidates are found by audio fingerprinting, so it catches clips the normal visual scan misses; by default each audio match is then visually confirmed by comparing frames at the matched offset.

It runs as an optional second phase after the normal visual duplicate scan, using an audio fingerprinting pipeline (Chromaprint-style chroma extraction + sliding-window Hamming similarity matching). Matched pairs appear in the duplicate list with a Clip Offset column showing where in the source the clip starts.

Enabling it

In Settings → Partial Clip Detection, check Enable Partial Clip Detection and adjust:

Setting Default Description
Min clip / source ratio (%) 10 Minimum clip duration as a percentage of the source duration. Clips shorter than this are ignored.
Min audio similarity (%) 80 Minimum average Hamming similarity for the sliding-window fingerprint match to be accepted.
Require visual confirmation on Reject audio matches whose frames at the matched offset don't also look similar.
Min visual similarity (%) 85 Minimum frame similarity for the visual confirmation step.

Note: Partial clip detection requires audio tracks in both files. Videos without audio are skipped — for those, see the visual variant under AI Matching below.

AI Matching (optional)

VDF can additionally compare videos with neural image embeddings (a DINOv2 vision model running via ONNX Runtime). The classic comparison stays authoritative — the AI pass only adds pairs it is confident about, so enabling it never hides results you would otherwise get. It is good at exactly the cases pixel-based methods miss:

  • Transformed copies — cropped, mirrored, zoomed, letterboxed, color-graded or otherwise heavily edited versions of the same video. Pairs found this way are marked with an AI chip in the results.
  • Visual partial detection — finds trimmed cuts and clips contained in longer recordings by matching sampled keyframes with a consistent time offset. Unlike the audio-based partial clip detection above, this also works on silent, muted and re-dubbed videos. Matches appear with the same Clip Offset column.

Enabling it

Both switches are independent and off by default:

Setting Where Default Description
AI matching (additional pass) Settings → Matching off Enables the embedding comparison on top of the selected classic mode.
AI similarity threshold (%) Settings → Matching 94 How similar two files' embeddings must be. Lower to ~92 to find more aggressively edited copies at a slightly higher false-positive risk.
Detect partial duplicates visually (AI) Settings → Partial Clip Detection off Dense keyframe matching for trimmed/embedded clips, no audio needed.
AI frame hit threshold (%) Settings → Partial Clip Detection 89 Per-keyframe similarity needed for a hit; at least 4 hits must agree on one time offset before two videos are paired. Raise it if unrelated videos get paired.

CLI: --ai-matching, --ai-percent, --ai-partial, --ai-partial-hit-percent.

Components, privacy & footprint

  • On first use VDF downloads two components (~100 MB once): the ONNX Runtime library from the official Microsoft release and the embedding model (integrity-checked against a pinned SHA256). They are stored next to the scan database. The GUI asks before downloading; CLI/Web/Docker download automatically when an AI option is enabled.
  • Everything runs locally on your CPU. No cloud services, no accounts, nothing is uploaded — the model analyzes your frames on your machine, full stop.
  • Cost: roughly 50 ms per file during hashing; embeddings are cached in the scan database (~2 KB per file), so rescans stay fast. Visual partial detection keeps its keyframe cache in a separate DenseEmbeddings.db sidecar (~25 KB per video) that cleans itself up.
  • Supported on all release platforms (Windows, Linux x64/ARM64, macOS Intel & Apple Silicon).

Downloads

Daily build — attachments are automatically rebuilt and replaced on every commit.

Prefer the classic interface? 4.1 introduces a redesigned interface. The final classic-UI build stays available on the 4.0.x release — databases and settings are compatible both ways.

Upgrading from 3.x: your scan database is migrated automatically on first load. Cached image hashes are recomputed on the next scan (image processing moved from ImageSharp to FFmpeg); video hashes are unaffected. Downgrading back to 3.x after the migration is not recommended. The last 3.x build remains available on the 3.0.x release.

Available packages per platform:

  • GUI-<platform> — desktop application
  • CLI-<platform> — command-line tool
  • Web-<platform> — self-contained web server

Desktop GUI

Requirements

FFmpeg and FFprobe are required. On first launch VDF attempts to download them automatically. Native FFmpeg binding requires FFmpeg 8.x shared libraries (not the master branch).

Windows

Download the latest FFmpeg GPL shared package from https://ffmpeg.org/download.html Extract ffmpeg.exe and ffprobe.exe into the same folder as VDF.GUI.exe, a subfolder named bin, or ensure they are on your PATH.

Linux

sudo apt-get update && sudo apt-get install ffmpeg

Then run:

chmod +x VDF.GUI
./VDF.GUI

Optional: add to your application menu

The Linux archive includes videoduplicatefinder.desktop and icon.png. To register the app with your desktop environment (GNOME, KDE, XFCE, etc.):

# Edit the Exec= and Icon= paths to match where you extracted the archive, e.g.:
sed -i "s|/opt/videoduplicatefinder|$(pwd)|g" videoduplicatefinder.desktop

# Install for the current user
mkdir -p ~/.local/share/applications
cp videoduplicatefinder.desktop ~/.local/share/applications/

The app will then appear in your application launcher with its icon.

macOS

brew install ffmpeg

Extract the archive — it contains Video Duplicate Finder.app. Double-click it to launch.

If macOS blocks the app with "cannot be opened because the developer cannot be verified", right-click the .app and choose Open, then confirm. You only need to do this once.

If macOS still refuses to launch the bundle (e.g. "library load disallowed by system policy" on macOS 14+ / Tahoe), clear the quarantine flag and re-sign every binary in the bundle ad-hoc:

xattr -cr "Video Duplicate Finder.app"
codesign --force --deep --sign - "Video Duplicate Finder.app"

CLI (Command-line Interface)

The CLI is useful for scripting, scheduled tasks, and headless servers where no display is available.

Requirements

Same as the GUI: FFmpeg and FFprobe must be on your PATH or in the same directory as the vdf-cli binary.

Installation

Download CLI-<platform> from the releases page and extract it.

On Linux/macOS, make the binary executable:

chmod +x vdf-cli

Usage

Scan and compare in one step

vdf-cli scan-and-compare --include /path/to/media

Scan multiple directories, save results as JSON

vdf-cli scan-and-compare \
  --include /mnt/movies \
  --include /mnt/series \
  --exclude /mnt/movies/extras \
  --format json \
  --output results.json

Common options

Flag Description Default
--include <path> Directory to scan (repeatable) required
--exclude <path> Directory to exclude (repeatable) —
--threshold <n> Hash difference threshold 5
--percent <n> Minimum similarity % to report 96
--parallelism <n> Parallel hashing threads 1
--include-images Also scan image files off
--use-phash Use perceptual hashing off
--partial-clip-detection Enable partial clip detection (audio fingerprinting) off
--partial-clip-min-ratio <n> Min clip/source duration ratio (0.0–1.0) 0.10
--partial-clip-similarity <n> Min audio fingerprint similarity (0.0–1.0) 0.80
--ai-matching AI matching pass (downloads components on first use) off
--ai-percent <n> AI similarity threshold (50–100) 94
--ai-partial Visual partial detection via AI keyframes (no audio needed) off
--ai-partial-hit-percent <n> Per-keyframe hit threshold (70–99) 89
--format json|text|csv Output format text
--output <file> Write results to file instead of stdout stdout
--settings <file> Load full settings from a JSON file —

Auto-mark and delete duplicates

# Dry run — shows what would be deleted, no changes made (default)
vdf-cli scan-and-compare --include /mnt/media --action lowest-quality --dry-run

# Move duplicates to trash (safer)
vdf-cli scan-and-compare --include /mnt/media --action lowest-quality --delete

# Permanently delete (use with care)
vdf-cli scan-and-compare --include /mnt/media --action lowest-quality --delete-permanent

Available --action strategies:

Strategy Keeps
lowest-quality Highest bitrate/resolution per group
smallest-file Largest file per group
shortest-duration Longest duration per group
worst-resolution Highest resolution per group
100-percent-only Only acts on 100% identical groups

Note: Automatic deletion is not recommended. Always review results with --dry-run first.

Web UI

The Web UI runs as a local web server and is accessed from your browser. It is designed for headless machines, NAS devices, and remote management.

Security note: The Web UI is password-protected but intended for local/Docker use only. Do not expose it to the internet.

Authentication

On first launch, a random password is generated and printed to the console:

============================================
  Web UI password:  aB3xK9mQ7p
============================================

Enter this password in your browser to log in. A "Remember me" cookie keeps you logged in for 30 days.

Docker users: Run docker logs vdf-web to see the password.

Environment variable Description
VDF_WEB_PASSWORD Set your own password instead of the auto-generated one
VDF_WEB_AUTH=false Disable authentication entirely

Requirements

FFmpeg and FFprobe are required. When running outside Docker, VDF.Web will attempt to download them automatically on first launch. You can also install them manually via your system package manager or place them on your PATH.

Installation (self-contained archive)

Download Web-<platform> from the releases page and extract it.

On Linux/macOS:

chmod +x VDF.Web
./VDF.Web

On Windows:

VDF.Web.exe

Then open http://localhost:5000 in your browser and enter the password shown in the console.

To change the port:

ASPNETCORE_URLS=http://+:8080 ./VDF.Web

Web settings and login credentials are saved to:

  • Windows: %APPDATA%\VDF\
  • Linux: ~/.config/VDF/ (or $XDG_CONFIG_HOME/VDF/)
  • macOS: ~/Library/Preferences/VDF/

The scan database (ScannedFiles.db) is stored next to the executable if that folder is writable; otherwise it falls back to:

  • Windows: %LOCALAPPDATA%\VDF\
  • Linux: ~/.local/state/VDF/ (or $XDG_STATE_HOME/VDF/)
  • macOS: ~/Library/Application Support/VDF/

Docker (Web UI)

Docker is the easiest way to run the Web UI on a NAS, home server, or any Linux machine. FFmpeg is included in the image — no separate installation needed.

Requirements

Quick start

docker run -d \
  --name vdf-web \
  -p 8080:8080 \
  -v vdf-db:/root/.config/VDF \
  -v vdf-state:/root/.local/state/VDF \
  -v /path/to/your/media:/media:ro \
  ghcr.io/0x90d/vdf-web:latest

Then open http://localhost:8080 in your browser. Check the password with docker logs vdf-web and enter it to log in. Inside the Web UI, add /media (or whatever path you mounted) as a scan directory.

To set your own password:

docker run -d \
  --name vdf-web \
  -p 8080:8080 \
  -e VDF_WEB_PASSWORD=mysecretpassword \
  -v vdf-db:/root/.config/VDF \
  -v vdf-state:/root/.local/state/VDF \
  -v /path/to/your/media:/media:ro \
  ghcr.io/0x90d/vdf-web:latest

docker compose (recommended for permanent installs)

  1. Download docker-compose.yml from this repository.

  2. Edit the file and add your media volume mounts. Optionally set your own password:

environment:
  - VDF_WEB_PASSWORD=mysecretpassword    # optional — otherwise check docker logs
volumes:
  - /mnt/nas/movies:/mnt/nas/movies:ro
  - /mnt/nas/series:/mnt/nas/series:ro
  1. Start the service:
docker compose up -d

  1. Open http://localhost:8080 in your browser and enter the password (check docker logs if you didn't set one).

  2. To update to the latest image:

docker compose pull && docker compose up -d

Volume reference

Volume Purpose
/root/.config/VDF Settings (web-settings.json) and login credentials — mount a named volume here so configuration persists across container updates
/root/.local/state/VDF Scan database (ScannedFiles.db) — mount a named volume here so hashed-file data persists across container updates
Your media paths Mount each media directory you want to scan. Read-only (:ro) is recommended.

Notes

  • The container image is built for linux/amd64 and linux/arm64 (Raspberry Pi / NAS ARM boards).
  • The image is published to GitHub Container Registry and updated automatically on every commit.

Screenshots (outdated)

License

Video Duplicate Finder is licensed under AGPLv3.

The optional AI components are downloaded separately on first use and carry their own licenses: ONNX Runtime (MIT) and the DINOv2-small embedding model (Apache-2.0). Neither is bundled with or linked into the release binaries.

Credits / Third Party

Building

  • .NET 10.x
  • Visual Studio 2022 or later is recommended

Contributing

  • Create a pull request for each addition or fix — do not merge them into one PR
  • Unless it refers to an existing issue, write into your pull request what it does
  • For larger PRs, open an issue for discussion first

About

Video Duplicate Finder - Crossplatform

Resources

Readme
Activity

Stars

3.4k stars

Watchers

60 watching

Forks

296 forks
Report repository

Releases 9

4.1.x Latest
Jul 7, 2026
+ 8 releases

Packages

 
 
 

Contributors

Languages