GitHub - 0x90d/videoduplicatefinder: Video Duplicate Finder - Crossplatform (original) (raw)

Video Duplicate Finder is a cross-platform software to find duplicated video (and image) files on hard disk based on similarity. Unlike other duplicate finders this one also finds duplicates which have a different resolution, frame rate and even watermarked.

Features

Partial Clip Detection

VDF can detect when a shorter video is a partial clip of a longer one — for example, a scene ripped from a movie, or a clip saved from a longer recording. Candidates are found by audio fingerprinting, so it catches clips the normal visual scan misses; by default each audio match is then visually confirmed by comparing frames at the matched offset.

It runs as an optional second phase after the normal visual duplicate scan, using an audio fingerprinting pipeline (Chromaprint-style chroma extraction + sliding-window Hamming similarity matching). Matched pairs appear in the duplicate list with a Clip Offset column showing where in the source the clip starts.

Enabling it

In Settings → Partial Clip Detection, check Enable Partial Clip Detection and adjust:

Setting Default Description
Min clip / source ratio (%) 10 Minimum clip duration as a percentage of the source duration. Clips shorter than this are ignored.
Min audio similarity (%) 80 Minimum average Hamming similarity for the sliding-window fingerprint match to be accepted.
Require visual confirmation on Reject audio matches whose frames at the matched offset don't also look similar.
Min visual similarity (%) 85 Minimum frame similarity for the visual confirmation step.

Note: Partial clip detection requires audio tracks in both files. Videos without audio are skipped.

Downloads

Daily build — attachments are automatically rebuilt and replaced on every commit.

Upgrading from 3.x: your scan database is migrated automatically on first load. Cached image hashes are recomputed on the next scan (image processing moved from ImageSharp to FFmpeg); video hashes are unaffected. Downgrading back to 3.x after the migration is not recommended. The last 3.x build remains available on the 3.0.x release.

Available packages per platform:

Desktop GUI

Requirements

FFmpeg and FFprobe are required. On first launch VDF attempts to download them automatically. Native FFmpeg binding requires FFmpeg 8.x shared libraries (not the master branch).

Windows

Download the latest FFmpeg GPL shared package from https://ffmpeg.org/download.htmlExtract ffmpeg.exe and ffprobe.exe into the same folder as VDF.GUI.exe, a subfolder named bin, or ensure they are on your PATH.

Linux

sudo apt-get update && sudo apt-get install ffmpeg

Then run:

chmod +x VDF.GUI ./VDF.GUI

Optional: add to your application menu

The Linux archive includes videoduplicatefinder.desktop and icon.png. To register the app with your desktop environment (GNOME, KDE, XFCE, etc.):

Edit the Exec= and Icon= paths to match where you extracted the archive, e.g.:

sed -i "s|/opt/videoduplicatefinder|$(pwd)|g" videoduplicatefinder.desktop

Install for the current user

mkdir -p ~/.local/share/applications cp videoduplicatefinder.desktop ~/.local/share/applications/

The app will then appear in your application launcher with its icon.

macOS

Extract the archive — it contains Video Duplicate Finder.app. Double-click it to launch.

If macOS blocks the app with "cannot be opened because the developer cannot be verified", right-click the .app and choose Open, then confirm. You only need to do this once.

If macOS still refuses to launch the bundle (e.g. "library load disallowed by system policy" on macOS 14+ / Tahoe), clear the quarantine flag and re-sign every binary in the bundle ad-hoc:

xattr -cr "Video Duplicate Finder.app" codesign --force --deep --sign - "Video Duplicate Finder.app"

CLI (Command-line Interface)

The CLI is useful for scripting, scheduled tasks, and headless servers where no display is available.

Requirements

Same as the GUI: FFmpeg and FFprobe must be on your PATH or in the same directory as the vdf-cli binary.

Installation

Download CLI-<platform> from the releases page and extract it.

On Linux/macOS, make the binary executable:

Usage

Scan and compare in one step

vdf-cli scan-and-compare --include /path/to/media

Scan multiple directories, save results as JSON

vdf-cli scan-and-compare
--include /mnt/movies
--include /mnt/series
--exclude /mnt/movies/extras
--format json
--output results.json

Common options

Flag Description Default
--include Directory to scan (repeatable) required
--exclude Directory to exclude (repeatable) —
--threshold Hash difference threshold 5
--percent Minimum similarity % to report 96
--parallelism Parallel hashing threads 1
--include-images Also scan image files off
--use-phash Use perceptual hashing off
--partial-clip-detection Enable partial clip detection (audio fingerprinting) off
--partial-clip-min-ratio Min clip/source duration ratio (0.0–1.0) 0.10
--partial-clip-similarity Min audio fingerprint similarity (0.0–1.0) 0.80
--format json|text csv Output format
--output Write results to file instead of stdout stdout
--settings Load full settings from a JSON file —

Auto-mark and delete duplicates

Dry run — shows what would be deleted, no changes made (default)

vdf-cli scan-and-compare --include /mnt/media --action lowest-quality --dry-run

Move duplicates to trash (safer)

vdf-cli scan-and-compare --include /mnt/media --action lowest-quality --delete

Permanently delete (use with care)

vdf-cli scan-and-compare --include /mnt/media --action lowest-quality --delete-permanent

Available --action strategies:

Strategy Keeps
lowest-quality Highest bitrate/resolution per group
smallest-file Largest file per group
shortest-duration Longest duration per group
worst-resolution Highest resolution per group
100-percent-only Only acts on 100% identical groups

Note: Automatic deletion is not recommended. Always review results with --dry-run first.

Web UI

The Web UI runs as a local web server and is accessed from your browser. It is designed for headless machines, NAS devices, and remote management.

Security note: The Web UI is password-protected but intended for local/Docker use only. Do not expose it to the internet.

Authentication

On first launch, a random password is generated and printed to the console:

============================================
  Web UI password:  aB3xK9mQ7p
============================================

Enter this password in your browser to log in. A "Remember me" cookie keeps you logged in for 30 days.

Docker users: Run docker logs vdf-web to see the password.

Environment variable Description
VDF_WEB_PASSWORD Set your own password instead of the auto-generated one
VDF_WEB_AUTH=false Disable authentication entirely

Requirements

FFmpeg and FFprobe are required. When running outside Docker, VDF.Web will attempt to download them automatically on first launch. You can also install them manually via your system package manager or place them on your PATH.

Installation (self-contained archive)

Download Web-<platform> from the releases page and extract it.

On Linux/macOS:

chmod +x VDF.Web ./VDF.Web

On Windows:

Then open http://localhost:5000 in your browser and enter the password shown in the console.

To change the port:

ASPNETCORE_URLS=http://+:8080 ./VDF.Web

Web settings and login credentials are saved to:

The scan database (ScannedFiles.db) is stored next to the executable if that folder is writable; otherwise it falls back to:

Docker (Web UI)

Docker is the easiest way to run the Web UI on a NAS, home server, or any Linux machine. FFmpeg is included in the image — no separate installation needed.

Requirements

Quick start

docker run -d
--name vdf-web
-p 8080:8080
-v vdf-db:/root/.config/VDF
-v vdf-state:/root/.local/state/VDF
-v /path/to/your/media:/media:ro
ghcr.io/0x90d/vdf-web:latest

Then open http://localhost:8080 in your browser. Check the password with docker logs vdf-web and enter it to log in. Inside the Web UI, add /media (or whatever path you mounted) as a scan directory.

To set your own password:

docker run -d
--name vdf-web
-p 8080:8080
-e VDF_WEB_PASSWORD=mysecretpassword
-v vdf-db:/root/.config/VDF
-v vdf-state:/root/.local/state/VDF
-v /path/to/your/media:/media:ro
ghcr.io/0x90d/vdf-web:latest

  1. Download docker-compose.yml from this repository.
  2. Edit the file and add your media volume mounts. Optionally set your own password:

environment:

  1. Start the service:
  2. Open http://localhost:8080 in your browser and enter the password (check docker logs if you didn't set one).
  3. To update to the latest image:

docker compose pull && docker compose up -d

Volume reference

Volume Purpose
/root/.config/VDF Settings (web-settings.json) and login credentials — mount a named volume here so configuration persists across container updates
/root/.local/state/VDF Scan database (ScannedFiles.db) — mount a named volume here so hashed-file data persists across container updates
Your media paths Mount each media directory you want to scan. Read-only (:ro) is recommended.

Notes

Screenshots (outdated)

License

Video Duplicate Finder is licensed under AGPLv3

Credits / Third Party

Building

Contributing