Detection & Deletion: DuplicateDuster

DuplicateDuster is a desktop application (Electron shell, Angular renderer, Node.js worker threads) that identifies exact duplicate files and exact duplicate folder trees by content hash, and reclaims the space they occupy through one of four user-selected mechanisms. It performs no network communication during scanning, hashing or reclaiming.

Two design decisions shape everything else. First, matching is exact-content only: it does not attempt similarity, perceptual or fuzzy matching of any kind. Second, the comparison is two-sided — the user nominates source folders (kept) and target folders (cleaned), and only files on the target side are ever selectable or removable.

A scan takes two independent input sets. The source set represents the copies being retained; the target set represents the locations being cleaned. Both are walked recursively, and matches are only ever paired across the two sets — a file is reported when its content also exists on the other side.

The consequence for safety is structural rather than procedural: the results screen renders source-side files with a keep badge and no selection control, so there is no interaction sequence that removes a file from the set the user nominated as canonical. This is the single property of the interface that must not regress.

File matching is based on SHA-256 content hashing. Files producing an identical digest are grouped as confirmed duplicates; because SHA-256 collisions are computationally infeasible to construct, a hash match is treated as proof of byte-for-byte identity. There is no probabilistic scoring and no tunable similarity threshold.

This has a direct consequence worth stating plainly: a file that has been re-saved, re-encoded, cropped or edited in any way — even by a single byte — will not match its original. DuplicateDuster implements no filename-similarity matching, no perceptual hashing and no near-duplicate detection.

Zero-byte files are excluded from consideration entirely. They match every other zero-byte file by definition, which is noise rather than a finding, and removing them reclaims nothing.

Hashing every file in scope would be the obvious implementation and the wrong one. Both sides are first indexed by exact byte size, and a file whose size does not also occur on the other side cannot be a duplicate of anything there — the renderer only ever pairs a source against a target — so it is skipped without ever being opened.

The result is identical to hashing everything, while avoiding reading the overwhelming majority of bytes on a typical drive. Progress is still reported for every file either way, so the progress bar reaches its stated total rather than jumping.

The scan runs on a worker thread so the Electron main process is never blocked. That thread owns the directory walk, the grouping and progress reporting, and dispatches hashing to a pool of separate hashing threads that read synchronously — which is what makes the parallelism worth having, since SHA-256 over a large file is the only genuinely slow part of a scan.

The pool is sized at three quarters of the machine's core count, capped at sixteen, on the reasoning that the remaining quarter still has to cover the dispatcher, the Electron main process and the renderer. Beyond sixteen, message round-trips cost more than an extra core returns. An environment variable pins the count for the one case where parallel reads lose: a mechanical drive, where seeking between eight open files is slower than reading one.

Work is distributed by demand rather than by a fixed split: batches (closed at 8 MB or 32 files, whichever comes first, so one very large file travels alone) are handed out as threads free up, with two batches queued per thread so no thread waits for its next assignment. A thread that dies returns its outstanding work to the queue; if every thread dies, the remaining files are hashed on the dispatcher. A scan always reaches a terminal state — completed, failed or cancelled — and the interface is never left with a spinner that cannot resolve.

For a trivial amount of work the pool is not started at all: below eight files and 64 MB, thread startup costs more than it saves.

Cancellation is signalled through a SharedArrayBuffer read directly by every thread, not through a message. Terminating the dispatcher leaves it no turn of the event loop in which to forward a message to its own hashing threads, and those threads outlive it — so the flag is the only thing that reliably reaches a thread in the middle of a synchronous read.

Beyond individual files, the scan builds for every directory under the scan roots the set of files it contains recursively, together with whether all of them were successfully hashed. One upward pass produces this for the whole tree: each file appends itself to every ancestor on the way to its root.

A directory's signature is a stable hash over the sorted digests of its recursive contents; a folder holding anything that could not be hashed produces no signature and is never reported as a match. Target folders whose signature equals a source folder's are reported as folder-level findings.

Only the topmost match in a nested run is reported. If a target `photos/` matches a source `backup/photos/`, then `photos/2024/` necessarily matches too, and listing every descendant would bury the useful finding under its own children.

Two directory names are skipped unconditionally: the recycle bin and System Volume Information. Walking the recycle bin in particular would offer to delete files the user has already deleted.

With the default `skipSystem` filter on, platform system roots are also excluded — SystemRoot, Program Files, ProgramData and their variants on Windows; /System, /Library, /usr, /bin, /sbin, /Applications on macOS; /bin, /sbin, /usr, /lib, /etc, /boot, /proc, /sys, /dev, /run on Linux. Directory junctions are guarded against by tracking already-visited real paths, so a link loop cannot produce an infinite walk.

Symbolic links are never followed and never reported. Resolving a link in a tool that deletes files risks deleting the wrong target.

User-controlled filters apply on top: minimum and maximum file size, an include list and an exclude list of extensions, and skip-hidden. A malformed or partial filter object degrades to the defaults rather than dropping every file.

A single unreadable file or directory must never abort a scan. Every filesystem operation is guarded; failures are recorded as warnings carrying the path and the underlying error code, and the walk continues. Warnings are reported alongside the results rather than surfacing as a failure dialog.

Four modes are available, selected in settings and applied to the whole batch. Recycle bin is the default and the only reversible one — the application previously deleted permanently with no way back, which for a tool whose job is removing files is the wrong default however clear the warning.

• Recycle bin — the file is moved to the platform's trash and is recoverable through the normal OS mechanism
• Permanent delete — a direct unlink, for the case where the recycle bin itself is the problem on a full volume
• Secure erase — three overwrite passes (0xFF, 0x00, then cryptographically random bytes), filename scrubbed through several renames, timestamps zeroed, then unlinked
• Hard link — the copy is replaced by a hard link to the source file, so both paths continue to resolve while the duplicate's blocks return to the filesystem

Overwriting uses positioned writes against an open file handle rather than a write stream: a stream buffers every chunk in memory when it signals backpressure, which on a multi-gigabyte file means the pass allocates the entire file size in RAM. Each pass is followed by an explicit sync, because passes left sitting in the page cache never reach the platter. On Linux, where `shred` is available, the implementation delegates to it (`shred -u -n 3 -z`), invoked with an argument array rather than a shell string so that filenames containing quotes or shell metacharacters cannot be executed.

This is explicitly not represented as an implementation of the DoD 5220.22-M standard, which specifies a different, certified procedure. Organisations with a compliance requirement for that certification should treat this as a strong general-purpose overwrite and verify independently.

On solid-state drives, wear-levelling firmware can relocate data to different physical cells and TRIM can mark blocks for erasure independently of an application-level overwrite. An overwrite pass on flash storage therefore provides a weaker guarantee than on a magnetic disk. For SSD-resident sensitive data, full-disk or filesystem-level encryption is a stronger control than file-level secure delete.

Hard links cannot cross volumes, and Windows reports the failure as a generic error well after the file has already been moved — so both paths are compared by device identifier up front, along with a size check and a guard against the two paths already being the same inode from an earlier run.

The replacement is ordered so that a failure cannot lose data: the duplicate is renamed aside, the link is created, and only then is the parked copy removed. If the link cannot be created for any reason, the rename is undone and the user still has their file; if even that fails, the error names the exact path where the file is currently sitting rather than failing silently.

Protected system locations are refused at delete time as well as during the walk, independently of what the scan produced or the user selected. Paths are resolved before comparison so that a traversal cannot slip past, and a separator is appended before the prefix test so that a directory such as ProgramData2 is not mistaken for something inside ProgramData. A bare drive or filesystem root is never a valid target.

One locked or already-removed file must not abort the rest of a batch, so every failure is collected with its reason and reported back rather than thrown. After a successful batch, directories the cleanup may have emptied are removed deepest-first using a non-recursive rmdir — which fails on a non-empty directory, and that failure is exactly the safety property wanted: anything still holding a file the user kept is left alone.

Scanning, grouping and the reclaimable-space report are unlimited on the free tier by design: the free tier exists to prove the number is real. What it gates is acting on that number at scale — a free build reclaims up to 50 files in a single run.

Activation posts the licence key with a machine fingerprint to the licensing service, which validates it and consumes a seat. Revalidation at startup is a check-only call, so a device deactivated in the licensing dashboard drops to the free tier, while an unreachable server changes nothing: a licence is revoked only on a definitive negative verdict, never on a network failure. Air-gapped machines activate from a cryptographically signed licence file bound to the device fingerprint, verified locally.

No file name, hash, path or content is transmitted off the host machine at any point in the scan, review or reclaim workflow. The application performs no network calls during those operations and is fully usable with networking disabled.

The only outbound requests it makes are licence validation and an update check against a static release feed. There is no telemetry, no analytics and no crash reporting.

Windows (NSIS installer), macOS (dmg and zip, Intel and Apple silicon) and Linux (AppImage) builds are produced from one codebase, and the application updates itself from a static feed. The interface ships in English, Serbian in Cyrillic and Latin script, and Russian.

The 1.0 beta is not code-signed on any platform, so SmartScreen and Gatekeeper will warn on first launch, and macOS in-place updates remain manual until signing is in place.