Detection & Deletion: DuplicateDuster

DuplicateDuster is a desktop application (Electron, with a Node.js worker-thread backend) that identifies exact duplicate files by content hash and optionally removes them with a multi-pass secure overwrite. It performs no network communication during scanning, hashing or deletion.

This document is intentionally precise about the boundary of the detection method: it matches identical content only, and does not attempt similarity, perceptual, or fuzzy matching of any kind.

Duplicate detection is based entirely on SHA-256 content hashing. Each file in the scan scope is read and hashed; files producing an identical digest are grouped as confirmed duplicates. Because SHA-256 collisions are computationally infeasible to construct, a hash match is treated as proof of byte-for-byte identity — there is no probabilistic scoring and no tunable similarity threshold.

This has a direct consequence: a file that has been re-saved, re-encoded, cropped, or edited in any way — even by a single byte — will not match its original. DuplicateDuster does not implement filename similarity matching, perceptual hashing, or any form of fuzzy or 'near-duplicate' detection.

File enumeration is recursive from the selected root path(s). Hashing is distributed across Node.js worker threads, parallelising I/O and CPU-bound hash computation across available cores rather than processing files sequentially. A scan is read-only with respect to the file system: no file is modified, moved, or removed during enumeration or hashing.

The scanner distinguishes symbolic links from regular files during traversal (using lstat rather than stat where appropriate), so a symlink is not mistakenly treated as if it were a duplicate of its target's content in a way that could lead to an unintended deletion.

Once hashing completes, files are presented grouped by matching hash. The interface always preserves a default 'keep' selection — at least one file per group is retained, and no deletion occurs without explicit user confirmation per the configuration in use.

When secure deletion is selected, DuplicateDuster performs a three-pass overwrite of the file's data before unlinking it: a pass of 0xFF bytes, a pass of 0x00 bytes, and a pass of cryptographically random bytes. The filename is also randomised and the file's timestamps are reset prior to removal, reducing the metadata trail left behind.

This is explicitly not represented as an implementation of the DoD 5220.22-M standard, which specifies a different, certified procedure. Organisations with a compliance requirement for that specific certification should treat this feature as a strong general-purpose overwrite, not a certified equivalent, and verify independently if certification is required.

Overwrite-based secure deletion was designed for the physical behaviour of magnetic storage, where overwriting a sector reliably destroys the previous data. On solid-state drives, wear-levelling firmware can relocate data to different physical cells, and TRIM can mark blocks for erasure independently of an application-level overwrite. As a result, an overwrite pass on an SSD provides a weaker guarantee than on a traditional hard disk. For SSD-resident sensitive data, full-disk or file-system-level encryption is a stronger control than file-level secure delete.

No file name, hash, path, or content is transmitted off the host machine at any point in the scan, review, or deletion workflow. The application performs no network calls during these operations, and is fully usable with networking disabled.