--- title: "How DuplicateDuster Works" description: "A scan in order: recursive enumeration, multi-threaded SHA-256 hashing, grouping by exact match, a full review step, then removal on your say-so." url: "https://royalsoftworks.com/products/duplicate-duster/how-it-works/" source: "https://royalsoftworks.com" format: "markdown" note: "Markdown rendering of the HTML page at `url`. Same content, same canonical URL." --- How it works # From pointing it at a folder to space reclaimed. A pipeline built to be fast on real hardware and to make the dangerous step — removing a file — the one that needs the most confirmation. 1. 01 ## Nominate the two sides Source folders hold what you're keeping. Target folders are the ones being cleaned. Either side can be one folder, several, or a single file, and the scan is recursive from each root you pick. 2. 02 ## Narrow it, if you want Optional filters: a minimum or maximum file size, an include or exclude list of extensions, skip hidden files, skip system folders. Zero-byte files are always excluded — they match each other by definition and removing them frees nothing. 3. 03 ## Both sides are walked and indexed by size Every file in scope is listed and grouped by exact byte size. Nothing is read or modified in this pass. Permission errors are recorded as warnings and the walk carries on — one unreadable folder must never end a scan. 4. 04 ## Only plausible candidates get hashed A file whose exact size appears nowhere on the other side cannot be a duplicate of anything there, so it is never opened. On a typical drive that eliminates the overwhelming majority of the bytes before hashing starts. 5. 05 ## Hash in parallel, SHA-256 The survivors are hashed across a worker pool — about three quarters of your cores, capped at sixteen — handed out in batches so no thread sits idle behind a large file. For a trivial amount of work the pool isn't started at all, because spinning up threads would cost more than it saves. 6. 06 ## Group files, then whole folders Identical hashes are grouped as confirmed byte-for-byte duplicates. The same data is then walked upward: any target folder whose entire recursive contents exist in the source is reported as a folder match — the topmost one only, so a match doesn't bury you in its own subfolders. 7. 07 ## Review everything Each group shows the source copy with a keep badge and no checkbox, and the target copies with full paths, sizes and dates. Search the results, expand or collapse everything, and watch the reclaimable total update as you select. 8. 08 ## Reclaim, four ways Recycle bin by default, or permanent delete, a three-pass secure erase, or replacing the copy with a hard link to the original. A confirmation names the file count, the total size and, in one sentence, exactly what is about to happen to those files. 9. 09 ## Clean up after itself Failures are collected and reported per file rather than aborting the batch, protected system paths are refused whatever was selected, and directories the cleanup left genuinely empty are removed, deepest first. 04 Reclaiming the space ## Four ways to get the gigabytes back. Deleting isn't always what you want. Sometimes the copy has to be unrecoverable; sometimes it needs to stay exactly where it is, under its own name, without costing you the disk space twice. ### Recycle bin default The copy goes where the OS puts deleted files. Reversible for as long as you haven't emptied it — which is why it's what the app starts on. ### Permanent delete opt-in Straight unlink, no bin. For the case where the recycle bin is the problem — a drive that's already full and can't hold the deleted copies too. ### Secure erase opt-in Three overwrite passes (0xFF, 0x00, random), the filename scrambled through several renames, timestamps zeroed, then unlinked. On Linux it hands off to shred where available. ### Hard link opt-in The copy is replaced by a hard link to the original: both filenames survive, both paths still open the file, and the duplicate's blocks go back to the filesystem. Same volume only — checked before anything moves. **The honest caveat on secure erase:**overwriting assumes the bytes you write land on the bytes you meant to destroy. On SSDs, wear levelling and TRIM mean that isn't guaranteed. This raises the cost of recovery substantially; it does not promise erasure on flash storage, and full-disk encryption is the real answer there. We say so in the app too. 05 Built for real drives ## The fastest hash is the one you never compute. A file whose exact byte size appears nowhere on the other side cannot be a duplicate of anything there — so it is never read at all. On a typical drive that removes the overwhelming majority of the bytes before hashing even starts. Size first then hash Both sides are indexed by exact size. Only files whose size occurs on the other side are ever opened, which is what makes a whole-drive scan finish in minutes rather than hours. Every core you can spare Hashing runs across a worker pool sized at three quarters of your cores, capped at sixteen. Work is handed out in batches as threads free up, so one thread drawing a 4 GB video doesn't leave the rest idle. Never stuck never silent An unreadable file is recorded as a warning and the scan carries on. If every hashing thread dies, the remaining files are hashed on the dispatcher instead — a scan always reaches a finish. ### Cancel actually cancels The stop signal is shared memory, not a message — so it reaches threads that are mid-read, in the middle of a batch, with no event loop turn to spare. ### Filters before the walk Minimum and maximum size, an include or exclude list of extensions, skip hidden, skip system folders. A scan you narrowed is a scan that finishes sooner. ### Spinning disks respected Parallel reads lose on mechanical drives, where seeking between eight files costs more than the extra cores win. The thread count can be pinned to one. ### Nothing blocks the window The whole scan lives on worker threads. The interface stays responsive while a million files are being indexed behind it. ## See exactly what 'secure erase' guarantees. And what it doesn't — stated plainly, including the SSD caveat most tools leave out. [Security](https://royalsoftworks.com/products/duplicate-duster/security/) [Read the whitepaper](https://royalsoftworks.com/products/duplicate-duster/whitepaper/)