osx - Find duplicate files on Mac OS X

18
2014-04
  • J. Pablo Fernández

    Is there an application that would search duplicate files in my Mac with an easy way to delete the duplicates?

  • Answers
  • 8088

    I'll just copy part of my other answer on a similar subject and also link to a possible duplicate as another reference.

    There are many commercial options, some may be better than the listed below, I haven't tried them all. Anyway, I'm listing my choice of apps considering which ones I was able to try. Coincidentally, this was not where I first heard about DupeCheck droplet, but thanks so much Systech for building it up! :)

    • TidyUp is a very well known app in this subject. You can specify where to scan for what kind of duplicates. It offers basic and advanced modes, several different strategies and criterias.

    • MrClean is a free tool that just scans for folders for duplicates and trash them. Very simplistic but efficient if you're sure on what you're doing.

    • Chipmunk scans duplicates and let you choose which ones you want to trash. It offers a node-view of folders and you can select to "delete all files in a folder that have duplicates elsewhere, or vice versa" as well as hand-picking. It may take very long to scan all files, but it does a very decent job after that.

    • DupeCheck "drop a file on it and it will use your Spotlight index to see if you have a potential duplicate somewhere." That's about this nice open source app. Not a great tool for space cleaning at once, but over time it helps you keep your space clean.

    • DuplicateFileSearcher from the website: "is a free powerful software utility that will help you to find and delete duplicate files on your computer. It can also be used to calculate MD5 and SHA hashes. The software runs in Windows, Linux, Solaris and MacOS.". Enough said.

  • admintech

    I've converted a bash script into a droplet. Drop a file on it, and it will use your Spotlight index to see if you have a potential duplicate somewhere on your system.
    Download the free DupeCheck droplet, unzip it, and find a good spot to put it. Drag and drop any file onto it, and it will use Spotlight to identify possible duplicates.

  • Tim

    I've written a simple python script FileDupeFinder.py that will find file duplicate files on OSX (and probably Linux/*nix) systems. Right now, you pass it a directory root path and a minimum file size and it will walk the directory tree and output a list of files that it believes to be duplicates. It will ignore files that are hard links of each other. You can tell it to exclude directories (right now it just excludes Backups.backupdb, as this script isn't ready to handle Time Machine wackiness) and tell it not to span filesystems (though these aren't currently paramaterized, they are easy enough to set in the script itself.

    This script is only minimally tested, but it has worked reasonably enough for me.

  • Debilski

    There is Araxis Find Duplicate Files which is pretty convenient.

  • SuperTempel

    Decloner is another one.

    You can choose the minimum file size you want to have checked, and it can eliminate all but one. Doesn't seem to be able to create hard links or symlinks, though, which I'd want.

  • shootie

    i recommended dupeGuru by Hardcoded Software, it's Open Source Fairware and free to use

  • Fr.

    The most convenient GUI that I have found for this task is Gemini, but a script might be enough for your needs. I personally scan for duplicates just to assess how much of my data is redundant (currently 3/50 GB, or 6% of the data), but I rarely delete anything in case it breaks the internal magic of my file chaos.

  • Martin

    I've been using TidyUp a lot on MacOS X to find duplicate files and sort them out.

    The GUI is not necessarily 100% intuitive, but the application is very powerful, offers a lot of options and choices for defining what a duplicate is (same name?, same size, same content, ...) and it also has a powerful interface for displaying duplicate groups and deciding what to do with the duplicates (or some of them).

    I never regretted buying it and it was very helpful for me.


  • Related Question

    osx - Permanently deleting files on Mac OS
  • Jonik

    A while back, as relatively new Mac OS X user, I was surprised to learn that you cannot easily delete files. Directly, that is, without moving them to the trash first. On Windows and Linux this can obviously be done with ease, but not so on the Mac.

    I noticed this when trying clear up files from a USB memory stick — removing the files ("move to trash") does not free up space; that happens only after emptying the whole system-wide Trash. Not particularly convenient! (It seems stupid to have to empty the whole trashcan just to make some space on the USB stick. There might be gigabytes of stuff in there, and this sort of defeats its purpose - what if you'd actually need to restore something from the trash some day.)

    So, what's your way of getting around this? Have you bought a 3rd party application like RAW Trash for $16.95 just to delete files, or do you diligently empty the trashcan whenever needed? Or did I miss something? Also, can you convince me that this is actually the way it should be — that users shouldn't be able to fiddle with the filesystem easily? :)


  • Related Answers
  • Daan

    I'm not so happy with the way Mac OS X handles this either. If I really want to delete something, especially from USB memory sticks, I usually fire up Terminal and rm the files manually.

  • Community

    Since so many people seem to want it, I just created a Service to do this, for Snow Leopard, called Delete Immediately. It shows up in the Services menu and the context menu in Finder.

    You can download it from GitHub. Uncompress the .zip archive and move the resulting "Delete Immediately.service" to the Services folder in your library folder, ~/Library/Services/. You may need to log out and back in (and/or enable it in the Services section of the Keyboard preferences) for the system to recognize the service.

    I'd also suggest checking out the README file on the main page on GitHub. And if you want to contribute to the code, localize it, or make an icon, feel free!

    Context menu

    Services menu

  • dbr

    To delete a specific file, without going through the trash..

    • open a Terminal
    • type rm (with a trailing space), or rm -r if you plan to remove a directory
    • drag and drop the file onto the Terminal window, which enters the full path to the dropped file
    • hit enter

    jtbandes's "Delete Immediately" service solution is much more elegant, but the "Terminal way" doesn't require any additional software (so is good when working on machines that you don't regularly use)

  • Arjan

    The Terminal-less way of solving this conundrum is to cast the following spell to appease the Apple gods, like so :

    Select the file to be sacrificed followed by reciting:

    ⌘ Delete

    ⌘ ⇧ ⌥ ⌫

    This will send the file to Trash, and then delete the whole Trash, thus should send the victim (and all other trashed items) to oblivion.

  • Andrew J. Brehm

    I use rm -rf in the command line. It deletes a lot faster and it deletes everything. A common problem I have with Windows is that deleting is a process rather than a point in time and I often find myself watching Windows delete folders and files for several minutes. Rm -rf is quick.

    You can also use rm -rf to delete a specific Trash. The trash can is a hidden folder named .Trash in the root of the relevant volume or directory.

    And yes, I think this is how it should be. Users shouldn't be able to delete files too quickly using Finder. Files should be recoverable from the Trash.

  • avstrallen

    Part of the beauty of Mac OS X is that while it's dead easy for inexperienced users to find their way around and get stuff done, power users can "fiddle with the system" by using the power of the underlying Unix via a command shell (i.e. Terminal.app).

    @Leauki is right about using the unix command rm, but BE VERY CAREFUL, particularly if using the -rf flags! There's no safety-net and you can do serious damage, up to and including deleting your own root filesystem!

    My recommendation is not to use absolute paths with rm, but to cd into the diretory you want to work with first, and then for example:

    rm -rf ./<subpath to file to delete> where the ./ in the path forces rm to operate only in the directory you're currently in.

  • fatboy

    I came up with the following applescript, which I bound to shift del with Keyboard Maestro, so it now works as in windows. With the item selected in Finder, shift del will run the script, which displays a dialog warning of the permanent deletion of the item with its name. Clicking OK completes the delete.

    I don't know much about shell commands, and some of the previous posters warned of the danger of using rm -rf, which is something I have done in the script. It seems to work as predicted for me so far though.

    tell application "Finder"
        set myPosixPath to selection as text --returns an alias path
        set myPosixPath to POSIX path of myPosixPath --set it to posix style path with backslashes
    
        --identify whether it's a file or folder. Only for the warning dialog.
        if character -1 of myPosixPath is "/" then
            set itemType to "folder"
        else
            set itemType to "file"
        end if
    
        --display a warning
        display dialog myPosixPath & "
    
    This cannot be undone." with title "Really permanently delete this " & itemType & "?"
    do shell script "rm -rf " & quoted form of myPosixPath --execute a shell script to delete the item
    end tell
    
  • Bruce McLeod

    I think that this is done for user consistency, and safety. I think that is better to keep files for longer than required and use space instead of being gung-ho and deleting them straight away.

    This is a bit annoying though when you are working with USB sticks and you move them to a windows machine and go ... where is all my space ... and what is this annoying .trash folder.

  • neoneye

    Use muCommander for deleting files (it's free)

    Instead of using the Finder then perhaps use another file manager such as muCommander. It can delete files and move files to the trashcan as well. It's much safer than pasting "rm -rf" into a Terminal, where you risk pasting the wrong commands causing damages that are irreversible.

  • Simon

    If you really want to use a third party App, a much cheaper alternative to RAW Trash that you mentioned is File Shredder. (US Mac App Store $3.99)

    It offers several ways to securely delete files:

    • A one-pass normal deletion that just overwrites your files.
    • A seven-pass DoD compliant pass makes sure most people can't get into your files.
    • A 35-pass Gutmann deletion ensures that your files are truly gone when you delete them.

    This might be a bit overkill for most people, although if you are looking for a cheaper way than RAW Trash to securely delete files FileShredder gets the job done.