linux - Are there limits to file length piped from find to xargs or using find -exec +

07
2014-07
  • user1663987

    In question [ Find and delete all the directories named "test" in linux ] on this site, the best answer talks about using these two commands:

    find . -name test -type d -print0|xargs -0 rm -r --
find . -name test -type d -exec rm -r {} +

    because they will call rm with a list of directory instead of invoking it many times individually.

    Since I cannot comment there due to low reputation, I ask here in a new question:

    Is there any limit on the number of files that can be passed to rm using these techniques (aside from realistic system resource bounds)?

    From the shell, a command like 'rm *' can exceed the shell's maximum command-line length, but do limits like that apply to this usage of find + or via a pipe to xargs?

    • Answers
  • mtak

    In short, no.

    The long answer: - Find will run the command specified by exec for every match, so if your find turns up 20 files, it will run 20 seperate instances of rm. - xargs will determine the maximum command length for your shell and add arguments within these limits as you can see with the output of xargs --show-limits mtak@frisbee:~$ xargs --show-limits Your environment variables take up 4050 bytes POSIX upper limit on argument length (this system): 2091054 POSIX smallest allowable upper limit on argument length (all systems): 4096 Maximum length of command we could actually use: 2087004 Size of command buffer we are actually using: 131072


    • Related Question

    linux - piping find to rsyncrypto
  • Richard

    I am setting up rsyncrypto to create a backup of my user directory. I read that to exclude directories I need to exclude with find and pipe to rsyncrypto.

    I am trying to exclude all .svn directories from getting encrypted and synced.

    Here is the find command I'm using. I don't see any .svn directories in the output:

    find ~/Documents -type d ( -name .svn ) -prune -o -print

    But when I pipe the output to rsyncrypto I see lots of .svn directories getting encrypted.

    find ~/Documents -type d ( -name .svn ) -prune -o -print | rsyncrypto -vc --trim=3 --filelist - /tmp/Documents/ Documents.keys backup.crt

    Can someone help me rewrite the command to exclude from rsyncrypto any files/folders that have .svn in their path?


    • Related Answers
  • Gilles

    Your original find command prints all directories except for those under a .svn directory, in particular it prints ~/Documents. Experimentally, rsynccrypto recurses into all existing directories in a filelist, even though (like you, I think) I understand the documentation to say that it should only recurse into directories with a / appended.

    So you need to print only files that are not directories. This means you must restrict -print with -type f (to back up only regular files) or ! -type d (to back up all non-directories, including symbolic links).

    find ~/Documents -type d \( -name .svn -o -name .DS_Store \) -prune \
              -o \! -type d -print

    Note that this won't back up the modification time and ownership of directories, or empty directories. This will include empty directories as well:

    find ~/Documents -type d \( -name .svn -o -name .DS_Store \) -prune \
              -o \! -type d -print \
              -o -type d -empty -print