rsync - How to write files in specific order?

07
2014-07
  • Bernie

    Okay, here's a weird problem -- My wife just bought a 2014 Nissan Altima. So, I took her iTunes library and converted the .m4a files to .mp3, since the car audio system only supports .mp3 and .wma. So far so good. Then I copied the files to a DOS FAT-32 formatted USB thumb drive, and connected the drive to the car's USB port, only to find all of the tracks were out of sequence. All tracks begin with a two digit numeric prefix, i.e., 01, 02, 03, etc. So you would think they would be in order. So I called Nissan Connect support and the rep told me that there is a known problem with reading files in the correct order. He said the files are read in the same order they are written. So, I manually copied a few albums with the tracks in a predetermined order, and sure enough he was correct.

    So I copied about 6 albums for testing, then changed to the top level directory and did a "find . >music.txt". Then I passed this file to rsync like this:

    rsync -av --files-from=music.txt . ../Marys\ Music\ Sequenced/
    

    The files looked like they were copied in order, but when I listed the files in order of modified time, they were in the same sequence as the original files:

    ../Marys Music Sequenced/Air Supply/Air Supply Greatest Hits> ls -1rt
    01 Lost In Love.mp3
    04 Every Woman In The World.mp3
    03 Chances.mp3
    02 All Out Of Love.mp3
    06 Here I Am (Just When I Thought I Was Over You).mp3
    05 The One That You Love.mp3
    08 I Want To Give It All.mp3
    07 Sweet Dreams.mp3
    11 Young Love.mp3
    

    So the question is, how can I copy files listed in a file named music.txt, and copy them to a destination, and ensure the modification times are in the same sequence as the files are listed?

  • Answers
  • Chelsea

    Could be done with xargs:

    cat music.txt | xargs -I% rsync -av "./%" "../Marys\ Music\ Sequenced/%"
    

    An alternative would be to do something like this (assuming everything is sequential):

    ls "../Marys\ Music\ Sequenced/" | xargs -I% touch "%"
    
  • snapshoe

    Couldn't you do something like this?

    copy files
    for each file in music.txt
        touch destination file
        sleep 1
    
  • mpy

    You asked two question, one clearly stated and one only implicitly:

    1. How can I copy files listed in a file named music.txt, and copy them to a destination, and ensure the modification times are in the same sequence as the files are listed?

    2. How can I achieve that my MP3 player reads the files in alphabetical order from an USB stick (formatted with FAT filesystem)?

    But IMHO the second one is your real one. I think, the Nissan guy sent you on the wrong track. (See XY Problem)

    But enough prattled... now I try to answer both questions:

    1. Your rsync approach is nearly the solution to the first question, only you used the -a parameter. Rsync's -a is equivalent to -rlptgoD, which especially includes -t:

      man rsync (...)

      This tells rsync to transfer modification times along with the files and update them on the remote system.

      Definitely not what you want. As you copy to a FAT filesystem (whicht neither support owner, groups, permission, links, etc.) I see no good reason for most of the other options provided by -a at all, so try

      rsync -rv --files-from=music.txt . ../Marys\ Music\ Sequenced/
      
    2. To get your MP3 files read in alphabetical order I suggest another approach. I doubt that the files are read in the same order they are written as stated by the Nissan technician, but instead in the order they are listed in the file allocation table. And exactly for rearranging that order there is a neat tool, called fatsort.

      First, make sure your USB stick is not mounted and find the corresponding device (e.g. using cat /proc/partitions), lets say it is /dev/sdi1. Then run fatsort:

      fatsort -cn /dev/sdi1
      

      This sorts the file entries case insensitive and in so called natural order, which includes alphabetically order.1)

      It's very fast and you do not need to copy all files over and over again, if you add only one other file.


    1) I cannot test the command right now, but if I remember correctly, -n is what you want. If not, please consult the help (fatsort -h) yourself.


  • Related Question

    rsync - Why every OS still can't resume file transferring?
  • Questioner

    I mean, Windows 7, Ubuntu 9.10 and Snow Leopard... All newest and top of line desktop operating system still use regular copy to transfer files, as it seems to me from experience.

    Instead of using some technique like rsync, sftp, or whatever is used to backup time machine, when you want to copy a file over network, or large amount of data through USB, or even to a really big pen drive, you have the eminent risk of having to start all over again, if you want convenience of course.

    So, why they insist on going like that?

    edit: since this still got no answer even today, I'm bringing it to most relevant discourse site I could find: http://discuss.howtogeek.com/t/why-every-os-still-cant-resume-file-transferring/16832


  • Related Answers
  • Seasoned Advice (cooking)

    Probably because there are downsides to those like lots of I/O operations and a performance hit.

    A straight copy is probably better off performance, I/O, and system wise.