windows server 2008 - Sync local files to S3 similar to Robocopy

07
2014-07
  • Yuck

    I am looking for a way to synchronize an entire local folder structure to Amazon S3, similar to how one might synchronize two folders using Robocopy.

    Whatever solution I come up with needs to be scheduled to run periodically from the Windows Task Scheduler. So anything that requires a GUI to perform the synchronization is not a viable solution.

    Standalone Windows .EXE command line utility for Amazon S3 & EC2 looked promising, but seems to have been abandoned and would not work when I tried to use it. Possibly a difference in the way that security is handled now compared to that software's most recent release.

    • Answers
  • Sean W.

    According to this post you should be able to access the files via WebDav.

    Accessing Amazon S3 via WebDav

    Therefore, in theory, the Server / Computer which you would like to upload files from could have a Batch File setup on it to map the WebDav folder to a drive letter on that machine and then execute a robocopy command from the local drive and folder to the newly mapped drive and folder. This batch file then may be attached to the Windows Task Scheduler at the desired intervals.


    • Related Question

    windows server 2008 - Delete dupe files from source, move only missing across to target in Robocopy
  • Oskar Kjellin

    We copied all our files to a new storage server recently. We didn't want to move at the time because we weren't sure if files would get lost.

    The problem now is that we have files on both places.

    How can we move only the files that do not exist in the target and for those that exist in both places we delete it from the source? It is Windows Server 2008.


    • Related Answers
  • Roald van Doorn

    Use a compare tool like Beyond Compare. It can compare folders and shows you what files exist and don't exit in the folders and subfolders. Also, it shows if there are any differences between files that exist on both locations. beyond compare can also copy and/or move files from one folder to another.

  • Community 
    robocopy OLDDIR NEWDIR /S /XO

    This leaves newer files in NEWDIR alone. Files in NEWDIR that are the same or older will get overwritten.

    robocopy OLDDIR NEWDIR /MIR /XO

    This does something similar - watch out for extra file deletion on this one.

    The semantics of the /XO command are backwards.

    I just set up a set of source and destination directories and proved it to myself, as no documentation explains the true meaning of "Exclude Newer". Newer what? Newer source? Newer destination? What?

    PROOF:

    1. Do the copy.

    2. Edit a file in NEWDIR giving it a new time stamp.

    3. Do the copy again using /XO. The new time stamp on the file in NEWDIR remains.

    4. Do the copy again using /XN. The new time stamp on the file in NEWDIR reverts.

  • Charles Gargent

    Use the robocopy tool with the /xn switch. I believe Server 2008 comes with robocopy as standard now.

    /XC :: eXclude Changed files.

    /XN :: eXclude Newer files.

    /XO :: eXclude Older files.

    /XX :: eXclude eXtra files and directories.

    /XL :: eXclude Lonely files and directories.

    /MOV :: MOVe files (delete from source after copying).

    /MOVE :: MOVE files AND dirs (delete from source after copying).

    so the command will be something like

    robocopy OLDDIR NEWDIR /MOVE /XN

    this will move over all the old files but wont overwrite new ones

    You can always use the /L switch to do a dry run this will show you what will happen without actually do anything. Some things may be different on 2008, I am using the XP version.