linux - Watch a filesystem and rsync changes in a cross-platform way with minimal dependencies
2014-04
John Feminella
I'd like to start a long-running Ruby or Python script, and have the script watch a local directory for changes to files and new files. Periodically, the script will trigger a one-way rsync to a remote network location.
This must function on both Linux (Ubuntu 13.10+) and Windows (Windows 7+) environments. Importantly, I'd like to minimize the number of dependencies required, and I'd like to install the same thing on every system by simply dropping some files in a directory -- no installer.
Is it possible to get the system I'm describing want without an installer? If so, how?
paj
As a possible solution to the first part of your question, you should take a look at inotify-tools, this contains the tool inotifywait which watches a directory (or set of directories recursively) and outputs notifications of changes (these can be restricted to certain event types, e.g. file creation, deletion, modification etc.). More information can be found on the man page.
I'm not familiar with programming (python or otherwise) on Windows machines, but there has been a port of inotifywait for Windows in the form of inotify-win which, after having produced the executable, you can presumably run from a script via subprocess with minimal platform-specific changes from the Linux version.
For the second part of your question, you could go for SFTP using paramiko, however, it seems that the dependency on PyCrypto may make this option a bit challenging from a Windows point of view.
I realise this isn't a complete answer to all the parts of your question, but hopefully there is something here that you find useful.
Console
I would like my external drives to be readable and writable from Linux, Mac OS X and Windows.
FAT32 works, but the 4 GB file size limit is a showstopper these days. Are there any alternatives?
Peter Mortensen
As Breakthrough said, use NTFS. In both Mac OS and Linux, you can use NTFS-3G to enable read/write to an NTFS partition; on Mac OS you also need MacFuse, but that's it.
These projects are free, open-source and mature. I've used this setup on a Mac and I've had no problems accessing data from an NTFS partition.
EmmEff
The simple answer is- no. There is no lowest common denominator across these OSes aside from FAT32.
By lowest common denominator, I mean built-in filesystems. For add-ons, you're on your own.
Marco
UDF is a candidate. It works out-of-the-box on linux >= 2.6.31, Windows >= Vista, MacOS >= 9 and on many BSDs.
Note: UDF comes in different versions, which are not equally supported on all platforms, see Wikipedia - Compatibility.
Related question: Using UDF on a USB flash drive
Breakthrough
Well you have two solutions. Many Linux distributions include tools for reading and writing to NTFS drives...
An alternative would be to use Ext2. There is a windows utility which integrates the filesystem with the Windows operating system. I think this would be your ideal solution:
It installs a pure kernel mode file system driver Ext2fs.sys, which actually extends the Windows operating system to include the Ext2 file system. Since it is executed on the same software layer at the Windows NT operating system core like all of the native file system drivers of Windows (for instance NTFS, FASTFAT, or CDFS for Joliet/ISO CD-ROMs), all applications can access directly to Ext2 volumes. Ext2 volumes get drive letters (for instance O:). Files, and directories of an Ext2 volume appear in file dialogs of all applications. There is no need to copy files from or to Ext2 volumes in order to work with them.
Will
Mount your external drives to a server with NFS and Samba.
lajuette
try exFAT, which becomes available for more and more OSes. Accordings to the linked wikipedia article (see sources) there's an open source kernel module for linux in development. OS X supports it since 10.6.5, Windows supports it since Vista. There are updates for oler Microsoft OSes.
exFAT supports large files.
Peter Mortensen
FAT32 is something that you can be sure of to work almost anywhere.
I struggle from the file size limit, which by today's standards isn't that large anymore. Since exFAT isn't available on Linux yet, I was looking for alternatives, and it's really hard to find something suitable.
UDF was once meant to be a cross platform and cross media filesystem, but it kinda got forgotten. There is an option to format UDF for hard drives, which is quite suitable for removable drives, but from what I experienced, support on Windows is minimal, if at all. I don't know if Windows 7 supports UDF drives other than BluRay discs.
I settled on using NTFS for my external drives, that need to be plugged in into Windows computers, as well as Linux computers. For my removable drives, that are mainly, if not only used on Linux computers, I use XFS.
The same problem applies to encryption as well: I use LUKS on Linux, which has some support on Windows. TrueCrypt can't be integrated into Linux systems too well, compared to LUKS, so I settled on that one.