mac - Apple has "sparse bundles" but not sparse files?

mac filesystems sparsebundle

05
2013-08

user52250

Backstory:

I use a Mac (10.6.4), and to this day when I download a torrent using Vuze, no matter what configuration I try Vuze allocates space for the entire file before it starts downloading said file. For a long time, a quick google led me to believe it was a Vuze bug, but after years have passed and nothing has changed a more detailed search tells me that it's because HFS+ doesn't support sparse files.

Question:

The sparse file wiki says that Apple's HFS+ doesn't support sparse files, and Apples' Time Machine and FileVault uses "sparse images" and "sparse bundles".

I understand what a sparse bundle is (image split into 8Meg chunks) but aside from that, is there a difference between these "sparse images" and just regular sparse files? How can HFS+ not support sparse files and yet still support sparse images and sparse bundles?

Answers

Gordon Davisson

It's because the sparseness is handled at a different level of abstraction for a sparse image vs. a sparse file. If you're storing a file "directly" on disk, the filesystem (e.g. HFS+) handles mapping the sequence of bytes that make up the file into a sequence of bytes on disk. A sparse file is one that has gaps in its sequence of bytes, and a filesystem that supports this will skip those missing bytes when it maps the file to disk.

When you're using a disk image, the mapping is quite a bit more complicated: you have the bytes in the file being mapped (by the inner filesystem) to bytes in the (virtual) disk. These are then mapped (by the image mounter) to bytes in the disk image (file), which are then mapped (by the outer filesystem) to bytes on the actual disk. The image mounter handles sparse mappings (and AIUI only between/beyond files, not within them), but that doesn't imply that the other layers (the filesystems) can. As far as the outer filesystem is concerned, the image file is just a (non-sparse) sequence of bytes.

BTW, the image mounter can also add other features that the filesystem alone doesn't deal with. For instance, Mac disk images have supported compression roughly forever, but native support for compression wasn't added to HFS+ until OS X v10.6.

Related Answers

nagul

Try Unison.

Unison is a file-synchronization tool for Unix and Windows. It allows
two replicas of a collection of files and directories to be stored on
different hosts (or different disks on the same host), modified
separately, and then brought up to date by propagating the changes in
each replica to the other.

It lets you specify directories to sync - these will be the folders you want to cache. You can also finely control the direction of sync. It will not auto-run when you connect to the drive. You'll need to trigger that yourself, depending on the Linux distro you run and how you/it connects to the remote drive. e.g if you use the mount command, wrap it in a script that calls unison after the mount. There's a brief write-up of how to use it on the Ubuntu community documentation site.

Unison is a command-line tool that also has a GUI of the same name. If you're on Ubuntu, both packages are included in the universe but not installed by default.

$sudo apt-get install unison unison-gtk

It is also available for the Mac via ports (Fink, Macports etc).

Features:

 * Unison runs on both Windows (95, 98, NT, 2k, and XP) and Unix
   (OSX, Solaris, Linux, etc.) systems. Moreover, Unison works across
   platforms, allowing you to synchronize a Windows laptop with a
   Unix server, for example.
 * Unlike a distributed filesystem, Unison is a user-level program:
   there is no need to modify the kernel or to have superuser
   privileges on either host.
 * Unlike simple mirroring or backup utilities, Unison can deal with
   updates to both replicas of a distributed directory structure.
   Updates that do not conflict are propagated automatically.
   Conflicting updates are detected and displayed.
 * Unison works between any pair of machines connected to the
   internet, communicating over either a direct socket link or
   tunneling over an encrypted ssh connection. It is careful with
   network bandwidth, and runs well over slow links such as PPP
   connections. Transfers of small updates to large files are
   optimized using a compression protocol similar to rsync.
 * Unison has a clear and precise specification, described below.
 * Unison is resilient to failure. It is careful to leave the
   replicas and its own private structures in a sensible state at all
   times, even in case of abnormal termination or communication
   failures.
 * Unison is free; full source code is available under the GNU Public
   License.

Richard Hoskins

FileSync will do what you want on the Macintosh.

Simon

I have had good success with ifolder, which is cross platform solution for bi-directional file sharing. It appears to be stable. Its a client-server solution.

Also, you could look at http://offlinefs.sourceforge.net/wiki/, I haven't tried this, but it looks interesting.

LKM

This is not entirely what you asked for, but depending on what specifically you want to achieve you might want to consider using a version control system such as subversion). Your files won't be synchronized automatically, but you will get a version history and some other neat features in return.

Home

mac - Apple has "sparse bundles" but not sparse files?