Please explain wasted space on an exFAT formatted external hard drive

29
2014-01
  • JohnB

    This question already has an answer here:

  • Answers
  • DragonLord

    The default allocation unit (cluster) size in exFAT is much larger than the default allocation unit size in NTFS. exFAT defaults to 256KB, as you noted above, but NTFS defaults to 4KB for a volume of this size (see http://support.microsoft.com/kb/314878).

    Because allocation of storage must occur in whole clusters, a file that is not a multiple of the cluster size will result in wasted space on the storage device. This wasted space depends on the size and is generally larger when the cluster size is larger and when there are many files on the device. This condition is called "internal fragmentation", and the wasted space is called "slack space".

    For example, a 257KB file will require two allocation units to store on the exFAT volume you formatted, hence requiring 512KB on disk. Such a file would require only 260KB on a file system with 4KB clusters (the default for NTFS). For more information, see this Wikipedia article.

    Most file systems are based on a block device, which is a level of abstraction for the hardware responsible for storing and retrieving specified blocks of data, though the block size in file systems may be a multiple of the physical block size. In classical file systems, a single block might contain only a part of a single file. This leads to space inefficiency due to internal fragmentation, since file lengths are often not multiples of block size, and thus the last block of files will remain partially empty. This will create slack space, which averages half a block per file.

    Try formatting the drive with a smaller allocation unit size. This may slightly slow I/O operations, but will greatly reduce the amount of wasted disk space.

  • dan_linder

    The size of the allocation unit will impact the usable space on the disk. A larger allocation unit will be better if you have large files. A smaller allocation unit is better for small files.

    A way to think about it is to think of your hard drive like a blank notebook, and each page is an "allocation unit" with a set size it can hold. A small notebook page (i.e. small allocation unit) is very convenient if you have a lot of small bits of information to save. The same area of paper can be cut into many more pages and lots of small notes can be stored. If a note happens to be larger than a single page, the last line of every page is reserved to point to the next page to continue the note. If your note is only a single page, the last line is never used, but it is always reserved just in case that note gets larger.

    If you use a large notebook page size, each page stores a lot more information without needing to flow onto another page. If it does, it still only uses the last line of the page to point to the next page of the note. The drawback is that if you have a lot of files that only use a small part of the page, still take up a full page in the notebook.

    If you want to get the most usable space on this drive, I'd suggest using the largest allocation unit that makes sense for your data.

  • Lưu Vĩnh Phúc

    The link DragonLord provided above is not good in any way. It talked about FAT12/16 which is much inferior to exFAT. 16 bit is not enough for addressing clusters so FAT16 volume size tops out at 4GB, 4-16GB is only supported with sectors larger than 512 bytes. Current implementation of exFAT uses 25 bit for cluster indexing so it's most comparable to FAT32. But exFAT uses 64 bit for file size so it supports files larger than 4GB.

    The default cluster size for exFAT is described here

    In general cluster size above 8KB is not recommended. And use exFAT for HDD is not recommended too, except for exchanging files with other systems which can't understand NTFS, because exFAT is specifically designed for flash memories which have limited write cycle. exFAT doesn't have journaling to reduce flash wear, which trades for some less reliability. The appropriate cluster size for most case is 4KB (which is one of the reasons that NTFS's default cluster size is almost 4KB), except for partitions that most stores very large files. So if the external hard drive is not for plugging into some old HTPCs, media players or devices which can't read NTFS, you should use NTFS


  • Related Question

    windows 7 - Trouble using Cavier 2TB external HDD as internal drive
  • Robert

    I recently got a 2TB Caviar "Greendrive" external hard drive. I tested it, and it works fine. Because of the low price I got it at, and the fact I dislike externals, I removed it from it's enclosure, and it looks just like any other internal HDD, nothing special. When I hook it up, and boot Windows 7 (64x), it doesnt show up at all in the available drives. I can't reformat it while it is hooked up internally because of the recognition problem, but it works just fine when I put the exchange equipment from the enclosure on, and run it from USB. Can anyone tell me where a driver or diagnostic tool might be that can help me solve this? I would rather not format it if possible.

    When booting, it recognizes the HD, has the correct amount of memory listed, and refers to my primary and the 2TB as 5th and 6th masters respectively.


  • Related Answers
  • harrymc

    Does the BIOS detect the disk when it is installed internally ?
    If the drive is marked as disabled, you might need to change it manually to Automatic.

    Once any BIOS problem is fixed, regard in Disk Management if the drive is at all detected. You might need to reformat it and create a partition manually, maybe even assign it a drive letter.

  • ta.speot.is

    You may need to format the drive and assign it a mount point in diskmgmt.msc.