ubuntu - What ZFS-layout for mixed redundancy?

I need to get all our home content off individual machines and onto a central server. What I'd like to have is the metaphorical "server under the stairs".

Stuff we need:

• expandable storage. I want to be able to add extra disc as we go along, with minimal maintenance required. Currently we've got about 3Tb of files we need to host, and that's likely to grow by another Tb every 6-12 months based on recent history. I need to be able to add additional disc with minimal pain
• needs to store all the media (i.e. photos, video, music) we have, and run services to serve the various devices we have in the house to playback (e.g. DAAP so we can play stuff through iTunes, ccxstream so we can play stuff over XBMC). DAAP and ccxstream are needed now, but we also need to support new standards as they emerge (so a closed-box solution isn't going to work)
• RAID 5, or something broadly equivalent (e.g. RAID-Z)
• BitTorrent client
• ssh, NFS, Samba access
• snapshot capability (as in ZFS), so we can snapshot individual file systems regularly and rollback when my kids delete their school assignments the day before they're due...
• ability to recover quickly from power outages (it's not unusual for us to have power outages that last longer than our UPS' batteries)
• FOSS software
• a modern distributed version control system running on the box, such as Mercurial

Stuff I'd like to have on the server, but can live without:

• PVR capability, so I could record TV to the box
• Web server. We currently run a small Web server on a very old box, and I'd ideally like to turn the old box off and move the content to the new server just to save some electricity
• Nagios + mrtg

I've been looking at using a EEE Box as the server, primarily because I can get them cheap and they don't consume much power. The choice of OS and file system is more difficult, from what I've found:

• I've got most experience with various Linux distros, but am happy to use another Unix
• FreeBSD and OpenSolaris seem to be the best choices for hosting ZFS
• OpenSolaris' hardware support is nowhere near as good as e.g. Ubuntu
• btrfs, while looking very good, doesn't seem ready for prime-time yet
• ZFS doesn't let you (easily?) add new discs to a RAID5 or RAID-Z
• reading around, it seems that ZFS is a bit short of tools for recovering lost data

At the moment, I'm leaning towards running FreeNAS+ZFS, but I'm concerned about the requirement to be able to add new disc on a fairly regular basis to an existing RAID-Z.

Can anyone provide some recommendations, or share experiences?

Thanks in advance

FreeBSD and OpenSolaris seem to be the best choices for hosting ZFS

With OpenSolaris, ZFS is usually versions/features/bug-fixes ahead.

OpenSolaris' hardware support is nowhere near as good as e.g. Ubuntu

Hardware support is getting much better with recent OpenSolaris builds but as long as your hardware is supported that shouldn't really matter.

ZFS doesn't let you (easily?) add new discs to a RAID5 or RAID-Z

You cannot add a single disk to a raidz but you can add another raidz to the pool where your first raidz is. The only drawback is you need to add multiple disks at the same time.

reading around, it seems that ZFS is a bit short of tools for recovering lost data

The point is ZFS doesn't lose data by design so recovering tools are of little purpose, outside the self-healing built in ones.

I don't know how to expand the RAID 5 part every few months but here's what I believe:

• You don't want an EEE box simply because you can't shove hard drives in there
• Perhaps get a cheap mid size case, low voltage single core AMD and matching mobo, a good 500W PSU (use an online power calculator to figure out the exact rating you need), and a RAID 5 PCI(E) controller
• Ubuntu will probably have all the codecs you need but FreeNAS sounds good too
• For document backup, try rdiff-backup on a nightly backup cycle (won't do snapshots, but incremental can save space)

My answer was to buy a BBS2 and install Ubuntu, at which point I was able to do what I wanted in terms of software and services. The basic box has room for 5 internal disks and if you chose it an eSATA port to which you can connect an external storage unit with room for 5 more disks. With today's disk capacities that up to 20 TB of raw (about 14 to 15 TB usable by the time you take RAID and filesystem overheads into account) storage.

The ability to take snapshots then depends on using LVM2, then you'll want to use software RAID over the top, which supports growing the array.

I've recently built a system that has some of the mentioned features.

It's based on an Atom Supermicro-System:
www.supermicro.com/products/system/1U/5015/SYS-5015A-H.cfm?typ=H

One 2-eSata-Port SIL3132-based PCIe-SATA Card
like this one: www.cooldrives.com/2poespcrasai.html

One (2 can be connected) SATA Port Multiplier for 5 Disks (Sil 3726)
like this one: www.cooldrives.com/cosapomubrso.html

The System run's FreeBSD 8.0 from a smal 2,5" internal SATA-Disk and has currently 3 1TB Disk connected to the Portmultiplier. The Filesystem on the external Disks is ZFS. One important thing is the following line in /boot/loader.conf
siis_load="YES"

I also have run iozone on this system an it performs not bad for 3 energy saving 5600rpm Drives

You should also think about the start current from the Disks. I have taped Pin 11 (pinouts.ru/Power/sata-power_pinout.shtml) of the Power connectors so the drives don't spin up simultaneously.

If you don't want it in an 19"-Case you can use the second PCIe Port on the Board for a 2nd Controller with 2 Ports for additional 10 Disks.

Some points from your Post:

• Adding Disks to the ZFS-Pool is realy easy
• I don't stream Media to Devices but there should be some piece of Software in the Ports collection (www.freshports.org/audio/mt-daapd/)
• The PVR-Part can be the hardest because of hardwaresupport (http://www.freshports.org/multimedia/mythtv)
• www.freshports.org/devel/mercurial/ -> Version controll
• ZFS is short of recoverytools because you don't realy need them. While testing I've tried to corrupt the Filesystem but no luck with this. :-)

SMB, SSH, NFS, Webserver, Nagios, MRTG/Cacti, BT are no real Problems

Good Luck with your decision and your Backup

I went with FreeBSD, ZFS

Antec P183 case, 400W Silverstone ST40EF PSU, ASRock A330ION ION Atom 330 mini ITX motherboard, LSI SAS3081E 8 port sas/sata pci-e 8x card, 4GB ram

6 2TB WD Green drives, 5 drive raidz, when that's full, buy another 5 drives (3GB will be cheap enough by then hopefully) When that's full, replace the old 2TB drives with 4TB drives, repeat until holographic storage.

7 3.5" drive slots, 4 5.25" (will have to convert to 3.5" somehow later)

It's a bit slow sometimes, I went for low power atom, but zfs plus other stuff might need more cpu.

ION 2 platform is useless though, not enough PCI-E lanes

Pfiouu, you want an all-in-the-box server in fact.

The point who should be the harder to decide is the storage. If you're sure about the amount of Hard drive memory required each years, you could made you server on a classic PC with 6 SATA connector and a RAID Controller over the Motherboard.

If you need more than 1 TB/Years in plus of those you already have, you should see over the professional solutions, but it's really expensive.

My 5 cents:

Look about deduplication or similar thing-a-magic to reduce the existing 3 TB and to reduce the growth of a TB every 6-12 Months.

How would you backup this amount of TB's to be on the safe side if something hurts your box?

In my opinion you are safer when storing your data over a few PC's in your LAN to eliminate to one point of failure.