Author Topic: Case with 9+ hard drives & OpenSolaris/ZFS advice  (Read 2201 times)

Offline geoffreak

  • Member
  • Posts: 1258
  • Yes I yam!
    • Anime Bite
Case with 9+ hard drives & OpenSolaris/ZFS advice
« on: October 13, 2010, 02:27:36 AM »
So my Drobo randomly glitched up and "forgot" everything, so I'm thinking about getting a more serious solution. From what I can find ZFS is the cheapest and most reliable solution (RAID cards are much too expensive and offer none of the awesome features provided by ZFS).
 
First I need a computer that can house my 9 hard drives (4x 2TB, 5x 1TB) and maybe more in the future. I don't plan on gaming or outputting video with the computer, just file storage. ZFS requires a moderately powerful processor and a decent amount of memory to run well, but I don't want to buy top of the line for cost reasons. Cost is not the most important issue for me (it is still important though), but rather noise and power drain. I already have the hard drives, so there's not much I can do in that department, but if everything else was low powered and the PSU only provided enough to power the necessary parts, that would save big on the electric bills.
(Is there such a device that could hold hard drives and run an OS supporting ZFS, but connect to another computer via Firewire?)

OpenSolaris, while technically discontinued, provides the best free implementation of ZFS, so I'm thinking about going with that, but I don't know what hardware is supported by it.

I know I'm just kind of rambling, but any advice anyone could give would be much appreciated.

Offline kureshii

  • Former Staff
  • Member
  • Posts: 4485
  • May typeset edited light novels if asked nicely.
Re: Case with 9+ hard drives & OpenSolaris/ZFS advice
« Reply #1 on: October 13, 2010, 03:07:35 AM »
Best advice on my end is to wait for a stable, mature implementation of btrfs.

1) It is generally not advisable to use an OS that is out-of-development, or even discontinued, for a long period of time, especially for a non-testing machine. I don’t think this needs very much explaining.

2) You want something that is cheap, holds nine 3.5" hard drives, uses as little power as possible and can use ZFS (or comparable solutions) as a filesystem. If that is not a low-end PC, I don’t know what it is.

3) ZFS is not worth shoehorning your system hardware for, if you are a user looking for cheap. I say this as one who spent a year trying to decide what parts to use in a ZFS/OpenSolaris home server, and eventually gave up and decided to stick with plain ext3/Linux while waiting for btrfs to mature.

4) If you must have ZFS and no other filesystem, I would suggest looking for it on an OS that is still being actively developed, e.g. Nexenta or BSD. I’m not quite ready to trust terabytes of data to ZFS-on-FUSE, nor willing to wait for IllumOS to pick up speed and then stabilise.

[edit] Might be of interest.
« Last Edit: October 13, 2010, 03:19:08 AM by kureshii »

Offline Pentium100

  • Member
  • Posts: 528
Re: Case with 9+ hard drives & OpenSolaris/ZFS advice
« Reply #2 on: October 13, 2010, 03:28:04 AM »
Well, I cannot advise you about the OS, but I have this case and it's quite good - CoolerMaster Stacker810. It has 11 5.25" bays and comes with one 4-in-3 drive module (it fits 4 3.5" HDDs and takes up 3x5.25" bays, also as a 12cm fan). Buy two additional modules and you can place 12 drives in 9 bays. Two additional drives can be added with regular 3.5" to 5.25" adapters. Also, if you really need drive #15 you can remove the power etc buttons and have one more 5.25" bay.

The build quality is also good.

My PC with that case has 8 hard drives (2xSATA, 1xSCSI, 5xIDE), a DVDRW drive, a DDS-4 tape drive and three empty bays for one more 4-in-3 module.
BBT Ika Musume Fanclub Member #080586
Misaka Mikoto Fanclub Member:080586

Offline geoffreak

  • Member
  • Posts: 1258
  • Yes I yam!
    • Anime Bite
Re: Case with 9+ hard drives & OpenSolaris/ZFS advice
« Reply #3 on: October 13, 2010, 03:59:58 AM »
@kureshii
It's not that I won't use any other solution besides ZFS, it's just that I use a drobo and in theory, ZFS is the only better filesystem than BeyondRAID. This is more future planning than anything and I likely won't buy anything just yet.
I'm not picky in regards to OS so long as it is free and supports unix-styled commands. A nice GUI with at least some application support (ie Firefox) would be nice though. (Linux is my preference though, so I'll be watching btrfs development)
The only thing I need to know is what hardware is supported because I'll need to make sure to get a compatible motherboard and a JBOD SATA card (as most motherboards can't directly support this many drives)

@Pentium100
Sadly it appears that the case you have specified is no longer sold (at least at NewEgg), but the 4-in-3 module is cheap and should work in any case that has a lot of 5.25" bays. Thanks for the idea!
« Last Edit: October 13, 2010, 04:01:51 AM by geoffreak »

Offline kureshii

  • Former Staff
  • Member
  • Posts: 4485
  • May typeset edited light novels if asked nicely.
Re: Case with 9+ hard drives & OpenSolaris/ZFS advice
« Reply #4 on: October 13, 2010, 04:47:56 AM »
I am a little confused by your expectations; ZFS is a filesystem indeed, but BeyondRAID is a RAID implementation, not a filesystem, as far as I can tell (the Drobo probably has its own filesystem implementation, but I do think when you speak of BeyondRAID you are comparing it to RAID-Z, not ZFS). Perhaps what you mean to say is “RAID-Z is the only better RAID implementation than BeyondRAID”?

Here I would like to point out that Linux has its own RAID implementation in the form of mdadm, and it is already at a rather mature stage. When used with LVM2, the logical volume management module, it easily matches ZFS/RAID-Z feature-wise, although it is still susceptible to the RAID-5 write hole.

One critical thing to note about ZFS, as of the time of writing, is that it does not support RAID expansion. What this means is that if you already have a 4-disk RAID-Z/RAID-Z2 array, you cannot expand it at a later date by adding one more disk to the array. This was one of the critical flaws of ZFS (for a home user; let us not speak of enterprise usage here) that led me to decide on LVM2 instead (yes, mdadm/lvm2 on linux allows you to do RAID expansion).

I had been keeping some tabs on development in this area; last I read is that Adam managed to come up with a sketch for implementing this. But now that he and many other ex-Sun engineers have left Oracle for other projects, I find it highly unlikely that Oracle will ever add this feature to ZFS, considering the lack of enterprise interest in this area (enterprises often expand their storage pools by adding an entire array, rather than expanding existing ones. Such a feature is low on their priority list).

If you ever decide to expand your RAID-Z array, you are going to have to back up all data on it first, and rebuild the array from scratch. Not a minor undertaking for a non-enterprise user by any means if you have data running in the terabytes, and no spare disks.

It just occurred to me that there is no cheap motherboard (~$100) in existence (that I know of) that has at least nine SATA ports. Higher-end motherboards have up to ten (with a secondary on-board controller), but those are often meant for higher-power systems (X58 chipsets and the like), which runs against your idea of a low-power file server.

Even with such a ten-port SATA system, there is something to be said for not putting all your disks on the southbridge; apart from the AMD 890GX and Nvidia nForce series, neither of which are particularly low-power, pretty much all southbridge interconnects have a maximum bi-directional bandwidth of 1GB/s, and I have not heard of any consumer-level southbridge managing over 700MB/s. If you are going to try to implement FireWire and/or gigabit LAN connectivity, that southbridge interconnect is going to get very crowded. Even getting a cheap four-port SATA controller on the northbridge PCIe slots would alleviate that load quite a bit.

A cheap motherboard with a decent southbridge (ICH10 or maybe SB750+), plus an additional disk controller (RAID or otherwise), is a cheaper and more power-efficient solution. Here I readily assume you will be running the file server with onboard graphics.
« Last Edit: October 13, 2010, 04:59:22 AM by kureshii »

Offline geoffreak

  • Member
  • Posts: 1258
  • Yes I yam!
    • Anime Bite
Re: Case with 9+ hard drives & OpenSolaris/ZFS advice
« Reply #5 on: October 13, 2010, 05:08:44 AM »
@kureshii
Thanks for the tips. While I did not know that ZFS didn't offer any kind of expansion feature for RAID-Z, I personally don't mind not having it. My plan is to have the 4x 2TB drives in a RAID-Z array, and the 5x 1TB drives in a RAID-Z2 array, while pooling both arrays together (I assume I can do this and that ZFS can't effectively use drives of different sizes in the same array).

Also, I meant that ZFS (and more specifically RAID-Z) is better for data integrity than BeyondRAID for the simple reason that (short of hardware failure) your data can't become corrupted. The point of going to ZFS for me is to make sure my data is safe, but a bit of a speed boost would be nice (Drobos are slow)

I don't mind getting a couple of cheap RAID cards to compensate lack of motherboard SATA ports. This will reduce costs and power consumption as you said.

Offline halfelite

  • Member
  • Posts: 1153
Re: Case with 9+ hard drives & OpenSolaris/ZFS advice
« Reply #6 on: October 13, 2010, 06:59:17 AM »
I run a 10bay norco case can be found on newegg for 70 bucks http://www.newegg.com/Product/Product.aspx?Item=N82E16811219030&cm_re=norco-_-11-219-030-_-Product. and I also run a 20bay norco case that is a little more expensive. http://www.newegg.com/Product/Product.aspx?Item=N82E16811219030&cm_re=norco-_-11-219-030-_-Product Since you want to go with ZFS you need a nice performing cpu/mobo combo. As all the disc I/O will fall to the cpu.

When I built my first system I went with the
Norco case - 79$
Cheap mobo/cpu combo - $40 on sale at newegg
2 gigs or ram - 30$
Some semi cheap 600watt psu dont remember the price.
areca  raid card with 2gb of cache - $600
areca bbu - 100$
9 1tb drives.

and here are the stats it puts out using hdparm Its running raid 6
Code: [Select]
hdparm -tT /dev/sdb

/dev/sdb:
 Timing cached reads:   1896 MB in  2.00 seconds = 947.94 MB/sec
 Timing buffered disk reads:  988 MB in  3.03 seconds = 325.79 MB/sec
« Last Edit: October 13, 2010, 07:00:54 AM by halfelite »

Offline Pentium100

  • Member
  • Posts: 528
Re: Case with 9+ hard drives & OpenSolaris/ZFS advice
« Reply #7 on: October 13, 2010, 06:18:02 PM »

@Pentium100
Sadly it appears that the case you have specified is no longer sold (at least at NewEgg), but the 4-in-3 module is cheap and should work in any case that has a lot of 5.25" bays. Thanks for the idea!

Oh well, I built that PC a log time ago. Anyway, I looked at what NewEgg has to offer (I don't usually go there as I don't live in the USA) and saw this case. This one has 5 3.5" bays and 6 5.25" bays. Add two of those modules and you get 13 hard drives.

As for the speed - gigabit Ethernet limits the transfer rate to 100MB/s in theory. In practice you will get a bit less, but unless you are planning on installing games over network or do a lot of IO intensive work (live video encoding) you should be fine. I have no experience with ZFS, so no advice about that.
BBT Ika Musume Fanclub Member #080586
Misaka Mikoto Fanclub Member:080586

Offline rostheferret

  • Member
  • Posts: 1584
Re: Case with 9+ hard drives & OpenSolaris/ZFS advice
« Reply #8 on: October 13, 2010, 07:06:56 PM »
Surely a rack mount server case would work better here? He doesn't need a full case with all the expansion slots and the like, Full Towers are designed more with gaming in mind...

This I spotted with a minutes googling. With that 4-in-3 it'll fit 12 HDDs. Alternatively, you can go with a more reliable brand such as this which can fit 14 HDDs. The key thing here is the size; HDDs generate far less heat than graphics cards, and would be far lass CPU intensive so all the fans aren't neccessary. They're more easily stackable, meaning you can just get another one when this one is full and plonk it on top, and the second one even comes with a specially designed PSU.

Actually, I missed halfelites posts but that one looks pretty good too, though I'm unfamiliar with the brand.

Offline geoffreak

  • Member
  • Posts: 1258
  • Yes I yam!
    • Anime Bite
Re: Case with 9+ hard drives & OpenSolaris/ZFS advice
« Reply #9 on: October 13, 2010, 09:19:40 PM »
Thanks for the case suggestions. Anyone have any recommendations for motherboards and CPUs?

Offline Pentium100

  • Member
  • Posts: 528
Re: Case with 9+ hard drives & OpenSolaris/ZFS advice
« Reply #10 on: October 13, 2010, 10:58:03 PM »
Surely a rack mount server case would work better here? He doesn't need a full case with all the expansion slots and the like, Full Towers are designed more with gaming in mind...

This I spotted with a minutes googling. With that 4-in-3 it'll fit 12 HDDs.

You found the case of my main PC :) Yea, it would fit 12 drives, I think, but 2 of them would be vertical. I think I read somewhere that vertical drives do not last as long as horizontal ones. And those drives would not have a fan in front of them.
BBT Ika Musume Fanclub Member #080586
Misaka Mikoto Fanclub Member:080586

Offline rostheferret

  • Member
  • Posts: 1584
Re: Case with 9+ hard drives & OpenSolaris/ZFS advice
« Reply #11 on: October 14, 2010, 08:45:43 AM »
Surely a rack mount server case would work better here? He doesn't need a full case with all the expansion slots and the like, Full Towers are designed more with gaming in mind...

This I spotted with a minutes googling. With that 4-in-3 it'll fit 12 HDDs.

You found the case of my main PC :) Yea, it would fit 12 drives, I think, but 2 of them would be vertical. I think I read somewhere that vertical drives do not last as long as horizontal ones. And those drives would not have a fan in front of them.

Possibly, due to the manner the disc spins (it would introduce uneven gravitational forces between the two ends of the disc) but tbh, I wouldn't expect it to make a significant difference. I wouldn't expect heat to be a major issue either - my old PC didn't have more than a couple of fans and it was fine - but if it is, it's not hard to tape a fan somewhere :P

Offline bork

  • Member
  • Posts: 251
  • Huh ?
Re: Case with 9+ hard drives & OpenSolaris/ZFS advice
« Reply #12 on: October 14, 2010, 03:23:34 PM »
Disk orientation - had several systems with over 200 disks each, all standing on their ends (Auspex).  Most common large disk array chassis hold the disk drives on their edge.

Just make sure that you have proper power and cooling.   

Offline per

  • Member
  • Posts: 114
Re: Case with 9+ hard drives & OpenSolaris/ZFS advice
« Reply #13 on: October 15, 2010, 08:40:00 PM »
ZFS is definately worth it.
The checksums, and total ease of administration is unbeatable if you want a sub $100k solution.

Anyway, personally I have one of these:

http://www.mullet.se/product.html?product_id=332663

It's a supermicro chassi and motherboard, and should be available elsewhere too.

That´s around $2300 (ouch, the dollar is low, it used to be $1600 when I bought it, and the price in SEK is more or less the same. :-)) for the complete system, everything except drives included, making the drives the by far most expensive part.

Perhaps not the cheapest solution, but not all that much more expensive than low-end systems, and having 16 hot-swap slots with status-lights is really convenient if you ever have to exchange a drive.

Offline geoffreak

  • Member
  • Posts: 1258
  • Yes I yam!
    • Anime Bite
Re: Case with 9+ hard drives & OpenSolaris/ZFS advice
« Reply #14 on: October 19, 2010, 02:32:17 PM »
I think the case I've decided on is this one. I still have no idea what controller card(s) I should get though. I need something that is cheap, supports Ubuntu/OpenSolaris, and allows the OS to recognize each of the drives (JBOD without concatenation). The difficult part will be to not limit bandwidth too much and find a motherboard that has the necessary expansion card ports.

Offline bork

  • Member
  • Posts: 251
  • Huh ?
Re: Case with 9+ hard drives & OpenSolaris/ZFS advice
« Reply #15 on: October 19, 2010, 03:00:02 PM »
Whatever controller you get...

Look at the chip-set on the card, most use the Silicon Image chip, and find out the Bios rev its running.  Make sure that the Bios on the Sata controller can handle what you want it to do correctly; check if there are any updates to the Bios to fix problems found in the earlier Bios rev's.

A controller I had bought has the Sil3114 chip on it and by using the Silicon Image's Site found that there is a revision for the RAID bios.
http://www.siliconimage.com/support/faq.aspx follow to support.  There is also a version of the Bios for their chips that has no RAID capability if you just what plain disks, depends on what you want it to do.

I have also found that some controllers out of the box do not have a Bios on them that will recognize disks 1T and over, computer freezes at boot-up or just does not see the disk.  A lot of controllers have been on the shelf long enough that you could run into this problem and you will need a Bios update to resolve this.

In doing you updates - Run the update programs in DOS, stay away from the Window update method.  Create a Free DOS boot disk and put the put the update software and the image file on a second disk.

Cheap controllers - stay away from Sabrent.  Whatever they did to make their controllers make it impossible to rev their bios.  Spent almost two days on trying to get on to work before chucking the card and keeping the cables.  Off the shelf, they will not handle a drive larger than 1T.  My preference for Sata controllers are the SATA II from SIIG.

Possible configuration you might want to look at if you are slot limited on the mother board, both use the Silicon Image chips.
SIIG SATA II PCIe controller - http://www.siig.com/ViewProduct.aspx?pn=SC-SAE012-S2
Sata port multiplier, one for each controller port - http://www.addonics.com/products/host_controller/ad5sapm-e.asp
The port multiplier can also be installed into a second chassis, running a eSATA cable from the chassis with the controller.  A two case design were the you have a mother board in one and the disks in the second.

If you are running RAID under software, the system will need to be running before it can mount the RAID.  You will need one disk that is not part of the RAID that has everything needed to allow a full start up (boot, swap, /, /dev, ...). 
« Last Edit: October 19, 2010, 04:08:19 PM by bork »

Offline geoffreak

  • Member
  • Posts: 1258
  • Yes I yam!
    • Anime Bite
Re: Case with 9+ hard drives & OpenSolaris/ZFS advice
« Reply #16 on: October 19, 2010, 09:26:43 PM »
@bork
Thanks for the tips, but that controller card is limited to a bandwidth of 250MB/s due to the one lane PCI Express. This is fine for two hard drives, but multiplying the ports will just slow down everything a lot.
I will keep a port multiplier in mind though.

Offline bork

  • Member
  • Posts: 251
  • Huh ?
Re: Case with 9+ hard drives & OpenSolaris/ZFS advice
« Reply #17 on: October 20, 2010, 01:16:35 AM »
If your doing a file server, your limited to Ethernet speeds of 100Mb/s on 100Mb/s Ethernet and about 250Mb/s on a 1G (with a one lane PCI card).  Assuming your running desktop hardware and not server class hardware.

Also your disks can not sustain a read/write speed of what that SATA cable is capable of, there is a fair amount of idle time on the cable while transferring data with a disk.
« Last Edit: October 20, 2010, 01:32:49 AM by bork »

Offline geoffreak

  • Member
  • Posts: 1258
  • Yes I yam!
    • Anime Bite
Re: Case with 9+ hard drives & OpenSolaris/ZFS advice
« Reply #18 on: October 20, 2010, 03:03:24 AM »
While you are correct that I will not be able to access it as fast as I can read it, when using ZFS it would be necessary to have fast internal speeds for data maintenance.

Offline _Jitsu_

  • Member
  • Posts: 41
Re: Case with 9+ hard drives & OpenSolaris/ZFS advice
« Reply #19 on: October 25, 2010, 05:55:46 AM »


50 disc raid anyone? :D

Terryfying contraption.