Author Topic: 48 TB array (Read 4024 times)

-yasu- · « **on:** May 17, 2009, 06:46:05 PM »

Hi folks and dams, my friend and I are trying to build a home server and we ran into a snag. We are going to run a 48 TB non raid array SATA II config all in one huge custom made box.

We got the controller: http://www.newegg.com/Product/Product.aspx?Item=N82E16816151037

and 24 of these: http://www.newegg.com/Product/Product.aspx?Item=N82E16822136344

The biggest threat is heat, and how to power this monster.

Suggestions?

bloody000 · « **Reply #1 on:** May 17, 2009, 07:11:37 PM »

Hard drive eats around 10W each. the actual problem is to get a PSU with 24 sata power connectors. but of course you can also use 2 or 3 PSUs and jump start the other 2.

heat is a non-issue if the case have good airflow. e.g. intakes are in front of the drives, fans are at the back of the case sucking air out.

Ryokushin · « **Reply #2 on:** May 17, 2009, 08:03:44 PM »

When you figure out the casing situation as to what you're gonna do with them all I think it'd be easier for people to help you out here. I'll give you an example. A buddy of mine was building his computer--well asking me to help. And with what he was getting he was gonna need some fans--well, I found a good casing (and a good power supply with a great fan as well), and the casing had 4 fans in it for enough airflow. It's like Antarctica in there. Anyway--if you find a case or what you're going to put everything in, post it and I think you'll get more responses. Good luck--PS. Those harddrives, are sexy.

(PS--if you're custom making the case, it might be a bit more difficult, but I'd get a crapton of fans or enough airflow)
(PPS--read all of the reviews, as well, there are a couple of comments that have cons, even though the majority are pros, but there are a couple of people who have a lot to say when it comes to the cons of it. Take those into consideration)

MarchHare · « **Reply #3 on:** May 17, 2009, 09:58:21 PM »

ask houkouonchi. hes a network admin

dankles · « **Reply #4 on:** May 17, 2009, 10:27:25 PM »

Quote from: -yasu- on May 17, 2009, 06:46:05 PM

Hi folks and dams, my friend and I are trying to build a home server and we ran into a snag. We are going to run a 48 TB non raid array SATA II config all in one huge custom made box.

We got the controller: http://www.newegg.com/Product/Product.aspx?Item=N82E16816151037

and 24 of these: http://www.newegg.com/Product/Product.aspx?Item=N82E16822136344

The biggest threat is heat, and how to power this monster.

Suggestions?

WOW! Do you have a case for it already? You'll probably need to do some custom moding to get the case to work like you'd need.
What are you going to use this for? and what OS? And why not RAID?

Calverin · « **Reply #5 on:** May 17, 2009, 11:18:09 PM »

With that many drives and that much space I wouldn't go without using RAID 6. You only lose 4TB of total storage capacity with RAID 6, as opposed to the 48TB you would lose if one drive fails with RAID 0. What case are you using?

Slykester · « **Reply #6 on:** May 18, 2009, 12:16:43 AM »

Why non-raid? non-raid as in spanning or non-raid as in JBOD+drive extender. And, out-of-curiosity, what do you plan to use this server for?

-- edit, was drooling over teh possible uses -- didn't realize those were the green editions

I would say, to be safe, ensure you have | ~168 watts, 20A on 5v, 37A on 12v | reserved for your hard-drives. Use high quality splitters if you're going to use a single power supply at a time. If using one power supply, make sure you have a spare.

Make sure you balance the rail loading where necessary while splitting.

-yasu- · « **Reply #7 on:** May 18, 2009, 03:21:48 AM »

On the front end, we plan to use the server to host live security feeds for small companies that don't want to deal with maintaining their own cctv data storage. (VHS tapes are outdated =p) We are starting this from our garage and hoping this will take off and give us room for expansion. We are starting at the local level (community wise).

The nature of our use might require us to hot swap a drive in order to extract information and bring it on site quickly, so no spanning, striping, or RAID config as much as I would like to. I took in all your considerations and as time goes by, we will adopt the RAID 6 setup on a second build. We are going to use a steel cage to construct a 4x6 rack with the board and main power supply behind it. as for cooling, 6x120mm ball bearing fans 3 intake, 3 exhaust. As for power, I'm trying to contact a vendor that makes a 1000w power supply. Building vs. Buying an enterprise rack is cost effective for us on paper, so we are going down that road for now.

Slykester · « **Reply #8 on:** May 18, 2009, 03:50:45 AM »

I see, so you plan the JBOD route.. interesting business plan

halfelite · « **Reply #9 on:** May 18, 2009, 05:44:27 PM »

I would run at least 2 or 3 PSU's make sure the amps on the 12v rail are high. The areca cards support different spin up times so that isnt an issue with running more then one psu. I would have a dedicated 110 line in your house for it. I would not risk running it with anything else.

Also Would be good to use more then one for redundancy. One huge psu runs the risk of dyeing and you are out of luck on downtime. Running 4 mini psu,s will be more then enough. Also something like this would work for you.

http://www.newegg.com/Product/Product.aspx?Item=N82E16817147071

122A on the 12v rail is plenty to spin up all 28 drives. and keep them going.

Also what room air control do you have. A standard bedroom with central air would not keep the room cool. You would have to look into extra room cooling for the ambient temperature Either having a floor unit that exhausts out a window or getting a little higher end. Rack cooling is only as good as the room temp.

Slykester · « **Reply #10 on:** May 18, 2009, 09:07:17 PM »

o0 where do you get 122A on 12v from?

halfelite · « **Reply #11 on:** May 19, 2009, 01:07:59 AM »

Quote from: Slykester on May 18, 2009, 09:07:17 PM

o0 where do you get 122A on 12v from?

Its in the specs. Your standard psu as anywhere from 15-26. since that is 4 psu together it gives 122 amps on the 12v rail.

rl9009 · « **Reply #12 on:** May 19, 2009, 04:21:24 AM »

My suggestion: Build a redundant power supply system like the Dell PowerEdge systems do. You'll like it after you get one.

And yeah, since your controller does have a huge amount of RAM you might want to get a Backup Battery Unit for the controller itself and get a three or four high powered UPSes powered by at least 4 marine car batteries each.

That's how I'd do it for redundancy... It's up to you...

geoffreak · « **Reply #13 on:** May 19, 2009, 04:51:32 AM »

I have to warn you that if you use the consumer grade drives like the one you linked to, they don't do well in a heavy-use environment. Make sure you get server grade drives that have a higher MTBU (mean time between failures). Don't forget that with 24 drives, at least one will fail, so be prepared for that.

In regards to the power supply, a redundant power supply is a must if your server is going to be used in an always-on environment. Also, no PSU I'm aware of will come with enough SATA power ports, so you may need to rewire one. Just make sure you check the voltage and amperage of a working SATA power port before you start and compare it to the results in order to make sure that you don't have to buy new drives.

wolkec · « **Reply #14 on:** May 19, 2009, 03:33:50 PM »

Id rather have 4 tb of flash drive than 38 of normal one.
My suggestion is to wait

Tho i dont dl much anime, its a comp ment to play games

halfelite · « **Reply #15 on:** May 19, 2009, 06:13:49 PM »

Quote from: rl9009 on May 19, 2009, 04:21:24 AM

My suggestion: Build a redundant power supply system like the Dell PowerEdge systems do. You'll like it after you get one.

And yeah, since your controller does have a huge amount of RAM you might want to get a Backup Battery Unit for the controller itself and get a three or four high powered UPSes powered by at least 4 marine car batteries each.

That's how I'd do it for redundancy... It's up to you...

Without the bbu for the controller the card does not let you use the fancy cache ahead. Which is a huge speed loss. So i agree with the bbu for the controller its only 99 bucks.

rl9009 · « **Reply #16 on:** May 20, 2009, 12:20:09 AM »

Quote from: geoffreak on May 19, 2009, 04:51:32 AM

I have to warn you that if you use the consumer grade drives like the one you linked to, they don't do well in a heavy-use environment. Make sure you get server grade drives that have a higher MTBU (mean time between failures). Don't forget that with 24 drives, at least one will fail, so be prepared for that.

WD's Hard Drives tend to last so I don't see a problem there.

halfelite · « **Reply #17 on:** May 20, 2009, 05:12:55 AM »

Quote from: rl9009 on May 20, 2009, 12:20:09 AM

Quote from: geoffreak on May 19, 2009, 04:51:32 AM
I have to warn you that if you use the consumer grade drives like the one you linked to, they don't do well in a heavy-use environment. Make sure you get server grade drives that have a higher MTBU (mean time between failures). Don't forget that with 24 drives, at least one will fail, so be prepared for that.

WD's Hard Drives tend to last so I don't see a problem there.

Yes but consumer grade drives dont hold up well to 28 drives all spinning at once. The vibrations might through the heads off. If it was for a business i would spend the extra money on SA drives.

-yasu- · « **Reply #18 on:** May 20, 2009, 05:15:30 AM »

Quote from: halfelite on May 18, 2009, 05:44:27 PM

I would run at least 2 or 3 PSU's make sure the amps on the 12v rail are high. The areca cards support different spin up times so that isnt an issue with running more then one psu. I would have a dedicated 110 line in your house for it. I would not risk running it with anything else.

Also Would be good to use more then one for redundancy. One huge psu runs the risk of dyeing and you are out of luck on downtime. Running 4 mini psu,s will be more then enough. Also something like this would work for you.

http://www.newegg.com/Product/Product.aspx?Item=N82E16817147071

122A on the 12v rail is plenty to spin up all 28 drives. and keep them going.

Also what room air control do you have. A standard bedroom with central air would not keep the room cool. You would have to look into extra room cooling for the ambient temperature Either having a floor unit that exhausts out a window or getting a little higher end. Rack cooling is only as good as the room temp.

Your methods are very close to google's solution to power redundancy! they run a 12v tied into every rack in their shipping crate servers which is a cheap but brilliant idea. Air control consists of a central air climate control, with a stand alone AC unit that will be also tied into a backup UPS system in case power goes out. Theres also a big fan to help move the air around. Running a big PSU has its cons, but for the most part it is convenient. For server redundant power, I am going to figure out a way to utilize several car batteries on a relay. The main problem with that is the nature of car batteries tend to be unclear. I read on a site somewhere that 12v car batteries aren't really 12v, but 6 cells each at 2.1v makes 12.6 volts fully charged. I'll do an amp reading on them when I get a chance.

halfelite · « **Reply #19 on:** May 20, 2009, 08:33:21 AM »

car batteries are not the way to go. They do not do good for float charge. And will only last a few months you will be spending to much on batteries.