Author Topic: Backups of Media? How do you handle it? (Read 4241 times)

hyperlight · « **on:** March 28, 2011, 02:50:49 AM »

I found myself in an interesting discussion about backups and redundancy here:

http://bakabt.me/details.php?id=158739&page=1#comments

As to not clutter the post I have decided to continue this discussion in the forums. In regards to backup cost, 1-2TBs is decently easy to backup. However it becomes exceedingly cost prohibitive as that value increases. I personally have stopped backing up my media RAID5 arrays as the costs are too high associated with it. Currently my media libraries are at 30TBs and increasing at 8TB a year give or take. I currently just backup my mission critical data and irrecoverable stuff, which is around 4TBs.

I don't know about everyone else, but I find just having disk redundancy enough. I don't have to worry about accidental deletion as IMO if your dumb enough to do that you deserve it. As for natural disasters fire ect, I think its so rare its not worth considering. I just keep insurance to replace the equipment.

What are everyones thoughts on this? Does anyone have a cheap backup solution that they employ for their media? I considered cloud based backup, but after doing cost analysis I found that it would costs thousands a month to keep 30TB in the cloud, not to mention many months to actually upload that.

Tatsujin · « **Reply #1 on:** March 28, 2011, 03:40:20 AM »

Quote from: hyperlight on March 28, 2011, 02:50:49 AM

I found myself in an interesting discussion about backups and redundancy here:

http://bakabt.me/details.php?id=158739&page=1#comments

As to not clutter the post I have decided to continue this discussion in the forums. In regards to backup cost, 1-2TBs is decently easy to backup. However it becomes exceedingly cost prohibitive as that value increases. I personally have stopped backing up my media RAID5 arrays as the costs are too high associated with it. Currently my media libraries are at 30TBs and increasing at 8TB a year give or take. I currently just backup my mission critical data and irrecoverable stuff, which is around 4TBs.

I don't know about everyone else, but I find just having disk redundancy enough. I don't have to worry about accidental deletion as IMO if your dumb enough to do that you deserve it. As for natural disasters fire ect, I think its so rare its not worth considering. I just keep insurance to replace the equipment.

What are everyones thoughts on this? Does anyone have a cheap backup solution that they employ for their media? I considered cloud based backup, but after doing cost analysis I found that it would costs thousands a month to keep 30TB in the cloud, not to mention many months to actually upload that.

All anime are put in my externals. With the exception of all incoming anime from using torrents which is handled by one hard drive that has one partition.

The other internal hard drives have divided partitions, with one handling all music, the other all installation files (anything I install basically I take the installation file and put it in there for future installation usage, with the exception of games), all work related to PhotoShop, websites, designing, etc have their own separate partition, all images have their own partition (including manga, h-manga, any snap shots from anime, and all images that are or will be used for PhotoShop), all INSTALLED games have their own partition separated from the Primary OS partition which I only have the OS and all programs related like PhotoShop installed, CCCP and what not. Hmm ... I think that's basically it.

That's how I handle my HDDs.

Freedom Kira · « **Reply #2 on:** March 28, 2011, 04:54:41 AM »

I don't get how the costs are "too high" for RAID 5. RAID 5 is the most cost-effective solution, as you only need one hot spare disk out of any number of disks. When you back up your data, you need one disk for every other disk.

For instance, RAID 5 with four disks means you utilize the space of three disks and use one for parity. If you used four disks for storage and backup, you utilize the space of two disks and copy the same data onto the remaining two disks (like with RAID 1, just without the faster read benefits). Of course, you shouldn't use RAID 5 as a backup medium, you use it as a storage one, as it provides additional benefits of faster I/O. Use individual disks as backup media.

Of course, just having RAID 5 is not enough to provide good security against data loss. Always keep an extra backup of your most important data as necessary. RAID is, after all, just a way to improve failure tolerance.

As for myself, I just keep a RAID 5 array (4 x 1.5TB) for storage of my anime and etc. I don't have a backup of it.

tomoya-kun · « **Reply #3 on:** March 28, 2011, 05:29:33 AM »

I have two RAID0 arrays of 5 x 3TB which are mirrored in case anything is lost. I used to only have 1, but it failed.

hyperlight · « **Reply #4 on:** March 28, 2011, 06:04:16 AM »

Quote from: Freedom Kira on March 28, 2011, 04:54:41 AM

I don't get how the costs are "too high" for RAID 5.

When I say costs are too high, I mean costs are too high to back it up.

Xiong Chiamiov · « **Reply #5 on:** March 28, 2011, 06:07:14 AM »

Backups are for lesser mortals. When I accidentally trashed my LVM, I had to redownload a bunch of stuff - other than my ISP getting a little angry, it wasn't particularly an issue. Not worth the cost to have enough disks to make a backup for me.

tomoya-kun · « **Reply #6 on:** March 28, 2011, 06:16:35 AM »

Quote from: Xiong Chiamiov on March 28, 2011, 06:07:14 AM

Backups are for lesser mortals. When I accidentally trashed my LVM, I had to redownload a bunch of stuff - other than my ISP getting a little angry, it wasn't particularly an issue. Not worth the cost to have enough disks to make a backup for me.

Not everyone has the same bandwidth as you. For 55$ a month I have 60Gb available to download. To redownload a couple terrabytes would be expensive, might as well have a backup at 1$ per overage.

Tatsujin · « **Reply #7 on:** March 28, 2011, 06:22:19 AM »

Quote from: tomoya-kun on March 28, 2011, 06:16:35 AM

Quote from: Xiong Chiamiov on March 28, 2011, 06:07:14 AM
Backups are for lesser mortals. When I accidentally trashed my LVM, I had to redownload a bunch of stuff - other than my ISP getting a little angry, it wasn't particularly an issue. Not worth the cost to have enough disks to make a backup for me.

Not everyone has the same bandwidth as you. For 55$ a month I have 60Gb available to download. To redownload a couple terrabytes would be expensive, might as well have a backup at 1$ per overage.

Who's your provider? Where do you live?

tomoya-kun · « **Reply #8 on:** March 28, 2011, 06:30:04 AM »

Quote from: Tatsujin on March 28, 2011, 06:22:19 AM

Quote from: tomoya-kun on March 28, 2011, 06:16:35 AM
Quote from: Xiong Chiamiov on March 28, 2011, 06:07:14 AM
Backups are for lesser mortals. When I accidentally trashed my LVM, I had to redownload a bunch of stuff - other than my ISP getting a little angry, it wasn't particularly an issue. Not worth the cost to have enough disks to make a backup for me.

Not everyone has the same bandwidth as you. For 55$ a month I have 60Gb available to download. To redownload a couple terrabytes would be expensive, might as well have a backup at 1$ per overage.
Who's your provider? Where do you live?

Toronto, Ontario. Bell legacy plan.

Wonderful... the 30$ charge is added to the 60 as overage.

Tatsujin · « **Reply #9 on:** March 28, 2011, 06:37:16 AM »

Quote from: tomoya-kun on March 28, 2011, 06:30:04 AM

Quote from: Tatsujin on March 28, 2011, 06:22:19 AM
Quote from: tomoya-kun on March 28, 2011, 06:16:35 AM
Quote from: Xiong Chiamiov on March 28, 2011, 06:07:14 AM
Backups are for lesser mortals. When I accidentally trashed my LVM, I had to redownload a bunch of stuff - other than my ISP getting a little angry, it wasn't particularly an issue. Not worth the cost to have enough disks to make a backup for me.

Not everyone has the same bandwidth as you. For 55$ a month I have 60Gb available to download. To redownload a couple terrabytes would be expensive, might as well have a backup at 1$ per overage.
Who's your provider? Where do you live?

Toronto, Ontario. Bell legacy plan.

Wonderful... the 30$ charge is added to the 60 as overage.

Contact them and contact other Internet providers in your area and see their plans about Business Class and see what Tiers they offer. It should cost you roughly about 20% more to get unlimited bandwidth and to have all ports open and possibly if your lucky you get static IP which is somewhat good. Just make sure you confirm the unlimited bandwidth and the all ports being opened. I would do that first thing in the morning if I were you.

Pentium100 · « **Reply #10 on:** March 28, 2011, 10:34:39 AM »

LOL at bandwidth caps

Anyway:

My way of storing files is a bit complex, but it evolved kind-of naturally.

I have a lot (>10) of hard drives (total capacity about 3TB, largest drive 750GB, second-largest 500GB) but most of them are of varying ages and capacity so I cannot use RAID (well, I could in theory, if I wanted a very inefficient array and was willing to back up all of my data to create that array).

After I collect 100-200GB of anime that I watched, I record it to LTO tape (LTO1 = 100GB, LTO2 = 200GB), same thing with live action TV shows and movies that I download. I keep anime and live-action stuff on different tapes. I then gradually (when I need the space) delete the files from my hard drives. I also use LTO tapes for backing up (i.e. I keep a copy on the hard drive) big files.

To back up system drives of my PCs I use DDS4 tape (as LTO is too fast for backing up a lot of small files from my hard drives). I also use DDS4 to temporarily put away some anime or movies if I really need the space for something else (why not LTO? the tapes are more expensive per tape and I have some DDS4 tapes and the drive already).

For small, but important files (some photos, various documents) that need to be preserved for a long time, I use Magneto-Optical disks.

For home videos, I keep the miniDV tapes and also copy the videos to VHS or SVHS tapes.

kathy025 · « **Reply #11 on:** March 28, 2011, 12:22:49 PM »

I don't backup. Not because I don't want to, I can't afford to. That's why I just make sure I buy a high-performance external HDDs and do regular maintenance to minimize risks of corruption.

I lost over around 600-700GB anime in my MAXTOR 1TB before but it wasn't so bad since it gave me a reason to replace the old fansubs with DVD/BD rips. I now keep my entire anime collection in my SEAGATE 2TB.

Sosseres · « **Reply #12 on:** March 28, 2011, 04:58:33 PM »

I don't backup. No reason to, failure rate of HDDs is so low that most likely only one or zero of my HDDs will have failed before I replace them with larger discs as needed. If they do fail I just download it again, I have no critical data on my hdds. Closest to data I want to have a backup of is my bookmarks in firefox...

I have had one full disc crash. Lost 200GB anime. 2 of those series was hard to replace, so I didn't.

halfelite · « **Reply #13 on:** March 28, 2011, 05:08:06 PM »

For movies/anime/tv I run a raid6 NAS as its things I find not to important if I loose. There is about 20tb worth. And I also keep keepsake stuff like photos and home movies on it but they get backed up to an offsite server as well. I always have raid6 running at home incase a drive or two fails then important stuff is also backed up to offsite locations.

I used to run another raid0 10tb server to backup media but have since dropped it.

Zirro · « **Reply #14 on:** March 28, 2011, 05:28:28 PM »

(click to show/hide)

Current solution with three pairs of Western Digital External Elements hard drives set up with RAID 1 for redundancy. Four of them is 2 TB, and the other two are on 1.5 TB. Finally, there's the old LaCie in the middle which I intend to replace later on. It has room for 1 TB. If I need more space in the future, I'll probably have to get some kind of server-solution.

Tatsujin · « **Reply #15 on:** March 28, 2011, 05:30:53 PM »

Can you run RAID on external hard drives? If so, does it require specific HDD's to implement the feature and possibly other equipment? I'm sure there is. My knowledge on RAID is close to 0.

Zirro · « **Reply #16 on:** March 28, 2011, 05:40:47 PM »

Quote from: Tatsujin on March 28, 2011, 05:30:53 PM

Can you run RAID on external hard drives? If so, does it require specific HDD's to implement the feature and possibly other equipment? I'm sure there is. My knowledge on RAID is close to 0.

I do it on regular external hard drives through the disk utility in OS X. I've also heard of RAID-specific equipment, but I suppose those are only needed when dealing with more advanced levels of RAID.

tomoya-kun · « **Reply #17 on:** March 28, 2011, 05:43:51 PM »

Quote from: Tatsujin on March 28, 2011, 05:30:53 PM

Can you run RAID on external hard drives? If so, does it require specific HDD's to implement the feature and possibly other equipment? I'm sure there is. My knowledge on RAID is close to 0.

You can through software raid. I've got my two raid arrays combined into one array.

Freedom Kira · « **Reply #18 on:** March 28, 2011, 06:10:56 PM »

Quote from: hyperlight on March 28, 2011, 06:04:16 AM

When I say costs are too high, I mean costs are too high to back it up.

If you've got RAID 5, a backup isn't really necessary unless you have sensitive data on it that you can't afford to lose. That's why I consider it cost-effective.

Tri_Edge · « **Reply #19 on:** March 28, 2011, 09:52:33 PM »

I started using backups just recently.
I always thought nothing could go wrong until something went wrong and ended costing me 300GB of sensitive data,which i have yet to fully recover.
Since then i've been running 4 x2TB in raid 5 for backup in a external raid box.
Better safe than sorry.