Moving large amounts of data between HDDs... CRC "fail"?

[Sakura90]

Hi

Seeing the great success that my other thread was... let's make a new one


I've experienced this for a long time now. It happens only when I move large amounts of data (around 1TB) from one HDD to another. As I have everything formatted in the usual "anime fashion" [Group/Source] Title [xxxxxxxx], after copying I do a full CRC check of all files. And the results often have one CRC mismatch. I don't recall seeing more than 1 fail per "batch". It's only one of the files, always a large one (10-20GB).

But it isn't a "fail" really. After the whole check, I go check again and again the "failed" file. But always after that the CRC matches. I even check a couple of times after using the drive everyday. The file never fails again. Nor does anything else.


This only occurs in the case I described, it can be from a 500GB to more than 1TB copies. It's never with a less amount. I always keep backup of every single bit I have, and despite those fails, I never lost anything. It's a "one time" fail that never "fails" again.

And I don't remember well now, but I could had had the same "issue" only by doing a hash check of all files in a 2TB drive, without anything new being copied. I should do a full check these days overnight and see what happens

Why does that happen? I've seen sometimes the HDDs specifications that they say something like a failure in 10^50 bits (to say a number). Does that have something to do? But the writes never fail actually, all files check ok. Even if one seems to fail, it passes if I check again and again.

Maybe it's a bug in the program? I use AnimeChecker, as it's the only one I know that can read the CRC to check from filenames.



It's weird. Any ideas? As this happens usually and I never really lost anything, I just check the failed file again and then sleep in peace . But curiosity kills me <.<

Btw, this happen while using the HDDs in a dock, external enclosure or even as regular internals. Drives are all "green" though, WD Green and Samsung Spinpoint F4 are what I use.



P.S.: Anyone knows of another program that can calculate a CRC and check it against the number in the filename?

[Freedom Kira]

I use RapidCRC. It's never given me problems, but at the same time I haven't used it extensively, and definitely not for so many files at once.

[lapa321]

It's normal. Our anime just uses so much data that we easily blow through any kind of error margin or data safeguards built into our systems.

Someone recommended FastCopy, it's a pretty good utility that's got a verify option.

[Big Teeth]

But better run a memorychecker for 24 hours on your system.
Or remove half of the memory from your system.
Do your copying around. Still troubles?
Swap the half memory and copy around again.

Have been there with a 3 GB RAM system. Troubles all over the place.
Using only 1 or 2 GB and everything went fine.

It turns out your files are oke, so why rerererecheck them again? :S

[rkruger]

Sounds like a memory problem to me.
As Big Teeth suggested, you should run a program to check the memory.
I suggest booting Memtest86 (http://www.memtest.org/) from a CD.

[megido-rev.M]

It's normal. Our anime just uses so much data that we easily blow through any kind of error margin or data safeguards built into our systems.

Not even: I've used checksum checks on single 1~8GiB files and X00GiB sets and they never failed on me like that.

As far as my OS-level disk subroutines knowledge goes, the disk device must return the exact data it contains no matter what, being nonvolatile and all. There is also the fact that a disk loads that data into memory, through DMA and such. So, like the guys above me mention, it's possibly a memory corruption issue-- mind anyone, checksum checks are computationally expensive (read: RAM).

Anyway, if you would like to be minimalist for checksum checking you could try HashCheck, completely stable as far as I know.

[Southrop]

What do you use to copy the files? Default OS copy manager? I recommend trying TeraCopy if you use Windows. It's faster and better than the default, and it automatically does CRC checking/testing after copying or moving files. Prediction times are also way better xD

[Pzc]

If you decide to do some diagnostics I recommend
the aptly named Ultimate Boot CD. There's some
useful stuff on it, including memtest. It can also be
booted from USB if so desired.

[lapa321]

Between my archives, download server, NAS, primary and secondary harddrives. I regularly move several hundred Gigs up to a terabyte of anime. More if a friend comes over with his harddrive to trade his collection. And despite mutiple diagnostic tests and even several hours of burn in tests with the computer coming out clean. CRC errors will still occasionally happen.

These are the kinds of copy operations that takes several hours to a day (hence, why i recommend burn tests on top of diagnostics). In fact i just finished archiving a 300Gigabyte batch yesterday. It took an hour to move from the download to the processing harddrive, and another 5-8 hours packing them into ISOs and archiving to an external backup drive.

I've run into a bad memory when i was copying anime from a friends computer. It was only ten gigs but several files were corrupted, that one was pretty easy to determine. But a 50% chance of a single error in a terrabyte worth of data? Whatever the cause, it's not showing up on any of my diagnostics.

BTW, the utilities i use when moving files for archiving: FastCopy and Beyond Compare.

[kitamesume]

im gonna join in into this discussion...

anyway, im getting errors with network transfer via LAN, the files that were transferred becomes incomplete like missing parts or corrupted parts when i recheck it with utorrent(using force re-check). any clues what causes this? or maybe my router is dying =/

edit: btw, USB-to-external drive transfers doesn't cause any of these issues at all =o

[boxer4]

For me, invariably when there's a single byte corruption in a file when transferring large amounts of data from disk to disk (whether HDD to optical, etc.) I've always rootcaused it to bad RAM.  Don't know why but it happened...  This also could mean bad motherboard that's not compatible with RAM but still it's a memory issue.

I have had bad disk controllers, CPUs, etc. too, but they tend to fail in other ways other than bad CRCs in large files...

I've yet to have a LAN adaptor fail in such way to cause bad data to be copied... we'll see...

[zzzYuyu]

It's always a memory problem. I have the very same problem with rtorrent and rutorrents automove feature, it usually reported uncompleted chunks when moving large completed files.
Funny thing is both of my ram sticks are healthy, it's just the ram controller that tends to cause errors when both ram sticks are plugged in.

[megido-rev.M]

Data transfer in general always involves loading and copying buffers, which would involve RAM in all cases. Network transfers are more fragile, and the probability of corruption has to do with how well the network channels can keep packets intact-- unlike disks, which have direct memory access (DMA) and such. Even then, it's all on the controllers and buses to keep things intact.

[boxer4]

Newer LAN cards are also "DMA" or rather whatever PCI does to write straight to memory.
LAN, because it's implied that theres a significant distance to go, is usually CRC-checked and this is a form of error detection.  The likelyhood of corruption over LAN due to the wire is very low due to the automatic retransmits if the wire data is bad.  However the chance of a bad DMA copy from the LAN chip to main memory is possible... but once again this is a RAM issue not a network card issue.

SATA and UDMA are also CRC checked. so I guess for most things, RAM/chipset is the weakest part of the link (CPU caches are ECC checked!)

Time to buy machines with ECC RAM?  People keep on saying this but costs keep on entering the equation, when you buy good RAM it makes ECC checks virtually unnecessary...

Definitely get a RAM check program, memtest86+ spotted errors on many of the machines I had corruption issues on.  But not all - I still have one machine that shows up clean on memtest86+ but still crashes randomly...  CPU issue possibly.

[megido-rev.M]

I use USB WLAN connections, so take that!

[Pagonis]

Get a RAID and forget about moving data arround for ever (replacing old smaller HDDs with new big ones one by one and rebuilding array doesn't count - it also protects from missmatched bits while "moving" such way).

If you're holding a lot of data that is dear to you and using one HDD or software (or crappy consumer level integrated) RAID-1, that's just wrong.

[kitamesume]

again with raid? we're talking about moving files from one place to another, doesn't necessarily mean its being moved to another storage. like you're packing something to watch on your laptop to be brought with you, or are you thinking of bringing your 10kg PC including the monitor w/e with you?

[Pagonis]

again with raid? we're talking about moving files from one place to another, doesn't necessarily mean its being moved to another storage. like you're packing something to watch on your laptop to be brought with you, or are you thinking of bringing your 10kg PC including the monitor w/e with you?

Erm. No. I just bring my laptop. Plenty of wi-fi hot spots to stream from my raid.

Also, 16 gB flash drive should hold enough anime for a quick trip.

[kitamesume]

thats why we're discussing why CRC fails from time to time through network transfers.

[lapa321]

thats why we're discussing why CRC fails from time to time through network transfers.

If you can recommend an application (one for each machine) that you can do a network (PC<->PC) burn test with. Especially one that will also do a burn test with an NAS harddrive (PC<->Remote harddrive), let me know. The one i have only does loopbacks and local.