When it Rains, it Pours. 8TB HDD Just Died

Octavean

So yeah, with prices higher than I can recall seeing for some time, I had a WD Red 8TB HDD fail. The drive in question was in a RAID 0 array with four other 8TB drives. Risky I know but the data in question was fully backed up in two different locations.

I’d like to get it back online ASAP. I sort of have a spare 8TB drive but I won’t setup another RAID 0 array. Therefore, I’ll need to buy at least one 8TB drive or take the hit on capacity.

I do have eight WD Red 8TB HDDs in an offsite location but that’s in a quasi decommissioned Synology DS1815+ that serves as a backup dataset. So best not to mess with it. Although, it is using SHR2 (Synology Hybrid RAID2 AKA quasi RAID 6) so in theory I could pull two drives and the it’ll still be fine,…..stupid risky though.

I guess I’ll start looking for some new 8TB WE Red drives. This is going to be expensive,….

urbanriot

Octavean every spring i lose a few drives as the temperatures increase.

In my main PC I lost a drive in my “user profile” RAID5 array, a collection of 600GB 10k Raptor drives.

In my home Plex server I lost a 6TB WD Red.

In my home movie server I lost an 8TB WD Red.

Fortunately my last home server is compromised of a bunch of shucked 14TB Hitachi drives so I’m going to migrate the data there temporarily and to my main PC which I’m going to flip over from 4 × 600GB 10K Raptor drives to 4 × 4TB 7,200 RPM Lenovo Enterprise drives (Seagate I think), and then probably shutter one of those home servers and maybe smush together the drives. I think I’m going to back everything up and just redo it all.

Normally I’d replace drives as they failed but with the price of drives due to the AI boom, I’m going to ride it out.

Octavean I guess I’ll start looking for some new 8TB WE Red drives.

Is “WE” 7,200 RPM or ~ 5,600 RPM? I find the slower drives with their lower heat levels last longer in home servers.

Thane_Bitter

Octavean That really sucks but still, RAID 0

Perhaps you could do RAID 10, I know, double the cost (for drives, more hardware costs) but far less pain during a failure (ie only the cost of a drive vs the cost of a drive + restoration for backups + downtime/time lost + infinite sadness/frustration screaming “WHY ME DAMNIT! WHY!!”)

Octavean the price jumped up to $279. Auhggggg!!!

Got to love market pricing, “Oh hello Sir, I see you have been looking for this item, well the price has gone up Muh-ahahahah!”

Octavean

urbanriot That was supposed to be WD Red and they are 5400RPM IIRC. The 8TB drives have been fairly rugged and durable thus far but I guess it was bound to happen sooner or later. The WD Red 3TB and 4TB had a higher failure rate.

Octavean

Just an FYI, I was looking at a similar WD 8TB drive for $250 US. I gave it a few days, talked to my Wife about it and when I was ready to make the purchase (which was sitting in my cart already) the price jumped up to $279. Auhggggg!!!

egeezer

Thane_Bitter

Just curious, since there are several RAID options. I’d think that hardware controllers rather than software would be better performance and reliability wise.

There’s a dearth of technical information on controllers beyond saying which RAID configurations they support. For example, how do they cache,
do they support disk write journaling, do they support cacheing the last write operation if a disk or controller failure occurs during a write.

With a controller failure, I’d hope the failed write(s) would be saved in a separate cache that could be moved to the replacement controller. That and disk journaling was a requirement for financial midrange and mainframe systems even back in the 1990s.

Thane_Bitter

egeezer That was my take when I maked my storage system some time ago (gasp! 20 years wtf!), but that was before I learned about RAID-Z - with the hardware based systems (raid controller) if a disk died you replaced it, if the controller died you should be able to install a new one and it reads the data from the drive set (often limited to the same brand, same model, same firmware). If one had deep enough pockets you could install several cards in a machine and more into another machine and have pooled storage and other fun stuff but all that was not cheap.

RAIDZ was so much cheaper and since you could knit different sized drives into a larger functional pool and didn’t need a costly proprietary controller most of the work was done by the cpu and system memory rather than a controller.

In reality it feels like that carnival trick in which someone has a lot of spinning plates they have to keep spinning, move your data to bigger drives, bigger capacity system, and then onto bigger drives, each move could be the last of your data if you haven’t backed it up (or even if you have but only have one copy and it is NFG).

Octavean

I’ve had different RAID configurations for this hardware before. So some changes have been made in order to swap drives all the while thinking it’ll be temporary.

Anyway, the hardware in question is a Mediasonic ProBox 8 bay RAID enclosure (USB / eSATA). This is an older model that supports a number of different RAID levels that include 2x RAID 0, 2x RAID 5, RAID 50 and 2x RAID 10. So the drives and arrays are divided into two groups of 4 essentially. IIRC there is no noteworthy software option just dip switches.

Anyway, I might have misdiagnosed the problem. I’m not 100% sure its a drive failure now. I noticed the array was technically still seen by the NAS but it wasn’t presenting any folder structure. So, I checked the permissions and that seemed OK. All I did before was look at the front LEDs. An error was indicated and one of the drive LED’s was out so Ipso facto. However, its a possibility that the Mediasonic ProBox itself has failed instead. I checked the connection to the NAS and opted to turn off the Mediasonic ProBox. I turned it on today and oddly enough all drives seemed to be present but an error is still indicated and it shuts down automatically.

I’ll test the drives individually. Judging by it’s startup behavior I still suspect a drive failure but it pays to be sure.

With a different brand of DAS (4 bay) I had an external RAID enclosure fail. This was a RAID 5 array and the fix was simple, just swap the drives into a new enclosure of the same make / model. Unfortunately I don’t have a spare Mediasonic ProBox.

Octavean

OK, so I pulled the 8TB drive in question (drive 3) from the array. Put it in a SATA to USB adapter and connected it to my main PC. Nothing, it won’t show up in the Disk Management utility at all. It spins up, makes the typical sounds and gets a little warm but nothing beyond that. So I think it shuffled off its mortal coil

I also checked my spare 8TB WD Red drive and there wasn’t much on it so I didn’t have to move any files around. I put the drive in the ProRAID / ProBox and it cleared the drive error. I just have to configure it properly and it should work again.

I need at least one more 8TB drive but 3 more would be ideal.

Octavean

That same 8TB drive I was looking at for ~$250 direct from WD is now at $289. For F@#k sake!!!

Octavean

Little update. I took the hit and bought an 8TB drive with similar specs at ~$289. It should arrive sometime today according to the tracking info.

I’m out of town for a few more days but when I can I’ll test the new drive to be sure it’s fit for purpose. That SMART test should take a day or so. After that I’ll setup the drive in the new array as a 2x RAID 5 or more likely RAID 50. A 2x RAID 5 (3 drives each) with 8TB drives would look to the system like two 16TB drives. A RAID 50 would look like a single 32TB drive. I’d rather have a 6 drive RAID 5 array but this inclosure doesn’t do that.

I’ll source another two 8TB drives ASAP to setup an 8 drive array in RAID 50 or look into a newer inclosure.

Octavean

The price for the same drive is now back down to ~$269. F@#k me!!!

real_aurgathor

Octavean

If you can wait it should get cheaper. At some point, supply should catch up with demand.

Not sure if you do that, but I always buy a spare or two when I set up a RAID system – when I set up my RAID 6 system with 8 drives several years ago, I purchased 10 drives at that time to ensure that I always have a correct spare on hand.

Given the current HDD prices, looks like I’m going to use at least one of those spares soon. 🙄

Of course buying many drives at the same time can be expensive, though in retrospect I should’ve done that when I bought some 16 TB drives.

Octavean

real_aurgathor Yeah, I often do have a spare drive. I had one 8TB that wasn’t technically being used for anything. Pivoting from the RAID 0 (RAID 00) requires more drives than I had on hand though.

I also had a 14TB drive fail not too long ago. I had to replace it with a spare 16TB drive. That basically cleaned out my reserves.

vaxvms

real_aurgathor in retrospect I should’ve done that when I bought some 16 TB drives.

Hindsight is 20/20. 🙄

real_aurgathor

Octavean Pivoting from the RAID 0 (RAID 00) requires more drives than I had on hand though.

I haven’t used RAID 0 in long time since SSDs made it a somewhat moot point, although there are some people who even RAID 0 NVME drives! 😉

I only have RAID 5 and 6 ATM, so a single spare should be sufficient most of the time, though drive failures during rebuild are not unheard of, though I never experienced that. However, I plan to se-up a JBOD array or two using some relatively smaller drives to use them for low importance storage.

bennor

Laughed out loud at WD sending me an email the other day offering a sale on WD Red drive, I think it was 8TB, that was “deal” at $450. Nope. Like others are saying if possible see if you can wait. Hopefully production will eventually exceed AI/data center demand and prices might start dropping on the consumer side.

Octavean

bennor It’s my hope that come this Black Friday some more reasonable options will be up for offer. I’m debating returning the drive and ordering the cheaper one but I’m not sure it’s worth the $20 difference if it slows the process of getting back the array.

Octavean

real_aurgathor RAID 0 wasn’t for speed . The external RAID enclosure uses USB 3.0 or eSATA neither one of which is particularly fast. The point of using RAID 0 was to recuperate the drive capacity that otherwise would have gone to redundancy.

Octavean

I’m doing an extended test on the new 8TB drive now. Western Digital Data Lifeguard Diagnostics is citing just over 10 hours for an extended test. This seems really fast but perhaps I just got used to running this test on 16TB and 20TB drives which should take at least a day or two (24+ hr).

If this works out I can proceed to creating a new RAID 50 array and reloading about 30TB of data. That’s just to get back to where I was. I really could use another two 8TB drives though for roughly 48TB of usable space.

Edit:

Maybe the WD utility is giving a erroneous ETA. We will see,…