D-Link Forums

The Graveyard - Products No Longer Supported => D-Link Storage => DNS-323 => Topic started by: lozinthevilla on May 21, 2010, 12:06:00 PM

Title: Is there a howto for recovering from harddisk failure?
Post by: lozinthevilla on May 21, 2010, 12:06:00 PM
Hello

I have been reading all your woes with hard disk failures and wrong drives being formatted and RAID 1 being degraded etc.

As one of my disks has recently gone 'pop', taking advice from the good people on this forum I backed up as much data I could rescue before swapping out the dead disk.

I have two Seagate ST31500341AS disks in my dns 323 and firmware 1.08.

I followed the manufacturer's instructions and I was 'not let down' to find that the dns 323 had decided to format both drives to create a stable RAID 1. :)

Fortunately I have my backup, so no tears. Just a little effort in putting data back on.

At the moment the dns 323 is now telling me the Sync Time Remaining:    259.8 minute(s). Which I find puzzling as it said the disks would be formatted. So what is it Syncing I wonder?

I must say I have found the whole process quite chaotic and a voyage into the unknown to say the least and leads me to wonder...

1. How long has this disk/RAID 1 problem been an issue?
2. Do you think D-Link do anything about the situation soon?
3. Is there a 'howto' on disk swapping/RAID1 restoration for anyone on firmware 1.08?

Thanks
loz
Title: Re: Is there a howto for recovering from harddisk failure?
Post by: gunrunnerjohn on May 21, 2010, 12:38:58 PM
Personally, I've decided that RAID-1 on this box really isn't worth the effort.  It slows things down, and since you really need an external backup anyway, what's the real point?  The only protection you get is from a disk failure, and that shouldn't be an everyday occurrence.

I have RAID-1 on my primary NAS that is actually in use for live data, I want that to be available in case of a drive failure.  The two D-Link boxes are just used for backup, so I formatted them in EXT3 for the added reliability and just use a single drive in them.
Title: Re: Is there a howto for recovering from harddisk failure?
Post by: dosborne on May 22, 2010, 03:55:59 AM
I had a new HDD fail and I tried reformatting and rebuilding a few times. I was quite impressed with the way the dlink software handled it and I had no data loss running v1.08.

I wonder if it has something to do with left vs right, primary vs master etc.

Even 2 clean HDD's have a short sync time. Any access to the shares during this time significantly extends the sync rebuild time.
Title: Re: Is there a howto for recovering from harddisk failure?
Post by: gunrunnerjohn on May 22, 2010, 06:02:55 AM
I tested the RAID by yanking a disk, I tried each side.  It appeared to function, but I've never had an actual hard drive failure in either of my D-Link NAS boxes.
Title: Re: Is there a howto for recovering from harddisk failure?
Post by: JoeHombre on May 22, 2010, 06:41:04 AM
I have experienced this exact same problem - twice.  The first time (fw 1.06) I had just purchased the DNS-323 and was a first-time user of RAID-1.  When a HD failed and I installed a new one, I expected a Re-Sync would take place and the RAID array would be established on the replacement HD.  That did not happen, instead both HDs were formatted as RAID-1 and all previous data was gone (of course I had a backup).  I assumed that this was my fault due to my lack of experience with RAID-1 and with the DSN-323.

Now, three years later (fw 1.08), I removed one of my two 500GB Samsung HDs (HD502IJ), from the Left Slot and replaced it with a 1TB Samsung (HD103SJ).  This should have resulted in a 500GB RAID-1 and a 500GB JBOD which is what I wanted.  Again I expected the RAID-1 array on the 500GM HD (Right Slot) to be transferred to the 1TB HD but didnít happen.  Just like the first time both HDs were formatted as RAID-1 and data previously on the 500GB HD was lost (again I had a backup).

In both the first time and second time attempting a Re-Sync, the DNS-323 menu (TOOLS; RAID) said that ďthe newly installed HD would be reformatted).  This is what I expected and proceeded to format.  But rather that formatting only the replaced HD, both HDs were formatted.

Anyone know if this a bug or what Iím missing here?

JH
Title: Re: Is there a howto for recovering from harddisk failure?
Post by: gunrunnerjohn on May 22, 2010, 07:44:32 AM
You should thank your lucky stars you have a backup! :D
Title: Re: Is there a howto for recovering from harddisk failure?
Post by: JoeHombre on May 22, 2010, 08:45:34 AM
Gunrunnerjohn, I have read many of your post where you employ folks to make backups and that RAID is not a substitute for backups.  I agree with you.  So itís not luck that saved me rather itís the same philosophy as yours.

My issue for D-Link is that on the DNS-323, when changing a HD, RAID-1 Re-Sync did not work as I expected.  Therefore, given that RAID-1 is not a substitute for backing up data - why doesnít the DNS-323 Re-Sync a RAID-1 HD with a replacement HD?  Or is it that contrary to expectations, the DNS-323 does not perform this function?

JH
Title: Re: Is there a howto for recovering from harddisk failure?
Post by: gunrunnerjohn on May 22, 2010, 08:54:21 AM
Like I said, I tested this with the DNS-323 and DNS-321, and it worked as expected.  I have seen a number of people here that have had your experience with both drives getting nuked, so obviously there's some *****s in the armor of the RAID-1 recovery process. :)

One reason I chose to abandon RAID on these devices and simply have extra layers of backup.  In addition, my Synology DS209 running RAID-1 is far faster, so it's a better choice for primary access.  The D-Link NAS units are used for unattended backups from the main NAS.
Title: Re: Is there a howto for recovering from harddisk failure?
Post by: fordem on May 22, 2010, 04:13:55 PM
Anyone know if this a bug or what Iím missing here?

JH
Were the "replacement" drives new, or at least clean and free of partitions and data, or were they used?
Title: Re: Is there a howto for recovering from harddisk failure?
Post by: JoeHombre on May 23, 2010, 12:57:35 AM
Fordem,

Initially, yes, in both cases the replacement HDs were new -- never formatted.

With my current attempt to upgrade from two 500GB drives to a 1TB matched with one of the 500GB drives, I have the second 500GH with the original RAID-1 array still present.  So after the first 500GB HD and 1TB HD reformatted both, I was able to try the second 500GB HD with the 1TB HD that was formatted in my first attempt at RAID-1.  However, I stopped short of proceeding when I saw nothing new that would indicate it would work so still have a 500GB HD with the previous RAID-1 array intact.

JH
Title: Re: Is there a howto for recovering from harddisk failure?
Post by: jamieburchell on May 23, 2010, 02:33:36 AM
This should have resulted in a 500GB RAID-1 and a 500GB JBOD which is what I wanted.  Again I expected the RAID-1 array on the 500GM HD (Right Slot) to be transferred to the 1TB HD but didnít happen.

How can you have a RAID array and JBOD on two drives?
Title: Re: Is there a howto for recovering from harddisk failure?
Post by: JoeHombre on May 23, 2010, 03:31:06 AM
In my case, I paired a 1TB HD with a 500GB HD.  RAID-1 will set the largest size it can which means a 500GB RAID-1 Volume and convert the remaining space to JBOD which means 500GB for my situation.  Therefore, Volume 1 is 500GB RAID-1 and Volume 2 is 500 GB JBOD.
Title: Re: Is there a howto for recovering from harddisk failure?
Post by: gunrunnerjohn on May 23, 2010, 05:32:16 AM
Actually, the extra 500gigs would be a plain disk partition, JBOD implies two disks.
Title: Re: Is there a howto for recovering from harddisk failure?
Post by: JoeHombre on May 23, 2010, 06:36:24 AM
Below is copy of info from my 323's menu showing Volume _1 Type as RAID 1 and Volume_2 Type as JBOD.  I have set Volume_1 as drive N: and Volume_2 as drive O:

So far I have only copied files to N:


HARD DRIVE INFO :
Total Drive(s):  2

--------------------------------------------------------------------------------
 
Volume Name:  Volume_1
Volume Type:  RAID 1
Sync Time Remaining:  Completed
Total Hard Drive Capacity:  490402 MB
Used Space:  60462 MB
Unused Space:  429939 MB



--------------------------------------------------------------------------------
Volume Name:  Volume_2
Volume Type:  JBOD
Total Hard Drive Capacity:  493851 MB
Used Space:  73 MB
Unused Space:  493777 MB
Title: Re: Is there a howto for recovering from harddisk failure?
Post by: fordem on May 23, 2010, 08:06:19 AM
Actually, the extra 500gigs would be a plain disk partition, JBOD implies two disks.

As JoeHombre as pointed out - D-Link would term the "non-RAID" volume in his configuration JBOD, what I would like to point out is that there are several different interpretations of the acronym JBOD, and in it's most basic interpretation it implies multiple independent disks which D-Link has termed standard volumes.

Prior to my acquiring my DNS-323 a few years ago, the above interpretation is the one I was familiar with, rather than D-Link's "concatenated" interpretation.
Title: Re: Is there a howto for recovering from harddisk failure?
Post by: JoeHombre on May 24, 2010, 05:39:17 AM
There seems to be some conditions where the 323 does not reestablish a RAID 1 when a HD is replaced.

In a RAID 1 format, does anyone know the 323ís assumption when a HD is replaced?

For example, if the Right HD fails or is just replaced with a different one -- does the 323 assume that the new HD will be inserted in the same (in this example) Right Slot?

What if the HD in the Left Slot that did not fail or is being replaced is moved to the Right slot and the new HD is installed in the Left Slot -- does this matter in terms of reconstituting the RAID 1 array?

Thanks for any insight.

JH
Title: Re: Is there a howto for recovering from harddisk failure?
Post by: gunrunnerjohn on May 24, 2010, 06:23:46 AM
Moving the remaining drive will screw things up.  I've actually tried this...
Title: Re: Is there a howto for recovering from harddisk failure?
Post by: JoeHombre on May 24, 2010, 08:19:49 AM
At one point I did move the HD not being replaced to the other slot.

After shuting-down the 323, I disconnected the power chord too so the 323ís circuit card would not be powered.  I considered this a safe practice procedure but maybe doing so caused the 323 to lose memory of the Slots were the HDs were located.

You'd think that DLink could find a way to allow relocating the HDís Slot and/or removing the power chord without losing the info needed to reestablish a RAID1 array.  If this is in fact a no-no, DLink  should so state the proper procedure to be followed as dosborne asked.

Thanks, Gunrunnerjohn.

JH
Title: Re: Is there a howto for recovering from harddisk failure?
Post by: gunrunnerjohn on May 24, 2010, 09:57:15 AM
We have a few "ought to be mentioned" things that have been talked about here. :D
Title: Re: Is there a howto for recovering from harddisk failure?
Post by: fordem on May 24, 2010, 10:50:28 AM
At one point I did move the HD not being replaced to the other slot.

After shuting-down the 323, I disconnected the power chord too so the 323ís circuit card would not be powered.  I considered this a safe practice procedure but maybe doing so caused the 323 to lose memory of the Slots were the HDs were located.

You'd think that DLink could find a way to allow relocating the HDís Slot and/or removing the power chord without losing the info needed to reestablish a RAID1 array.  If this is in fact a no-no, DLink  should so state the proper procedure to be followed as dosborne asked.

Thanks, Gunrunnerjohn.

JH
First - there is a specific process for replacing a failed drive in a RAID array - this can be found in the FAQ on D-Link's support web page.

Second - the serial numbers of the drives in a RAID array and the respective slot locations are stored, I believe on the drives themselves - this permits for example, a functional RAID array pair in a defunct DNS-323 to be transferred into a replacement DNS-323.  If you power the unit, off, swap the drives around and then power it back on, it will detect this and should ask you to insert them correctly, however, I don't think they catered for a situation where a drive would fail and the user would swap the drives around simultaneously.

Also I'd like to point out here, that in most environments using RAID, swapping physical drive locations is a NO-NO -

It's generally considered a "sure-fire" way to create a problem resulting in the loss of data.  Newer hardware RAID controllers do permit "disk roaming", older ones do not, and most support professionals will go to great lengths to ensure that disks are replaced in the poisitions they were removed from, because this is what they are taught to do.

Title: Re: Is there a howto for recovering from harddisk failure?
Post by: JoeHombre on May 24, 2010, 07:50:06 PM
Thanks, Fordem.  This is very helpful info.  I'll go read the FAQ on replacing RAID1 HDs.

JH
Title: Re: Is there a howto for recovering from harddisk failure?
Post by: lozinthevilla on May 30, 2010, 07:00:05 AM
Quote
there is a specific process for replacing a failed drive in a RAID array - this can be found in the FAQ on D-Link's support web page.

After many days of transferring my data back to my dns323 after a RAID1 failure, guess what? A disk has failed again in the left side, again! This was a brand new disk and the same breed as the other (Seagate ST31500341AS)

I have followed the FAQ to the letter just like last time and I think its behaving like it should!!!

I popped a WD HD in the left side, switched on, followed the prompts and it appears to be formatting it, instead of nuking my RAID1! :)

Fingers crossed it doesn't nuke my RAID1 after its formatted the WS HD!  ;D

Having another failure so soon after the last one, (two weeks), does this me I may have a faulty DNS unit now?

Or could it be a faulty new hard disk?


l

P.S. Just checked my email : the dns323 is telling me the right hand side disk has failed!  I'm going back to tape! :'(
Title: Re: Is there a howto for recovering from harddisk failure?
Post by: jamieburchell on May 30, 2010, 10:39:19 AM
I'd be tempted to pop all these suspect drives out and insert them directly in a PC and run manufacturers diagnostics on them. That will help better determine the health of the drives.

If they're OK, and resetting the NAS to factory defaults and re-formatting doesn't help it sounds like a hardware issue with the NAS. Try reseating the drives, blowing dust off connectors etc.

If you can, backup your data before doing anything.
Title: Re: Is there a howto for recovering from harddisk failure?
Post by: lozinthevilla on May 30, 2010, 11:23:42 AM
Quote
If you can, backup your data before doing anything.

Oh yes! I have learned my lesson well! :) After reading a lot of postings here, I have felt the pain of so many!

Thanks for the pointers jamieburchell I shall get these disks in a PC and give them a whirl!

I have a windows7 pc, these disks are formatted to ext3, will the disk utilities still work on them does anyone know?
Title: Re: Is there a howto for recovering from harddisk failure?
Post by: gunrunnerjohn on May 30, 2010, 11:43:57 AM
You'll need a driver.  Here's a couple of the popular ones.

http://www.fs-driver.org/

http://sourceforge.net/projects/ext2fsd/

Three Ways To Access Linux Partitions (ext2/ext3) From Windows (http://www.howtoforge.com/access-linux-partitions-from-windows)
Title: Re: Is there a howto for recovering from harddisk failure?
Post by: jamieburchell on May 30, 2010, 12:51:05 PM
Oh yes! I have learned my lesson well! :) After reading a lot of postings here, I have felt the pain of so many!

Thanks for the pointers jamieburchell I shall get these disks in a PC and give them a whirl!

I have a windows7 pc, these disks are formatted to ext3, will the disk utilities still work on them does anyone know?

As far as I know, you only need those Windows drivers if you need to read data off the disk or write data to the disk. You don't need them for running manufacturers diagnostics as tests are performed at a lower level. You can usually get bootable diagnostics too bypassing windows altogether.

If you do decide to use the drivers above to read data from the drives you should mount them as read only to avoid screwing up your drives and/or confusing your NAS.
Title: Re: Is there a howto for recovering from harddisk failure?
Post by: gunrunnerjohn on May 30, 2010, 01:20:07 PM
Right, I assumed he'd want to recover any data first. :)
Title: Re: Is there a howto for recovering from harddisk failure?
Post by: jamieburchell on May 30, 2010, 01:22:52 PM
:) think (hope) he's already got a backup
Title: Re: Is there a howto for recovering from harddisk failure?
Post by: gunrunnerjohn on May 30, 2010, 01:23:17 PM
Never hurts to offer the option. :D
Title: Re: Is there a howto for recovering from harddisk failure?
Post by: lozinthevilla on June 04, 2010, 02:56:11 AM
Hi everyone!

Just to let you know how the disk swap/format/RAID1 rebuild went...

Like a dream! I followed exactly the FAQ (like I did the first time around), and the new WD disk (HD2) was formatted and then the RAID1 rebuild kicked in straight after that.

Now the disk that I replaced has been tested with seatools and I can find nothing wrong with it at all!

Also the disk (HD1) that told me it had failed has stopped telling that it has failed and seems absolutely fine, although I am keeping an eye on it on a daily basis.

I also tested the disk that failed a few weeks ago -  and that comes back OK too!  ???

So now I have 2 x 1.5Gb HDs spare... How lucky am I?  ;D

Thank you very much for all your help and directions - I would surely have lost my data without your help!

I still think the 323 is a great bit of kit!

Title: Re: Is there a howto for recovering from harddisk failure?
Post by: jamieburchell on June 04, 2010, 07:09:42 AM
I wonder why the DNS-323 said they had failed if they are physically OK. Data corruption? I wonder what a fsck on the drives would report?