Re: Screwed my RAID array...

> "Also sprach Bas van Schaik:"
> 
>>> For some reason, enbd stopped responding,
> 
> It´ll always stop respondig for a reason! Usally it is for network
> outage. The default configuration is for it to block until reconnect.
True, but (1) the network seems very stable: it has been up for a week
or so and it's a gigabit LAN. Of course, this is no garantuee. I used
the "-e" option to drop the default (block until reconnect), because it
should be the job of the RAID array to respond to disk/host/network
failures.


>>> causing my RAID5 array "md1"
>>> to hang.
> 
> Well, that is what is supposed to happen when your disk is not
> responding! Try sticking a scewdriver in your hard disk ... (joke!).
Typical * don't try this at home :P *


>>> No way to assemble it again using mdadm,
> 
> 
> There is, but I do not know it! I always use the mkraid rool in that
> situation. It has a really-really-do-what-I-say and no-argumentz switch
> which is extremely useful for reassembling a raid array, in conjunction
> with --dangerous-no-resync. Then it just puts your raid all together
> again the way it should be. I have some yoyo like scsi disks that need
> this treatment every week.
> 
> 
>>> it keeps complaining about "too
> 
> 
> It is correct. So override it. Ask the author for the secret do what I
> say switch. Danged if I ever could see it in the man page.
OK, I found it. It's just "-f" or "--force". However, if LVM is running
on top of RAID, it's impossible to stop the RAID device ("device or
resource busy"). It's also impossible to stop LVM, because it doesn't
recognize the LVM volume: it can't read the RAID device.
However, a reboot will do the trick: the RAID array will not be restored
at boot time and LVM will (of course) not start. After starting the
enbd-clients, an forced assemble can be issued:

mdadm --assemble --force /dev/md1 --force /dev/nda --force /dev/ndb
--force /dev/ndc --force /dev/ndd

after starting LVM, everything works again! I'll have to do some more
testing (with real data) and check if nothing ends up being corrupt, but
this seems to work.



>>> few devices". Luckily, there was no important data on the RAID-device,
>>> it was built for testing purposes. However, I would like to know (and
>>> maybe it's nice to have it documented on this mailinglist, using this
>>> thread) how to act in such situation.
> 
> 
> First - DO NOTHING.
> 
> 
> Almost all errors are human-caused. You caused all but one in this
> situation, as far as I can see.
True.


>>> Afterwards, I think it would have been better to kick the client
>>> processes one-by-one. But, I expect mdadm would mark the nd* device
>>> belonging to the kicked client process as faulty. Kicking a second
>>> client process will therefore corrupt my RAID5 array.
> 
> 
> Your first problem is that you are probably not running with the FR1
> patch (for 2.6?). It would have put the enbd device in a mode where it
> does not block on fault, and would have let enbd talk to the raid
> device about their respective states of mind.
Also true, but currently there's only a kernel patch available for FR1
(kernel 2.6), no module. First I want to get enbd running and after
that, I'll try to build a module from the FR1 code to replace the
default md-modules.

> 
> 
>>> By the way: I've checked /proc/nbdstat before killing/kicking anything,
> 
> Does not exist. Maybe nbdinfo.
You're absolutely right. It's /proc/mdstat and /proc/nbdinfo.

>>> all seemed fine. All devices like this:
>>> 
>>
>>>> > State:      verify, rw, merge requests, enabled, show_errs, last error
>>>> > 0, lives 0, bp 0
> 
> 
> That is not the interesting part ... look lower down.
OK, I'll post that the next time when I'm in trouble.


> Well, sounds like a good moment not to do that, then!  I think your
> echo 0 was fine - it would have done no harm. If only you had known how
> to put the poor shattered raid back together again!  :-( .
> 
> mkraid --force --dangerous-no-resync ....
So, with the mdadm-tools it's:

mdadm --assemble --force /dev/md1 --force /dev/nda --force /dev/ndb
--force /dev/ndc --force /dev/ndd

or, if you're lucky:

mdadm --assemble --force /dev/md1

-- Bas