Re: Screwed my RAID array...

"Also sprach Bas van Schaik:"
> For some reason, enbd stopped responding,

It´ll always stop respondig for a reason! Usally it is for network
outage. The default configuration is for it to block until reconnect.

> causing my RAID5 array "md1"
> to hang.

Well, that is what is supposed to happen when your disk is not
responding! Try sticking a scewdriver in your hard disk ... (joke!).

> I thought just letting the enbd-client processes would solve
> the problem, so I issued:
> 
> > echo -n 0 > /proc/nbdstat

That should be fine - it will generate errors for five seconds, instead
of blocking requests till kingdom come. So now your raid array thinks
all its disks have errored out?

> (FYI: my configuration has one "frontend" running enbd-clients to four
> "backends")
> 
> This appearently wasn't a very good idea: md1 was completely down.

Well of course!

> No
> way to assemble it again using mdadm,

There is, but I do not know it! I always use the mkraid rool in that
situation. It has a really-really-do-what-I-say and no-argumentz switch
which is extremely useful for reassembling a raid array, in conjunction
with --dangerous-no-resync. Then it just puts your raid all together
again the way it should be. I have some yoyo like scsi disks that need
this treatment every week.

> it keeps complaining about "too

It is correct. So override it. Ask the author for the secret do what I
say switch. Danged if I ever could see it in the man page.

> few devices". Luckily, there was no important data on the RAID-device,
> it was built for testing purposes. However, I would like to know (and
> maybe it's nice to have it documented on this mailinglist, using this
> thread) how to act in such situation.

First - DO NOTHING.


Almost all errors are human-caused. You caused all but one in this
situation, as far as I can see.

> Afterwards, I think it would have been better to kick the client
> processes one-by-one. But, I expect mdadm would mark the nd* device
> belonging to the kicked client process as faulty. Kicking a second
> client process will therefore corrupt my RAID5 array.

Your first problem is that you are probably not running with the FR1
patch (for 2.6?). It would have put the enbd device in a mode where it
does not block on fault, and would have let enbd talk to the raid
device about their respective states of mind.

> By the way: I've checked /proc/nbdstat before killing/kicking anything,

Does not exist. Maybe nbdinfo.

> all seemed fine. All devices like this:
> 
> > State:      verify, rw, merge requests, enabled, show_errs, last error
> > 0, lives 0, bp 0

That is not the interesting part ... look lower down.

> So, I couldn't determine which client process was hanging, I couldn't

Client processes do not hang - if something has stopped it has to be in
kernel, so look in dmesg. But it sounds to me like a network outage.

> even determine what exactly was the problem. md1 was just not responding
> to anything, and "mdadm --detail /dev/md1" or "cat /proc/mdstat" hung up
> too.

Well, sounds like a good moment not to do that, then!  I think your
echo 0 was fine - it would have done no harm. If only you had known how
to put the poor shattered raid back together again! :-(.

mkraid --force --dangerous-no-resync ....

Peter