Re: Weird enbd failure while updating md superblock

Hi all,

Still getting some errors, once in a while. Lately, one of my machines
"hung" while using enbd and I found the following errors in
/var/log/kern.log

> Jul 21 19:02:39 localhost kernel: ENBD #1163[0]: enbd_rollback (0):
> rollback req cc9eab8c!
> Jul 21 19:02:54 localhost kernel: ENBD #1465[0]: enbd_ack (0): fatal:
> Bad handle cc9eab8c != 00000000!
> Jul 21 19:02:54 localhost kernel: ENBD #1472[0]: enbd_ack (0):
> ignoring ack of req cc9eab8c which slot lacks
> Jul 21 19:32:13 localhost kernel: ENBD #1163[1]: enbd_rollback (2):
> rollback req cc9eab8c!
> Jul 21 19:32:30 localhost kernel: ENBD #1465[1]: enbd_ack (2): fatal:
> Bad handle cc9eab8c != 00000000!
> Jul 21 19:32:30 localhost kernel: ENBD #1472[1]: enbd_ack (2):
> ignoring ack of req cc9eab8c which slot lacks
I don't get those errors on one of my other ENBD-"clusters", which
worries me a bit. Is it possible that one of my exporting servers is to
slow to do the job? Or is the network instable? And: how do I determine
which enbd-client process is generating this errors?

Regards,

Bas






Bas van Schaik wrote:
> Hi Peter,
>
>   
>>> Lately, one of my RAID5-arrays on top of ENBD ran out of sync, causing
>>> it to run in degraded mode (2 of 3 disks available). No problem, of
>>> course, I just hotadded the third device (ndc) again and a resync was
>>> initiated. However, this resync failed while updating the md superblock
>>> on one of the up-and-running devices. From the clientside syslog:
>>>
>>>
>>>       
>>>> Jun 13 04:00:12 localhost kernel: ENBD #1163[0]: enbd_rollback (0):
>>>> rollback req c3bc7b8c!
>>>> Jun 13 04:00:12 localhost kernel: ENBD #1163[1]: enbd_rollback (1):
>>>> rollback req c41f65ec!
>>>> Jun 13 04:00:12 localhost kernel: ENBD #1163[2]: enbd_rollback (2):
>>>> rollback req ced2404c!
>>>> Jun 13 04:00:12 localhost kernel: ENBD #1163[3]: enbd_rollback (3):
>>>> rollback req c419f04c!
>>>>         
>> Timeout (I'm a little surprised the device was in that mode, rather
>> than erroring on timeouts - it indicates you're not using the fr1 patch
>> part that alters md to talk to the enbd driver about what mode they
>> think each other should be in; it works for raid5 too).
>>     
> True, I didn't take the time to patch my kernel with it (for kernel 2.6
> there's only a kernel patch available, for 2.4 also a module, if I'm right)
>
>
>   
>>> (...)
>>> Time currently seems to differ about 1m10s.
>>>       
>> Are you saying these are simultaneous, despite the apparent time
>> difference in the logs.
>>     
> If I compare the two times, correcting them with the given difference, I
> would say the client saw a problem exactly 10 seconds before the server
> did. However, the given time difference was computed after two non-clean
> reboots, so I cannot guarantee it's accurate at all...
>
>
>   
>>> Currently I'm running a new resync (hdc and ndb are in-sync, ndc
>>> out-of-sync), I'll report any problems back to the list. Does anyone
>>> have any idea what could be the cause of this? Currently, this setup is
>>> just a "backup of a backup", but it should be stable... By the way: I'm
>>> using the semi-latest 2.4.33pre, which is stable on another "cluster".
>>>       
>> It looks as though one side or the other lost the network connection.
>> There's no actual "error" logged here, just reactions to network
>> outages. Without knowing the exact time difference (is it exactly 70s)
>> it's hard to say what happened first. I suppose the client saw the net
>> disappear first in the middle of some transaction and shut down. The
>> server also saw the same transaction fail and shut down on timeout.
>>     
> Knowing that there's no actual enbd error gives me confidence and hope :)
>
>
>   
>> If you were doing a rsync, I would suspect that it filled memory
>> somehow, leaving nothing for the tcp buffers, leading to timeouts (I
>> run rsync every night on many machines for many gigabytes of backups,
>> and it can get stuck on its own, so I do rsync on each subdir on its
>> own, and set a timeout - the "stuckness" is cured by a STOP CONT signal
>> pair, but I can't be bothered to do anything about programming it). If
>> you were doing a raid5 resync on its own, well, I dunno.
>>     
> Hmmm... The filling memory-thing would be almost impossible: I'm
> resyncing now and both machines have at least 750MB free (some RAM, and
> a bunch of swap), which should be enough! I'll wait for the result of
> the current resync, maybe it was just a stupidity of me which I can't
> recall. Anyhow, thanks for your feedback, it's really appreciated!
>
> -- Bas
>
>
>
> _______________________________________________
> ENBD mailing list
> [email protected]
> http://lists.community.tummy.com/mailman/listinfo/enbd
>