Re: Weird enbd failure while updating md superblock

Hi Peter,

>>Lately, one of my RAID5-arrays on top of ENBD ran out of sync, causing
>>it to run in degraded mode (2 of 3 disks available). No problem, of
>>course, I just hotadded the third device (ndc) again and a resync was
>>initiated. However, this resync failed while updating the md superblock
>>on one of the up-and-running devices. From the clientside syslog:
>>
>>
>>>Jun 13 04:00:12 localhost kernel: ENBD #1163[0]: enbd_rollback (0):
>>>rollback req c3bc7b8c!
>>>Jun 13 04:00:12 localhost kernel: ENBD #1163[1]: enbd_rollback (1):
>>>rollback req c41f65ec!
>>>Jun 13 04:00:12 localhost kernel: ENBD #1163[2]: enbd_rollback (2):
>>>rollback req ced2404c!
>>>Jun 13 04:00:12 localhost kernel: ENBD #1163[3]: enbd_rollback (3):
>>>rollback req c419f04c!
> 
> 
> Timeout (I'm a little surprised the device was in that mode, rather
> than erroring on timeouts - it indicates you're not using the fr1 patch
> part that alters md to talk to the enbd driver about what mode they
> think each other should be in; it works for raid5 too).
True, I didn't take the time to patch my kernel with it (for kernel 2.6
there's only a kernel patch available, for 2.4 also a module, if I'm right)


>> (...)
>>Time currently seems to differ about 1m10s.
> Are you saying these are simultaneous, despite the apparent time
> difference in the logs.
If I compare the two times, correcting them with the given difference, I
would say the client saw a problem exactly 10 seconds before the server
did. However, the given time difference was computed after two non-clean
reboots, so I cannot guarantee it's accurate at all...


>>Currently I'm running a new resync (hdc and ndb are in-sync, ndc
>>out-of-sync), I'll report any problems back to the list. Does anyone
>>have any idea what could be the cause of this? Currently, this setup is
>>just a "backup of a backup", but it should be stable... By the way: I'm
>>using the semi-latest 2.4.33pre, which is stable on another "cluster".
> 
> 
> It looks as though one side or the other lost the network connection.
> There's no actual "error" logged here, just reactions to network
> outages. Without knowing the exact time difference (is it exactly 70s)
> it's hard to say what happened first. I suppose the client saw the net
> disappear first in the middle of some transaction and shut down. The
> server also saw the same transaction fail and shut down on timeout.
Knowing that there's no actual enbd error gives me confidence and hope :)


> If you were doing a rsync, I would suspect that it filled memory
> somehow, leaving nothing for the tcp buffers, leading to timeouts (I
> run rsync every night on many machines for many gigabytes of backups,
> and it can get stuck on its own, so I do rsync on each subdir on its
> own, and set a timeout - the "stuckness" is cured by a STOP CONT signal
> pair, but I can't be bothered to do anything about programming it). If
> you were doing a raid5 resync on its own, well, I dunno.
Hmmm... The filling memory-thing would be almost impossible: I'm
resyncing now and both machines have at least 750MB free (some RAM, and
a bunch of swap), which should be enough! I'll wait for the result of
the current resync, maybe it was just a stupidity of me which I can't
recall. Anyhow, thanks for your feedback, it's really appreciated!

-- Bas