Re: problem with enbd init on standby

(can you summarize a bit? I'm not sure where the problem definition is
here :-)


"Also sprach u.schmeling:"
> my standby config is between host A and host B.

Hey, snap!

> I'm using raid1 on A 
> consisting of a local device and a nbd device from standby.

Well, I think I know what you mean. A is primary and serves a raid
composed of its local disk and an nbd device exported from B, which is
secondary.

> After 
> switching over to B, I use B's local devices and put them into a raid1. 

So the server with the primary has died, and you now switch to the
server with the secondary. You set up a (degraded) raid.


> The intention is, to get it synchronized to the A's nbds again.

Not sure I follow ... this is when A comes back to life? At that point
you want to synchronize A's local disk FROM B. So you would export A's
local disk from A via enbd, and incorporate the enbd device into B's
raid, and it will sync.

That may well be the fastest way round (B writes to A over enbd).  But
it is not the same way round as originally, when A was writing to B
over enbd.

My first question would be "why change the direction of the enbd"?

Just curious. It is pleasingly symmetric. That's why, I'd suppose.

Doing it the other way round would require A reading from B's local
disk over enbd, while it served from the same raid as before. But now
the "enbd-half" of the raid mirror would be the most uptodate half, and
A would be reading and copying to its local disk.


> However 
> when starting the server on A and client on B I get the log looks like this:

Hummm ... this sounds like the sync of A by incorporating it over enbd
into a raid on B.


> enbd-client 12195: <#2409> expect result from looking for write command bksz is 4
> enbd-client 12195: <#2427> expect writing data for command bksz
> enbd-client 12195: <#2434> expect client (-1) suggested bksz 1024 to server
> enbd-client 12195: <#2437> expect have 0 after offset 4, need 5 more in buf
> enbd-client 12195: <#2366> fill Warning - read only 0/1B on descriptor 5: Invalid argument

Looks like the server either died or when crazy at this point. I
presume read() returned EINVAL on fd 5, or at least that's the current
value of errno.

A strace would be helpful, but really the story is on the server side!


> enbd-client 12195: <#2448> expect skipped 11B@4B to EOL
> enbd-client 12195: <#2451> expect Warning - client read half line <bksz 13107rqto >
> enbd-client 12195: <#2824> introduction client (-1) suggested blksize 1024 to server
> enbd-client 12195: <#2835> introduction client (-1) received suggestion 
> blksize 13107
> enbd-client 12195: client (-1) negotiated blksize 13107

Uhhhhhhh. I think we desync there. Looks like a minor bug in the error
handling sequence. But it's minor since it knows something is wrong and
it is going to fail the intro anyway.

> enbd-client 12195: <#2405> expect try looking for write command rqto
> enbd-client 12195: <#2272> command key rqto not found
> enbd-client 12195: <#2409> expect result from looking for write command 
> rqto is -1
> enbd-client 12195: <#2541> expect client (-1) fails in expect sequence
> enbd-client 12195: <#2999> introduction client (-1) introduction sequence ends FAIL

Well, that's interesting. 

I haven't the faintest idea what is happening, but a strace on both the
server and the client and that precise point is what I would want. To
me it looks as though the server died at that point. One question in my
mind is whether the server matches the client (compiled from the same
source archive). The intro handshake is pretty robust, and accepts
missing items and extra items and so on

And is the blocksize really 13107? That's just crazy enough to be true!


Peter