Re: ENBD Issues with client crash / weird errors....

"Also sprach Jonathan Carron:"
(something in a windows charset, not terribly readably formatted either)!

Hi - can you put some spaces in your description? It makes my eyes
blur! I'll try extracting what I need to see, so you can get the idea:

> running gentoo with kernel 2.6.12.

What ARCHITECTURE?

Is enbd definitely working then on 2.6.12?  I got the impression there
were still a couple of minor issues somewhere (I can't use recent 2.6
kernels for very long on my laptop as they lock it up under moderate
graphics use such as moving the mouse ...  ).  I am running 2.6.12.3 at
the moment and I see occasinal reconnect troubles as I return from
hiernation, but they're cured by a quick swat of a zero to
/proc/nbdinfo, so I haven't investigated.


>   Anyhow at first I would mount the /dev/nda on the client
> machine as /dev/md2

You are using enbd as a raid1 component? In that case you need FR1.

> my data to the raid array, cp –rv /src/ /dst/. This starts and seems
> to be going perfect no errors in dmesg. TCP Track showed 4 connections
> floating about about 800kb/sec to about 1Mbps ish. Great, what I

Sounds a bit slow ... oh, I suppose that might be 100BT half-duplex.

> expected.  I come back later and the /dev/md2 has about 22 gb on it,
> and doesn't seem to be filling up anymore.

Well, why not? (What does raid say it is doing?)

> Am I trying to copy too much data?

No, though it sounds a mite strange as a way to spend time. What is the
other component of the raid? If it is local, why not write to it with
the raid in degraded mode, then add in the enbd component and let raid
take care of syncing, all in the background?

> Im worried if I just put the drive in and leave it running,
> what happens if one does indeed fail :| TCPtrack shows the connections
> are no longer active (moving data).

But what does enbd say it is doing? /proc/nbdinfo?

> I do however still see 4 enbd
> processed running.

But what does enbd say?

> I cannot unmount or stop the raid device, the

Well, I don't see why you would want to - but it sounds as though the
connection has broken and enbd is blocking until the connection comes
back. That's what it is supposed to do, unless told otherwise.

> terminal just hangs.

Sounds right.

> Kill –9 will not kill the enbd-clients at all.

I would hope not! Why do that? They are just waiting in kernel for the
network to come back up, I suppose. Anyway, what does the kernel say
about all this? dmesg? It sounds as though smething has oopsed! Or at
any rate as though all the requests in the world were stacked up in
kernel somewhere! dmesg and nbdinfo woud let me say.

> And sometimes I've noticed the network is not working (cannot ping IP

There you are.

> address of that machine from any other machine on that network). Halt

There!

> or reboot don't work. Process is locked solid! :S… (The server is

I would hope so. 

> fine, I can reconnect no problem after having to hard reset the
> client)

But why?  If you want to error out the stacked kernel requests you would
write a 0 to /proc/nbdinfo.  But as it is it sounds as though the
networking has died and the enbd device is doing what it should do
unless you tell it otherwise - waiting for the network to come back.

The real question is why is networking dead?


> Im totally lost. I've tried different nics, and no luck.

Why do you think the nics are at fault? What does the kernel say?

> Hard was is
> known to be in good shape. Have tested drives and RAM. Im using enbd
> 2-4.32 . Again, I could not get 2.4.31 to compile.

I don't think it is supposed to cmpile against 2.6.12?


<snip>
> test 5 success:  0 incorrect blocks
> flushing buffers..done
> ioctl 1 (write direct) testing write illegal ..success (fail Invalid argument)
> ioctl 1 (write direct) testing write 0x469b392d ..FAIL (Invalid argument)   <- FAIL??

Yes, a bit strange.

> ioctl 2 (read indirect) testing read 0x469b392d ..FAIL (Invalid argument)   <- ?
> ioctl 3 (write + read indirect) testing illegal ..success (fail
> Invalid argument)
> ioctl 3 (write + read indirect) testing w/r 0x250d15c7 ..FAIL (Invalid
> argument)  <- ?
> ioctl 4 (large read indirect) testing ..FAIL (Invalid argument)   <- ?
> ioctl 5 (large write + read indirect) testing ..FAIL (Invalid argument)  <-?
> <snip>

Looks bad to me.



dmesg output (when copying data to the disk, usually only get about 5
lines or so. )

EVEN after this happens it continues to copy just fine…. Strange.

> <snip>
> ENBD #1168[0]: enbd_rollback (1): rollback req e3fb7398!
> ENBD #1168[1]: enbd_rollback (2): rollback req e04ed79c!
> ENBD #1168[2]: enbd_rollback (3): rollback req e3eaa79c!

Well, you have issues. It has given up and timed out waiting for
requests. The server is dead. Or the net is dead.


> ENBD #1469[0]: enbd_ack (3): fatal: Bad handle e3eaa79c != 00000000!
> ENBD #1476[0]: enbd_ack (3): ignoring ack of req e3eaa79c which slot lacks
> ENBD #1469[1]: enbd_ack (2): fatal: Bad handle e04ed79c != 00000000!

It looks like the request acks came in much later, and there was no
longer a request there.

> ENBD #1476[1]: enbd_ack (2): ignoring ack of req e04ed79c which slot lacks
> ENBD #1469[2]: enbd_ack (1): fatal: Bad handle e3fb7398 != 00000000!
> ENBD #1476[2]: enbd_ack (1): ignoring ack of req e3fb7398 which slot lacks
> ENBD #1168[3]: enbd_rollback (0): rollback req dff734b0!
....
....

> ENBD #1168[23]: enbd_rollback (2): rollback req dfc7dac8!
> ENBD #1168[24]: enbd_rollback (3): rollback req dfc04184!
> ENBD #1469[21]: enbd_ack (1): fatal: Bad handle e1c24398 != 00000000!

Yes, serious timeout issues.

>ioctl 1 (write direct) testing write illegal ..success (fail Invalid argument)
>ioctl 1 (write direct) testing write 0x32745e95 ..FAIL (Invalid argument)
>ioctl 2 (read indirect) testing read 0x32745e95 ..FAIL (Invalid argument)
>ioctl 3 (write + read indirect) testing illegal ..success (fail
>Invalid argument)
>ioctl 3 (write + read indirect) testing w/r 0x7c6728e9 ..FAIL (Invalid argument)
>ioctl 4 (large read indirect) testing ..FAIL (Invalid argument)
>ioctl 5 (large write + read indirect) testing ..FAIL (Invalid argument)
>echo done

All the ioctl stuff is bad. Is the enbd_ioctl module loaded?


Peter