Re: ENBD Issues with client crash / weird errors....

On 8/22/05, Peter T. Breuer <[email protected]> wrote:
> "Also sprach Jonathan Carron:"
> (something in a windows charset, not terribly readably formatted either)!

 -  sorry about that I think it was a result of me copying and pasting
the message to work and sending it from there through terminal
services (all windoze at work).

> 
> Hi - can you put some spaces in your description? It makes my eyes
> blur! I'll try extracting what I need to see, so you can get the idea:
> 
> > running gentoo with kernel 2.6.12.
> 
> What ARCHITECTURE?
>From what I can see a lot of what I wrote was cut out some how so I
appoligize for that.

Two boxes. Client is a i686 , Pentium 3 600 mhz 100 mhz fsb with dual
processors, kernel is compiled for smp. support.

Linux DELL.localdomain 2.6.12-gentoo-r6 #2 SMP Mon Aug 15 17:42:38 EDT
2005 i686 Pentium III (Katmai) GenuineIntel GNU/Linux

 Server is a AMD Althon 1.4 at 266 mhz fsb. Not sure what "i" arch
this is ? 586 ot 686? But their definately both X86.

AMD mnt # uname -a
Linux AMD 2.6.12-gentoo-r6 #1 SMP Thu Aug 11 10:50:41 EDT 2005 i686
AMD Athlon(tm) Processor AuthenticAMD GNU/Linux



> 
> Is enbd definitely working then on 2.6.12?  I got the impression there
> were still a couple of minor issues somewhere (I can't use recent 2.6
> kernels for very long on my laptop as they lock it up under moderate
> graphics use such as moving the mouse ...  ).  I am running 2.6.12.3 at
> the moment and I see occasinal reconnect troubles as I return from
> hiernation, but they're cured by a quick swat of a zero to
> /proc/nbdinfo, so I haven't investigated.
> 
> 
> >   Anyhow at first I would mount the /dev/nda on the client
> > machine as /dev/md2
> 
> You are using enbd as a raid1 component? In that case you need FR1.
- - well at the start I was. Recently it was also tried as mkfs.xfs
/dev/nda1 and then
mount /dev/nda1 /mnt ... so no raid there.  Just a block dev. same
issue. Want to get this up and going before I try to raid anything.

> 
> > my data to the raid array, cp –rv /src/ /dst/. This starts and seems
> > to be going perfect no errors in dmesg. TCP Track showed 4 connections
> > floating about about 800kb/sec to about 1Mbps ish. Great, what I
> 
> Sounds a bit slow ... oh, I suppose that might be 100BT half-duplex.

 - yes you are correct, dumb hub, no switched ether, setting to 100
Full doesn't work well with inter-operability with some of my other
machines.  Have a 100 Switch , but it is not managed either. Both
produce the same issue.

> 
> > expected.  I come back later and the /dev/md2 has about 22 gb on it,
> > and doesn't seem to be filling up anymore.
> 
> Well, why not? (What does raid say it is doing?) 

 - well there is nothing to rebuild at the moment (only one disk, not
two), but I didn't check :(

> 
> > Am I trying to copy too much data?
> 
> No, though it sounds a mite strange as a way to spend time. What is the
> other component of the raid? If it is local, why not write to it with
> the raid in degraded mode, then add in the enbd component and let raid
> take care of syncing, all in the background?
> 
> > Im worried if I just put the drive in and leave it running,
> > what happens if one does indeed fail :| TCPtrack shows the connections
> > are no longer active (moving data).
> 
> But what does enbd say it is doing? /proc/nbdinfo?

I will definately get this information next time. When the box kaks I
have no network connectivity and couldn't previously ssh in. Ive setup
console access now.


> 
> > I do however still see 4 enbd
> > processed running.
> 
> But what does enbd say?

- only from what the dmesg output is all I have so far :(

> 
> > I cannot unmount or stop the raid device, the
> 
> Well, I don't see why you would want to - but it sounds as though the
> connection has broken and enbd is blocking until the connection comes
> back. That's what it is supposed to do, unless told otherwise.
> 
> > terminal just hangs.
> 
> Sounds right.

- ok ? I wasn't aware of that, my mistake.

> 
> > Kill –9 will not kill the enbd-clients at all.
> 
> I would hope not! Why do that? They are just waiting in kernel for the
> network to come back up, I suppose. Anyway, what does the kernel say
> about all this? dmesg? It sounds as though smething has oopsed! Or at
> any rate as though all the requests in the world were stacked up in
> kernel somewhere! dmesg and nbdinfo woud let me say.

- well I did want to get my box up and running! Didn't think about the
echo 0 >/proc/nbdinfo.



> 
> > And sometimes I've noticed the network is not working (cannot ping IP
> 
> There you are.

Ive replaced the onboard 3com nic with an intel one. I do not see a
loss of network  connectivity... I can ssh files back and forth now,
however it still freezes. (by this I mean both boxes have the 4
processes running (enbd-client on the client, and enbd-server on the
server).


> 
> > address of that machine from any other machine on that network). Halt
> 
> There!

also corrected. still no go this setup.

> 
> > or reboot don't work. Process is locked solid! :S… (The server is
> 
> I would hope so.

hmm ok. 

> 
> > fine, I can reconnect no problem after having to hard reset the
> > client)
> 
> But why?  If you want to error out the stacked kernel requests you would
> write a 0 to /proc/nbdinfo.  But as it is it sounds as though the
> networking has died and the enbd device is doing what it should do
> unless you tell it otherwise - waiting for the network to come back.
> 
> The real question is why is networking dead?

bad nic or nic kernel driver...  this issue has been resolved. I have
network connectivitiy, but it still fails.

> 
> 
> > Im totally lost. I've tried different nics, and no luck.
> 
> Why do you think the nics are at fault? What does the kernel say?

only thing I can tell you is from dmesg (what ive recorded), what
other information can I obtain that would help ?


> 
> > Hard was is
> > known to be in good shape. Have tested drives and RAM. Im using enbd
> > 2-4.32 . Again, I could not get 2.4.31 to compile.
> 
> I don't think it is supposed to cmpile against 2.6.12?

that would explain it!   *doh*

> 
> 
> <snip>
> > test 5 success:  0 incorrect blocks
> > flushing buffers..done
> > ioctl 1 (write direct) testing write illegal ..success (fail Invalid argument)
> > ioctl 1 (write direct) testing write 0x469b392d ..FAIL (Invalid argument)   <- FAIL??
> 
> Yes, a bit strange.

I thought so too.

> 
> > ioctl 2 (read indirect) testing read 0x469b392d ..FAIL (Invalid argument)   <- ?
> > ioctl 3 (write + read indirect) testing illegal ..success (fail
> > Invalid argument)
> > ioctl 3 (write + read indirect) testing w/r 0x250d15c7 ..FAIL (Invalid
> > argument)  <- ?
> > ioctl 4 (large read indirect) testing ..FAIL (Invalid argument)   <- ?
> > ioctl 5 (large write + read indirect) testing ..FAIL (Invalid argument)  <-?
> > <snip>
> 
> Looks bad to me.
> 

 yes

> 
> 
> dmesg output (when copying data to the disk, usually only get about 5
> lines or so. )
> 
> EVEN after this happens it continues to copy just fine…. Strange.
> 
> > <snip>
> > ENBD #1168[0]: enbd_rollback (1): rollback req e3fb7398!
> > ENBD #1168[1]: enbd_rollback (2): rollback req e04ed79c!
> > ENBD #1168[2]: enbd_rollback (3): rollback req e3eaa79c!
> 
> Well, you have issues. It has given up and timed out waiting for
> requests. The server is dead. Or the net is dead.
> 
> 
> > ENBD #1469[0]: enbd_ack (3): fatal: Bad handle e3eaa79c != 00000000!
> > ENBD #1476[0]: enbd_ack (3): ignoring ack of req e3eaa79c which slot lacks
> > ENBD #1469[1]: enbd_ack (2): fatal: Bad handle e04ed79c != 00000000!
> 
> It looks like the request acks came in much later, and there was no
> longer a request there.

time out, perhaps not re est. the connection ?

> 
> > ENBD #1476[1]: enbd_ack (2): ignoring ack of req e04ed79c which slot lacks
> > ENBD #1469[2]: enbd_ack (1): fatal: Bad handle e3fb7398 != 00000000!
> > ENBD #1476[2]: enbd_ack (1): ignoring ack of req e3fb7398 which slot lacks
> > ENBD #1168[3]: enbd_rollback (0): rollback req dff734b0!
> ....
> ....
> 
> > ENBD #1168[23]: enbd_rollback (2): rollback req dfc7dac8!
> > ENBD #1168[24]: enbd_rollback (3): rollback req dfc04184!
> > ENBD #1469[21]: enbd_ack (1): fatal: Bad handle e1c24398 != 00000000!
> 
> Yes, serious timeout issues.

again ill re capture the issues , the error messages that appeared are the same.



> 
> >ioctl 1 (write direct) testing write illegal ..success (fail Invalid argument)
> >ioctl 1 (write direct) testing write 0x32745e95 ..FAIL (Invalid argument)
> >ioctl 2 (read indirect) testing read 0x32745e95 ..FAIL (Invalid argument)
> >ioctl 3 (write + read indirect) testing illegal ..success (fail
> >Invalid argument)
> >ioctl 3 (write + read indirect) testing w/r 0x7c6728e9 ..FAIL (Invalid argument)
> >ioctl 4 (large read indirect) testing ..FAIL (Invalid argument)
> >ioctl 5 (large write + read indirect) testing ..FAIL (Invalid argument)
> >echo done
> 
> All the ioctl stuff is bad. Is the enbd_ioctl module loaded?

yes

DELL # lsmod
Module                  Size  Used by
enbd                  194080  0
ipv6                  250784  14
rtc                    13100  0
usbcore               114108  1

and also tried it with
lsmod
Module                  Size  Used by
enbd_ioctl              4564  0
enbd                  194080  1 enbd_ioctl
ipv6                  250784  14
rtc                    13100  0
usbcore               114108  1

not sure I should be doing that however.

Now... I went and swapped the client and server roles. So now the AMD
box is the client. and the Dell 600 mhz is the server. I did get the
same strange error messages while doing the "make test" , however I
was *able* to copy over all 200 gb with out and errors being reported
to the console or dmesg.  I have not yet verified CRC's or file
integrity yet though.    Anyways on to figuring out why this is
crashing! .... I will start the process again tonite and capture some
more information and better document.

If you have another chance, can you send me a quick note of the output
your interested in seeing and I will make sure its included!


Thank you for your time Peter!

Regards,

Jonathan


> 
> 
> Peter
> _______________________________________________
> ENBD mailing list
> [email protected]
> http://lists.community.tummy.com/mailman/listinfo/enbd
>