Re: ENBD Issues with client crash / weird errors....

On 8/24/05, Peter T. Breuer <[email protected]> wrote:
> "Also sprach Jonathan Carron:"
> > Im totally lost as to how to install the enbd fr1 module. When I
> > download the module off
> > ftp://oboe.it.uc3m.es/pub/Programs/fr1-2.current.tgz , it doesn't work
> > because of linux 2.6. Ok... so looking in
> > nbd-2.4.32/kernel/linux-2.6.x  I find the kernel patches. Looked in
> > the make file, it looks like it should go in the root of the kernel
> > directory  (where I have it extracted).
> 
> 
> Eh? NO - don't invent procedures! Just do what it says. Read the
> README.

cat /nbd-2.4.32/README | grep -i "patch"  
<nothing>
?
cat /nbd-2.4.32/INSTALL | grep -i "patch"  
<nothing> 
?


I see in INSTALL
"
BUILD instructions

It should be sufficient to type "make config all" in this directory.
But check first

  0) that you have the kernel sources installed

  1) that the kernel source directory LINUXDIR in the Makefile is correct

  2) that you (a) set SMP=1 in the Makefile if your target kernel is SMP;
     and (b) that there isn't an old config.cache hanging around in the nbd
     subdirectory when you run the make config.

"..... nothing about patches
 
> 
> > I backed up the directory and
> > tried that and its overwriting the Makefile for the kernel. I know im
> > doing something very wrong but I don't know what.
> 
> Not reading the instructions.

What instructions ?!!?

DELL doc # cat DIARY  |grep -i "patch"
  was the xfs patch that I applied.
DELL doc # pwd
/root/nbd-2.4.32/nbd/doc
DELL doc # cat PROTOCOL | grep -i "patch"


Its not that im not looking, I've read the files & docs with that I
have , there is nothing here that is telling me what I need to do to
patch.

> 
> > the nbd-2.4.32/kernel/patches directory has no patches to apply to
> > kernel 2.6.12 ... latest is 2.6.8.... and I don't meet the criteria,
> > im not sure applying one is a good idea, nor am I running debian, and
> > I don't know what im doing at this point :)
> 
> 
> I don't understand what you are doing or why you think you should be
> doing whatever you think you should be doing...

> 
> The enbd 2.4.32 kernel patches directory has plenty of patches to apply
> to 2.6.12 (all of them would work, more or less), but why should you do
> such a thing?  The code is standalone and elsewhere - the patches are
> really only to the kernel configuration system, and you can use them as
> you wish, and besides, there is a big notice saying

its the less part that worries me. Thats ok though, I won't patch anything .

> 
>   THESE_ARE_NOT_KERNEL_PATCHES

ahem.

DELL patches # cat THESE_ARE_NOT_KERNEL_PATCHES  | head -n 20
Well, oh yes they are - but you are not supposed to apply these to the
kernel, with two notable exceptions.

These patches are historical snapshots.  They are not up to date, though
they might well work.  They were made by simply combining the kernel code
you will find in the linux subdirectory near here with certain constant
patches for the CONFIGURATION mechanisms of the linux kernel.

The two exceptions to when you may apply these patches:

   1) The latest 2.6 patch is up to date and represents the best and
      only code for 2.6 kernels.

=> so you want me to apply 2.6.8 to 2.6.12 ?  
=> I don't know how ? patch -p <  ???


   2) if you are running a debian-patch install on an old kernel, these
      ARE the patches that will be used (and good luck to you).

=> nope, gentoo

There is a third exception, of course:

   3) if you know what you are doing.

=> at this point, no. 

 
> in that directory.
> 
> >
> > > Are you running with "show_errors" set, by the way? It probably would be
> > > helpful if you were to do so (as well as appropriate for FR1, when yu
> > > use that).
> >
> > Can you give me some more information on how to  show errors ?  I
> 
> I don't understand exactly what you mean - "show_errs" is a module
> param. It can be set through modprobe and /proc/nbdinfo and /proc/sys/dev
> and so on.

I wasn't aware that there was this option. Now I know. Ill set it .

> 
> > don't see  it on the command line --help ?  (and I'm not as familiar
> > with things in the Linux world as you, unfortunately, but I'm trying:)
> 
> You MUST be familiar with linux in order to be a sysadmin.

familiar yes
but I am not a Master Linux Certified Engineer! 

> 
> > > But you described serverside issues, no? You showed traces which said
> > > the server was not replying!

The reason the server looked like it was not replying is because the
client could not reach it. It _does not_ mean that the server crashed!
 Ive stated many times that the server has never had a problem! 
Client! always :(


> > >
> > >
> >
> > No - - server has always been fine. the machine that runs enbd-client
> > is the the one which locks up .
> 
> But the log showed that the server was not replying. What makes you
> think the server is OK when the log says it is NOT?

yes server is up. Reset client, connect again , and life goes on
without a hitch until the next crash.

> 
> >
> > > What do you mean by "crashed"? A really crashed kernel will leave an oops.
> > >
> >
> > well with a monitor and keyboard pushing enter does *nothing* ... no
> > alt + F1 feature , no shift pg up and down. Just *dead*.   Again I
> 
> What does sysreq tell you?
again, no response.

ctrl + break = nil.


> 
> > > That's usually nothing much to do with the machine. The question is what
> > > is the kernel state at this point.  Show debug info from the machine you
> > > say is locked or crashed at this point ..
> >
> > How  ?  I cannot perform any commands.
> 
> You use the alt sysreq monitor set of commands.  A serial console would
> be fine for such things (you can do a net console too, but I have never
> used one so I have no experience).

can you elaborate a little bit perhaps ? I havea  serial console as
well as a real console...

> 
> > SSH is dead. I have a serial
> > console, that's DEAD, and the KB console is dead. (no response).
> 
> The serial console should respond to "break" as sysreq.

nope.

> 
> > At least before w/o playing with /proc/kerne/vm I could type, so again
> > I think that was the issue.
> 
> If you can make a difference by playing there, it would seem to be a
> possible focus of invstigation. What is different with the machine
> you say everything works on, there?

Ok this will answer the below question as well.


Scenario 1: (current)

I have two computers
Computer #1 (hostname=dell) which is running ENBD-Client + module
- Dual P3 processor system.
- 512 MB RAM

Computer #2 (hostname=amd) which is running ENBD-Server
- AMD Althon 1.4 Ghz
- 512 MB Ram



Now, with computer #1 (dell) running the client and computer #2(amd)
running the server I am experiencing the issues.  the computer
#1(dell) has all the hard drives in it and I want to take a drive from
computer #2 (amd) and add it to an array on the dell.

This is the scenario above.

Scenario 2: (below)

Now last week sometime I swapped the roles of the 2 pc's. This is not
how its setup and I really do not want it setup this way either. Im
just mentioning this because when the roles are reversed the enbd
system works, perfectly.

SO computer #1 (dell) has the enbd-server running on it
Computer #2 (amd) has the client + module running on it.

Copy 160 gb of data to a raid device on computer #2 - -  no problem. 4
daemons and all! works.

But like I said this situation isn't ideal. The amd is very loaded
down and performs poorly because of that for SMB share. Thus the DELL
(dual processor ) is used, because it has no other task.   So scenario
2 works , but doesn't fit.

Scenario 1 is what im trying to get working.


> 
> Is one of these machines SMP? How many cpus does /proc/cpuinfo show?
> 
Yes See above.

DELL proc # cat cpuinfo
processor       : 0
vendor_id       : GenuineIntel
cpu family      : 6
model           : 7
model name      : Pentium III (Katmai)
stepping        : 3
cpu MHz         : 448.998
cache size      : 512 KB
fdiv_bug        : no
hlt_bug         : no
f00f_bug        : no
coma_bug        : no
fpu             : yes
fpu_exception   : yes
cpuid level     : 2
wp              : yes
flags           : fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge
mca cmov pat pse36 mmx fxsr sse
bogomips        : 884.73

processor       : 1
vendor_id       : GenuineIntel
cpu family      : 6
model           : 7
model name      : Pentium III (Katmai)
stepping        : 3
cpu MHz         : 448.998
cache size      : 512 KB
fdiv_bug        : no
hlt_bug         : no
f00f_bug        : no
coma_bug        : no
fpu             : yes
fpu_exception   : yes
cpuid level     : 2
wp              : yes
flags           : fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge
mca cmov pat pse36 mmx fxsr sse
bogomips        : 894.97


Also Some of the logs from yesterday were recoverable ... I saw them
after I rebooted and looked at them. Ill send them to you after this
message.

In regards to running it with 1 daemon controlling the speed, its
stull running but looks to be successful .. has not crashed yet, but
it has not finished either.


TCPTrack

 Client                Server                State        Idle A Speed
 192.168.0.5:41872     192.168.0.180:1036    ESTABLISHED  0s   * 1 MB/s

 * I saw speeds from 800 - 1 Mb / sec. This was done during this
command... whew this could take all night :)

dd if=/dev/zero of=/dev/nda1

After a few minutes of the above command

DELL ~ # free
             total       used       free     shared    buffers     cached
Mem:        579924     514400      65524          0     383736      89612
-/+ buffers/cache:      41052     538872
Swap:       996020          0     996020

2 hours later
free
             total       used       free     shared    buffers     cached
Mem:        579924     573356       6568          0     519948       8480
-/+ buffers/cache:      44928     534996
Swap:       996020        824     995196

The next day (still going)
DELL proc # free
             total       used       free     shared    buffers     cached
Mem:        579924     573476       6448          0     521204       8244
-/+ buffers/cache:      44028     535896
Swap:       996020        824     995196

(we know who DELL and AMD are) 
Settings:  /etc/enbd.conf , on each respective machine.

client   0  /dev/nda AMD 1035 -n 1            -m -b 4096

server  0       1035 /dev/hda1



when I wipe hda1 I get  (I did this because you said without a
filesystem) so no ext2 / 3 /xfs ..etc.

 unable to read partition table
Buffer I/O error on device nda, logical block 1
ENBD #883[5]: enbd_open DISK CHECK wanted in open on device nda
ENBD #4532[5]: enbd_media_changed MEDIA CHANGED called
ENBD #4538[5]: enbd_media_changed REMOTE CHECK done locally, no remote ioctl!
ENBD #4636[10]: enbd_revalidate REVALIDATE called on nda
ENBD #4642[10]: enbd_revalidate asked to revalidate remotely dead drive nda
ENBD #4636[11]: enbd_revalidate REVALIDATE called on nda
ENBD #4642[11]: enbd_revalidate asked to revalidate remotely dead drive nda
 nda:<4>printk: 7 messages suppressed.
Buffer I/O error on device nda, logical block 0
ldm_validate_partition_table(): Disk read failed.
 unable to read partition table


So I have to make a partition again to get enbd to work. If I don't, I get 
 fdisk /dev/nda

Unable to read /dev/nda

Not sure if thats by design or not.


Thanks again

Jonathan