Re: DRBD failover issue, kernal panic no init found

I also just want to verify that running x86_64 on the Opterons does not
require any modification to the drbd package.  The kernel we are using
is the kernel-ssi-source-2.4.21-27.0.2.EL_ssi_2.x86_64.rpm.


Larry Brown
Network Engineer

-----Original Message-----
From: [email protected]
[mailto:[email protected]] On Behalf Of
Brown, Larry
Sent: Monday, July 10, 2006 9:29 AM
To: Roger Tsang; [email protected]
Subject: Re: [SSI-users] DRBD failover issue, kernal panic no init found

I am working on this same server with Vijay and can add some additionals
info...

The scripts you refer to such as rc.sysrecover are part of the
installation instructions to be copied from the drbd folder and have the
prior versions overwritten.  This was done so the drbd versions are in
place.

I checkd the log on the "slave" and it too reads:

drbd0: drbdsetup [131257]: cstate Unconfigured --> StandAlone
drbd0: drbdsetup [131260]: cstate StandAlone --> Unconnected
drbd0: drbd0_receiver [131261]: cstate Unconnected --> WFConnection
drbd0: Registering drbd0 with CLMS subsystem
drbd0: drbd0_receiver [131261]: cstate WFConnection --> WFReportParams
drbd0: Handshake successful: DRBD Network Protocol version 74
drbd0: Connection established.

I went into the drbd.o code and added a printk line at the beginning of
the nodedown function to echo back that it is being hit.  When we killed
the master the slave gave its typical kernel panic message and my printk
showed up one line below the kernel panic message on the console.  I
thought this unusual but it gives me the feeling that drbd is trying to
do its thing when the kernel gives up the ghost trying to "find init".




Larry Brown
Network Engineer

-----Original Message-----
From: [email protected]
[mailto:[email protected]] On Behalf Of
Roger Tsang
Sent: Saturday, July 08, 2006 11:53 AM
To: Swami, Vijay
Cc: [email protected]
Subject: Re: [SSI-users] DRBD failover issue, kernal panic no init found

Have you checked whether you get the DRBD register message on your
*slave* machine?

Roger

On 7/8/06, Vijay Swami <[email protected]> wrote:
> I am running drbd-ssi. I checked and I do get the DRBD register 
> message, and the other two when the machine boots. I was kind of 
> hoping that was the problem, as it makes sense. :)
>
> Here is the interesting thing... when I fail the *slave*, I do see the

> proper DRBD messages on the master, i.e. the DRBD nodedown. However, 
> that doesn't happen when I fail the *master*.
>
> Also, this is RHEL3, using drbd-ssi for FC2. Although one is based off

> the other, I'm wondering if there is something weird going on there. 
> For instance, the drbd-ssi mkinitrd flat out does not work for me out 
> of the box. If I do mkinitrd --cfs --drbd the initrd it generates will

> not boot. It doesn't even seem to be executing linuxrc properly. I.E. 
> The first line of linuxrc I put echo 'Test', and it prints: "Kernel
Panic:
> No init found" before even printing that line.
>
> Very odd.
>
> I'm about to try Debian Sarge on the machine, and see how that goes 
> with DRBD. Or perhaps hack FC2 kernel to work with my hardware (mainly

> SCSI driver is needed).
>
> I suspect while FC2, OpenSSI 1.2.2, drbd-ssi-1.2.2-20050712 are a 
> known working combo, there might be something off with RHEL3.
>
> /vijay
>
> On Sat, 2006-07-08 at 00:48 -0400, Roger Tsang wrote:
> > Hi,
> >
> > I see your screenshot.  I don't see any drbd0 nodedown messages 
> > there, so there are a few possibilities:
> >
> > 1.  You are running vanilla drbd.
> > 2.  You are running drbd-ssi, but drbd-ssi never registered your 
> > device as a CLMS service.  Check that drbd registered with OpenSSI 
> > by looking for the following console messages when you first start
drbd.
> >
> > drbd0: drbdsetup [197114]: cstate Unconfigured --> StandAlone
> > drbd0: drbdsetup [197117]: cstate StandAlone --> Unconnected
> > drbd0: drbd0_receiver [197118]: cstate Unconnected --> WFConnection
> > drbd0: Registering drbd0 with CLMS subsystem
> >
> >
> > Roger
> >
> >
> > On 7/7/06, Vijay Swami <[email protected]> wrote:
> > > I don't think its an SSIfailover, or rc.sysrecover issue. It seems

> > > that it tries to re-spawn INIT, without failing over DRBD first. 
> > > If I try cfs_setroot ext3 /dev/drbd/0 on a node, nothing seems to 
> > > happen. Is that correct behavior?
> > >
> > > I'm attaching a JPEG screen-shot of the error. Hopefully it goes 
> > > through.
> > >
> > > /vijay
> > >
> > > On Fri, 2006-07-07 at 17:56 -0400, Roger Tsang wrote:
> > > > I already migrated to SSI-1.9.x with the latest DRBD which is 
> > > > based on drbd-0.7.20 but haven't checked in the update yet.  The

> > > > latest one in CVS is based on drbd-0.7.19 and works fine unless 
> > > > you changed your drbd devices' al-extents parameter.  In that 
> > > > case you would want drbd-ssi based on drbd-0.7.20.
> > > >
> > > > It seems to me, without much info, your SSIfailover and 
> > > > rc.sysrecover system scripts weren't setup properly.  You have 
> > > > to modify them for drbd for root filesystem failover.  The
> > > > drbd-ssi-1.2.2-20050712 tarball should include a sample of these
two system scripts.
> > > >
> > > > Roger
> > > >
> > > >
> > > > On 7/7/06, Vijay Swami <[email protected]> wrote:
> > > > > Roger,
> > > > >
> > > > > Its drbd-ssi-1.2.2-20050712. It seems like as soon as the 
> > > > > master goes down, the slave does not resort to using its own 
> > > > > copy. I don't see any
> > > > > EXT3 FSCK messages like I've seen by searching the list for 
> > > > > successful fail over messages. The error is most certainly it 
> > > > > can't find /sbin/init probably because once the connection to 
> > > > > the master dies, its not properly using its 'own' copy.
> > > > >
> > > > > I'm wondering if this is related to the Debian /dev/drbd0 vs 
> > > > > /dev/drbd/0 issue that I've read about. I noticed you said you

> > > > > ran into this problem on FC2, which is what RHEL3 is based on.

> > > > > Where you using OpenSSI 1.2.x or 1.9.x and which version of
DRBD?
> > > > >
> > > > > Thanks.
> > > > >
> > > > > /vijay
> > > > >
> > > > > On Thu, 2006-07-06 at 22:50 -0400, Roger Tsang wrote:
> > > > > > Are you using drbd-ssi (openSSI modified drbd) and have 
> > > > > > enabled SSI-failover service?
> > > > > >
> > > > > > If you like to try the latest drbd-ssi I'll be making a 
> > > > > > drbd-ssi tarball based on drbd-0.7.20, hopefully sometime
this week.
> > > > > >
> > > > > > Roger
> > > > > >
> > > > > >
> > > > > > On 7/6/06, Vijay Swami <[email protected]> wrote:
> > > > > > > Pertinent details:
> > > > > > >  * 2 node cluster
> > > > > > >  * RHEL3
> > > > > > >  * Kernel/OpenSSI ver: 2.4.21-27.0.2.EL_ssi_2 (install 
> > > > > > > from RPM)
> > > > > > >  * drbd version 0.7.11 (api:77)
> > > > > > >  * SunFire X4100 2 CPU AMD Opteron hardware
> > > > > > >
> > > > > > > drbd.conf snippet:
> > > > > > >
> > > > > > >  on host1 {
> > > > > > >     device     /dev/drbd/0;
> > > > > > >     disk       /dev/sda2;
> > > > > > >     nodenum    1;
> > > > > > >     address    192.168.1.1:7788;
> > > > > > >     meta-disk  /dev/sda3[0];
> > > > > > >
> > > > > > >
> > > > > > >   on host2 {
> > > > > > >     device    /dev/drbd/0;
> > > > > > >     disk      /dev/sda2;
> > > > > > >     nodenum   2;
> > > > > > >     address   192.168.1.2:7788;
> > > > > > >     meta-disk /dev/sda3[0];
> > > > > > >
> > > > > > > /etc/fstab:
> > > > > > >
> > > > > > > /dev/drbd/0 / ext3
> > > > > > > defaults,chard,errors=remount-ro,node=1:2 0 1
> > > > > > > /dev/sda1 /boot ext3 defaults,node=1:2 1 2
> > > > > > > none    /dev/pts        devpts  gid=5,mode=620,node=*   0
0
> > > > > > > none    /proc   proc    defaults,node=* 0       0
> > > > > > > #none                    /dev/shm                tmpfs
defaults
> > > > > > > 0 0
> > > > > > >
> > > > > > > df output:
> > > > > > >
> > > > > > > Filesystem           1K-blocks      Used Available Use%
Mounted on
> > > > > > > /dev/1/drbd/0         10080520   2329744   7238708  25% /
> > > > > > > /dev/1/sda1             248895     61612    174433  27%
/boot
> > > > > > >
> > > > > > > The root file system is configured with DRBD, and working 
> > > > > > > great. Each node can act as an init node when booted up 
> > > > > > > individually. The other node joins the cluster, and things
work great.
> > > > > > >
> > > > > > > However, I'm having trouble with the failover.
> > > > > > >
> > > > > > > If I 'unplug' the master node, the slave recognizes the 
> > > > > > > master has gone down, and attempts to take over. It prints

> > > > > > > DRBD timeout messages to the console (as expected), then:
> > > > > > >
> > > > > > > ipcnameserver ready
> > > > > > > Kernel Panic: no init found. Cannot restart
> > > > > > >
> > > > > > > .. and then it reboots.
> > > > > > >
> > > > > > > I have a feeling its something fairly simple here, as DRBD

> > > > > > > itself works fine when the nodes are booted. I'm assuming 
> > > > > > > its looking for /sbin/init to start on the slave node but 
> > > > > > > can't find it? I don't know why since DRBD itself works on

> > > > > > > either node acting as an init node when they are booted
regardless of sequence.
> > > > > > >
> > > > > > > Any ideas?
> > > > > > >
> > > > > > > Thanks.
> > > > > > >
> > > > > > > /vijay
> > > > > > >
> > > > > > > Using Tomcat but need to do more? Need to support web
services, security?
> > > > > > > Get stuff done quickly with pre-integrated technology to 
> > > > > > > make your job easier Download IBM WebSphere Application 
> > > > > > > Server v.1.0.1 based on Apache Geronimo
> > > > > > > http://sel.as-us.falkag.net/sel?cmd=lnk&kid=120709&bid=263
> > > > > > > 057&dat=121642
> > > > > > > _______________________________________________
> > > > > > > Ssic-linux-users mailing list 
> > > > > > > [email protected]
> > > > > > > https://lists.sourceforge.net/lists/listinfo/ssic-linux-us
> > > > > > > ers
> > > > > > >
> > > > > >
> > > > >
> > > >
> > >
> > >
> > >
> >
>

Using Tomcat but need to do more? Need to support web services,
security?
Get stuff done quickly with pre-integrated technology to make your job
easier Download IBM WebSphere Application Server v.1.0.1 based on Apache
Geronimo
http://sel.as-us.falkag.net/sel?cmd=lnk&kid=120709&bid=263057&dat=121642
_______________________________________________
Ssic-linux-users mailing list
[email protected]
https://lists.sourceforge.net/lists/listinfo/ssic-linux-users



------------------------------------------------------------------------
-
Using Tomcat but need to do more? Need to support web services,
security?
Get stuff done quickly with pre-integrated technology to make your job
easier Download IBM WebSphere Application Server v.1.0.1 based on Apache
Geronimo
http://sel.as-us.falkag.net/sel?cmd=lnk&kid=120709&bid=263057&dat=121642
_______________________________________________
Ssic-linux-users mailing list
[email protected]
https://lists.sourceforge.net/lists/listinfo/ssic-linux-users



-------------------------------------------------------------------------
Using Tomcat but need to do more? Need to support web services, security?
Get stuff done quickly with pre-integrated technology to make your job easier
Download IBM WebSphere Application Server v.1.0.1 based on Apache Geronimo
http://sel.as-us.falkag.net/sel?cmd=lnk&kid=120709&bid=263057&dat=121642