Re: Constant core dumps...

[email protected] wrote:

 >
 >I did get core dumps even using glibc 2.2.2 on RH 7.1. It happened when I
 >mistakenly kept the default value of '/dev/ttyd2' for the reset port. When
 >restarting, `/' was quickly filled up with core dumps. I fixed this by
 >manually replacing '/dev/ttyd2' by `meatware' in the cdb files (in
 >`/var/failsafe' ?). However, when restarting, the nodes I had defined for
 >my test cluster had disappeared. (Is there any other special configuration
 >to make `meatware' work? I didn't see anything in the archieves.)
 >
Adding to your problem about the system controller port, I have found
out(don't know if this is a bug or not) that if you don't define or
enable the reset port, you'll get big problems.(I don't have any power
controllers yet & i'm running LFS on two nodes)

After looking at all the logs, when ha_cmsd detects that a node isn't
responding it tries to send a reset message to that node via ha_crsd.
  Since ha_crsd can't send the message(no controller), the reset message
times out, and ha_cmsd ends up confused.  The downed node status is
changed to unknown.  This causes the entire cluster state to switch to
unknown(whereby I couldn't even move a resource group from the downed
node to another healthy node).

What is even worse is that ha_cmsd goes lonely and restarts(via
ha_cmond).  ha_gcd detects ha_cmsd is lonely, and restarts too.  After
ha_cmsd restarts, it goes lonely again, and the whole process of
restarting is done again.  I believe because the cluster state is
unknown, it is unable to determine an active membership.

All this time, ha_srmd is waiting for ha_gcd to respond, and ha_fsd is
waiting for ha_srmd to respond. Deadlock, because ha_gcd is constantly
restarting due to ha_cmsd.  That's why I can't move resource groups.

Granted, if I had a power controller, none of this would have
happened...but if a single controller fails, wouldn't this would put the 
entire cluster into an unknown state?  I had the "Powerfail" option 
enabled on both nodes, but still no avail.

As a temporary work-around, I hacked in some code in crsd_pending.c,
where reset messages are being sent, and compiled FailSafe again.  The
code just checks if my network (crossover) link(between 2 nodes) is
down, and if so, replies back with a success.  So when I manually turn
off one of the nodes, I can mimic a 'failure'.

BTW, Can anyone recommend good external system controllers or stonith
devices that work properly with FailSafe?

-- Kashif