Re: Spurious failovers

On Thu, 2014-11-13 at 14:58 +0000, John Horne wrote:

> We had another failover last night, at a slightly different time
> (03:42). What we saw (by monitoring the VRRP traffic with tcpdump) was a
> 'hang' on the master server for around 20 seconds. During that time, the
> secondary server took the VIP but when the interface became active again
> the original master server took the VIP back again. So, in effect, we
> had two failovers (although I gather that the second failover may have
> been due to keepalived comparing the IP addresses and selecting the
> higher one as master given that the priorities are equal.)
> 
Well in the end we decided to run an strace on the keepalive processes
overnight. The problem occurred again last night, and the culprit seems
to have been within a 'select' statement:

03:48:37.966268 select(1024, [4 6 10], [], [], {1, 0}) = 0 (Timeout)
<22.928987>

(The strace command was: strace -tt -T -ff -p ...)
As can be seen the select is called with a timeout value of 1 second,
but it times out at nearly 23 seconds!

This statement occurs in 'lib/scheduler.c ' at line 581:

ret = select(FD_SETSIZE, &readfd, &writefd, &exceptfd, &timer_wait);

I see no problem with the select call, so I can only assume that the
problem is something deep inside either the C library or the kernel.

I can only think that the next step is to try and replace the select
call with poll to see if that makes any difference. Not sure how easy
that will be though.




John.

-- 
----------------------------------------------------
John Horne                   Tel: +44 (0)1752 587287
Plymouth University, UK


------------------------------------------------------------------------------
Download BIRT iHub F-Type - The Free Enterprise-Grade BIRT Server
from Actuate! Instantly Supercharge Your Business Reports and Dashboards
with Interactivity, Sharing, Native Excel Exports, App Integration & more
Get technology previously reserved for billion-dollar corporations, FREE
http://pubads.g.doubleclick.net/gampad/clk?id=157005751&iu=/4140/ostg.clktrk