Re: Best approach to prevent split brain

Hello Petr,

you can't do that properly with two instances.

For reliable split brain detection you need a consensus algorithm. VRRP isn't 
one. You have to use something like zookeeper, etcd, or consul, and use the 
consensus implemented by that layer to drive the states of your application 
(and maybe keepalived when used as the IP failover solution).

Note that any consensus algorithm needs at least 3 instances, because any 
consensus needs a qualified (>50%) majority.

best regards
  Patrick

On Monday 29 August 2016 08:54:28 Petr Postulka wrote:
> Hi All,
> 
> we started using keepalived and so far it looks very promising. We would
> like to use it for VIP failover/HA but we are missing one important point
> which is how one can avoid split brain ...
> 
> We already posted the question on Github and pqarmitage answered it with a
> note that follow up should be posted onto this mailing list.
> 
> For initial question and response please see
> https://github.com/acassen/keepalived/issues/408
> 
> To follow up pqarmitage answer .. we were told to not use separate
> communication link for VRRP but to use the same interface where VIP is
> running - this really makes sense and thank you pqarmitage for that advice.
> But if I understand it correctly then even with that configuration there
> could be a state that both application instances are running in MASTER mode
> and therefore some clients can connect to one instance and some other
> clients to the second instance and because our application has a state
> which has to be replicated we always require that there is only one and
> exactly one MASTER in a pair and then 0 or one HOT STANDBY instance.
> 
> To give you a better picture of our use case here is the full description:
> 
> - We have two separate boxes on which we are running our application. This
> application is listening for incoming connections and doing some job. The
> incoming connections are TCP based and the application has a state. To
> support HA of our application we implemented a state replication between
> two instances of our application. The instance can run in two modes:
> 
> A) Master - the incoming TCP-based connections are established and it is
> processing all client requests and doing some job. Every state change is
> replicated to second (Hot Standby) instance.
> 
> B) Hot standby - application is running but the TCP-based connections are
> not established. Hot standby instance is only listening for replicated
> state and updated from master instances. Once the master application stops
> working hot standby application should switch to MASTER mode, start
> listening for incoming TCP-based connections and start processing the
> requests.
> 
> Can you provide us with more information how exactly we should configure
> keepalived to best meet our requirements and more importantly to prevent
> any false state which could cause some unexpected behavior of our platform?
> 
> Thank you very much for any additional advice.
> 
> Kind regards,
> Petr


------------------------------------------------------------------------------