UDP packet loss when server health check fails

I'm using keepalived to distribute DNS requests (UDP port 53) to a
group of DNS servers.  The farm is using source hashing.  Environment
is RHEL7.2, with the stock keepalived and IPVS.

I'm testing what happens when a health check fails and one host is
taken out of the farm.  My test bed has two farm servers and one
keepalived server.  The keepalived server is using two ethernet
adapters in a bond interface as its primary adapter.  For testing,
I'm using a fake health check of "sh -c '! test -f FLAGFILE'", which
returns success as long as the file doesn't exist, and I create the
fileto provoke a health check failure.

Using dnsperf to generate a stream of 10,000 queries/second, I create the
flag file and keepalived reports taking the real server out of the farm.
dnsperf then reports losing around 100 queries (around 10 milliseconds)
during the transition.

I ran tcpdump to capture the traffic, and I can see on successful queries
the packet is received on bond0 and then re-transmitted out on bond0 with
the destination server's MAC address.  The configuration is using direct
server response so the source and destination IP addresses in the packet
are unchanged when it is transmitted to the farm server.

I checked several of the query-ids that dnsperf reported as missing. tcpdump
saw all of their query packets arriving, but did not show any of them
getting re-sent out the interface.

My questions are:

* Is it realistic to expect that no packets will be dropped during a farm
reconfigure transition?

* If it's not realistic, what can I do to minimize the drops?  10 ms is
not a lot by some standards, but in my environment it could be 100-200
requests that I'd rather see answered.

* My theory is either keepalived dropped the requests, IPVS dropped them,
or something further down the network stack dropped them.  I looked into
the IPVS counters in /proc but didn't see anything that keeps track of errors.
Can anyone suggest other avenues of visibility into finding where the
requests or responses are being lost?

Thanks,

	-- Ed

------------------------------------------------------------------------------
Check out the vibrant tech community on one of the world's most
engaging tech sites, Slashdot.org! http://sdm.link/slashdot