debugging page allocation failure

I have a two keepalived + haproxy pairs that are experiencing memory 
problems.  Other systems, including those running haproxy without 
keepalived do no experience the same problems.  The failover 
keepalived/haproxy pair is also unaffected.  The problem takes two forms:

  * Memory used grows over time.  Memory used by individual userspace 
applications do now account for this growth. http://imgur.com/PToSG0m
  * Every few weeks the primary will fail.  When this happens many 'page 
allocation failure' failures occur, with some 'BUG: soft lockup' 
springled in.  This lasts for a few minutes during which I can not ssh 
into the box.  The allocation failures are associated with a variety of 
processes including snmpd and sh.

I'm posting here because the problem only seems to occur on boxes where 
I am running keepalived.  However, I'm unsure if it is a keepalived 
'bug', the kernel, or something else.  Pointers in the right direction 
would be appreciated.


Versions:
  * Centos6
  * keepalived: 1.2.13
  * kernel: 2.6.32-504.1.3.el6.x86_64

This isn't the latest kernel at this moment, but we have upgraded the 
kernel and keepalived in the past without success.

Example:
snmpd: page allocation failure. order:0, mode:0x20
Pid: 1556, comm: snmpd Not tainted 2.6.32-504.1.3.el6.x86_64 #1
Call Trace:
  <IRQ>  [<ffffffff8113438a>] ? __alloc_pages_nodemask+0x74a/0x8d0
  [<ffffffff81497098>] ? ip_local_deliver+0x98/0xa0
  [<ffffffff8116c69a>] ? alloc_pages_current+0xaa/0x110
  [<ffffffffa018a063>] ? igb_alloc_rx_buffers+0xe3/0x1b0 [igb]
  [<ffffffff814606b0>] ? napi_skb_finish+0x50/0x70
  [<ffffffffa018cb36>] ? igb_poll+0x726/0x1010 [igb]
  [<ffffffffa018cf56>] ? igb_poll+0xb46/0x1010 [igb]
  [<ffffffff814b59e0>] ? tcp_delack_timer+0x0/0x270
  [<ffffffff814b3b19>] ? tcp_send_ack+0xd9/0x120
  [<ffffffff814620a3>] ? net_rx_action+0x103/0x2f0
  [<ffffffff8107d8b1>] ? __do_softirq+0xc1/0x1e0
  [<ffffffff810eaa90>] ? handle_IRQ_event+0x60/0x170
  [<ffffffff8107d90f>] ? __do_softirq+0x11f/0x1e0
  [<ffffffff8100c30c>] ? call_softirq+0x1c/0x30
  [<ffffffff8100fc15>] ? do_softirq+0x65/0xa0
  [<ffffffff8107d765>] ? irq_exit+0x85/0x90
  [<ffffffff81533b65>] ? do_IRQ+0x75/0xf0
  [<ffffffff8100b9d3>] ? ret_from_intr+0x0/0x11
  <EOI>  [<ffffffff8152ce8e>] ? _spin_lock+0x1e/0x30
  [<ffffffff8128c1e5>] ? _atomic_dec_and_lock+0x55/0x80
  [<ffffffff811a995a>] ? dput+0x9a/0x150
  [<ffffffff8118f969>] ? __fput+0x189/0x210
  [<ffffffff8118fa15>] ? fput+0x25/0x30
  [<ffffffff8118ac6d>] ? filp_close+0x5d/0x90
  [<ffffffff8118ad45>] ? sys_close+0xa5/0x100
  [<ffffffff8100b072>] ? system_call_fastpath+0x16/0x1b
snmpd: page allocation failure. order:0, mode:0x20
Pid: 1556, comm: snmpd Not tainted 2.6.32-504.1.3.el6.x86_64 #1
Call Trace:
  <IRQ>  [<ffffffff8113438a>] ? __alloc_pages_nodemask+0x74a/0x8d0
  [<ffffffffa0192076>] ? igb_xmit_frame+0x56/0xc0 [igb]
  [<ffffffff81173332>] ? kmem_getpages+0x62/0x170
  [<ffffffff81173f4a>] ? fallback_alloc+0x1ba/0x270
  [<ffffffff8117399f>] ? cache_grow+0x2cf/0x320
  [<ffffffff81173cc9>] ? ____cache_alloc_node+0x99/0x160
sh: page allocation failure. order:0, mode:0x20
Pid: 30722, comm: sh Not tainted 2.6.32-504.1.3.el6.x86_64 #1
Call Trace:

BUG: soft lockup - CPU#2 stuck for 67s! [haproxy:2047]

------------------------------------------------------------------------------
Dive into the World of Parallel Programming The Go Parallel Website, sponsored
by Intel and developed in partnership with Slashdot Media, is your hub for all
things parallel software development, from weekly thought leadership blogs to
news, videos, case studies, tutorials and more. Take a look and join the 
conversation now. http://goparallel.sourceforge.net/