keepalived + NIC bonding
John Skarbek <[email protected]> Fri, 3 Jul 2015 18:33:48 -0700
| Newsgroups | gmane.linux.keepalived.announce |
|---|---|
| Message-ID | <CAE20NMT89VnRKEMdp1EReZJ0U3jJM5D7F41pHj8A9RA96jASsg@mail.gmail.com> |
--===============5840164714898755657==
Content-Type: multipart/alternative; boundary=001a113640780649a9051a02aaf5
--001a113640780649a9051a02aaf5
Content-Type: text/plain; charset=UTF-8
Good Morning,
I'm wondering if anyone has any advice on configuration when using
keepalived plus the bonding of ethernet devices on a server. I've
discovered a problem with our implementation where despite an IP address
being applied to a bond interface, he's not receiving traffic. I'm still
in the midst of troubleshooting and suspect it may be arp related. Below
I've pasted our scenario with sanitized configs. In this case, IP address
10.1.1.102 does not respond to any traffic. Here's the important part of
our keepalived config:
vrrp_instance test {
state BACKUP
interface bond0
virtual_router_id 254
nopreempt
priority 02
advert_int 1
virtual_ipaddress {
10.1.1.101 dev bond0
10.1.1.102 dev bond0
}
}
In our case we've got quite a few NIC's that participate in a bond0
interface.
DEVICE="bond0"
BOOTPROTO="static"
GATEWAY="10.1.1.1"
IPADDR="10.1.1.100"
IPV6INIT=no
MTU="1500"
NETMASK="255.255.255.0"
NM_CONTROLLED=no
ONBOOT="yes"
TYPE="Ethernet"
BONDING_OPTS="mode=balance-alb miimon=100"
Using 'adaptive-load-balancing' I believe might be one portion of the
problem. The IP's get assigned to the expected interface:
10: bond0: <BROADCAST,MULTICAST,MASTER,UP,LOWER_UP> mtu 1500 qdisc noqueue
state UP
link/ether 14:fe:99:cb:73:43 brd ff:ff:ff:ff:ff:ff
inet 10.1.1.100/24 brd 10.1.1.255 scope global bond0
inet 10.1.1.101/24 brd 10.1.1.255 scope global bond0
inet 10.1.1.102/24 brd 10.1.1.255 scope global bond0
inet6 fe80::16fe:b5ff:fecb:7343/64 scope link
valid_lft forever preferred_lft forever
But when looking at the arp tables on our switches we end up with a varied
set of MAC addresses to IP addresses:
On the switch (switch1) who's active slave NIC is plugged into we get
something like this:
Internet 10.1.1.100 32 14fe.99cb.7343 ARPA Vlan10
Internet 10.1.1.101 2 14fe.99cb.7343 ARPA Vlan10
Internet 10.1.1.102 2 14fe.9944.aa9e ARPA Vlan10
On the switch (switch2) who's inactive slave NIC is plugged into we get
something like this:
Internet 10.1.1.100 33 14fe.99cb.7343 ARPA Vlan10
Internet 10.1.1.101 3 14fe.99cb.7343 ARPA Vlan10
Internet 10.1.1.102 3 14fe.99cb.7343 ARPA Vlan10
Notice that on switch1, he's got a different MAC address for the
troublesome IP address. This is the MAC address applied to the slave
interface, and not that of the bond0 interface.
I'm struggling to put together an appropriate packet capture. All I end up
seeing is something such as this:
13:03:11.270093 ARP, Request who-has 10.1.1.102 (ff:ff:ff:ff:ff:ff) tell
10.1.1.102, length 46
This single message is repeated about 10 times, in groups of 5 about 4
seconds apart.
I'm sure there's more information that can be gathered to assist with
troubleshooting, I'm all ears.
Secondly, is this the appropriate place to ask such a question? I wasn't
able to find another mailing list.
-- John
--001a113640780649a9051a02aaf5
Content-Type: text/html; charset=UTF-8
Content-Transfer-Encoding: quoted-printable
<div dir=3D"ltr"><div class=3D"gmail_signature"><div dir=3D"ltr"><span><div=
dir=3D"ltr" style=3D"margin-left:0pt"><span style=3D"font-size:12.80000019=
07349px">Good Morning,</span><div style=3D"font-size:12.8000001907349px"><b=
r></div><div style=3D"font-size:12.8000001907349px">I'm wondering if an=
yone has any advice on configuration when using keepalived plus the bonding=
of ethernet devices on a server.=C2=A0 I've discovered a problem with =
our implementation where despite an IP address being applied to a bond inte=
rface, he's not receiving traffic.=C2=A0 I'm still in the midst of =
troubleshooting and suspect it may be arp related.=C2=A0 Below I've pas=
ted our scenario with sanitized configs.=C2=A0 In this case, IP address 10.=
1.1.102 does not respond to any traffic.=C2=A0 Here's the important par=
t of our keepalived config:</div><div style=3D"font-size:12.8000001907349px=
"><br></div><div style=3D"font-size:12.8000001907349px"><br></div><div styl=
e=3D"font-size:12.8000001907349px"><br></div><div style=3D"font-size:12.800=
0001907349px"><div>vrrp_instance test {</div><div>=C2=A0 state BACKUP</div>=
<div>=C2=A0 interface bond0</div><div>=C2=A0 virtual_router_id 254</div><di=
v>=C2=A0 nopreempt</div><div>=C2=A0 priority 02</div><div>=C2=A0 advert_int=
1</div><div>=C2=A0 virtual_ipaddress {</div><div>=C2=A0 =C2=A0 10.1.1.101 =
dev bond0</div><div>=C2=A0 =C2=A0 10.1.1.102 dev bond0</div><div>=C2=A0 }</=
div><div>}</div><div><br></div><div>In our case we've got quite a few N=
IC's that participate in a bond0 interface. =C2=A0</div><div><div>DEVIC=
E=3D"bond0"</div><div>BOOTPROTO=3D"static"</div><div>GA=
TEWAY=3D"10.1.1.1"</div><div>IPADDR=3D"10.1.1.100"</div=
><div>IPV6INIT=3Dno</div><div>MTU=3D"1500"</div><div>NETMASK=3D&q=
uot;255.255.255.0"</div><div>NM_CONTROLLED=3Dno</div><div>ONBOOT=3D&qu=
ot;yes"</div><div>TYPE=3D"Ethernet"</div><div>BONDING_OPTS=
=3D"mode=3Dbalance-alb miimon=3D100"</div></div><div><br></div><d=
iv>Using 'adaptive-load-balancing' I believe might be one portion o=
f the problem.=C2=A0 The IP's get assigned to the expected interface:</=
div><div><br></div><div><div>10: bond0: <BROADCAST,MULTICAST,MASTER,UP,L=
OWER_UP> mtu 1500 qdisc noqueue state UP</div><div>=C2=A0 =C2=A0 link/et=
her 14:fe:99:cb:73:43 brd ff:ff:ff:ff:ff:ff</div><div>=C2=A0 =C2=A0 inet=C2=
=A0<a href=3D"http://10.1.1.100/24" target=3D"_blank">10.1.1.100/24</a>=C2=
=A0brd 10.1.1.255 scope global bond0</div><div>=C2=A0 =C2=A0 inet=C2=A0<a h=
ref=3D"http://10.1.1.101/24" target=3D"_blank">10.1.1.101/24</a>=C2=A0brd 1=
0.1.1.255 scope global bond0<br></div><div>=C2=A0 =C2=A0 inet=C2=A0<a href=
=3D"http://10.1.1.102/24" target=3D"_blank">10.1.1.102/24</a>=C2=A0brd 10.1=
.1.255 scope global bond0<br></div><div>=C2=A0 =C2=A0 inet6 fe80::16fe:b5ff=
:fecb:7343/64 scope link</div><div>=C2=A0 =C2=A0 =C2=A0 =C2=A0valid_lft for=
ever preferred_lft forever</div></div><div><br></div><div>But when looking =
at the arp tables on our switches we end up with a varied set of MAC addres=
ses to IP addresses:</div><div><br></div><div><div>On the switch (switch1) =
who's active slave NIC is plugged into we get something like this:</div=
><div><div>Internet =C2=A010.1.1.100 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=
=A0 32 =C2=A0 14fe.99cb.7343 =C2=A0ARPA =C2=A0 Vlan10</div><div>Internet =
=C2=A010.1.1.101 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A02 =C2=A0 1=
4fe.99cb.7343 =C2=A0ARPA =C2=A0 Vlan10</div><div>Internet =C2=A010.1.1.102 =
=C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A02 =C2=A0 14fe.9944.aa9e=C2=
=A0 ARPA =C2=A0 Vlan10</div></div></div><div><br></div><div>On the switch (=
switch2) who's inactive slave NIC is plugged into we get something like=
this:</div><div><div>Internet =C2=A010.1.1.100 =C2=A0 =C2=A0 =C2=A0 =C2=A0=
=C2=A0 =C2=A0 33 =C2=A0 14fe.99cb.7343 =C2=A0ARPA =C2=A0 Vlan10</div><div>=
Internet =C2=A010.1.1.101 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A03=
=C2=A0 14fe.99cb.7343 =C2=A0ARPA =C2=A0 Vlan10</div><div>Internet =C2=A010=
.1.1.102 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A03 =C2=A0 14fe.99cb=
.7343 =C2=A0ARPA =C2=A0 Vlan10</div></div><div><br></div><div>Notice that o=
n switch1, he's got a different MAC address for the troublesome IP addr=
ess.=C2=A0 This is the MAC address applied to the slave interface, and not =
that of the bond0 interface. =C2=A0</div><div><br></div><div>I'm strugg=
ling to put together an appropriate packet capture.=C2=A0 All I end up seei=
ng is something such as this:</div><div><br></div><div>13:03:11.270093 ARP,=
Request who-has 10.1.1.102 (ff:ff:ff:ff:ff:ff) tell 10.1.1.102, length 46<=
br></div><div><br></div><div>This single message is repeated about 10 times=
, in groups of 5 about 4 seconds apart.</div><div><br></div><div>I'm su=
re there's more information that can be gathered to assist with trouble=
shooting, I'm all ears.</div><div><br></div><div>Secondly, is this the =
appropriate place to ask such a question? I wasn't able to find another=
mailing list.</div><div><br></div><div>-- John</div></div></div></span></d=
iv></div>
</div>
--001a113640780649a9051a02aaf5--
--===============5840164714898755657==
Content-Type: text/plain; charset="us-ascii"
MIME-Version: 1.0
Content-Transfer-Encoding: 7bit
Content-Disposition: inline
------------------------------------------------------------------------------
Don't Limit Your Business. Reach for the Cloud.
GigeNET's Cloud Solutions provide you with the tools and support that
you need to offload your IT needs and focus on growing your business.
Configured For All Businesses. Start Your Cloud Today.
https://www.gigenetcloud.com/
--===============5840164714898755657==
Content-Type: text/plain; charset="us-ascii"
MIME-Version: 1.0
Content-Transfer-Encoding: 7bit
Content-Disposition: inline
_______________________________________________
Keepalived-announce mailing list
[email protected]
https://lists.sourceforge.net/lists/listinfo/keepalived-announce
--===============5840164714898755657==--