Re: Igfrp IP Distribution Function
Stephen Stocke <[email protected]> Tue, 4 Feb 2025 23:31:41 +0000
| Newsgroups | gmane.comp.hardware.netapp |
|---|---|
| Message-ID | <CAEGA2qZJ=eTzNMrVSxwWf4Nzus7k=-PKffUG_R=1z-B_jS7yZg@mail.gmail.com> |
--===============1791380735877941590== Content-Type: multipart/alternative; boundary="000000000000e88d76062d596918" --000000000000e88d76062d596918 Content-Type: text/plain; charset="UTF-8" Content-Transfer-Encoding: quoted-printable Hello Thank you to everyone who replied on and off list. I believe I have a good understanding of ifgrp configuration and the purpose of the various distribution functions. In this case I'm asking about the specifics of the IP distribution function: how the hash is actually calculated with a given set of source and destination IP addresses and the result used to select the egress interface. As John suggests, this may be a case where only Netapp can answer the question. Quoting from TR-4847 <https://www.netapp.com/media/19900-tr-4847.pdf> which Peter linked to (emphasis mine): IP: Second-best load distribution method, since the IP addresses of both sender (LIF) and client are used to deterministically select the particular physical link that a packet traverses. Although deterministic in the selection of a port, the *balancing is * *performed using an advanced hash function*. This has been found to work under a wide variety of circumstances, but particular selections of IP addresses might still lead to unequal load distribution. The only information I can find about the 'advanced hash function' is in an old 7-mode KB <https://kb.netapp.com/Legacy/ONTAP/7Mode/How_does_load_balancing_on_a_VIF_= work> which states (emphasis mine): *As of Data ONTAP 7.3.2, a multimode or LACP ifgrp/vif uses an implementation of a "SuperFastHash"*, utilizing the last 16 bits of the source and destination IP addresses (-b ip), the last 16 bits of the source and destination MAC addresses (-b mac), or the last 16 bits of the source and destination IP addresses in combination with the source and destination TCP port (-b port). The output of the algorithm results in a far more dynamic, more balanced distribution than the algorithm used in versions of Data ONTAP prior to 7.3.2. The result is still the same, however, in that each TCP stream will associate with only one interface, allowing for only one port's worth of bandwidth per TCP stream. Data ONTAP releases prior to 7.3.2: Documentation *prior to 7.3.2* states this as the formula: *((source_address XOR destination_address) % number_of_links)* When I was asked to look into a SnapMirror traffic imbalance, I started by calculating the IP hash using the 'standard' load balancing formula you would expect for source and destination IP. However, the results indicated different behaviour than what we are seeing. After doing more research today, I ran across the old KB quoted above and also this Github <https://github.com/qdrddr/ontap-lacp> page. However, it looks like the code was created to replicate empirical data rather than being the actual SuperFastHash algorithm used by Netapp. *Why do I care?* Well, mostly just academic curiosity about the 'better mouse trap' that Netapp built. The imbalance we are seeing isn't causing any issues. It was just one of those curious things that we investigated to understand why it was happening. Also, it will be much easier for me to tweak the intercluster LIF IPs to get the desired traffic balancing in the short term than waiting for a maintenance window to tear down the data ifgrps and rebuild them with the 'Port' distribution function. (We will rebuild them eventually.) Thanks again. I hope someone can shed a bit more light on the 'SuperFastHash' or whatever has replaced it in ONTAP 9. Best wishes Stephen On Tue, 4 Feb 2025 at 20:06, Peter Tas <[email protected]> wrote: > Neoria - Public > > Hello Stephen, > > This document, although already 12y old, is still a good reference. > > https://www.netapp.com/media/19900-tr-4847.pdf > > > > Check pp7 and the following page(s) regarding load balancing. > =E2=80=9CPort=E2=80=9D distri function is generally known to be the prefe= rred one when > configuring an ifgrp, but it may also depend on the brand/type of switch > stack to which you connected the nics of the controllers. > > > > Good luck & best regards, > > > > Peter Tas. > > > > > > > > > Neoria - Public > From: John Stoffel <[email protected]> > *Date: *Tuesday, 4 February 2025 at 18:29 > *To: *Stephen Stocke <[email protected]> > *Cc: *[email protected] <[email protected]> > *Subject: *Re: Igfrp IP Distribution Function > > >>>>> "Stephen" =3D=3D Stephen Stocke <[email protected]> writes: > > > Greetings fellow toaster admins I hope someone can shed some light > > on the IP distribution function for lacp ifgrps. > > You need to give more information on your setup, especially what kind > of switches you're using and how they're configured. > > > We have a four port, multi mode lacp ifgrp, a0a, using interfaces > > e2a-e2d. We observe SnapMirror traffic egressing port e2c. > > So? Why do you care? > > > The XOR of the last two bits of the source and destination IPs are > > either x0 or x3 so I am expecting traffic to egress either the first > > or fourth port in the ifgrp. > > > Assuming e2a is port 0, e2b is port 1, etc., I would expect traffic > > to egress either e2a (0x0) or e2d (0x3). > > Share your config (cli output) so we can look at it. > > > > What am I missing? I can=E2=80=99t find any details about the actual ha= shing > > function for IP distribution or port member indexing in an ifgrp to > > confirm my assumptions. > > > Both clusters are a single A250 HA pair running 9.15.1. > > Are you seeing performance problems? Are you seeing that your traffic > isn't being balanced across all your links? > > I guess I really don't understand the problem you're trying to solve, > unless oyu're just looking for info on why it works this way, which > might really be a Netapp only answer. > > _______________________________________________ > toasters mailing list -- [email protected] > To unsubscribe send an email to [email protected] > --000000000000e88d76062d596918 Content-Type: text/html; charset="UTF-8" Content-Transfer-Encoding: quoted-printable <div dir=3D"ltr">Hello<div><br></div><div>Thank you to everyone who replied= on and off list.</div><div><br></div><div>I believe I have a good understa= nding of ifgrp configuration and the purpose of the various distribution fu= nctions.=C2=A0 In this case I'm asking about the specifics of the IP di= stribution function: how the hash is actually calculated with a given set o= f source and destination IP addresses and the result used to select the egr= ess interface.=C2=A0 As John suggests, this may be a case where only Netapp= can answer the question.</div><div><br></div><div>Quoting from <a href=3D"= https://www.netapp.com/media/19900-tr-4847.pdf">TR-4847</a> which Peter lin= ked to (emphasis mine):</div><div><br></div><blockquote style=3D"margin:0 0= 0 40px;border:none;padding:0px"><div>IP: Second-best load distribution met= hod, since the IP addresses of both sender </div><div>(LIF) and client are = used to deterministically select the particular physical link that a </div>= <div>packet traverses. Although deterministic in the selection of a port, t= he <b><i>balancing is </i></b></div><div><b><i>performed using an advanced = hash function</i></b>. This has been found to work under a </div><div>wide = variety of circumstances, but particular selections of IP addresses might s= till</div>lead to unequal load distribution.=C2=A0</blockquote><div><br></d= iv><div>The only information I can find about the 'advanced hash functi= on' is in an old <a href=3D"https://kb.netapp.com/Legacy/ONTAP/7Mode/Ho= w_does_load_balancing_on_a_VIF_work">7-mode KB</a> which states (emphasis m= ine):</div><div><br></div><div><blockquote style=3D"margin:0 0 0 40px;borde= r:none;padding:0px"><div><b><i>As of Data ONTAP 7.3.2, a multimode or LACP = ifgrp/vif uses an implementation of a "SuperFastHash"</i></b>, ut= ilizing the last 16 bits of the source and destination IP addresses (-b ip)= , the last 16 bits of the source and destination MAC addresses (-b mac), or= the last 16 bits of the source and destination IP addresses in combination= with the source and destination TCP port (-b port). <br><br>The output of = the algorithm results in a far more dynamic, more balanced distribution tha= n the algorithm used in versions of Data ONTAP prior to 7.3.2.=C2=A0 The re= sult is still the same, however, in that each TCP stream will associate wit= h only one interface, allowing for only one port's worth of bandwidth p= er TCP stream. <br><br>Data ONTAP releases prior to 7.3.2:<br><br>Documenta= tion =C2=A0<b><i>prior to 7.3.2</i></b> states this as the formula:<br><br>= <b><i>((source_address XOR destination_address) % number_of_links)</i></b><= /div></blockquote></div><div><br></div><div>When I was asked to look into a= SnapMirror traffic imbalance, I started by calculating the IP hash using t= he 'standard' load balancing formula you would expect for source an= d destination IP. However, the results indicated different behaviour than w= hat we are seeing.=C2=A0 After doing more research today, I ran across the = old KB quoted above and also this <a href=3D"https://github.com/qdrddr/onta= p-lacp">Github</a> page.=C2=A0 However, it looks like the code was created = to replicate empirical data rather than being the actual SuperFastHash algo= rithm used by Netapp.</div><div>=C2=A0</div><div><b>Why do I care?</b></div= ><div>Well, mostly just academic curiosity about the 'better mouse trap= ' that Netapp built.=C2=A0 The=C2=A0 imbalance we are seeing isn't = causing any issues. It was just one of those curious things that we investi= gated to understand why it was happening.=C2=A0 Also, it will be much easie= r for me=C2=A0 to tweak the intercluster LIF IPs to get the desired traffic= balancing in the short term than waiting for a maintenance window to tear = down the data ifgrps and rebuild them with the 'Port' distribution = function.=C2=A0 (We will rebuild them eventually.)</div><div><br></div><div= >Thanks again.=C2=A0 I hope someone can shed a bit more light on the 'S= uperFastHash' or whatever has replaced it in ONTAP 9.</div><div><br></d= iv><div>Best wishes</div><div>Stephen</div></div><br><div class=3D"gmail_qu= ote gmail_quote_container"><div dir=3D"ltr" class=3D"gmail_attr">On Tue, 4 = Feb 2025 at 20:06, Peter Tas <<a href=3D"mailto:[email protected]">Pet= [email protected]</a>> wrote:<br></div><blockquote class=3D"gmail_quote" = style=3D"margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);pa= dding-left:1ex"><div class=3D"msg-5497054979085322493"> <div lang=3D"en-BE" style=3D"overflow-wrap: break-word;"> <p style=3D"font-family:Calibri;font-size:14pt;color:rgb(0,128,0);margin:5p= t;font-style:normal;font-weight:normal;text-decoration:none" align=3D"Left"= > Neoria - Public<br> </p> <br> <div class=3D"m_-5596649041429618504WordSection1"> <p class=3D"MsoNormal"><span lang=3D"EN-GB" style=3D"font-family:Calibri,sa= ns-serif">Hello Stephen,<br> <br> This document, although already 12y old, is still a good reference.<u></u><= u></u></span></p> <p class=3D"MsoNormal"><span lang=3D"EN-GB" style=3D"font-family:Calibri,sa= ns-serif"><a href=3D"https://www.netapp.com/media/19900-tr-4847.pdf" target= =3D"_blank">https://www.netapp.com/media/19900-tr-4847.pdf</a><u></u><u></u= ></span></p> <p class=3D"MsoNormal"><span lang=3D"EN-GB" style=3D"font-family:Calibri,sa= ns-serif"><u></u>=C2=A0<u></u></span></p> <p class=3D"MsoNormal"><span lang=3D"EN-GB" style=3D"font-family:Calibri,sa= ns-serif">Check pp7 and the following page(s) regarding load balancing. <br> =E2=80=9CPort=E2=80=9D distri function is generally known to be the preferr= ed one when configuring an ifgrp, but it may also depend on the brand/type = of switch stack to which you connected the nics of the controllers.<u></u><= u></u></span></p> <p class=3D"MsoNormal"><span lang=3D"EN-GB" style=3D"font-family:Calibri,sa= ns-serif"><u></u>=C2=A0<u></u></span></p> <p class=3D"MsoNormal"><span lang=3D"EN-GB" style=3D"font-family:Calibri,sa= ns-serif">Good luck & best regards,<u></u><u></u></span></p> <p class=3D"MsoNormal"><span lang=3D"EN-GB" style=3D"font-family:Calibri,sa= ns-serif"><u></u>=C2=A0<u></u></span></p> <p class=3D"MsoNormal"><span lang=3D"EN-GB" style=3D"font-family:Calibri,sa= ns-serif">Peter Tas.<u></u><u></u></span></p> <p class=3D"MsoNormal"><span lang=3D"EN-GB" style=3D"font-family:Calibri,sa= ns-serif"><u></u>=C2=A0<u></u></span></p> <p class=3D"MsoNormal"><span lang=3D"EN-GB" style=3D"font-family:Calibri,sa= ns-serif"><u></u>=C2=A0<u></u></span></p> <p class=3D"MsoNormal"><span lang=3D"EN-GB" style=3D"font-family:Calibri,sa= ns-serif"><u></u>=C2=A0<u></u></span></p> <div id=3D"m_-5596649041429618504mail-editor-reference-message-container"> <div> <div> <div style=3D"border-right:none;border-bottom:none;border-left:none;border-= top:1pt solid rgb(181,196,223);padding:3pt 0cm 0cm"> <p class=3D"MsoNormal" style=3D"margin-bottom:12pt"><b><span style=3D"color= :black"><br> </span></b></p> <p style=3D"font-family:Calibri;font-size:14pt;color:rgb(0,128,0);margin:5p= t;font-style:normal;font-weight:normal;text-decoration:none" align=3D"Left"= > Neoria - Public<br> </p> From: <span style=3D"color:black">John Stoffel <<a href=3D"mailto:john@s= toffel.org" target=3D"_blank">[email protected]</a>><br> <b>Date: </b>Tuesday, 4 February 2025 at 18:29<br> <b>To: </b>Stephen Stocke <<a href=3D"mailto:[email protected]" target= =3D"_blank">[email protected]</a>><br> <b>Cc: </b><a href=3D"mailto:[email protected]" target=3D"_blank">toast= [email protected]</a> <<a href=3D"mailto:[email protected]" target=3D= "_blank">[email protected]</a>><br> <b>Subject: </b>Re: Igfrp IP Distribution Function<u></u><u></u></span> <p></p> </div> <div> <p class=3D"MsoNormal"><span style=3D"font-size:11pt">>>>>> = "Stephen" =3D=3D Stephen Stocke <<a href=3D"mailto:scstocke@gm= ail.com" target=3D"_blank">[email protected]</a>> writes:<br> <br> > Greetings fellow toaster admins I hope someone can shed some light<br> > on the IP distribution function for lacp ifgrps.<br> <br> You need to give more information on your setup, especially what kind<br> of switches you're using and how they're configured.=C2=A0 <br> <br> > We have a four port, multi mode lacp ifgrp, a0a, using interfaces<br> > e2a-e2d. We observe SnapMirror traffic egressing port e2c.<br> <br> So?=C2=A0 Why do you care?<br> <br> > The XOR of the last two bits of the source and destination IPs are<br> > either x0 or x3 so I am expecting traffic to egress either the first<b= r> > or fourth port in the ifgrp.<br> <br> > Assuming e2a is port 0, e2b is port 1, etc., I would expect traffic<br= > > to egress either e2a (0x0) or e2d (0x3).<br> <br> Share your config (cli output) so we can look at it. <br> <br> <br> > What am I missing? I can=E2=80=99t find any details about the actual h= ashing<br> > function for IP distribution or port member indexing in an ifgrp to<br= > > confirm my assumptions.<br> <br> > Both clusters are a single A250 HA pair running 9.15.1.<br> <br> Are you seeing performance problems?=C2=A0 Are you seeing that your traffic= <br> isn't being balanced across all your links?=C2=A0 <br> <br> I guess I really don't understand the problem you're trying to solv= e,<br> unless oyu're just looking for info on why it works this way, which<br> might really be a Netapp only answer.=C2=A0 <br> <br> _______________________________________________<br> toasters mailing list -- <a href=3D"mailto:[email protected]" tar= get=3D"_blank">[email protected]</a><br> To unsubscribe send an email to <a href=3D"mailto:[email protected]= arty.net" target=3D"_blank">[email protected]</a><u></u><u>= </u></span></p> </div> </div> </div> </div> </div> </div> </div></blockquote></div> --000000000000e88d76062d596918-- --===============1791380735877941590== Content-Type: text/plain; charset="us-ascii" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit Content-Disposition: inline _______________________________________________ toasters mailing list -- [email protected] To unsubscribe send an email to [email protected] --===============1791380735877941590==--