Re: git: 6e4b811009d6 - main - vtnet: disable hardware TCP LRO by default
Mickaël Maillot <[email protected]> Wed, 22 Jul 2026 15:34:50 +0200
| Newsgroups | gmane.os.freebsd.devel.net |
|---|---|
| Message-ID | <CAKT0JB6oPC6MM_h7RH6-98hihbrGieCC0hZMq-Jpsc651rPq1A@mail.gmail.com> |
--000000000000e7a2cb06573334c4 Content-Type: text/plain; charset="UTF-8" Content-Transfer-Encoding: quoted-printable Hi, I'd like to report that a problem still exists with vtnet LRO on 15.1-RELEASE =E2=80=94 but in the software LRO path, which remains enabled = by default (this commit only disables hardware LRO). Environment: FreeBSD 15.1-RELEASE-p1 guests (vtnet/virtio) on Proxmox/KVM, running haproxy in front of Linux backends. After upgrading the guests from 14.3 to 15.1, we started seeing intermittent mid-transfer TCP stalls on responses larger than ~100 KB (roughly 1 in 5 transfers under production load), ending in a 30 s application timeout. Packet captures on the FreeBSD side show the following sequence: 1. The FreeBSD receiver's window closes under application back-pressure (win goes down to 0-2 with wscale 8), the sender fits one last small segment into it. 2. FreeBSD then sends a window-update ACK reopening the window (win 257). 3. The Linux sender never acts on it: its subsequent zero-window probes keep echoing the previous timestamp, i.e. the window update was never received/processed, and it stays in persist mode. 4. FreeBSD does not respond to the zero-length probes, so the connection deadlocks until the application gives up. The window-update ACK is visible in the capture taken on the FreeBSD guest itself (BPF), so it is generated but apparently lost on the way out. The issue is correlated with software LRO: with LRO enabled we reproduced 10 failures out of 60 transfers; after ifconfig vtnet0 -lro (TSO4 still enabled) we got 0 out of 60, across several identical guests. Disabling LRO fleet-wide made the problem disappear entirely. 14.x guests with the same workload never showed this. Happy to provide the pcaps or test patches. Le mar. 30 sept. 2025 =C3=A0 19:14, Michael Tuexen <[email protected]> a = =C3=A9crit : > The branch main has been updated by tuexen: > > URL: > https://cgit.FreeBSD.org/src/commit/?id=3D6e4b811009d63f33c59d51f28fd4a03= 0ca90843e > > commit 6e4b811009d63f33c59d51f28fd4a030ca90843e > Author: Michael Tuexen <[email protected]> > AuthorDate: 2025-09-29 23:58:37 +0000 > Commit: Michael Tuexen <[email protected]> > CommitDate: 2025-09-30 00:03:49 +0000 > > vtnet: disable hardware TCP LRO by default > > Hardware TCP LRO results in problems in settings with IP forwarding > being enabled. In case of nodes without IP forwarding, using > software LRO is also beneficial in general, since it can provide bett= er > information about what was received on the wire. > Therefore, disable hardware TCP LRO by default. > By tuning the loader tunable, this can be changed. > > PR: 263229 > Reviewed by: Timo V=C3=B6lker > MFC after: 3 days > Differential Revision: https://reviews.freebsd.org/D52684 > --- > share/man/man4/vtnet.4 | 22 +++++++++++++++++----- > sys/dev/virtio/network/if_vtnet.c | 2 +- > 2 files changed, 18 insertions(+), 6 deletions(-) > > diff --git a/share/man/man4/vtnet.4 b/share/man/man4/vtnet.4 > index 636ce4cc9b60..fd41b36882a6 100644 > --- a/share/man/man4/vtnet.4 > +++ b/share/man/man4/vtnet.4 > @@ -22,7 +22,7 @@ > .\" OUT OF THE USE OF THIS SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY = OF > .\" SUCH DAMAGE. > .\" > -.Dd September 26, 2025 > +.Dd September 30, 2025 > .Dt VTNET 4 > .Os > .Sh NAME > @@ -54,6 +54,18 @@ TCP segmentation offload (TSO), TCP large receive > offload (LRO), > hardware VLAN tag stripping/insertion features, a multicast hash filter, > as well as Jumbo Frames (up to 9216 bytes), which can be > configured via the interface MTU setting. > +.Pp > +Two flavors of TCP LRO are supported: > +Hardware TCP LRO, which is performed by the host providing TCP segments > larger > +than the MTU to the guest, and Software TCP LRO, which is performed by t= he > +network stack of the guest processing TCP segments in an optimized way. > +Only one flavour of TCP LRO should be used. > +Since hardware TCP LRO can have bad interactions with IP forwarding and > +software TCP LRO mitigates several drawbacks of hardware TCP LRO, the > default > +setting is to disable hardware TCP LRO. > +See the loader tunable > +.Va hw.vtnet. Ns Ar X Ns Va .lro_disable . > +.Pp > TCP/UDP receive checksum offload cannot be configured independently for > IPv4 > and IPv6. > Selecting an MTU larger than 1500 bytes with the > @@ -94,8 +106,8 @@ This tunable disables TCP segmentation offloading. > The default value is 0. > .It Va hw.vtnet.lro_disable > .It Va hw.vtnet. Ns Ar X Ns Va .lro_disable > -This tunable disables large receive offload. > -The default value is 0. > +This tunable disables hardware TCP LRO. > +The default value is 1. > .It Va hw.vtnet.mq_disable > .It Va hw.vtnet. Ns Ar X Ns Va .mq_disable > This tunable disables multiqueue. > @@ -119,11 +131,11 @@ This tunable sets the number of RX segments > processed in one pass. > The default value is 1024. > .It Va hw.vtnet.lro_entry_count > .It Va hw.vtnet. Ns Ar X Ns Va .lro_entry_count > -This tunable sets the software LRO entry count. > +This tunable sets the software TCP LRO entry count. > The default value is 128, the minimum value is 8. > .It Va hw.vtnet.lro_mbufq_depth > .It Va hw.vtnet. Ns Ar X Ns Va .lro_mbufq_depth > -This tunable sets the depth of the software LRO mbuf queue. > +This tunable sets the depth of the software TCP LRO mbuf queue. > The default value is 0. > .It Va hw.vtnet.altq_disable > This tunable disables ALTQ support, allowing the use of multiqueue > instead. > diff --git a/sys/dev/virtio/network/if_vtnet.c > b/sys/dev/virtio/network/if_vtnet.c > index 634ba0de2d55..471c6b3714b2 100644 > --- a/sys/dev/virtio/network/if_vtnet.c > +++ b/sys/dev/virtio/network/if_vtnet.c > @@ -281,7 +281,7 @@ static int vtnet_tso_disable =3D 0; > SYSCTL_INT(_hw_vtnet, OID_AUTO, tso_disable, CTLFLAG_RDTUN, > &vtnet_tso_disable, 0, "Disables TSO"); > > -static int vtnet_lro_disable =3D 0; > +static int vtnet_lro_disable =3D 1; > SYSCTL_INT(_hw_vtnet, OID_AUTO, lro_disable, CTLFLAG_RDTUN, > &vtnet_lro_disable, 0, "Disables hardware LRO"); > > > --000000000000e7a2cb06573334c4 Content-Type: text/html; charset="UTF-8" Content-Transfer-Encoding: quoted-printable <div dir=3D"ltr">Hi,<br><br>I'd like to report that a problem still exi= sts with vtnet LRO on 15.1-RELEASE =E2=80=94 but in the software LRO path, = which remains enabled by default (this commit only disables hardware LRO).<= br><br>Environment: FreeBSD 15.1-RELEASE-p1 guests (vtnet/virtio) on Proxmo= x/KVM, running haproxy in front of Linux backends. After upgrading the gues= ts from 14.3 to 15.1, we started seeing intermittent mid-transfer TCP stall= s on responses larger than ~100 KB (roughly 1 in 5 transfers under producti= on load), ending in a 30 s application timeout.<br><br>Packet captures on t= he FreeBSD side show the following sequence:<br><br>1. The FreeBSD receiver= 's window closes under application back-pressure (win goes down to 0-2 = with wscale=C2=A08), the sender fits one last small segment into it.<br>2. = FreeBSD then sends a window-update ACK reopening the window (win=C2=A0257).= <br>3. The Linux sender never acts on it: its subsequent zero-window probes= keep echoing the previous timestamp, i.e. the window update was never rece= ived/processed, and it stays in persist mode.<br>4. FreeBSD does not respon= d to the zero-length probes, so the connection deadlocks until the applicat= ion gives up.<br><br>The window-update ACK is visible in the capture taken = on the FreeBSD guest itself (BPF), so it is generated but apparently lost o= n the way out.<br><br>The issue is correlated with software LRO: with LRO e= nabled we reproduced 10 failures out of 60 transfers; after ifconfig vtnet0= -lro (TSO4 still enabled) we got 0 out of 60, across several identical gue= sts. Disabling LRO fleet-wide made the problem disappear entirely. 14.x gue= sts with the same workload never showed this.<br><br>Happy to provide the p= caps or test patches.</div><br><div class=3D"gmail_quote gmail_quote_contai= ner"><div dir=3D"ltr" class=3D"gmail_attr">Le=C2=A0mar. 30 sept. 2025 =C3= =A0=C2=A019:14, Michael Tuexen <<a href=3D"mailto:[email protected]">tu= [email protected]</a>> a =C3=A9crit=C2=A0:<br></div><blockquote class=3D"= gmail_quote" style=3D"margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(20= 4,204,204);padding-left:1ex">The branch main has been updated by tuexen:<br= > <br> URL: <a href=3D"https://cgit.FreeBSD.org/src/commit/?id=3D6e4b811009d63f33c= 59d51f28fd4a030ca90843e" rel=3D"noreferrer" target=3D"_blank">https://cgit.= FreeBSD.org/src/commit/?id=3D6e4b811009d63f33c59d51f28fd4a030ca90843e</a><b= r> <br> commit 6e4b811009d63f33c59d51f28fd4a030ca90843e<br> Author:=C2=A0 =C2=A0 =C2=A0Michael Tuexen <[email protected]><br> AuthorDate: 2025-09-29 23:58:37 +0000<br> Commit:=C2=A0 =C2=A0 =C2=A0Michael Tuexen <[email protected]><br> CommitDate: 2025-09-30 00:03:49 +0000<br> <br> =C2=A0 =C2=A0 vtnet: disable hardware TCP LRO by default<br> <br> =C2=A0 =C2=A0 Hardware TCP LRO results in problems in settings with IP forw= arding<br> =C2=A0 =C2=A0 being enabled. In case of nodes without IP forwarding, using<= br> =C2=A0 =C2=A0 software LRO is also beneficial in general, since it can prov= ide better<br> =C2=A0 =C2=A0 information about what was received on the wire.<br> =C2=A0 =C2=A0 Therefore, disable hardware TCP LRO by default.<br> =C2=A0 =C2=A0 By tuning the loader tunable, this can be changed.<br> <br> =C2=A0 =C2=A0 PR:=C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 = =C2=A0 =C2=A0 =C2=A0263229<br> =C2=A0 =C2=A0 Reviewed by:=C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 Timo V= =C3=B6lker<br> =C2=A0 =C2=A0 MFC after:=C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 3 = days<br> =C2=A0 =C2=A0 Differential Revision:=C2=A0 <a href=3D"https://reviews.freeb= sd.org/D52684" rel=3D"noreferrer" target=3D"_blank">https://reviews.freebsd= .org/D52684</a><br> ---<br> =C2=A0share/man/man4/vtnet.4=C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 | 22 = +++++++++++++++++-----<br> =C2=A0sys/dev/virtio/network/if_vtnet.c |=C2=A0 2 +-<br> =C2=A02 files changed, 18 insertions(+), 6 deletions(-)<br> <br> diff --git a/share/man/man4/vtnet.4 b/share/man/man4/vtnet.4<br> index 636ce4cc9b60..fd41b36882a6 100644<br> --- a/share/man/man4/vtnet.4<br> +++ b/share/man/man4/vtnet.4<br> @@ -22,7 +22,7 @@<br> =C2=A0.\" OUT OF THE USE OF THIS SOFTWARE, EVEN IF ADVISED OF THE POSS= IBILITY OF<br> =C2=A0.\" SUCH DAMAGE.<br> =C2=A0.\"<br> -.Dd September 26, 2025<br> +.Dd September 30, 2025<br> =C2=A0.Dt VTNET 4<br> =C2=A0.Os<br> =C2=A0.Sh NAME<br> @@ -54,6 +54,18 @@ TCP segmentation offload (TSO), TCP large receive offloa= d (LRO),<br> =C2=A0hardware VLAN tag stripping/insertion features, a multicast hash filt= er,<br> =C2=A0as well as Jumbo Frames (up to 9216 bytes), which can be<br> =C2=A0configured via the interface MTU setting.<br> +.Pp<br> +Two flavors of TCP LRO are supported:<br> +Hardware TCP LRO, which is performed by the host providing TCP segments la= rger<br> +than the MTU to the guest, and Software TCP LRO, which is performed by the= <br> +network stack of the guest processing TCP segments in an optimized way.<br= > +Only one flavour of TCP LRO should be used.<br> +Since hardware TCP LRO can have bad interactions with IP forwarding and<br= > +software TCP LRO mitigates several drawbacks of hardware TCP LRO, the defa= ult<br> +setting is to disable hardware TCP LRO.<br> +See the loader tunable<br> +.Va hw.vtnet. Ns Ar X Ns Va .lro_disable .<br> +.Pp<br> =C2=A0TCP/UDP receive checksum offload cannot be configured independently f= or IPv4<br> =C2=A0and IPv6.<br> =C2=A0Selecting an MTU larger than 1500 bytes with the<br> @@ -94,8 +106,8 @@ This tunable disables TCP segmentation offloading.<br> =C2=A0The default value is 0.<br> =C2=A0.It Va hw.vtnet.lro_disable<br> =C2=A0.It Va hw.vtnet. Ns Ar X Ns Va .lro_disable<br> -This tunable disables large receive offload.<br> -The default value is 0.<br> +This tunable disables hardware TCP LRO.<br> +The default value is 1.<br> =C2=A0.It Va hw.vtnet.mq_disable<br> =C2=A0.It Va hw.vtnet. Ns Ar X Ns Va .mq_disable<br> =C2=A0This tunable disables multiqueue.<br> @@ -119,11 +131,11 @@ This tunable sets the number of RX segments processed= in one pass.<br> =C2=A0The default value is 1024.<br> =C2=A0.It Va hw.vtnet.lro_entry_count<br> =C2=A0.It Va hw.vtnet. Ns Ar X Ns Va .lro_entry_count<br> -This tunable sets the software LRO entry count.<br> +This tunable sets the software TCP LRO entry count.<br> =C2=A0The default value is 128, the minimum value is 8.<br> =C2=A0.It Va hw.vtnet.lro_mbufq_depth<br> =C2=A0.It Va hw.vtnet. Ns Ar X Ns Va .lro_mbufq_depth<br> -This tunable sets the depth of the software LRO mbuf queue.<br> +This tunable sets the depth of the software TCP LRO mbuf queue.<br> =C2=A0The default value is 0.<br> =C2=A0.It Va hw.vtnet.altq_disable<br> =C2=A0This tunable disables ALTQ support, allowing the use of multiqueue in= stead.<br> diff --git a/sys/dev/virtio/network/if_vtnet.c b/sys/dev/virtio/network/if_= vtnet.c<br> index 634ba0de2d55..471c6b3714b2 100644<br> --- a/sys/dev/virtio/network/if_vtnet.c<br> +++ b/sys/dev/virtio/network/if_vtnet.c<br> @@ -281,7 +281,7 @@ static int vtnet_tso_disable =3D 0;<br> =C2=A0SYSCTL_INT(_hw_vtnet, OID_AUTO, tso_disable, CTLFLAG_RDTUN,<br> =C2=A0 =C2=A0 =C2=A0&vtnet_tso_disable, 0, "Disables TSO");<b= r> <br> -static int vtnet_lro_disable =3D 0;<br> +static int vtnet_lro_disable =3D 1;<br> =C2=A0SYSCTL_INT(_hw_vtnet, OID_AUTO, lro_disable, CTLFLAG_RDTUN,<br> =C2=A0 =C2=A0 =C2=A0&vtnet_lro_disable, 0, "Disables hardware LRO&= quot;);<br> <br> <br> </blockquote></div> --000000000000e7a2cb06573334c4--