Re: IPv6 path MTU discovery not working (large number of retransmited TCP data packets)

I think I experienced this too recently. Was looking into the countermeasures against icmp6 too big attacks on netbsd. But I thought maybe I was doing something wrong because I was manually crafting the icmp pakets into active connections. Maybe we would need to look into it.

> On 26 May 2025, at 4:41 PM, Christof Meerwald <[email protected]> wrote:
> 
> Noticed a lot of retransmitted data packets showing up in "netstat
> -ss", and after some experimentation it appears to be some issue with
> path MTU discovery on IPv6 (or the lack of it).
> 
> Test VPS is running current: NetBSD 10.99.14 (GENERIC) #0: Sun May 25 16:59:07 UTC 2025  [email protected]:/usr/src/sys/arch/amd64/compile/GENERIC amd64
> but I have also seen it on NetBSD 10.1.
> 
> The NetBSD VPS is using vioif0 with MTU 1500, but I am actually
> connecting to the IPv6 address via an HE tunnel (so the MTU on the
> path is lower).
> 
> On the NetBSD VPS I am running:
> 
> dd if=/dev/zero bs=16k count=1 | nc -6 -l 12345
> 
> And on a Linux machine (also with MTU 1500) I am then using
> 
> nc -6 green.cmeerw.net 12345 >/dev/null
> 
> I would expect that after the first few packets with a too big MTU, it
> will then use the lower MTU from an ICMP6 packet too big. However, it
> appears that NetBSD tries to retransmit the first few packets at full
> speed for a few seconds until finally falling back to a lower MTU.
> tcpdump shows this:
> 
> 15:48:31.482763 IP6 2001:470:597a:bbb:419:1719:4348:1725.43226 > 2a03:d9c2:100:49::b5d.12345: Flags [S], seq 2776544030, win 64800, options [mss 1440,sackOK,TS val 1231408504 ecr 0,nop,wscale 7], length 0
> 15:48:31.489363 IP6 2a03:d9c2:100:49::b5d.12345 > 2001:470:597a:bbb:419:1719:4348:1725.43226: Flags [S.], seq 3901012170, ack 2776544031, win 32768, options [mss 1440,nop,wscale 3,sackOK,TS val 1 ecr 1231408504], length 0
> 15:48:31.535870 IP6 2001:470:597a:bbb:419:1719:4348:1725.43226 > 2a03:d9c2:100:49::b5d.12345: Flags [.], ack 1, win 507, options [nop,nop,TS val 1231408569 ecr 1], length 0
> 15:48:31.536080 IP6 2a03:d9c2:100:49::b5d.12345 > 2001:470:597a:bbb:419:1719:4348:1725.43226: Flags [.], seq 1:1429, ack 1, win 4140, options [nop,nop,TS val 1 ecr 1231408569], length 1428
> 15:48:31.536116 IP6 2a03:d9c2:100:49::b5d.12345 > 2001:470:597a:bbb:419:1719:4348:1725.43226: Flags [.], seq 1429:2857, ack 1, win 4140, options [nop,nop,TS val 1 ecr 1231408569], length 1428
> 15:48:31.536137 IP6 2a03:d9c2:100:49::b5d.12345 > 2001:470:597a:bbb:419:1719:4348:1725.43226: Flags [.], seq 2857:4285, ack 1, win 4140, options [nop,nop,TS val 1 ecr 1231408569], length 1428
> 15:48:31.544930 IP6 2001:470:0:221::2 > 2a03:d9c2:100:49::b5d: ICMP6, packet too big, mtu 1460, length 1240
> 15:48:31.545013 IP6 2a03:d9c2:100:49::b5d.12345 > 2001:470:597a:bbb:419:1719:4348:1725.43226: Flags [.], seq 1:1429, ack 1, win 4140, options [nop,nop,TS val 1 ecr 1231408569], length 1428
> 15:48:31.545040 IP6 2a03:d9c2:100:49::b5d.12345 > 2001:470:597a:bbb:419:1719:4348:1725.43226: Flags [.], seq 1429:2857, ack 1, win 4140, options [nop,nop,TS val 1 ecr 1231408569], length 1428
> 15:48:31.545058 IP6 2a03:d9c2:100:49::b5d.12345 > 2001:470:597a:bbb:419:1719:4348:1725.43226: Flags [.], seq 2857:4285, ack 1, win 4140, options [nop,nop,TS val 1 ecr 1231408569], length 1428
> 15:48:31.545430 IP6 2001:470:0:221::2 > 2a03:d9c2:100:49::b5d: ICMP6, packet too big, mtu 1460, length 1240
> 15:48:31.545465 IP6 2a03:d9c2:100:49::b5d.12345 > 2001:470:597a:bbb:419:1719:4348:1725.43226: Flags [.], seq 1:1429, ack 1, win 4140, options [nop,nop,TS val 1 ecr 1231408569], length 1428
> 15:48:31.545484 IP6 2a03:d9c2:100:49::b5d.12345 > 2001:470:597a:bbb:419:1719:4348:1725.43226: Flags [.], seq 1429:2857, ack 1, win 4140, options [nop,nop,TS val 1 ecr 1231408569], length 1428
> 15:48:31.545502 IP6 2a03:d9c2:100:49::b5d.12345 > 2001:470:597a:bbb:419:1719:4348:1725.43226: Flags [.], seq 2857:4285, ack 1, win 4140, options [nop,nop,TS val 1 ecr 1231408569], length 1428
> 15:48:31.545981 IP6 2001:470:0:221::2 > 2a03:d9c2:100:49::b5d: ICMP6, packet too big, mtu 1460, length 1240
> 15:48:31.546016 IP6 2a03:d9c2:100:49::b5d.12345 > 2001:470:597a:bbb:419:1719:4348:1725.43226: Flags [.], seq 1:1429, ack 1, win 4140, options [nop,nop,TS val 1 ecr 1231408569], length 1428
> 15:48:31.546035 IP6 2a03:d9c2:100:49::b5d.12345 > 2001:470:597a:bbb:419:1719:4348:1725.43226: Flags [.], seq 1429:2857, ack 1, win 4140, options [nop,nop,TS val 1 ecr 1231408569], length 1428
> 15:48:31.546052 IP6 2a03:d9c2:100:49::b5d.12345 > 2001:470:597a:bbb:419:1719:4348:1725.43226: Flags [.], seq 2857:4285, ack 1, win 4140, options [nop,nop,TS val 1 ecr 1231408569], length 1428
> 15:48:31.553659 IP6 2001:470:0:221::2 > 2a03:d9c2:100:49::b5d: ICMP6, packet too big, mtu 1460, length 1240
> 15:48:31.553729 IP6 2a03:d9c2:100:49::b5d.12345 > 2001:470:597a:bbb:419:1719:4348:1725.43226: Flags [.], seq 1:1429, ack 1, win 4140, options [nop,nop,TS val 1 ecr 1231408569], length 1428
> 15:48:31.553750 IP6 2a03:d9c2:100:49::b5d.12345 > 2001:470:597a:bbb:419:1719:4348:1725.43226: Flags [.], seq 1429:2857, ack 1, win 4140, options [nop,nop,TS val 1 ecr 1231408569], length 1428
> 15:48:31.553768 IP6 2a03:d9c2:100:49::b5d.12345 > 2001:470:597a:bbb:419:1719:4348:1725.43226: Flags [.], seq 2857:4285, ack 1, win 4140, options [nop,nop,TS val 1 ecr 1231408569], length 1428
> 15:48:31.554185 IP6 2001:470:0:221::2 > 2a03:d9c2:100:49::b5d: ICMP6, packet too big, mtu 1460, length 1240
> 15:48:31.554219 IP6 2a03:d9c2:100:49::b5d.12345 > 2001:470:597a:bbb:419:1719:4348:1725.43226: Flags [.], seq 1:1429, ack 1, win 4140, options [nop,nop,TS val 1 ecr 1231408569], length 1428
> 15:48:31.554238 IP6 2a03:d9c2:100:49::b5d.12345 > 2001:470:597a:bbb:419:1719:4348:1725.43226: Flags [.], seq 1429:2857, ack 1, win 4140, options [nop,nop,TS val 1 ecr 1231408569], length 1428
> 15:48:31.554255 IP6 2a03:d9c2:100:49::b5d.12345 > 2001:470:597a:bbb:419:1719:4348:1725.43226: Flags [.], seq 2857:4285, ack 1, win 4140, options [nop,nop,TS val 1 ecr 1231408569], length 1428
> 15:48:31.554747 IP6 2001:470:0:221::2 > 2a03:d9c2:100:49::b5d: ICMP6, packet too big, mtu 1460, length 1240
> 15:48:31.554780 IP6 2a03:d9c2:100:49::b5d.12345 > 2001:470:597a:bbb:419:1719:4348:1725.43226: Flags [.], seq 1:1429, ack 1, win 4140, options [nop,nop,TS val 1 ecr 1231408569], length 1428
> 15:48:31.554799 IP6 2a03:d9c2:100:49::b5d.12345 > 2001:470:597a:bbb:419:1719:4348:1725.43226: Flags [.], seq 1429:2857, ack 1, win 4140, options [nop,nop,TS val 1 ecr 1231408569], length 1428
> 15:48:31.554816 IP6 2a03:d9c2:100:49::b5d.12345 > 2001:470:597a:bbb:419:1719:4348:1725.43226: Flags [.], seq 2857:4285, ack 1, win 4140, options [nop,nop,TS val 1 ecr 1231408569], length 1428
> 15:48:31.555303 IP6 2001:470:0:221::2 > 2a03:d9c2:100:49::b5d: ICMP6, packet too big, mtu 1460, length 1240
> 15:48:31.555337 IP6 2a03:d9c2:100:49::b5d.12345 > 2001:470:597a:bbb:419:1719:4348:1725.43226: Flags [.], seq 1:1429, ack 1, win 4140, options [nop,nop,TS val 1 ecr 1231408569], length 1428
> 15:48:31.555355 IP6 2a03:d9c2:100:49::b5d.12345 > 2001:470:597a:bbb:419:1719:4348:1725.43226: Flags [.], seq 1429:2857, ack 1, win 4140, options [nop,nop,TS val 1 ecr 1231408569], length 1428
> 15:48:31.555373 IP6 2a03:d9c2:100:49::b5d.12345 > 2001:470:597a:bbb:419:1719:4348:1725.43226: Flags [.], seq 2857:4285, ack 1, win 4140, options [nop,nop,TS val 1 ecr 1231408569], length 1428
> 15:48:31.555823 IP6 2001:470:0:221::2 > 2a03:d9c2:100:49::b5d: ICMP6, packet too big, mtu 1460, length 1240
> 15:48:31.555856 IP6 2a03:d9c2:100:49::b5d.12345 > 2001:470:597a:bbb:419:1719:4348:1725.43226: Flags [.], seq 1:1429, ack 1, win 4140, options [nop,nop,TS val 1 ecr 1231408569], length 1428
> 15:48:31.555885 IP6 2a03:d9c2:100:49::b5d.12345 > 2001:470:597a:bbb:419:1719:4348:1725.43226: Flags [.], seq 1429:2857, ack 1, win 4140, options [nop,nop,TS val 1 ecr 1231408569], length 1428
> 15:48:31.555903 IP6 2a03:d9c2:100:49::b5d.12345 > 2001:470:597a:bbb:419:1719:4348:1725.43226: Flags [.], seq 2857:4285, ack 1, win 4140, options [nop,nop,TS val 1 ecr 1231408569], length 1428
> 15:48:31.556376 IP6 2001:470:0:221::2 > 2a03:d9c2:100:49::b5d: ICMP6, packet too big, mtu 1460, length 1240
> 15:48:31.556410 IP6 2a03:d9c2:100:49::b5d.12345 > 2001:470:597a:bbb:419:1719:4348:1725.43226: Flags [.], seq 1:1429, ack 1, win 4140, options [nop,nop,TS val 1 ecr 1231408569], length 1428
> 15:48:31.556428 IP6 2a03:d9c2:100:49::b5d.12345 > 2001:470:597a:bbb:419:1719:4348:1725.43226: Flags [.], seq 1429:2857, ack 1, win 4140, options [nop,nop,TS val 1 ecr 1231408569], length 1428
> 15:48:31.556446 IP6 2a03:d9c2:100:49::b5d.12345 > 2001:470:597a:bbb:419:1719:4348:1725.43226: Flags [.], seq 2857:4285, ack 1, win 4140, options [nop,nop,TS val 1 ecr 1231408569], length 1428
> 15:48:31.556905 IP6 2001:470:0:221::2 > 2a03:d9c2:100:49::b5d: ICMP6, packet too big, mtu 1460, length 1240
> 15:48:31.556939 IP6 2a03:d9c2:100:49::b5d.12345 > 2001:470:597a:bbb:419:1719:4348:1725.43226: Flags [.], seq 1:1429, ack 1, win 4140, options [nop,nop,TS val 1 ecr 1231408569], length 1428
> 15:48:31.556958 IP6 2a03:d9c2:100:49::b5d.12345 > 2001:470:597a:bbb:419:1719:4348:1725.43226: Flags [.], seq 1429:2857, ack 1, win 4140, options [nop,nop,TS val 1 ecr 1231408569], length 1428
> 15:48:31.556975 IP6 2a03:d9c2:100:49::b5d.12345 > 2001:470:597a:bbb:419:1719:4348:1725.43226: Flags [.], seq 2857:4285, ack 1, win 4140, options [nop,nop,TS val 1 ecr 1231408569], length 1428
> 15:48:31.557516 IP6 2001:470:0:221::2 > 2a03:d9c2:100:49::b5d: ICMP6, packet too big, mtu 1460, length 1240
> 15:48:31.557549 IP6 2a03:d9c2:100:49::b5d.12345 > 2001:470:597a:bbb:419:1719:4348:1725.43226: Flags [.], seq 1:1429, ack 1, win 4140, options [nop,nop,TS val 1 ecr 1231408569], length 1428
> 15:48:31.557567 IP6 2a03:d9c2:100:49::b5d.12345 > 2001:470:597a:bbb:419:1719:4348:1725.43226: Flags [.], seq 1429:2857, ack 1, win 4140, options [nop,nop,TS val 1 ecr 1231408569], length 1428
> 15:48:31.557584 IP6 2a03:d9c2:100:49::b5d.12345 > 2001:470:597a:bbb:419:1719:4348:1725.43226: Flags [.], seq 2857:4285, ack 1, win 4140, options [nop,nop,TS val 1 ecr 1231408569], length 1428
> 
> this continues for a few seconds until:
> 
> 15:48:38.629056 IP6 2a03:d9c2:100:49::b5d.12345 > 2001:470:597a:bbb:419:1719:4348:1725.43226: Flags [.], seq 1:1429, ack 1, win 4140, options [nop,nop,TS val 15 ecr 1231408569], length 1428
> 15:48:38.629130 IP6 2a03:d9c2:100:49::b5d.12345 > 2001:470:597a:bbb:419:1719:4348:1725.43226: Flags [.], seq 1:1209, ack 1, win 4140, options [nop,nop,TS val 15 ecr 1231408569], length 1208
> 15:48:38.629572 IP6 2001:470:0:221::2 > 2a03:d9c2:100:49::b5d: ICMP6, packet too big, mtu 1460, length 1240
> 15:48:38.629607 IP6 2a03:d9c2:100:49::b5d.12345 > 2001:470:597a:bbb:419:1719:4348:1725.43226: Flags [.], seq 1:1209, ack 1, win 4140, options [nop,nop,TS val 15 ecr 1231408569], length 1208
> 15:48:38.630118 IP6 2001:470:0:221::2 > 2a03:d9c2:100:49::b5d: ICMP6, packet too big, mtu 1460, length 1240
> 15:48:38.630152 IP6 2a03:d9c2:100:49::b5d.12345 > 2001:470:597a:bbb:419:1719:4348:1725.43226: Flags [.], seq 1:1209, ack 1, win 4140, options [nop,nop,TS val 15 ecr 1231408569], length 1208
> 15:48:38.630739 IP6 2001:470:0:221::2 > 2a03:d9c2:100:49::b5d: ICMP6, packet too big, mtu 1460, length 1240
> 15:48:38.630772 IP6 2a03:d9c2:100:49::b5d.12345 > 2001:470:597a:bbb:419:1719:4348:1725.43226: Flags [.], seq 1:1209, ack 1, win 4140, options [nop,nop,TS val 15 ecr 1231408569], length 1208
> 
> and even that continues for half a second:
> 
> 15:48:39.133856 IP6 2a03:d9c2:100:49::b5d.12345 > 2001:470:597a:bbb:419:1719:4348:1725.43226: Flags [.], seq 1:1209, ack 1, win 4140, options [nop,nop,TS val 16 ecr 1231408569], length 1208
> 15:48:39.134355 IP6 2001:470:0:221::2 > 2a03:d9c2:100:49::b5d: ICMP6, packet too big, mtu 1460, length 1240
> 15:48:39.134390 IP6 2a03:d9c2:100:49::b5d.12345 > 2001:470:597a:bbb:419:1719:4348:1725.43226: Flags [.], seq 1:1209, ack 1, win 4140, options [nop,nop,TS val 16 ecr 1231408569], length 1208
> 15:48:39.511721 IP6 2001:470:597a:bbb:419:1719:4348:1725.43226 > 2a03:d9c2:100:49::b5d.12345: Flags [.], ack 1209, win 498, options [nop,nop,TS val 1231416205 ecr 15], length 0
> 15:48:39.511803 IP6 2a03:d9c2:100:49::b5d.12345 > 2001:470:597a:bbb:419:1719:4348:1725.43226: Flags [.], seq 1209:2417, ack 1, win 4140, options [nop,nop,TS val 16 ecr 1231416205], length 1208
> 15:48:39.511830 IP6 2a03:d9c2:100:49::b5d.12345 > 2001:470:597a:bbb:419:1719:4348:1725.43226: Flags [.], seq 2417:3625, ack 1, win 4140, options [nop,nop,TS val 16 ecr 1231416205], length 1208
> 
> So we end up with something like:
> 
> 17943 data packets (25439300 bytes) retransmitted
> 
> just from those 16 kB.
> 
> 
> Christof
> 
> -- 
> https://cmeerw.org                             sip:cmeerw at cmeerw.org
> mailto:cmeerw at cmeerw.org                   xmpp:cmeerw at cmeerw.org

A scoffer seeks wisdom in vain, but knowledge is easy for a man of understanding.
Emmanuel