Re: [List] ipfw nat on 14.4 not working

[2026-07-15 19:28] Frank Leonhardt <[email protected]>
> On 25/06/2026 11:05, Philipp wrote:
> > Hi
> >
> > Since I updated to 14.4 release my nat setup only works for icmp. I hav=
e
> > one public interface and two epair interfaces. The epairs (vnet42,
> > vnet43) are connected to a bridge and on the bridge are a bunch of jail=
s.
> >
> > I can connect to the service of the jails from the host itself. But the
> > ports redirects don't work and the jails can't use tcp/udp to hosts on
> > the internet.
> >
> > The public interface has the address 80.190.133.201 and the vnet interf=
ace
> > have the address 10.188.42.1/24 and 10.188.43.1/24. the vnet* epairs ar=
e
> > connected to a bridge. The jails are created by ezjail and the jib[0] t=
ool
> > is used to create epairs and attach to this bridge. I haven't had time =
to
> > look at ether_gen_addr so it's off.
> >
> > the ipfw.rules looks like this:
> >
> > #!/bin/sh
> > # Flush out the list before we begin.
> > ipfw -q -f flush
> >
> > pif=3D"em1"
> >
> > #public interface
> > pif=3Dem1
> >
> > ipfw nat 123 delete
> > ipfw nat 123 config ip 80.190.133.201 redirect_port tcp 10.188.42.84:80=
 80 redirect_port tcp 10.188.42.1:22 22
> > # redirect_port tcp 10.188.43.54:2222 2222 \
> > # redirect_port tcp 10.188.43.55:2223 2223 \
> > # redirect_port tcp 10.188.42.84:443 443 \
> > # redirect_port tcp 10.188.42.86:25 25 \
> > # redirect_port tcp 10.188.42.86:587 587 \
> > # redirect_port tcp 10.188.42.46:993 993 \
> > # redirect_port tcp 10.188.42.46:4190 4190
> >
> > #${fwcmd} add 00120 nat 123 ip4 from any to any via $pif
> > ipfw add 00120 nat 123 ip from 10.0.0.0/8 to any via $pif #out via $pif
> > ipfw add 00121 nat 123 ip from any to 80.190.133.201 via $pif #in via $=
pif
> > ipfw add 00122 check-state
> > # ICMP
> > ipfw add 00130 allow icmp from any to any
> > ipfw add 00131 allow icmp6 from any to any
> >
> > ipfw add 00132 allow ip from 10.0.0.0/8 to any keep-state
> >
> > #ipfw add 00140 skipto 200 ipv6 from 2a01:138:9000:bc00::/56 to any out=
 via $pif
> > #ipfw add 00141 skipto 200 ipv6 from me6 to any out via $pif
> > #ipfw add 00142 skipto 200 ipv6 from any to 2a01:138:9000:bc00::/56 in =
via $pif
> > #ipfw add 00143 skipto 200 ipv6 from any to me6 in via $pif
> > #ipfw add 00144 skipto 200 ipv4 from any to me in via $pif
> > #ipfw add 00145 skipto 200 ipv4 from me to any out via $pif
> > #ipfw add 00150 drop ipv6 from any to any via $pif
> > #ipfw add 00151 drop all from any to any frag in via $pif
> > #ipfw add 00152 drop ipv4 from any to any via $pif
> >
> > ipfw add 00210 allow tcp from any to any 22 via $pif # keep-state
> > ipfw add 00211 allow tcp from any to any 25 in via $pif # keep-state
> > ipfw add 00212 allow tcp from any to any 587 in via $pif # keep-state
> > ipfw add 00213 allow tcp from any to any 993 in via $pif #keep-state
> > ipfw add 00214 allow tcp from any to any 2222 in via $pif # keep-state
> > ipfw add 00215 allow tcp from any to any 4190 in via $pif # keep-state
> > ipfw add 00216 allow tcp from any to any 2223 in via $pif # keep-state
> > ipfw add 00217 allow udp from any to any 60000-61000 in via $pif
> > ipfw add 00218 allow tcp from any to any 80 in via $pif
> > ipfw add 00219 allow tcp from any to any 443 in via $pif
> > ipfw add 00220 allow all from any to any out via $pif keep-state
> > #ipfw add 00221 allow all from any to any in via $pif keep-state
> > #ipfw add 00230 allow tcp from any to 2a01:138:9000:bc00::/56 in via $p=
if
> > #ipfw add 00231 allow udp from any to 2a01:138:9000:bc00::/56 in via $p=
if
> > #ipfw add 00232 allow tcp from any to 80.190.133.201 in via $pif
> > #ipfw add 00233 allow udp from any to 80.190.133.201 in via $pif
> >
> > # vnet42
> > ipfw add 41999 allow all from 10.188.42.0/24 to any in via vnet42 keep-=
state
> > ipfw add 42000 allow all from 2a01:138:9000:bc42::/64 to any in via vne=
t42 keep-state
> > #ipfw add 42001 allow tcp from 10.188.42.0/24 to any in via vnet42 keep=
-state
> > #ipfw add 42002 allow udp from 10.188.42.0/24 to any in via vnet42 keep=
-state
> > ipfw add 42003 allow all from any to any 80 via vnet42 keep-state
> > ipfw add 42004 allow tcp from any to 2a01:138:9000:bc42::2e 993 out via=
 vnet42 keep-state
> > #ipfw add 42005 allow tcp from any to 10.188.42.46 993 out via vnet42 k=
eep-state
> > ipfw add 42006 allow tcp from any to 2a01:138:9000:bc42::15e4:2d4d 587 =
out via vnet42 keep-state
> > ipfw add 42007 allow tcp from any to 2a01:138:9000:bc42::15e4:2d4d 25 o=
ut via vnet42 keep-state
> > #ipfw add 42009 allow tcp from any to 10.188.42.86 25 out via vnet42 ke=
ep-state
> > #ipfw add 42010 allow tcp from any to 10.188.42.86 587 out via vnet42 k=
eep-state
> > ipfw add 42011 allow tcp from any to 2a01:138:9000:bc42::2e 4190 out vi=
a vnet42 keep-state
> > #ipfw add 42012 allow tcp from any to 10.188.42.46 4190 out via vnet42 =
keep-state
> > ipfw add 42013 allow all from any to 2a01:138:9000:bc42::53 53 out via =
vnet42 keep-state
> > ipfw add 42014 allow tcp from any to 2a01:138:9000:bc42::54 443 out via=
 vnet42 keep-state
> >
> > # vnet43
> > #ipfw add 42999 allow all from 10.188.43.0/24 to any in via vnet43 keep=
-state
> > ipfw add 43000 allow all from 2a01:138:9000:bc43::/64 to any in via vne=
t43 keep-state
> > ipfw add 43002 allow all from 10.188.43.0/24 to any in via vnet43 keep-=
state
> > ipfw add 43003 allow udp from any to 10.188.43.0/24 60000-61000 out via=
 vnet43 keep-state
> > ipfw add 43004 allow udp from any to 2a01:138:9000:bc43::/64 60000-6100=
0 out via vnet43 keep-state
> > ipfw add 43013 allow tcp from any to 10.188.43.0/24 22 out via vnet43 k=
eep-state
> > ipfw add 43014 allow tcp from any to 2a01:138:9000:bc43::/64 22 out via=
 vnet43 keep-state
> > ipfw add 43023 allow tcp from any to 10.188.43.0/24 2223 out via vnet43=
 keep-state
> > ipfw add 43024 allow tcp from any to 2a01:138:9000:bc43::/64 2223 out v=
ia vnet43 keep-state
> > # git ssh port
> > ipfw add 43100 allow tcp from any to 2a01:138:9000:bc43::36 2222 out vi=
a vnet43 keep-state
> > ipfw add 43101 allow tcp from any to 10.188.43.54 2222 out via vnet43 k=
eep-state
> >
> > # catch all
> > ipfw add 64999 reject ip6 from any to any
> > ipfw add 65000 allow tcp from any to any
> > ipfw add 65001 allow udp from any to any
> >
> > This is a bit messi at the moment because I had done some changes and
> > tests because of the nat problem. The strange thing is I have tested
> > if I redirect the port 22 to the vnet interface of the host and this
> > works. It just doesn't work for the jails.
> >
> > Any idea what is going wrong or what change I have missed?
> >
> > Philipp
> >

Thanks for the hints. I have now changed my nat to pf so I can't test
right away but I try to switch back.

> I believe this is a known problem with checksum offloading and NAT.
>
> Try:
>
> ifconfig epairXa -txcsum -rxcsum -tso -lro
> ifconfig em1 -txcsum -rxcsum -tso -lro

With this hint I found 503bf058cd0a which adds checksum offloading
to epair(4). So I would guess disabling txcsum and rxcsum fixes the
issue. This also explaines why it worked on 14.3.

Sadly this is not good documented. I only know about one part in the
handbook to set net.inet.tcp.tso=3D"0"[0].

> Problems with ipfw and NAT (which required a special kernel module, =

> which I believe is included now) was one of the things that prompted me =

> to switch to pf, where everything is wonderful.

I still can't get my head around the last match style of pf, thats
why I prefere ipfw (even with the ungly rule numbers).

Philipp

[0] https://docs.freebsd.org/en/books/handbook/firewalls/#in-kernel-nat