another netfilter-initiated martian source

Hi!

I'm having, it looks like, an endless amount of issues with
netfilter hooking in the wrong order into the linux network
stack.  This time, it is the netfilter "reject" action, which
triggers "martian source" detection in the network stack, in
a weird way.

I've two "external" network interfaces (pointing to two ISPs),
with policy routing set up for them (different IPs are routed
to/via different ISPs), -- eth-mts and eth-rinet.  The main
routing table has no default route, there are two additional
routing tables (with just one "default" route pointing to one
or another ISP).  And a few rules like 'from foo lookup bar',
routing specific IPs via one or another ISP.

Also there's an "internal" interface, eth-tls, with 192.168.x
addresses behind it.  With NAT for certain internal hosts.

The policy rules:

   from 192.168.177.2 lookup route-mts
   from all lookup route-rinet

  route-mts:
   default via $route-mts dev eth-mts
  route-rinet:
   default via $route-rinet dev eth-rinet

The relevant netfilter rules:

  ip filter chain forward {
    type filter hook forward priority filter
    ct state {established,related} accept
    iifname eth-tls oifname eth-mts saddr 192.168.177.2 <some-condition> 
accept
    iifname eth-tls oifname eth-mts saddr 192.168.177.2 reject log 
prefix "tls->ext reject: "
  }

  ip nat chain postrouting {
    type nat hook postrouting priority srcnat
    oifname eth-mts ip saddr 192.168.177.2 counter snat to $mts-ip
  }

With these, the following packets are seen on eth-tls interface
(192.168.177.5 is this router box):

listening on eth-tls, link-type EN10MB (Ethernet), snapshot length 
262144 bytes
10:13:12.353416 IP 192.168.177.2.59152 > 16.15.192.246.443: Flags [S], 
seq 1254965270, win 64240, options [mss 1460,sackOK,TS val 1778537536 
ecr 0,nop,wscale 7], length 0
10:13:12.353617 IP 192.168.177.5 > 192.168.177.2: ICMP 16.15.192.246 tcp 
port 443 unreachable, length 68

So far, this all works as expected.

And at the same time, I see this in dmesg:

Jul 03 10:13:12 gate kernel: tls->ext reject: IN=eth-tls OUT=eth-mts 
MAC=00:90:27:30:6d:1c:d6:4d:61:a4:e3:03:08:00 SRC=192.168.177.2 
DST=16.15.192.246 LEN=60 TOS=0x00 PREC=0x00 TTL=63 ID=58986 DF PROTO=TCP 
SPT=41754 DPT=443 WINDOW=64240 RES=0x00 SYN URGP=0
ul 03 10:13:12 gate kernel: IPv4: martian source 192.168.177.2 from 
16.15.192.246, on dev eth-rinet
Jul 03 10:13:12 gate kernel: ll header: 00000000: 00 90 27 30 6d 1c d6 
4d 61 a4 e3 03 08 00

and this logging makes no sense whatsoever.

This *seems* to be triggered by the reject packet (ICMP), which should
be injected by the netfilter into the network stack.  Why it causes
"martians detection" is the first question - it should not, since
this packet is not sent by any external host.

Next, why it is logged as coming from eth-rinet interface??  It is
not coming from any interface, and it has nothing to do with this
interface either - initially it should've been routed to eth-mts,
not eth-rinet.

And finally, eth-rinet iface is configured to drop martians (in
addition to logging it) - rp_filter is set to 1.  Yet, this ICMP packet
is actually sent to the originating host.

What is going on here?

It seems that interaction between netfilter code and the network
stack should be reviewed.  There are a few rather serious bugs in
there, it looks like, - all are about the order of the hooks between
netfilter and network/routing stack.  We're almost there, but not
quite there yet :)

Thanks,

/mjt