Re: GRE-NAT broken - SOLVED

On 01/30/2018 03:37 PM, Grant Taylor wrote:
> It seems as if something is intercepting the packets.  -  I doubt that 
> it's the NAT module, but I can't rule it out.

Well, I think I ran into the same problem in my tests.

Spoiler:  I did manage to overcome it.

I think the Connection Tracking was part of my (initial?) problem.

> Wait.  tcpdump shows that packets are entering one network interface but 
> they aren't leaving another network interface?

I was seeing this behavior too.

> That sounds like something is filtering the packets.

I think connection tracking (thus NAT) was (at least) part of the culprit.

> I feel like the kicker is that the traffic is never making it out of the 
> local system to the far side.  As such the far side never gets anything, 
> much less replies.

I don't know if this was the case for my testing or not.  I did all of 
my testing from the far side in.

> Ya, the [UNREPLIED] bothers me.  As does the fact that you aren't seeing 
> the traffic leaving the host's external interface.

The [UNREPLIED] was the kicker for me.

> I'd look more into the TRACE option (target) that you seem to have 
> enabled in the raw table.  That should give you more information about 
> the packets flowing through the kernel.

I ended up not using TRACE.

I'm not sure why I did a "conntrack -D", but as soon as I did, my long 
running ping started working.

Upon retesting I can confirm that "conntrack -D" was required to make 
things work.

Further testing and using "conntrack -L" showed that there were some 
connection tracking states that were in an [UNREPLIED] state.  I think 
that "conntrack -D" cleared the stale connections and allowed things to 
start working.

> My hunch is that the packets aren't making it out onto the wire for some 
> reason.  Thus the lack of reply.

After the testing that I did, I suspect that packets did make it onto 
the wire, but were swallowed by connection tracking, thus NAT as you had 
originally thought.

> I'll see if I can't throw together a PoC in Network namespaces this 
> evening to evaluate if NATing GRE works.  -  I'd like to test NATing 
> different sets of endpoints (1:1) and NATing multiple remote endpoints 
> to one local endpoint (many:1).

I threw together a Proof of Concept using Network Namespaces, using a 
pair of OVSs (bri1 & bri2) and a pair of vEths (between R3 / R2 and R2 / 
R1).

I was able to establish a initially establish a GRE tunnels between H1 / 
H3 and H2 / H4.

After figuring out the the connection tracking problem I was also able 
to bring up additional GRE tunnels between H1 / H4 and H2 / H3.

Take a look at the attached GRE-NAT.sh script.  -  I do take some 
liberties and set up aliases to make things easier.  (Read: I'm lazy and 
don't want to type any more characters than I have to.)

alias vsctl='ovs-vsctl'
alias h1='ip netns exec h1'
alias h2='ip netns exec h2'
alias h3='ip netns exec h3'
alias h4='ip netns exec h4'
alias r1='ip netns exec r1'
alias r2='ip netns exec r2'
alias r3='ip netns exec r3'

The network between:

H1 / H2 / R3 is Test-Net-1, 192.0.2.0/24
R3 / R2 is Test-Net-2, 198.51.100.0/24
R2 / R1 is Test-Net-3, 203.0.113.0/24
R1 / H3 / H4 is RFC 1918 private, 192.168.0.0/24

I addressed the GRE tunnels as RFC 1918 private, 10.<Left #>.<Right 
#>.<Device #/24.

R3 & R1 are numbered the way that they are so that their device # 
doesn't conflict with something local.

I did manage to get the PoC to work without needing to issue the 
"conntrack -D" command by simply moving the NAT rules earlier in the 
script before I tried to establish the tunnels.

I can only surmise that there was some sort of bad state that connection 
tracking learned that couldn't fix itself.  -  This was sort of random 
and unpredictable, much like what you're saying.  -  It also likely has 
to do with what end talks first.

I found that I could get things to start working if I issued the 
following command:

(ip netns exec) r1 conntrack -D

Ultimately I was able to issue the following commands:

h1 ping -c 4 10.1.3.3
h1 ping -c 4 10.1.4.4

h2 ping -c 4 10.2.3.3
h2 ping -c 4 10.2.4.4

h3 ping -c 4 10.1.3.1
h3 ping -c 4 10.2.3.2

h4 ping -c 4 10.1.4.1
h4 ping -c 4 10.2.4.2

I /think/ that this is what you were wanting to do.  And, I think you 
were correct all along in that NAT ~> connection tracking was in fact 
messing with you.

Anyway, have fun with the PoC.  Ask if you have any questions about what 
/ why / how I did something.

Oh, ya, I did have the following GRE related modules loaded:

# lsmod | grep -i gre
nf_conntrack_proto_gre    16384  0
nf_nat_proto_gre          16384  0
ip_gre                    24576  0
ip_tunnel                 28672  1 ip_gre
gre                       16384  1 ip_gre

I'm running kernel 4.9.76-gentoo-r1.

> You might be onto something about the first packet.  At least as far as 
> what connection tracking sees.

I think the kicker has to do with connection tracking learning state on 
the first packet.



-- 
Grant. . . .
unix || die

alias vsctl='ovs-vsctl'
alias h1='ip netns exec h1'
alias h2='ip netns exec h2'
alias h3='ip netns exec h3'
alias h4='ip netns exec h4'
alias r1='ip netns exec r1'
alias r2='ip netns exec r2'
alias r3='ip netns exec r3'
ip netns add h1
ip netns add h2
ip netns add h3
ip netns add h4
ip netns add r1
ip netns add r2
ip netns add r3
vsctl add-br bri1
vsctl add-br bri2
vsctl add-port bri1 h1 -- set interface h1 type=internal
vsctl add-port bri1 h2 -- set interface h2 type=internal
vsctl add-port bri1 r3 -- set interface r3 type=internal
vsctl add-port bri2 h3 -- set interface h3 type=internal
vsctl add-port bri2 h4 -- set interface h4 type=internal
vsctl add-port bri2 r1 -- set interface r1 type=internal
ip link set h1 netns h1
ip link set h2 netns h2
ip link set h3 netns h3
ip link set h4 netns h4
ip link set r1 netns r1
ip link set r3 netns r3
ip link add r2 type veth peer name r3
ip link set r2 netns r3
ip link set r3 netns r2
ip link add r2 type veth peer name r1
ip link set r2 netns r1
ip link set r1 netns r2
h1 ip link set lo up
h1 ip link set h1 up
h2 ip link set lo up
h2 ip link set h2 up
h3 ip link set lo up
h3 ip link set h3 up
h4 ip link set lo up
h4 ip link set h4 up
r1 ip link set lo up
r1 ip link set r1 up
r1 ip link set r2 up
r2 ip link set lo up
r2 ip link set r1 up
r2 ip link set r3 up
r3 ip link set lo up
r3 ip link set r3 up
r3 ip link set r2 up
h1 ip addr add 192.0.2.1/24 dev h1
h2 ip addr add 192.0.2.2/24 dev h2
h3 ip addr add 192.168.0.3/24 dev h3
h4 ip addr add 192.168.0.4/24 dev h4
r1 ip addr add 192.168.0.1/24 dev r1
r1 ip addr add 203.0.113.1/24 dev r2
r1 ip addr add 203.0.113.3/24 dev r2
r1 ip addr add 203.0.113.4/24 dev r2
r2 ip addr add 203.0.113.2/24 dev r1
r2 ip addr add 198.51.100.2/24 dev r3
r3 ip addr add 198.51.100.3/24 dev r2
r3 ip addr add 192.0.2.3/24 dev r3
h1 ip route add default via 192.0.2.3
h2 ip route add default via 192.0.2.3
h3 ip route add default via 192.168.0.1
h4 ip route add default via 192.168.0.1
r1 ip route add default via 203.0.113.2
r2 ip route add 192.0.2.0/24 via 198.51.100.3
r3 ip route add 203.0.113.0/24 via 198.51.100.2

r1 iptables -t nat -A PREROUTING -d 203.0.113.3 -p 47 -j DNAT --to-destination 192.168.0.3
r1 iptables -t nat -A PREROUTING -d 203.0.113.4 -p 47 -j DNAT --to-destination 192.168.0.4
r1 iptables -t nat -A POSTROUTING -s 192.168.0.3 -d 192.0.2.1 -p 47 -j SNAT --to-source 203.0.113.3
r1 iptables -t nat -A POSTROUTING -s 192.168.0.4 -d 192.0.2.2 -p 47 -j SNAT --to-source 203.0.113.4

h1 ip tunnel add h3 mode gre local 192.0.2.1 remote 203.0.113.3
h1 ip link set h3 up
h1 ip addr add 10.1.3.1/24 dev h3
h1 ip tunnel add h4 mode gre local 192.0.2.1 remote 203.0.113.4
h1 ip link set h4 up
h1 ip addr add 10.1.4.1/24 dev h4

h2 ip tunnel add h3 mode gre local 192.0.2.2 remote 203.0.113.3
h2 ip link set h3 up
h2 ip addr add 10.2.3.2/24 dev h3
h2 ip tunnel add h4 mode gre local 192.0.2.2 remote 203.0.113.4
h2 ip link set h4 up
h2 ip addr add 10.2.4.2/24 dev h4

h3 ip tunnel add h1 mode gre local 192.168.0.3 remote 192.0.2.1
h3 ip link set h1 up
h3 ip addr add 10.1.3.3/24 dev h1
h3 ip tunnel add h2 mode gre local 192.168.0.3 remote 192.0.2.2
h3 ip link set h2 up
h3 ip addr add 10.2.3.3/24 dev h2

h4 ip tunnel add h1 mode gre local 192.168.0.4 remote 192.0.2.1
h4 ip link set h1 up
h4 ip addr add 10.1.4.4/24 dev h1
h4 ip tunnel add h2 mode gre local 192.168.0.4 remote 192.0.2.2
h4 ip link set h2 up
h4 ip addr add 10.2.4.4/24 dev h2