Re: Questions on tunneling multiple connections between two servers with Linux

This is a cryptographically signed message in MIME format.

--------------ms080209040905030901020705
Content-Type: text/plain; charset=UTF-8; format=flowed
Content-Transfer-Encoding: 7bit

On 12/15/23 06:05, Martin Maurer wrote:
> Hello,

Hi,

> I hope this mailing list is still read even when around 4 years no more 
> messages have been exchanged.

The mailing list is still a thing.  I've seen messages in '23 on the 
LARTC mailing list.  --  Including a discussion about keeping it or not. 
  My understanding is that the result was to migrate to the new system.

> Or has the mailing list moved to some newer place and this one is now 
> obsolete?

Not that I'm aware of.

> I have some questions on tunneling with Linux:
> 
> I have two servers: A and B.
> 
> Server B has multiple interface (e.g. 4 interfaces, name it wwan0, 
> wwan1, wwan2, wwan3) going to different networks outside the current 
> network.

Maybe your examples below have been simplified but it doesn't look like 
they go outside what I would assume to be the current network; 
192.168.181.0/24.

Typical caveats about assumptions especially when you have /30 on your 
tunnels.

> Between server A and B there shall be one or more tunnels, so that 
> server A is able to select which data goes over which interface on 
> server B.

Would you please elaborate on how this selection is being made?  It 
might be germane.  At least specify if it's basic destination based 
routing or if it's more complex policy based routing.

> 1) I tried GRE. Configured 4 IPv4 IP adresses on server A and 4 IPv4 IP 
> addresses on server B.

Okay.

> For each pair of IP addresses I installed a GRE tunnel between A and B.
> 
> I named the tunnels conn0 to conn3:
> 
> #!/bin/bash
> sysctl -w net.ipv4.conf.all.forwarding=1
> sysctl -w net.ipv6.conf.all.forwarding=1
> ip tunnel del conn0
> ip tunnel add conn0 mode gre local 192.168.181.1 remote 192.168.181.2 
> dev eth0
> ip addr add 10.0.0.1/30 dev conn0
> ip link set conn0 up
> ...
> 
> On server B I added forwarding rules like this (4 interfaces shown)

I think we might be using different definitions of "forwarding rule".

> sudo iptables-F

I think that `-F` is going to flush the `filter` table which is the 
default.  --  Maybe I'm mistaken and it also flushes all tables.

> sudo iptables-P INPUT ACCEPT
> sudo iptables-P FORWARD ACCEPT
> sudo iptables-P OUTPUT ACCEPT

Basic reset of default part of the filter table.

> sudoiptables-t nat-A POSTROUTING -o wwan0 -j MASQUERADE

You /might/ be dealing with unwanted state in the `nat` table.  (See 
comment about `-F` above.)

> sudoiptables-A FORWARD -i conn0 -o wwan0 -j ACCEPT
> 
> sudoiptables-A FORWARD -i wwan0 -o conn0 -m 
> state--stateESTABLISHED,RELATED-j ACCEPT
> ...


These seem both simple enough.

But they also seem redundant as I don't see anything that would deviate 
from the default policy of ACCEPT set above.

> Could this work? Or I am missing something?

Maybe.

I don't know what your routing tables look like on systems A or B nor do 
I know how you are testing.

I feel the need to say that FORWARDING is ostensibly messing with routed 
traffic.  As such, it is dependent on if systems A and B think they need 
to route the traffic between different subnets.

I'm not sure if you are trying to deal with bridged / unrouted traffic.

That being said, iptables does have an option to filter bridged traffic 
too.  But that's quite a bit more complex and would require a much more 
complete understanding of your configurations, tests cases, and what 
your end goal is.

> Do I really need 4 IP addresses on server A and server B for the 4 
> tunnels? Or can I do it with just 1 real IPv4 address on each side?

I assume that you're asking about GRE tunnels and eliding other types of 
tunneling.

My understanding is that GRE supports some form of key (think 
identifying tag, not encryption).  As such, I think that you might be 
able to establish different tunnels using different keys between the 
same two endpoint IPs.

We did this at ${OLD_DAY_JOB} between Juniper switches.  I /think/ that 
Linux supports doing it too, but I've never tried.

> "Inside" addresses relevant, or can i just use the same for each tunnel 
> (e.g. 10.0.0.1 and 10.0.0.2)

If you want to use basic destination based routing, you will almost 
certainly need different addresses inside the tunnel.

More complex policy based routing and / or iptables route manipulation 
/may/ be able to do this with duplicate IPs inside the tunnel.

That being said, I would HIGHLY suggest that you not introduce the 
complexity of duplicate IPs in tunnels without good reason to do so. 
There are many options to choose from to avoid duplicate IPs.

Heck, I think I'd go for unnumbered interfaces / device routes before 
I'd go the duplicate IP route.

N.B. reverse path filtering may bite you here if you aren't careful.

I'd also suggest a good understanding of the strong / weak host model as 
vagaries of that could also bite you.

I almost guarantee that you will end up needing to use a sniffer to 
debug this.  tcpdump and Wireshark / tshark are your friends, trust them.

#trustTheBitsOnTheWire

> I see with wireshark that sometimes packets are not exchanged over the 
> right tunnel. But I have not yet analyzed what is the problem.

Which tunnel may have a lot to do with routing.

Aside:  That's what I get for typing replies as I read the message.  As 
soon as I say something about sniffer, your next statement is saying you 
are using one.  <ASCII thumbs up>

> 2) A suggestion was to use a stream id with IP-in-IP.
> Can it really used for such a use case? Is the stream id used for 
> tunneling purposes at all? Is it supported by Linux,> to fill in 
> one sending tunneling endpoint and using it on receiving tunneling 
> endpoint?

I'm not sure what stream id is in this context.  I've seen it at 
multiple layers but can't recall exactly what it would be in this context.

Sure, there are ways to have various things look at a specific id, but 
said IDs are almost always transient / ephemeral in nature and thus not 
good candidates to use in a static firewall configuration.

I also think that dynamically updating iptables has it's own set of 
problems.

If you want a dynamic firewall I'd highly suggest things like iptables' 
recent match extension.

But I'd strongly suggest relying on routing, perhaps policy based 
routing, and not dynamically alter the firewall without really good reason.

Read: really good reason being "it won't work without doing this" or "i 
am making a decision to do this contrary to others advice".

> 3) Assuming in next step I also need encryption on tunnels:

Okay.

I was wondering if encryption would be needed.

> Better to switch over to some different approach like VPN?
Aside:  What is /private/ in a VPN???

Private may simply be something that's not shared with someone else.  Or 
it may be something that's encrypted such that only those with the 
decryption material can see it.

GRE /is/ a virtual /private/ network in that it's for a specific set of 
traffic and shouldn't have other traffic.

Also, VPN can be a LOT of different things.

Based on the examples tunnels above, both A and B seem to be on a shared 
network segment.  As such I would wonder if VLANs might work.

VXLAN and MPLS might also be an option.  Though they both usually have a 
bit of other complexity / daemons that they bring with them, though I'm 
not confident that they are strictly required.

I am confident that there are many other networking technologies that 
the Linux kernel supports that can fall into the VPN category 
(unencrypted and / or encrypted).

Then there is native Linux support for IPsec which can be wrapped around 
things.

You could use IPsec /Transport/ Mode to protect the GRE tunnels that you 
have.  --  I don't know if that would change the multiple IP pairs 
requirement or if Linux's transform (xform) support has the capability 
of differentiating GRE keys (tags).

You could use Linux's native IPsec /Tunnel/ Mode in place of any other 
tunneling technology.

I would probably suggest avoiding newer VPNs, which tend to require more 
user space interaction (at least for configuration), like OpenVPN or 
WireGuard.  Sure, they will work, but there is a bit more that they need 
to work than I think is strictly necessary.

Aside:  I had IPsec /transport/ mode working between my public systems 
in a full mesh with shell scripts and kernel support.  It worked great.

Further aside:  SNATing complicates the /need/ for tunnel mode vs 
transport mode.  I successfully SNATed clients at home to my home 
router's public IP -- as one oft does in a SOHO configuration -- and 
easily connected to my VPS via the aforementioned full mess of IPsec 
/transport/ mode.  SNATing happened before IPsec xfrm so it was quite 
happy to use /transport/ to connect from my home's SOHO's public IP to 
my VPS's public IP. }:-)

> Or is there some "secure GRE" or "secure IP-in-IP"? Or possible to use 
> stunnel, or stunnel together with GRE or IP-in-IP? Or not preferred way?

IPsec can be used to protect GRE or IP-in-IP traffic.

N.B. My understanding is that IP-in-IP is a Linux specific counterpart 
that is roughly comparable to GRE.  I think that Linux's support for 
IP-in-IP pre-dates Linux's support for GRE.  I've always tried to favor 
GRE over IP-in-IP in order to be more compatible with other things.

stunnel falls into that additional complexity and / or dependency on 
user-space things for network communications.  --  It's okay if you need 
/ want it being aware of the dependency.  But I'd try to avoid it in 
this case.

> Are there other variants instead of VPN which I shall look at?

I don't know enough about what you're trying to accomplish so I can't say.

But based on your email, I'd try multiple GRE tunnels between the same 
pair of IPs using GRE keys (tags) to differentiate the tunnels and I'd 
use IPsec /transport/ mode to protect said GRE tunnels.  Or at least 
that's what I'd try first.  If that didn't work and I couldn't figure 
out how to get past it, then I'd explore other options.

> 4) A question which came up was "MTU". The above tunnels eats some bytes 
> from MTU.

Ya.  That's going to be a problem in just about everything you do.

One advantage of the conn# GRE tunnel interfaces is that you can set a 
small enough MTU that will mean that the outer tunnel / encryption won't 
need to be fragmented.

If you are on a LAN, you might be able to explore Jumbo Frames to be 
able to pass larger tunnel / encrypted frames / packets (which term 
depends on the technology) so that you can pass full 1500 B frames 
through the tunnel.

I /think/ that some tunneling technologies support fragmenting things so 
that you can carry a 1500 B frame in multiple tunneled / encrypted 
packets / frames on the next network that only supports 1500 B frames.

But this is getting quite complex.

I'd suggest lowering the MTU on the conn# interfaces to 1400 (or 
whatever is necessary) to avoid the fragmentation issue and allow 
routing to do what it has been doing for 40+ years.

That is unless you have an explicit reason to NOT do this as in what 
you're trying to do won't support this.  But if that's the case, please 
share more details about what you're working on.

> What happens if e.g. MTU gets reduced by GRE or IP-in-IP by 24 or 20 
> bytes and a packet is sent which exceeds the reduced MTU.

That depends on your configuration.

> Is it fragmented automatically? Do I need to take care that sent 
> packets are now really smaller than 1476 or 1480 bytes?

You should address this and avoid fragmentation if your sending & 
receiving applications suffer from fragmentation.

> Is there perhaps some documentation, webpages or books which can help 
> me with info on such topics?

Yes, lots, at least 20 years worth.  But much of it is background and 
not much of it is a recipe.

> Many thanks in advance!

You're welcome.



-- 
Grant. . . .

--------------ms080209040905030901020705
Content-Type: application/pkcs7-signature; name="smime.p7s"
Content-Transfer-Encoding: base64
Content-Disposition: attachment; filename="smime.p7s"
Content-Description: S/MIME Cryptographic Signature

MIAGCSqGSIb3DQEHAqCAMIACAQExDzANBglghkgBZQMEAgEFADCABgkqhkiG9w0BBwEAAKCC
C0owggUyMIIEGqADAgECAhEA0VX8K1B6bsGDH67kXwHPWzANBgkqhkiG9w0BAQsFADCBljEL
MAkGA1UEBhMCR0IxGzAZBgNVBAgTEkdyZWF0ZXIgTWFuY2hlc3RlcjEQMA4GA1UEBxMHU2Fs
Zm9yZDEYMBYGA1UEChMPU2VjdGlnbyBMaW1pdGVkMT4wPAYDVQQDEzVTZWN0aWdvIFJTQSBD
bGllbnQgQXV0aGVudGljYXRpb24gYW5kIFNlY3VyZSBFbWFpbCBDQTAeFw0yMzExMTIwMDAw
MDBaFw0yNDExMTEyMzU5NTlaMCsxKTAnBgkqhkiG9w0BCQEWGmd0YXlsb3JAdG5ldGNvbnN1
bHRpbmcubmV0MIIBIjANBgkqhkiG9w0BAQEFAAOCAQ8AMIIBCgKCAQEAzOnBjTJUlBTzN81c
PlYErJc9kEbTI/hXq0NA6ZoG4VM6puYTEXtITANjgX+NRwwHjldESnC8dvh6Mx5ckEk9sWoD
l8Yr/dWhF3s4fGxAX5ziOeuBI/yX7rKJn6DOwclV3C6dyt3zrLB6LOiF4gA+lk/o3EbOwoPh
pW2MqAywy18OIvzfmEXKdya8E/uIP4v/8AHmtakxHfmZ33Krbwh2oia69esRKc7q2i3Jh+ar
Tf3PuZJETd86Sb0Lz1+3zAXcYko2/3G9O9AwtUSDvkx5IUKieG8R4a8HLwuUTBNIsJ0qOdmv
4hUjc3IsP0jN+xebTE4w7PheolE/OStiFshpKQIDAQABo4IB4zCCAd8wHwYDVR0jBBgwFoAU
CcDy/AvalNtf/ivfqJlCz8ngrQAwHQYDVR0OBBYEFPUkNRFsHVlNMgaz3G4kfNa8DU4VMA4G
A1UdDwEB/wQEAwIFoDAMBgNVHRMBAf8EAjAAMB0GA1UdJQQWMBQGCCsGAQUFBwMEBggrBgEF
BQcDAjBQBgNVHSAESTBHMDoGDCsGAQQBsjEBAgEKATAqMCgGCCsGAQUFBwIBFhxodHRwczov
L3NlY3RpZ28uY29tL1NNSU1FQ1BTMAkGB2eBDAEFAQIwWgYDVR0fBFMwUTBPoE2gS4ZJaHR0
cDovL2NybC5zZWN0aWdvLmNvbS9TZWN0aWdvUlNBQ2xpZW50QXV0aGVudGljYXRpb25hbmRT
ZWN1cmVFbWFpbENBLmNybDCBigYIKwYBBQUHAQEEfjB8MFUGCCsGAQUFBzAChklodHRwOi8v
Y3J0LnNlY3RpZ28uY29tL1NlY3RpZ29SU0FDbGllbnRBdXRoZW50aWNhdGlvbmFuZFNlY3Vy
ZUVtYWlsQ0EuY3J0MCMGCCsGAQUFBzABhhdodHRwOi8vb2NzcC5zZWN0aWdvLmNvbTAlBgNV
HREEHjAcgRpndGF5bG9yQHRuZXRjb25zdWx0aW5nLm5ldDANBgkqhkiG9w0BAQsFAAOCAQEA
PQ6Jlp6ozAXKe2VGjovH67KxmyxktTuxPbxyTMtA0WwiCBiWgpF7SVmJCN8501uAeVqhEUeW
66J11qs2jBuLqsaUjiA114qU1wMBD2tsDZI14Amn1C50YWTqBQDbrvwAqV2yYPOWF6/N8xHF
wXod6jsRnGjK541AkLQU0MjFPpL0orVEsPsV5TaUB3tDCKzzdRvZ0Ey7DlxgGCXeGstFkwDe
aJmBgHmD76iZ9Bil/TEsQ62zwcvq6w1df8+iODIgbx0WKGhrqn0coyeEtumaKh8Mya8KPyHI
6UqjcbkWbefnEduO/T1Dc9E9jBakSAgPUbUdRAZnpiVDLH7G2yOBJjCCBhAwggP4oAMCAQIC
EE2ULBDUO+CUCcWBLTorBk8wDQYJKoZIhvcNAQEMBQAwgYgxCzAJBgNVBAYTAlVTMRMwEQYD
VQQIEwpOZXcgSmVyc2V5MRQwEgYDVQQHEwtKZXJzZXkgQ2l0eTEeMBwGA1UEChMVVGhlIFVT
RVJUUlVTVCBOZXR3b3JrMS4wLAYDVQQDEyVVU0VSVHJ1c3QgUlNBIENlcnRpZmljYXRpb24g
QXV0aG9yaXR5MB4XDTE4MTEwMjAwMDAwMFoXDTMwMTIzMTIzNTk1OVowgZYxCzAJBgNVBAYT
AkdCMRswGQYDVQQIExJHcmVhdGVyIE1hbmNoZXN0ZXIxEDAOBgNVBAcTB1NhbGZvcmQxGDAW
BgNVBAoTD1NlY3RpZ28gTGltaXRlZDE+MDwGA1UEAxM1U2VjdGlnbyBSU0EgQ2xpZW50IEF1
dGhlbnRpY2F0aW9uIGFuZCBTZWN1cmUgRW1haWwgQ0EwggEiMA0GCSqGSIb3DQEBAQUAA4IB
DwAwggEKAoIBAQDKPO2UCkH/3vlGuejWO+bakr8rEE6qGryCvb4mHCkqKtLNnFCBP22ULvOX
qGfV9eNKjkypdR8i0yW2sxpepwRIm4rx20rno0JKuriIMpoqr03E5cWapdfbM3wccaNDZvZe
/S/Uvk2TUxA8oDX3F5ZBykYQYVRR3SQ36gejH4v1pXWuN82IKPdsmTqQlo49ps+LbnTeef8h
Nfl7xZ8+cbDhW5nv0qGPVgGt/biTkR7WwtMewu2mIr06MbiJBEF2rpn9OVXH+EYB7PmHfpsE
kzGp0cul3AhSROpPyx7d53Q97ANyH/yQc+jl9mXm7UHR5ymr+wM3/mwIbnYOz5BTk7kTAgMB
AAGjggFkMIIBYDAfBgNVHSMEGDAWgBRTeb9aqitKz1SA4dibwJ3ysgNmyzAdBgNVHQ4EFgQU
CcDy/AvalNtf/ivfqJlCz8ngrQAwDgYDVR0PAQH/BAQDAgGGMBIGA1UdEwEB/wQIMAYBAf8C
AQAwHQYDVR0lBBYwFAYIKwYBBQUHAwIGCCsGAQUFBwMEMBEGA1UdIAQKMAgwBgYEVR0gADBQ
BgNVHR8ESTBHMEWgQ6BBhj9odHRwOi8vY3JsLnVzZXJ0cnVzdC5jb20vVVNFUlRydXN0UlNB
Q2VydGlmaWNhdGlvbkF1dGhvcml0eS5jcmwwdgYIKwYBBQUHAQEEajBoMD8GCCsGAQUFBzAC
hjNodHRwOi8vY3J0LnVzZXJ0cnVzdC5jb20vVVNFUlRydXN0UlNBQWRkVHJ1c3RDQS5jcnQw
JQYIKwYBBQUHMAGGGWh0dHA6Ly9vY3NwLnVzZXJ0cnVzdC5jb20wDQYJKoZIhvcNAQEMBQAD
ggIBAEFEdQCrOcIV9d6OlW0ycWiMAN0X13ocEDiQyOOxvRcxkfO244K0oX7GzCGHYaqRbklC
szzNWVT4DZU/vYrLaeVEDUbCYg+Ci7vhNn9dNqscbzN0xKBoOuRVjPPWDechU70geT3pXCxp
wi8EXwl+oiz7xpYfY99JSs3E/piztTSxljHitcPr5yoWr9lbkFR8KU3+uGTZ11BfKfuSSaRr
ZFBv133SeY0d2AqvB9Dj2ZDaFZA0OQkkhfAqNgDpVRH99lQV4JSKx0N7/QAEtMj6OF5dRXV6
hhXuU3A0Eql4d0247oBpxvnfcmV95QfG8HP059hZSJe7T2wwC+IzXVDQO4xnnvrQJ07ZWemx
c/grFpgiG+o+pQxapF1bKftysi02Rl6uhdp5wbTeLeYzt2SI9oKSChwGDQQFixtkNnxuwbdr
TwvASwvViDPdIGzIQJrTBqriE5/9nzkXbDZmld8/7DyriJ/A73RIZllX4dH8mHqsRpU8NEX8
IQZWpHWGK5A5nVgvl7MxNfRlIvCvKZQTSnCL8oNqJgHXm6zCB4gBwDonM8V/2kuQAUVazVA3
I376eIWGwzjuqh3H88v7mNHzubLHm5h0ERCSQNz6UoHVZy3q5xeqbYSaxpDQz3lCNObL6sNa
OQNh3DcyzqZJYTcGfuLlmC3AIteAAh7lbybJszYnMYIENTCCBDECAQEwgawwgZYxCzAJBgNV
BAYTAkdCMRswGQYDVQQIExJHcmVhdGVyIE1hbmNoZXN0ZXIxEDAOBgNVBAcTB1NhbGZvcmQx
GDAWBgNVBAoTD1NlY3RpZ28gTGltaXRlZDE+MDwGA1UEAxM1U2VjdGlnbyBSU0EgQ2xpZW50
IEF1dGhlbnRpY2F0aW9uIGFuZCBTZWN1cmUgRW1haWwgQ0ECEQDRVfwrUHpuwYMfruRfAc9b
MA0GCWCGSAFlAwQCAQUAoIICWTAYBgkqhkiG9w0BCQMxCwYJKoZIhvcNAQcBMBwGCSqGSIb3
DQEJBTEPFw0yMzEyMTYxOTA3NDhaMC8GCSqGSIb3DQEJBDEiBCA6MinF/3/djZgrkejMoAFW
l8zp8j/0E3GTbVLfrcSJqzBsBgkqhkiG9w0BCQ8xXzBdMAsGCWCGSAFlAwQBKjALBglghkgB
ZQMEAQIwCgYIKoZIhvcNAwcwDgYIKoZIhvcNAwICAgCAMA0GCCqGSIb3DQMCAgFAMAcGBSsO
AwIHMA0GCCqGSIb3DQMCAgEoMIG9BgkrBgEEAYI3EAQxga8wgawwgZYxCzAJBgNVBAYTAkdC
MRswGQYDVQQIExJHcmVhdGVyIE1hbmNoZXN0ZXIxEDAOBgNVBAcTB1NhbGZvcmQxGDAWBgNV
BAoTD1NlY3RpZ28gTGltaXRlZDE+MDwGA1UEAxM1U2VjdGlnbyBSU0EgQ2xpZW50IEF1dGhl
bnRpY2F0aW9uIGFuZCBTZWN1cmUgRW1haWwgQ0ECEQDRVfwrUHpuwYMfruRfAc9bMIG/Bgsq
hkiG9w0BCRACCzGBr6CBrDCBljELMAkGA1UEBhMCR0IxGzAZBgNVBAgTEkdyZWF0ZXIgTWFu
Y2hlc3RlcjEQMA4GA1UEBxMHU2FsZm9yZDEYMBYGA1UEChMPU2VjdGlnbyBMaW1pdGVkMT4w
PAYDVQQDEzVTZWN0aWdvIFJTQSBDbGllbnQgQXV0aGVudGljYXRpb24gYW5kIFNlY3VyZSBF
bWFpbCBDQQIRANFV/CtQem7Bgx+u5F8Bz1swDQYJKoZIhvcNAQEBBQAEggEATAYkidvDuBp6
/P1EwATbaTmTTSUufeF00xqs4hpMoGMjmx6yBEuhTC1tVNUU0xwzCpZXfToHUcCrUlaaUZ7E
xWyM4ZA5KX6YkF48w8WUTfB2awI+9GsLd2+KVywobXdnuzse3XXsRUZE3G4TzIcLuYIa6Pq/
Ilroawh9r2aGoT0Z9LdN+LN97KTuT5Q5RF1JQ5vei0K8eaHLT22EbrYLtUcdMlCclgEeQyMe
heeHcClB/+5yMAKjtJBUXcurFdRF93UjoBLI75w4HpJIeav/l0dyhJOMVkTaKPmtgLE4uhSu
y4njYBKafugLsSOkW8Ru4se9ZLDZ8kYJbiY3b3udIAAAAAAAAA==
--------------ms080209040905030901020705--