Re: Dual-Homed/Triple-Subnet Bridge Challenge

Grant,

Many thanks for your prompt and detailed response. Let me step back and
give you some background before I answer your questions....

It seems to me that I have a VERY UNUSUAL server configuration
requirement. To solve it, I have progressed from examining the following
possible solutions:

1. Arp-filter
2. Bonding/Port Trunking 
3. Bridging

The system in question is an "Image Server" based around Ubuntu 6.06 LTS
(Dapper) Server Edition and Samba. It is essentially a repository for
hard disk images which can be pushed to (and pulled from) the server
using special purpose DOS-based client network boot disks.

These network boot disks use a DHCP address if available but will fall
back to a ZeroConf/LinkLocal/APIPA address in the 169.254.*.* range if a
DHCP Server is not available. This behaviour was originally prompted by
the requirement to be able to use a laptop and cross-over patch lead as
an "Imaging Server" when on the road (the laptop will fall back to a
Zeroconf address in the absence of DHCP). These
universal/general-purpose boot disks are tried-and-tested and I do not
want to modify them or have to maintain different versions for different
Image Server configurations.

My Image Server will, in the first instance, be configured for it's
"home network" for the purpose of software updates and general admin,
file transfers, etc. However, as far as imaging operations are
concerned, it needs to be a general purpose solution and cannot make any
assumptions about the presence or otherwise of a DHCP Server.

For example:

1. I may be imaging (or restoring) the DHCP Server itself (which will be
off-line during the imaging process).

2. I may be imaging (or restoring) on a subnet which does not support
DHCP by design.

3. I may be imaging (or restoring) using a direct/cross-over cable or a
spare switch in the lab.

4. I may even have my Image Server with me on the road.

I might also point-out that adding DHCP Server functionality to the
Image Server itself is not an option as it would conflict with an
existing DHCP server (if present).

In the absence of DHCP, my Image Server needs to match the behaviour of
my client boot disks. Fortunately, the range of "dynamic" Zeroconf
addresses used by the client is restricted so that I can confidently
assign a "static" fall-back address alias to the Imaging Server outside
this range but within the Zeroconf subnet.

My original /etc/network/interfaces file (in the SINGLE-HOMED case)
would have looked something like this:

auto eth0
iface eth0 inet static
address 192.168.3.13
netmask 255.255.255.0
network 192.168.3.0
broadcast 192.168.3.255
gateway 192.168.3.1
dns-nameservers 192.168.3.1

auto eth0:0
iface eth0:0 inet static
address 169.254.0.13
netmask 255.255.0.0
network 169.254.0.0
broadcast 169.254.255.255

This SINGLE-HOMED solution has worked exceptionally well and proven
itself over time.

Having recently added a second subnet to my network, I was faced with
the choice of installing a second Image Server on the new subnet or
changing my existing Image Server to a DUAL-HOMED system by adding a
second NIC. Apart from the cost saving, a single image repository is
always preferable from an administration point of view.

The DUAL-HOMED version of my /etc/network/interfaces file would have
looked something like this:

auto eth0
iface eth0 inet static
address 192.168.3.13
netmask 255.255.255.0
network 192.168.3.0
broadcast 192.168.3.255
gateway 192.168.3.1
dns-nameservers 192.168.3.1

auto eth0:0
iface eth0:0 inet static
address 169.254.0.13
netmask 255.255.0.0
network 169.254.0.0
broadcast 169.254.255.255

auto eth1
iface eth1 inet static
address 192.168.4.13
netmask 255.255.255.0
network 192.168.4.0
broadcast 192.168.4.255

auto eth1:0
iface eth1:0 inet static
address 169.254.0.14
netmask 255.255.0.0
network 169.254.0.0
broadcast 169.254.255.255

Of course, my fallback Zeroconf addresses will no longer work in this
scenario because eth0:0 and eth1:0 are within the same subnet and you
have what essentially boils down to a routing conflict.

I initially tried an arp-filter solution, which I was able to verify by
pinging the server from Zeroconf clients on each subnet. Unfortunately
Samba will not play nicely in this scenario, probably something to do
with netbios broadcasts, which are initiated from the server side. This
pointed to the need for a round-robin ARP broadcast mechanism across the
two interfaces.

This prompted me to try a bonding solution using mode 3 (broadcast). My
/etc/network/interfaces configuration then looked something like this: 

auto bond0
iface bond0 inet static
address 169.254.0.13
netmask 255.255.0.0
network 169.254.0.0
broadcast 169.254.255.255

auto bond0:0
iface bond0:0 inet static
address 192.168.3.13
netmask 255.255.255.0
network 192.168.3.0
broadcast 192.168.3.255
gateway 192.168.3.1
dns-nameservers 192.168.3.1

auto bond0:1
iface bond0 inet static
address 192.168.4.13
netmask 255.255.255.0
network 192.168.4.0
broadcast 192.168.4.255

The bonding solution worked perfectly...well almost so.

This solution falls down if I am imaging multiple Zeroconf clients
simultaneously on both subnets. As both sides of the Zeroconf subnet are
distinct, address conflicts may not get picked up during client address
negotiation. Admittedly, this will be a rare eventuality, particularly
since client addresses are generated randomly and seeded uniquely, but
nevertheless, it is not a strictly correct solution.

...and that brought me to my bridging "challenge" and your reply.

And so to your questions:

> I must ask, why not bind the 169.254.x.y/16 subnet to the 
> eth1:0 interface?

Because of routing conflict (as discussed above)
 
> Note that aliased interfaces are usually not ""real 
> interfaces in such 
> as they do not appear as an interface that can be used in 
> routing and / 
> or firewalling / filtering.  As such, you are effectively 
> dealing with 
> the real unaliased interface.

Sorry, I should have just said bridge from eth0 to eth1 (not eth0:0 to
eth1)

> Are you planing on having the eth1 network having multiple subnets on 
> it, i.e. the 169.254/16 subnet, or will devices on the 192.168.4/24 
> network route via 192.168.4.13 to get to the 169.254/16 subnet?

eth1 will interface to both 169.254.*.* and 192.168.4.*
eth0 will interface to both 169.254.*.* and 192.168.3.*

That's THREE subnets in total on TWO interfaces
with no routing between these subnets
but I do need to bridge between eth0 and eth1 for 169.254.*.* traffic
only

> For the sake of discussion I'm going to presume that you will not be 
> routing the 169.254/16 subnet and that devices will be directly 
> connected to eth1.
> 
> Build a bridge containing eth0 and eth1 and use EBTables to filter 
> traffic that is bridged based on the source / destination IP address. 
> For example:
> 
> ebtables -t filter -A FORWARD -s 169.254.0.0/16 -j ACCEPT 
> ebtables -t filter -A FORWARD -d 169.254.0.0/16 -j ACCEPT 
> ebtables -t filter -A FORWARD -j DROP
> 
> Seeing as how the only use of the bridge (as I see it) is to 
> bridge the 
> 169.254/16 traffic, you can and should have the bridge block 
> (DROP) all 
> other traffic.  Your services will still be bound to the eth0 
> and eth1 
> interfaces, thus not being dependent on the bridge.

That looks good and makes sense even to me :)
You say that my services will be bound to eth0 and eth1.
Does that mean all three subnets will be visible?
 
> I'd suggest that you not have IPTables see your bridged 
> traffic unless 
> you have a specific reason to do so.  Use EBTables to do your 
> filtering.

That's fine - I have no need for IP filtering (or forwarding) anyway.

> 
> One thing to keep in mind with this bridge is that you will have some 
> ARP broadcasts that are common that you may what to filter out.
> 

I will want ARP broadcasts on 169.254.*.* to permeate both interfaces
so that Zeroconf clients on either interface can detect conflicts during
address negotiation. But I don't want ARPs from other subnets getting
bridged.
Do I need extra ebtables commands to ensure this?

> > If this is indeed possible, will server performance
> > be seriously degraded by non-169.254.*.* traffic on 
> > either interface?
> 
> Unless you are talking about millions of packets, I don't 
> think you will 
> have a problem.
> 

My very basic understanding about Linux bridging is that each NIC is
placed
in promiscuous mode and the switches will also pass traffic which would
not otherwise be visible on the Image Server ports. In other words,
all traffic on each subnet is visible to the server and needs to be
processed (and ignored where applicable).

So let's say there is a heavy NFS backup happening on the 192.168.3.*
subnet
between two other servers. Will this traffic create a lot of unnecessary
processing overhead for my Image Server?

Many thanks again for your help


-------------------------------------------------------------------------
This SF.net email is sponsored by: Microsoft
Defy all challenges. Microsoft(R) Visual Studio 2008.
http://clk.atdmt.com/MRT/go/vse0120000070mrt/direct/01/