Re: net5501: FreeBSD ipfw and the elusive 75Mbps throughput

Christopher Hilton <[email protected]> Thu, 9 Jun 2016 01:00:02 -0400
Newsgroups	gmane.comp.hardware.soekris.technical
Message-ID	<[email protected]>
> On Jun 8, 2016, at 8:18 PM, Jed Clear <[email protected]> wrote:
> 
> I just climbed out of the bronze age of home networking (DSL) and now have "75Mbps service” from $BIG_CABLE_CO (iron age?).  Before the DSL was the bottle neck.  Now it appears the 5501 is the bottle neck.  My net5501-70 has long been running nanobsd (FreeBSD 9.3-R) and ipfw as my perimeter router-firewall-nat.  While I’m not expecting 75, especially in the evening, it’s not even close.  Note all the speeds mentioned are download speeds in Mbps.  The upload is much worse, but not bothered by that in this exercise.
> 
> When the cable modem was first brought up, a laptop directly on it pulled 56 with one of the speed test sites.  The cable modem channel power and SNR don’t look bad.  Putting the 5501 in-line dropped the speed to the 30s.  Some googling later and I discover FreeBSD’s polling feature.  So I added options DEVICE_POLLING to the kernel config (HZ was already 1000), baked a new image, set all the interfaces to polling and … it dropped like a rock to 5 Mbps.  Flipping off polling on the three interfaces brought it back to the 30s.
> 
> I tried the built in “simple” firewall rule set, and that did modestly better than my, perhaps overly complicated, rule set.  It got around 44.  I will work that later.
> 
> Anyway I’m a bit baffled by the negative results when enabling polling.  And any other advice on improving the performance through the 5501 would be appreciated.  I haven’t given up on self help, but need a break from google for a bit so will appeal to the collective wisdom of soekris-tech.
> 
> Thanks,
> 
> -Jed
> 
> PS: To add insult to injury, I just repeated the directly connected laptop experiment and clocked over 90.  :-(
> 

I don’t know about FreeBSD/ipfw but on OpenBSD/pf and the latest performance tweaks to both the vr driver and the pf firewall the best I could do with a Net5501-70, pf, and the vr driver based nics was 85Mbit/s. If I understand correctly, FreeBSD’s vr driver is more performant than OpenBSD’s but that may have changed. In 5.8, OpenBSD’s pf is much more performant than pf in FreeBSD 9-* and less buggy. Again, I don’t know about FreeBSD/ipfw. If you aren’t reaching 75Mbit/s now on the Net5501-70 you might be able to do so either by switching to pfSense or by switching to OpenBSD.

SUMMARY

At the end of the day I think that 75 ~ 85 MBit’s per second is the limit on the vr interface in net5501. I don’t know what the limit is on the em interface in the Net6501 because between the 1Gbit speed of the NIC and the PCIe bus I can’t afford to buy enough bandwidth to get close. If you only have one firewall/router I’d replace it with either a Net6501 or some other Intel Atom/PCIe/Intel Gigabit based solution.

DETAILS:

At the end of the day, I solved this by throwing money at the problem in three steps:

First I replace my Net5501 with a Net6501. That changed the ethernet driver from vr to em and the em driver is much more performant.

I have two firewalls serially so moving to the Net6501 just moved the problem upstream in my network.

So, Second, to address that, I put the Net5501-70 into a Soekris Rackmount case for the better power supply and put a dual em interface into a net5501-70 for the second firewall. This worked and was stable under OpenBSD 5.2. It became biweekly unstable when I upgraded the OS From OpenBSD 5.2 to OpenBSD 5.6 and to 5.8 meaning that I never saw an uptime greater than 14 days out of the OpenBSD 5.6/5.8 box with the dual em card in it. This box would spontaneously reboot under heavy traffic and I could never figure out the reason. I speculate that under heavy traffic loads the power supply can’t keep up with the PCI dual em card. Thus the dual em hangs the PCI bus and ultimately triggers the watchdog reboot on the Net5501-70.

Finally, I replaced the Net6501 with a 1U SuperMicro D525 Atom and moved the Net6501 upstream replacing the net5501. Since then the only reason that either firewall goes down is because I rebooted it.

Hope this helps,

--
Chris

     __o          "All I was trying to do was get home from work."
   _`\<,_           -Rosa Parks
___(*)/_(*)____.___o____..___..o...________ooO..._____________________
Christopher Sean Hilton                    [chris/at/vindaloo/dot/com]

_______________________________________________
Soekris-tech mailing list
[email protected]
http://lists.soekris.com/mailman/listinfo/soekris-tech
signature.asc (application/pgp-signature, 841 B)