Re: How multithreaded is networking in FreeBSD 15.1?

On 8/16/26 09:32, Brett Glass wrote:
> All:
> 
> I need to build a high-performance FreeBSD-based router that will handle 
> gigabits of traffic. It'll be connected via aggregated Ethernet 
> connections (at least upstream and maybe downstream too) and so will 
> need to be able to handle LACP-based LAGs efficiently. The router will 
> also use IPFW to firewall and filter packets as well as some kernel- 
> based NAT. (Not all traffic will go through NAT, but some will.)
> 
> I have ny choice of several computing platforms: Some with more cores, 
> the other with higher clock speed and fewer higher performance cores. 
> Which is likely to perform better under FreeBSD 15.1? I'm especially 
> concerned that, due to locks, the LAGs and the hashing algorithms that 
> split the packets between the interfaces could be a bottleneck.
> 
> Given the current multithreading in the network stack, IPFW, and the NAT 
> library, does it pay to go for lots of cores? Or due to locking is it 
> more efficient to storm on through with fewer very-high-performance 
> cores? What tuning will provide the best results in the latest kernel? 
> (There are plenty of tutorials on FreeBSD network tuning, but many go 
> back to MUCH earlier versions and may longer contain valid advice.)
> 
> --Brett Glass


Amdahl's law comes to mind:

     https://en.wikipedia.org/wiki/Amdahls_law

     "The overall performance improvement gained by optimizing a single 
part of a system is limited by the fraction of time that the improved 
part is actually used."


My theory for running mixed sequential/ parallel programs is that there 
is a speedup vs. efficiency sweet spot when the number of threads is 
numerically equal to Amdahl's maximum speedup value.  If I run the 
program with more threads, I will get more speedup but less efficiency. 
If I run the program with less threads, I will get less speedup but more 
efficiency.


Looking at the 95% parallelization curve on the "Speedup" vs. "Number of 
processors" graph on the above Wikipedia web page:

* 5% of the work is sequential.

* The maximum speedup is 100% / 5% = 20.

* If I run the program with 20 threads (number of processors), the graph 
shows a speedup of about 10.  So, efficiency is about 10/20 = 50%.

* For 40 threads, speedup is about 13 and efficiency is about 13/40 = 33%.

* For 10 threads, speedup is about 6 and efficiency is about 6/10 = 60%.


When shopping for processors, Hyperthreading complicates thread counts 
vs. core counts.  My experience for general-purpose workloads is that 1 
Hyperthreading core has about the same performance as 1.3 
non-Hyperthreading cores.  But, hand-coded assembly and/or microcode can 
have very good performance for specific Hyperthreading workloads.


Furthermore, if your processor has Hyperthreading and you disable 
Hyperthreading in the motherboard firmware Setup utility so that the 
processor runs faster, the frequency boost will be less than 30%.


So, if your math says 20 threads, a processor with 16 Hyperthreading 
cores could work.  But, a processor with 24 Hyperthreading cores should 
work better; now and in the future.


The above assumes the program is CPU bound, not memory bound or I/O bound.


I am uncertain how the above efficiency values relate to power 
consumption.  Blocked threads should be asleep and not consuming cycles, 
but more sleeping threads may be more work for the scheduler.


All that said, have you considered FreeNAS (now TrueNAS)?  It is derived 
from BSD and purpose-built for networking.  It could be less work and 
could provide better performance than building your own router by 
starting from FreeBSD:

     https://www.truenas.com/freenas/


I suggest benchmarking to evaluate your alternatives.


David