Re: Performance test for CUBIC in stable/14

On Wed, Oct 23, 2024 at 03:14:08PM -0400, Cheng Cui wrote:
>I see. The result of `newreno` vs. `cubic` shows non-constant/infrequent
>packet
>retransmission. So TCP congestion control has little impact on improving the
>performance.
>
>The performance bottleneck may come from somewhere else. For example, the
>sender CPU shows 97.7% utilization. Would there be any way to reduce CPU
>usage?

There are 11 VMs running on the bhyve server. None of them are very busy but the
server shows 
% uptime
  9:54p.m.  up 8 days,  6:08, 22 users, load averages: 0.82, 1.25, 1.74

The test vm vm4-fbsd14s:
% uptime
  9:55PM  up 2 days,  3:12, 5 users, load averages: 0.35, 0.31, 0.21

It has 
% sysctl hw.ncpu
hw.ncpu: 8

and
avail memory = 66843062272 (63746 MB)

so it's not short of resources.

A test just now gave these results:
- - - - - - - - - - - - - - - - - - - - - - - - -
Test Complete. Summary Results:
[ ID] Interval           Transfer     Bitrate         Retr
[  5]   0.00-20.04  sec  1.31 GBytes   563 Mbits/sec    0             sender
[  5]   0.00-20.06  sec  1.31 GBytes   563 Mbits/sec                  receiver
CPU Utilization: local/sender 94.1% (0.1%u/94.1%s), remote/receiver 15.5%
(1.5%u/13.9%s)
snd_tcp_congestion cubic
rcv_tcp_congestion cubic

iperf Done.

so I'm not sure how the utilization figure was synthesised, unless it's derived
from something like 'top' where 1.00 is 100%. Load when running the test got to
0.83 as observed in 'top' in another terminal. Five mins after the test, 
load in the vm is: 0.32, 0.31, 0.26
on the bhyve host: 0.39, 0.61, 1.11

Before we began testing, I was looking at the speed issue as being caused by
something to do with interrupts and/or polling, and/or HZ, somehting that linux
handles differently and gives better results on the same bhyve host.
Maybe rebuilding the kernel with a different scheduler on both the host and the
freebsd vms will give a better result for freebsd if tweaking sysctls doesn't
make much of a difference.

In terms of real-world bandwidth, I found that the combination of your modified
cc_cubic + rack gave the best results in terms of overall throughput in a
speedtest context, although it's slower to get to its max throughput than cubic
alone. I'm still testing with a webdav/rsync context (cubic against cubic+rack)

The next lot of testing after changing the scheduler will be on a KVM host, 
with various *BSDs as guests.

There may be a tradeoff of stability against speed I guess.
--