Re: Support for more than 256 CPU cores

On 5/5/23 6:38 AM, Ed Maste wrote:
> FreeBSD supports up to 256 CPU cores in the default kernel configuration
> (on Tier-1 architectures).  Systems with more than 256 cores are
> available now, and will become increasingly common over FreeBSD 14’s
> lifetime.  The FreeBSD Foundation is supporting the effort to increase
> MAXCPU, and PR269572[1] is open to track tasks and changes.
> 
> As a project we have scalability work ahead of us to make best use of
> high core count machines, but at a minimum we should be able to boot a
> GENERIC kernel on such systems, and have an ABI for the FreeBSD 14
> release that supports such a configuration.
> 
> Some changes have already been committed in support of increased MAXCPU,
> including increasing MAX_APIC_ID (commit c8113dad7ed4) and a number of
> changes to reduce bloat (such as commits 42f722e721cd, e72f7ed43eef,
> 78cfa762ebf2 and 74ac712f72cf).
> 
> The next step is to increase the maximum cpuset size for userland.
> I have this change open in review D39941[2] and an exp-run request in
> PR271213[3].  Following that the kernel change for increasing MAXCPU is
> in D36838[4].
> 
> Additional work on bloat reduction will continue after this change, and
> looking forward FreeBSD is going to need ongoing effort from the
> community and the FreeBSD Foundation to continue improving scalability.
> 
> [1] https://bugs.freebsd.org/269572
> [2] https://reviews.freebsd.org/D39941
> [3] https://bugs.freebsd.org/271213
> [4] https://reviews.freebsd.org/D36838

FWIW, I think it will be useful for main to run with a larger userspace
MAXCPU than kernel for at least a while so that we have better testing
of that configuration and to give headroom for bumping MAXCPU in the
kernel during the 14.x branch.

The only other viable path I think which would be more work would be to
rework cpuset_t in userspace to always use a dynamically sized mask.
This could perhaps be done in an API-preserving manner by making cpuset_t
an opaque wrapper type in userland and requiring CPU_* to indirect to
functions in libc, etc.  That's a fair bit more work however.

-- 
John Baldwin