Re: Question about SCHED_DEADLINE and cgroupv2/cpuset interaction

Hello Tobias,

On Mon, 2026-05-04 at 09:53 +0200, Tobias Jakobi wrote:
> Actually I saw this comment during my investigation. But I disregarded 
> it quickly, reasons see below.
> 
> Anyway, I saw this term "root_domain" being mentioned a few times in the 
> documentation 
> (https://www.kernel.org/doc/Documentation/scheduler/sched-deadline.rst). 
> But only there. It is not mentioned at all in the rest of the docs. Nor 
> have I found a definition for it. Might as well ask this here, what 
> exactly is this root_domain supposed to be? Yes, there is this comment 
> for struct root_domain in sched/sched.h, but honestly, without heavy 
> knowledge about the scheduler codebase this tells me almost nothing... :-(

The reason why you need a root cpuset is the same why you cannot simply taskset:
if all your tasks can migrate freely you can use as admission control total
utilisation < number of cpus (kinda, read more in the references if you're
interested), but if some tasks have constraints on what CPUs they can migrate
to, that formula is no longer sufficient.

The way this is solved is by splitting scheduling domain into independent
chunks, and a root partition does exactly that: if you give CPU2 to your root
partition, no other partition can use CPU2, so you can apply the formula above
only on that partition (with a single CPU).

Without setting the partition as root, you're saying all tasks in the cpuset can
only use CPU2, but CPU2 can be used by other cpusets as well, it's impossible to
guarantee anything on your cpuset in this case because you'd need to consider
also the others.

You should be able to find more about cpuset partitions in Documentation/admin-
guide/cgroup-v2.rst

> > In short that's to make the admission control able to give guarantees.
> > 
> > Setting the partition as root:
> > 
> > echo root > /sys/fs/cgroup/workload.slice/cpuset.cpus.partition
> > echo root > /sys/fs/cgroup/workload.slice/workload-
> > core2.slice/cpuset.cpus.partition
> > 
> > Fixes the issue in my case.
> > I don't really know how to specify that in systemd and I'm assuming
> > your version/configuration of systemd was somehow setting workload-
> > slice as root automatically, that's likely why it was working.
> > 
> > Could you try this on your side?
> 
> Sure, as I said above I disregarded the comment in syscalls.c after 
> fiddling with this.
> 
> So, first of all, when doing nothing on top of what systemd does, then 
> both cpuset.cpus.partition are member. What I did not mention in the 
> initial mail was that I tried to change the partition type. 
> Unsuccessfully however.
> 
> root@cm4-fat:/sys/fs/cgroup/workload.slice# echo root > 
> cpuset.cpus.partition
> root@cm4-fat:/sys/fs/cgroup/workload.slice# echo root > 
> workload-core2.slice/cpuset.cpus.partition
> root@cm4-fat:/sys/fs/cgroup/workload.slice# cat cpuset.cpus.partition
> root invalid (partition config conflicts with housekeeping setup)
> root@cm4-fat:/sys/fs/cgroup/workload.slice# cat 
> workload-core2.slice/cpuset.cpus.partition
> root invalid (Invalid cpu list in cpuset.cpus.exclusive)
> root@cm4-fat:/sys/fs/cgroup/workload.slice# grep . cpuset.*
> cpuset.cpus:2-3
> cpuset.cpus.effective:2-3
> cpuset.cpus.partition:root invalid (partition config conflicts with 
> housekeeping setup)
> cpuset.mems.effective:0
> root@cm4-fat:/sys/fs/cgroup/workload.slice# cd workload-core2.slice/
> root@cm4-fat:/sys/fs/cgroup/workload.slice/workload-core2.slice# grep . 
> cpuset.*
> cpuset.cpus:2
> cpuset.cpus.effective:2
> cpuset.cpus.partition:root invalid (Invalid cpu list in 
> cpuset.cpus.exclusive)
> cpuset.mems.effective:0

Here you got other issues though, by making a partition root, you are taking
exclusivity on it's CPUs, so system tasks cannot use those CPUs any longer,
pretty much like when you make them (domain) isolated.
The thing is that unlike isolated CPUs, root domains have load balancing among
the exclusive CPUs, that fights with the fact 2-3 are also isolated!

Now you have 2 ways how to solve this: as far as I could test, an isolated
partition counts as root, so you could keep your isolcpus and do:

  echo isolated > /sys/fs/cgroup/workload.slice/workload-core2.slice/cpuset.cpus.partition

OR you could ditch isolcpus entirely (nohz_full can stay) and use root for both,
since the workload domains are exclusive, this shouldn't really make a
difference.

> I then remembered reading something in the systemd documentation about 
> cgroup delegation and that changing anything cgroup related right under 
> systemd's nose was asking for trouble. I guess this is what "partition 
> config conflicts with housekeeping setup" is supposed to tell me here?

I don't think systemd is to blame here (it seems not to care about partition
types at all). But since it doesn't seem to support those it may be a good idea
to find alternatives (e.g. plain cgroup).

> After I wasn't getting anywhere here I focused on the bandwidth check. 
> And since the whole thing worked after disabling the check, I 
> disregarded the root_domain/partition idea.

Of course also disabling the bandwidth check (like you did initially) works, but
you don't really want that.

> But I guess the real cause is the root_domain then?
> 
> With best wishes,
> Tobias

I hope I didn't make it more confusing than it actually is!

Gabriele