Re: Question about SCHED_DEADLINE and cgroupv2/cpuset interaction

Hey Gabriele,

once again thanks for the help!

On 5/5/26 08:56, Gabriele Monaco wrote:
> Hello Tobias,
>
> On Mon, 2026-05-04 at 09:53 +0200, Tobias Jakobi wrote:
>> Actually I saw this comment during my investigation. But I disregarded
>> it quickly, reasons see below.
>>
>> Anyway, I saw this term "root_domain" being mentioned a few times in the
>> documentation
>> (https://www.kernel.org/doc/Documentation/scheduler/sched-deadline.rst).
>> But only there. It is not mentioned at all in the rest of the docs. Nor
>> have I found a definition for it. Might as well ask this here, what
>> exactly is this root_domain supposed to be? Yes, there is this comment
>> for struct root_domain in sched/sched.h, but honestly, without heavy
>> knowledge about the scheduler codebase this tells me almost nothing... :-(
> The reason why you need a root cpuset is the same why you cannot simply taskset:
> if all your tasks can migrate freely you can use as admission control total
> utilisation < number of cpus (kinda, read more in the references if you're
> interested), but if some tasks have constraints on what CPUs they can migrate
> to, that formula is no longer sufficient.
>
> The way this is solved is by splitting scheduling domain into independent
> chunks, and a root partition does exactly that: if you give CPU2 to your root
> partition, no other partition can use CPU2, so you can apply the formula above
> only on that partition (with a single CPU).
>
> Without setting the partition as root, you're saying all tasks in the cpuset can
> only use CPU2, but CPU2 can be used by other cpusets as well, it's impossible to
> guarantee anything on your cpuset in this case because you'd need to consider
> also the others.
>
> You should be able to find more about cpuset partitions in Documentation/admin-
> guide/cgroup-v2.rst

Yeah, that makes things a lot clearer! You need a disjoint union of the 
CPUs, so that you can keep using the scheduling algorithms efficiently.


>
>>> In short that's to make the admission control able to give guarantees.
>>>
>>> Setting the partition as root:
>>>
>>> echo root > /sys/fs/cgroup/workload.slice/cpuset.cpus.partition
>>> echo root > /sys/fs/cgroup/workload.slice/workload-
>>> core2.slice/cpuset.cpus.partition
>>>
>>> Fixes the issue in my case.
>>> I don't really know how to specify that in systemd and I'm assuming
>>> your version/configuration of systemd was somehow setting workload-
>>> slice as root automatically, that's likely why it was working.
>>>
>>> Could you try this on your side?
>> Sure, as I said above I disregarded the comment in syscalls.c after
>> fiddling with this.
>>
>> So, first of all, when doing nothing on top of what systemd does, then
>> both cpuset.cpus.partition are member. What I did not mention in the
>> initial mail was that I tried to change the partition type.
>> Unsuccessfully however.
>>
>> root@cm4-fat:/sys/fs/cgroup/workload.slice# echo root >
>> cpuset.cpus.partition
>> root@cm4-fat:/sys/fs/cgroup/workload.slice# echo root >
>> workload-core2.slice/cpuset.cpus.partition
>> root@cm4-fat:/sys/fs/cgroup/workload.slice# cat cpuset.cpus.partition
>> root invalid (partition config conflicts with housekeeping setup)
>> root@cm4-fat:/sys/fs/cgroup/workload.slice# cat
>> workload-core2.slice/cpuset.cpus.partition
>> root invalid (Invalid cpu list in cpuset.cpus.exclusive)
>> root@cm4-fat:/sys/fs/cgroup/workload.slice# grep . cpuset.*
>> cpuset.cpus:2-3
>> cpuset.cpus.effective:2-3
>> cpuset.cpus.partition:root invalid (partition config conflicts with
>> housekeeping setup)
>> cpuset.mems.effective:0
>> root@cm4-fat:/sys/fs/cgroup/workload.slice# cd workload-core2.slice/
>> root@cm4-fat:/sys/fs/cgroup/workload.slice/workload-core2.slice# grep .
>> cpuset.*
>> cpuset.cpus:2
>> cpuset.cpus.effective:2
>> cpuset.cpus.partition:root invalid (Invalid cpu list in
>> cpuset.cpus.exclusive)
>> cpuset.mems.effective:0
> Here you got other issues though, by making a partition root, you are taking
> exclusivity on it's CPUs, so system tasks cannot use those CPUs any longer,
> pretty much like when you make them (domain) isolated.
> The thing is that unlike isolated CPUs, root domains have load balancing among
> the exclusive CPUs, that fights with the fact 2-3 are also isolated!
>
> Now you have 2 ways how to solve this: as far as I could test, an isolated
> partition counts as root, so you could keep your isolcpus and do:
>
>    echo isolated > /sys/fs/cgroup/workload.slice/workload-core2.slice/cpuset.cpus.partition
>
> OR you could ditch isolcpus entirely (nohz_full can stay) and use root for both,
> since the workload domains are exclusive, this shouldn't really make a
> difference.
OK, so I've proceeded by ditching both isolcpus and nohz_full. Upon 
reading about the tradeoffs (overhead of kernel entry/exit) I decided 
that nohz_full is probably not really helping the team that writes our 
userspace application. An application that makes heavy use of 
socket-based IPC.

Anyway, with these two parameters removed I can setup the partition as root.

I have moved things around a bit, and this is how it currently looks like.

root@cm5-slim:/sys/fs/cgroup/workload.slice# grep . cpuset.*
cpuset.cpus:2-3
cpuset.cpus.exclusive.effective:2-3
cpuset.cpus.partition:root
cpuset.mems.effective:0-7
root@cm5-slim:/sys/fs/cgroup/workload.slice# cd workload-group0.slice
root@cm5-slim:/sys/fs/cgroup/workload.slice/workload-group0.slice# grep 
. cpuset.*
cpuset.cpus:2
cpuset.cpus.effective:2
cpuset.cpus.exclusive.effective:2
cpuset.cpus.partition:root
cpuset.mems.effective:0-7
root@cm5-slim:/sys/fs/cgroup/workload.slice/workload-group0.slice# cd 
../workload-group1.slice
root@cm5-slim:/sys/fs/cgroup/workload.slice/workload-group1.slice# grep 
. cpuset.*
cpuset.cpus:3
cpuset.cpus.effective:3
cpuset.cpus.exclusive.effective:3
cpuset.cpus.partition:root
cpuset.mems.effective:0-7

sched_rt_runtime_us has its default value and I can spawn processes with 
SCHED_DEADLINE in both workload-group0.slice and workload-group1.slice. 
So far, so good!

>
>> I then remembered reading something in the systemd documentation about
>> cgroup delegation and that changing anything cgroup related right under
>> systemd's nose was asking for trouble. I guess this is what "partition
>> config conflicts with housekeeping setup" is supposed to tell me here?
> I don't think systemd is to blame here (it seems not to care about partition
> types at all). But since it doesn't seem to support those it may be a good idea
> to find alternatives (e.g. plain cgroup).

Hmm, I'm not sure if going for plain cgroup is feasible for us. The 
system I'm currently testing on is not the final thing. That's going to 
be some bigger AMD Epyc where each CCD becomes one of the group slices. 
With a lot of processes running inside the slices. The current 
architecture design intends to let systemd do the heavy lifting here.

I think I'm going to at least ask the systemd folks if I'm looking for 
trouble here. At the moment I'm using a small helper service unit that 
sets up the slice/cgroup.

root@cm5-slim:~# systemctl cat workload-compute0.service
# /etc/systemd/system/workload-compute0.service
[Unit]
Description=Spawn compute workload on group 0
[email protected]
[email protected]

[Service]
Slice=workload-group0.slice
ExecStart=/usr/bin/chrt --sched-runtime 80000 --sched-period 200000 
--deadline 0 /usr/bin/dd if=/dev/urandom of=/dev/zero bs=1M

That is not exactly beautiful, but also not terribly ugly. Also all 
workload service units are auto-generated from a template anyway, so 
it's just a matter of adding something to the template.


>
>> After I wasn't getting anywhere here I focused on the bandwidth check.
>> And since the whole thing worked after disabling the check, I
>> disregarded the root_domain/partition idea.
> Of course also disabling the bandwidth check (like you did initially) works, but
> you don't really want that.
>
>> But I guess the real cause is the root_domain then?
>>
>> With best wishes,
>> Tobias
> I hope I didn't make it more confusing than it actually is!

Not at all! I agree though that it's not easy to wrap you around all 
this the first time around. :-D

With best wishes,
Tobias


>
> Gabriele
>