Re: Question about SCHED_DEADLINE and cgroupv2/cpuset interaction

Hello Gabriele,

thanks for your feedback!

On 4/29/26 16:47, Gabriele Monaco wrote:
> Hi Tobias,
>
> giving my 2 cents here.
>
> On Tue, 2026-04-28 at 15:52 +0200, Tobias Jakobi wrote:
>> For the RT work we have these slices.
>>
>> root@cm4-fat:/etc/systemd/system# cat workload.slice
>> [Slice]
>> AllowedCPUs=2-3
>>
>> root@cm4-fat:/etc/systemd/system# cat workload-core2.slice
>> [Slice]
>> AllowedCPUs=2
>>
> Applying the configuration above, I get both slices as member
> partitions (not root):
>
> head /sys/fs/cgroup/workload.slice/{,*/}cpuset.cpus.partition
> ==> /sys/fs/cgroup/workload.slice/cpuset.cpus.partition <==
> member
>
> ==> /sys/fs/cgroup/workload.slice/workload-core2.slice/cpuset.cpus.partition <==
> member
>
> Running your example in either slice doesn't work in my case.
> That's because in the admission control we "Don't allow tasks with an
> affinity mask smaller than the entire root_domain to become
> SCHED_DEADLINE" [1].

Actually I saw this comment during my investigation. But I disregarded 
it quickly, reasons see below.

Anyway, I saw this term "root_domain" being mentioned a few times in the 
documentation 
(https://www.kernel.org/doc/Documentation/scheduler/sched-deadline.rst). 
But only there. It is not mentioned at all in the rest of the docs. Nor 
have I found a definition for it. Might as well ask this here, what 
exactly is this root_domain supposed to be? Yes, there is this comment 
for struct root_domain in sched/sched.h, but honestly, without heavy 
knowledge about the scheduler codebase this tells me almost nothing... :-(

> In short that's to make the admission control able to give guarantees.
>
> Setting the partition as root:
>
> echo root > /sys/fs/cgroup/workload.slice/cpuset.cpus.partition
> echo root > /sys/fs/cgroup/workload.slice/workload-core2.slice/cpuset.cpus.partition
>
> Fixes the issue in my case.
> I don't really know how to specify that in systemd and I'm assuming
> your version/configuration of systemd was somehow setting workload-
> slice as root automatically, that's likely why it was working.
>
> Could you try this on your side?

Sure, as I said above I disregarded the comment in syscalls.c after 
fiddling with this.

So, first of all, when doing nothing on top of what systemd does, then 
both cpuset.cpus.partition are member. What I did not mention in the 
initial mail was that I tried to change the partition type. 
Unsuccessfully however.

root@cm4-fat:/sys/fs/cgroup/workload.slice# echo root > 
cpuset.cpus.partition
root@cm4-fat:/sys/fs/cgroup/workload.slice# echo root > 
workload-core2.slice/cpuset.cpus.partition
root@cm4-fat:/sys/fs/cgroup/workload.slice# cat cpuset.cpus.partition
root invalid (partition config conflicts with housekeeping setup)
root@cm4-fat:/sys/fs/cgroup/workload.slice# cat 
workload-core2.slice/cpuset.cpus.partition
root invalid (Invalid cpu list in cpuset.cpus.exclusive)
root@cm4-fat:/sys/fs/cgroup/workload.slice# grep . cpuset.*
cpuset.cpus:2-3
cpuset.cpus.effective:2-3
cpuset.cpus.partition:root invalid (partition config conflicts with 
housekeeping setup)
cpuset.mems.effective:0
root@cm4-fat:/sys/fs/cgroup/workload.slice# cd workload-core2.slice/
root@cm4-fat:/sys/fs/cgroup/workload.slice/workload-core2.slice# grep . 
cpuset.*
cpuset.cpus:2
cpuset.cpus.effective:2
cpuset.cpus.partition:root invalid (Invalid cpu list in 
cpuset.cpus.exclusive)
cpuset.mems.effective:0

I then remembered reading something in the systemd documentation about 
cgroup delegation and that changing anything cgroup related right under 
systemd's nose was asking for trouble. I guess this is what "partition 
config conflicts with housekeeping setup" is supposed to tell me here?

After I wasn't getting anywhere here I focused on the bandwidth check. 
And since the whole thing worked after disabling the check, I 
disregarded the root_domain/partition idea.

But I guess the real cause is the root_domain then?

With best wishes,
Tobias

>
> Cheers,
> Gabriele
>
> [1] -
> https://elixir.bootlin.com/linux/v7.0.1/source/kernel/sched/syscalls.c#L627
>
>> Note that core2 is a child of workload, i.e. on the cgroup level it
>> looks like this.
>>
>> root@cm4-fat:~# find /sys/fs/cgroup/ -name workload\*.slice
>> /sys/fs/cgroup/workload.slice
>> /sys/fs/cgroup/workload.slice/workload-core2.slice
>>
>> So what's exactly this problem I'm seeing? It's this here.
>>
>> root@cm4-fat:~# systemd-run --slice=workload-core2.slice --pty chrt
>> --sched-runtime 100000 --sched-period 200000 --deadline 0 echo this
>> works
>> Running as unit: run-p676-i976.service; invocation ID:
>> d59dc443f66f46a6ba2dfd125e04f830
>> Press ^] three times within 1s to disconnect TTY.
>> chrt: failed to set pid 0's policy: Operation not permitted
>>
>> What works however is this here.
>>
>> root@cm4-fat:~# systemd-run --slice=workload.slice --pty chrt
>> --sched-runtime 100000 --sched-period 200000 --deadline 0 echo this
>> works
>> Running as unit: run-p670-i970.service
>> Press ^] three times within 1s to disconnect TTY.
>> this works
>>
>> I double checked the slice settings, and they apply.
>>
>> root@cm4-fat:~# systemd-run --slice=workload.slice --pty bash -c
>> 'grep
>> Cpus_allowed_list /proc/$BASHPID/status'
>> Running as unit: run-p727-i1027.service
>> Press ^] three times within 1s to disconnect TTY.
>> Cpus_allowed_list:      2-3
>>
>> Once again for the core2 slice.
>>
>> root@cm4-fat:~# systemd-run --slice=workload-core2.slice --pty bash -
>> c
>> 'grep Cpus_allowed_list /proc/$BASHPID/status'
>> Running as unit: run-p732-i1032.service
>> Press ^] three times within 1s to disconnect TTY.
>> Cpus_allowed_list:      2
>>
>> This really puzzled here. Why would this work for AllowedCPUs=2-3,
>> but
>> not for AllowedCPUs=2?
>>
>> I then dove in the documentation and the code, trying to understand
>> where the -EPERM was coming from. I finally decided to disable the
>> bandwidth check that the kernel performs.
>>
>> root@cm4-fat:~# echo -1 > /proc/sys/kernel/sched_rt_runtime_us
>> root@cm4-fat:~# systemd-run --slice=workload-core2.slice --pty chrt
>> --sched-runtime 100000 --sched-period 200000 --deadline 0 echo this
>> works
>> Running as unit: run-p760-i1060.service
>> Press ^] three times within 1s to disconnect TTY.
>> this works
>>
>> And voila, suddenly everything works. I fail to understand though why
>> the bandwidth check fails here. I mean, 100000 / 200000, that's fifty
>> percent, so the check should pass.
>>
>> And apparantly the ratio itself doesn't seem to have any effect. I
>> enabled the check again and reduced the runtime so that the ratio is
>> just 5%. Still the same.
>>
>> root@cm4-fat:~# echo 950000 > /proc/sys/kernel/sched_rt_runtime_us
>> root@cm4-fat:~# systemd-run --slice=workload-core2.slice --pty chrt
>> --sched-runtime 1000 --sched-period 200000 --deadline 0 echo this
>> works
>> Running as unit: run-p838-i1138.service
>> Press ^] three times within 1s to disconnect TTY.
>> chrt: failed to set pid 0's policy: Operation not permitted
>>
>> I looked around some more, made sure that CONFIG_RT_GROUP_SCHED=n,
>> but
>> the more I think about, the more I get the impression that I totally
>> misunderstand how the bandwidth check works.
>>
>> To be explicit, by bandwidth check I mean the line that is found in
>> section 4.1. here:
>> https://www.kernel.org/doc/Documentation/scheduler/sched-deadline.rst
>>
>> Would be nice if someone could shed some light on this. Thanks in
>> advance!
>>
>> With best wishes,
>> Tobias Jakobi
>>