Re: Question about SCHED_DEADLINE and cgroupv2/cpuset interaction

On Fri, 2026-05-08 at 09:56 +0200, Tobias Jakobi wrote:
> OK, so I've proceeded by ditching both isolcpus and nohz_full. Upon 
> reading about the tradeoffs (overhead of kernel entry/exit) I decided 
> that nohz_full is probably not really helping the team that writes our 
> userspace application. An application that makes heavy use of 
> socket-based IPC.

Yeah that's better, nohz_full is basically asking the kernel to be out of your
way. It's useful mostly when you have a single userspace task per CPU and you
don't want interruptions.

I'd say unless you have a benchmark proving nohz_full behaves better with your
workload, just live without it.

> Anyway, with these two parameters removed I can setup the partition as root.
> 
> I have moved things around a bit, and this is how it currently looks like.
> 
> root@cm5-slim:/sys/fs/cgroup/workload.slice# grep . cpuset.*
> cpuset.cpus:2-3
> cpuset.cpus.exclusive.effective:2-3
> cpuset.cpus.partition:root
> cpuset.mems.effective:0-7
> root@cm5-slim:/sys/fs/cgroup/workload.slice# cd workload-group0.slice
> root@cm5-slim:/sys/fs/cgroup/workload.slice/workload-group0.slice# grep 
> . cpuset.*
> cpuset.cpus:2
> cpuset.cpus.effective:2
> cpuset.cpus.exclusive.effective:2
> cpuset.cpus.partition:root
> cpuset.mems.effective:0-7
> root@cm5-slim:/sys/fs/cgroup/workload.slice/workload-group0.slice# cd 
> ../workload-group1.slice
> root@cm5-slim:/sys/fs/cgroup/workload.slice/workload-group1.slice# grep 
> . cpuset.*
> cpuset.cpus:3
> cpuset.cpus.effective:3
> cpuset.cpus.exclusive.effective:3
> cpuset.cpus.partition:root
> cpuset.mems.effective:0-7
> 
> sched_rt_runtime_us has its default value and I can spawn processes with 
> SCHED_DEADLINE in both workload-group0.slice and workload-group1.slice. 
> So far, so good!
> 

Great!

> > 
> > > I then remembered reading something in the systemd documentation about
> > > cgroup delegation and that changing anything cgroup related right under
> > > systemd's nose was asking for trouble. I guess this is what "partition
> > > config conflicts with housekeeping setup" is supposed to tell me here?
> > I don't think systemd is to blame here (it seems not to care about partition
> > types at all). But since it doesn't seem to support those it may be a good
> > idea
> > to find alternatives (e.g. plain cgroup).
> 
> Hmm, I'm not sure if going for plain cgroup is feasible for us. The 
> system I'm currently testing on is not the final thing. That's going to 
> be some bigger AMD Epyc where each CCD becomes one of the group slices. 
> With a lot of processes running inside the slices. The current 
> architecture design intends to let systemd do the heavy lifting here.
> 
> I think I'm going to at least ask the systemd folks if I'm looking for 
> trouble here. At the moment I'm using a small helper service unit that 
> sets up the slice/cgroup.

I had a look at the systemd code and I don't see any reason why doing 

  ExecStart=/bin/sh -c 'echo root > /sys/fs/cgroup/cpuset/<name>.slice/cpuset.partition'

would cause any trouble.

I opened a PR to systemd to add an official way to do it though [1], anyway do
ask the systemd folks, they surely know better than I do ;)

[1] - https://github.com/systemd/systemd/pull/42013
> 
> root@cm5-slim:~# systemctl cat workload-compute0.service
> # /etc/systemd/system/workload-compute0.service
> [Unit]
> Description=Spawn compute workload on group 0
> [email protected]
> [email protected]
> 
> [Service]
> Slice=workload-group0.slice
> ExecStart=/usr/bin/chrt --sched-runtime 80000 --sched-period 200000 
> --deadline 0 /usr/bin/dd if=/dev/urandom of=/dev/zero bs=1M
> 
> That is not exactly beautiful, but also not terribly ugly. Also all 
> workload service units are auto-generated from a template anyway, so 
> it's just a matter of adding something to the template.
> 

Yeah, that's quite a pity systemd doesn't support the deadline scheduler..

> 
> > 
> > > After I wasn't getting anywhere here I focused on the bandwidth check.
> > > And since the whole thing worked after disabling the check, I
> > > disregarded the root_domain/partition idea.
> > Of course also disabling the bandwidth check (like you did initially) works,
> > but
> > you don't really want that.
> > 
> > > But I guess the real cause is the root_domain then?
> > > 
> > > With best wishes,
> > > Tobias
> > I hope I didn't make it more confusing than it actually is!
> 
> Not at all! I agree though that it's not easy to wrap you around all 
> this the first time around. :-D
> 
> With best wishes,
> Tobias

Happy to help.
Cheers,
Gabriele