Re: amflush and systemd-oomd

Am 05.02.23 um 15:39 schrieb Uwe Menges:
> Hi,
> 
> I just updated my backup server from Fedora 35 to 37.
> After the upgrade, I tried to run amflush.
> 
> The session running amflush was killed by systemd-oomd as per these
> lines in the journal:
> 
> systemd[1]: session-5.scope: systemd-oomd killed some process(es) in 
> this unit.
> systemd-oomd[7544]: Considered 4 cgroups for killing, top candidates were:
> systemd-oomd[7544]:         Path: /user.slice/user-0.slice/session-5.scope
> systemd-oomd[7544]:                 Memory Pressure Limit: 0.00%
> systemd-oomd[7544]:                 Pressure: Avg10: 56.78 Avg60: 40.05 
> Avg300: 14.53 Total: 53s
> systemd-oomd[7544]:                 Current Memory Usage: 30.0G
> systemd-oomd[7544]:                 Memory Min: 0B
> systemd-oomd[7544]:                 Memory Low: 0B
> systemd-oomd[7544]:                 Pgscan: 14100751
> 
> My workaround was to "systemctl edit systemd-oomd" and add "--dry-run" 
> as argument
> (https://www.freedesktop.org/software/systemd/man/systemd-oomd.service.html).
> A "systemctl cat systemd-oomd" now has this override appended:
> 
> # /etc/systemd/system/systemd-oomd.service.d/override.conf
> [Service]
> ExecStart=
> ExecStart=/usr/lib/systemd/systemd-oomd --dry-run
> 
> After "systemctl restart systemd-oomd", this makes the log look like 
> this when running amflush:
> 
> systemd-oomd[13095]: oomd dry-run: Would have tried to kill 
> /sys/fs/cgroup/user.slice/user-0.slice/session-3.scope with recurse=true
> systemd-oomd[13095]: Considered 4 cgroups for killing, top candidates were:
> systemd-oomd[13095]:         Path: /user.slice/user-0.slice/session-3.scope
> systemd-oomd[13095]:                 Memory Pressure Limit: 0.00%
> systemd-oomd[13095]:                 Pressure: Avg10: 54.15 Avg60: 35.63 
> Avg300: 15.82 Total: 1min 25s
> systemd-oomd[13095]:                 Current Memory Usage: 29.9G
> systemd-oomd[13095]:                 Memory Min: 0B
> systemd-oomd[13095]:                 Memory Low: 0B
> systemd-oomd[13095]:                 Pgscan: 40620400
> systemd-oomd[13095]:                 Last Pgscan: 40620400
> 
> And now amflush keeps running fine again.
> 
> I'm not sure what the right solution is, I just quickly wanted to get it
> running again, and my itch is scratched.
> 
> Maybe something needs to be tweaked in systemd-oomd, or the amanda
> package should include something to exempt amanda from systemd-oomd, or
> something else.
> 
> Yours, Uwe


A quick google finds this:

https://fedoraproject.org/wiki/Changes/EnableSystemdOomd#Can_we_exclude_certain_units_from_being_killed?

So maybe you add a systemd override conf for the amanda.service (or 
however it is called in F37).

You basically disabled systemd-oomd, it would have been even easier to 
run "systemctl disable --now systemd-oomd", I assume.

I am not sure if amflush triggers amandad, I would have to look that up 
(but I am on a train right now, so it's a bit complicated).

It should be rather easy to find the killed cgroups in the logs 
somewhere. From there you might find the related service.

I am using F37 too, but currently I don't have amanda installed on these 
machines.

Maybe it makes sense to come up with some fine-tuned service files and 
contact the Fedora developer maintaining Amanda.

I wonder when and if this feature comes to other distros as well, I 
haven't yet noticed it in Debian, for example.

regards, Stefan