earlyoom kills wrong processes first due to oom_score_adj

[vinc17fr]

I'm using earlyoom under Debian/unstable (upcoming Debian 13), mainly due to a bug in Firefox on YouTube, with the consequence that the associated firefox process suddenly takes more and more memory until OOM. This has happened again, where this process took more than 3 GB VmRSS memory. But earlyoom first killed other processes instead, in particular the important pipewire-pulse process:

Aug 04 23:41:26 qaa earlyoom[1308]: mem avail:  2051 of 20537 MiB ( 9.99%), swap free:    0 of  975 MiB ( 0.01%)
Aug 04 23:41:26 qaa earlyoom[1308]: low memory! at or below SIGTERM limits: mem 10.00%, swap 10.00%
Aug 04 23:41:26 qaa earlyoom[1308]: sending SIGTERM to process 1024668 uid 1000 "Web Content": oom_score 823, VmRSS 79 MiB, cmdline "/usr/lib/firefox/firefox -contentproc -isForBrowser -prefsHandle 0:44331 -prefMapHandle 1:283567 -jsInitHandle 2:242012 -parentBuildID 20250717180000 -sandboxReporter 3 -ipcHandle 4 -initialChannelId {af7f00c7-3771-49ed-8ae7-972b131e8b32} -parentPid 46 -"
Aug 04 23:41:26 qaa earlyoom[1308]: kill_release: pid=1024668: process_mrelease pidfd=4 success
Aug 04 23:41:26 qaa earlyoom[1308]: process 1024668 exited after 0.100 seconds
Aug 04 23:41:28 qaa earlyoom[1308]: mem avail:  2054 of 20543 MiB (10.00%), swap free:    0 of  975 MiB ( 0.01%)
Aug 04 23:41:28 qaa earlyoom[1308]: low memory! at or below SIGTERM limits: mem 10.00%, swap 10.00%
Aug 04 23:41:28 qaa earlyoom[1308]: sending SIGTERM to process 1026163 uid 1000 "Web Content": oom_score 823, VmRSS 79 MiB, cmdline "/usr/lib/firefox/firefox -contentproc -isForBrowser -prefsHandle 0:44331 -prefMapHandle 1:283567 -jsInitHandle 2:242012 -parentBuildID 20250717180000 -sandboxReporter 3 -ipcHandle 4 -initialChannelId {6c109649-b80d-4abc-8d7c-54085db3ad4e} -parentPid 46 -"
Aug 04 23:41:28 qaa earlyoom[1308]: kill_release: pid=1026163: process_mrelease pidfd=4 success
Aug 04 23:41:28 qaa earlyoom[1308]: process 1026163 exited after 0.100 seconds
Aug 04 23:41:28 qaa earlyoom[1308]: mem avail:  2034 of 20540 MiB ( 9.91%), swap free:    0 of  975 MiB ( 0.01%)
Aug 04 23:41:28 qaa earlyoom[1308]: low memory! at or below SIGTERM limits: mem 10.00%, swap 10.00%
Aug 04 23:41:28 qaa earlyoom[1308]: sending SIGTERM to process 1025338 uid 1000 "Web Content": oom_score 823, VmRSS 78 MiB, cmdline "/usr/lib/firefox/firefox -contentproc -isForBrowser -prefsHandle 0:44331 -prefMapHandle 1:283567 -jsInitHandle 2:242012 -parentBuildID 20250717180000 -sandboxReporter 3 -ipcHandle 4 -initialChannelId {95143ef1-8f9d-4a88-8da6-ee03ca80d612} -parentPid 46 -"
Aug 04 23:41:28 qaa earlyoom[1308]: kill_release: pid=1025338: process_mrelease pidfd=4 success
Aug 04 23:41:28 qaa earlyoom[1308]: process 1025338 exited after 0.100 seconds
Aug 04 23:41:28 qaa earlyoom[1308]: mem avail:  2036 of 20537 MiB ( 9.92%), swap free:    0 of  975 MiB ( 0.01%)
Aug 04 23:41:28 qaa earlyoom[1308]: low memory! at or below SIGTERM limits: mem 10.00%, swap 10.00%
Aug 04 23:41:28 qaa earlyoom[1308]: sending SIGTERM to process 2002 uid 1000 "pipewire-pulse": oom_score 800, VmRSS 40 MiB, cmdline "/usr/bin/pipewire-pulse"
Aug 04 23:41:28 qaa earlyoom[1308]: kill_release: pid=2002: process_mrelease pidfd=4 failed: Invalid argument
Aug 04 23:41:28 qaa systemd[1981]: pipewire-pulse.service: Consumed 17min 43.690s CPU time, 54.2M memory peak, 100K memory swap peak.
Aug 04 23:41:28 qaa earlyoom[1308]: process 2002 exited after 0.100 seconds
Aug 04 23:41:35 qaa earlyoom[1308]: mem avail:  2050 of 20519 MiB ( 9.99%), swap free:    0 of  975 MiB ( 0.02%)
Aug 04 23:41:35 qaa earlyoom[1308]: low memory! at or below SIGTERM limits: mem 10.00%, swap 10.00%
Aug 04 23:41:35 qaa earlyoom[1308]: sending SIGTERM to process 5446 uid 1000 "Isolated Web Co": oom_score 800, VmRSS 3257 MiB, cmdline "/usr/lib/firefox/firefox -contentproc -isForBrowser -prefsHandle 0:34285 -prefMapHandle 1:283567 -jsInitHandle 2:242012 -parentBuildID 20250717180000 -sandboxReporter 3 -ipcHandle 4 -initialChannelId {4ac59450-5248-4a8c-83ff-bced872d4463} -parentPid 46 -"
Aug 04 23:41:35 qaa earlyoom[1308]: kill_release: pid=5446: process_mrelease pidfd=4 success
Aug 04 23:41:35 qaa earlyoom[1308]: process 5446 exited after 0.100 seconds

Note: I'm using Debian's default options for the earlyoom process, which is

UID          PID    PPID  C STIME TTY          TIME CMD
earlyoom    1308       1  0 Jul28 ?        00:01:01 /usr/bin/earlyoom -r 3600

[hakavlad]

By the way, these processes have a high oom_score.
Are you sure they’re the wrong processes?

Also: Web Content processes have hight oom_score_adj by default

[hakavlad]

this process took more than 3 GB VmRSS memory. But earlyoom first killed other processes instead

Maybe this process has an oom_score_adj of zero, unlike the processes that were killed.

[vinc17fr]

Maybe this process has an oom_score_adj of zero, unlike the processes that were killed.

I don't see why this would be the case. I think that earlyoom should also output the oom_score_adj value. It would be easier to see what's going wrong. With the following command under zsh, I cannot see any "Isolated Web Co" process with an oom_score_adj score of zero.

for i in "${(f)$(ps -eo 'pid,command' c)}" ; do a=(${=i}); printf "%6s %s\n" "$(cat /proc/$a[1]/oom_score_adj 2>/dev/null)" "$a[2,-1]"; done

[hakavlad]

Both browsers set high oom_score_adj for its working processes (tab groups).

$ oom-sort -l0
oom_score oom_score_adj   UID     PID Name            VmRSS   VmSwap
--------- ------------- ----- ------- --------------- ------- --------
      870           300  1000 1729862 chromium          163 M      0 M 
      870           300  1000 1729953 chromium          155 M      0 M 
      870           300  1000 1729960 chromium          161 M      0 M 
      870           300  1000 1729986 chromium          187 M      0 M 
      869           300  1000 1729883 chromium          124 M      0 M 
      869           300  1000 1729905 chromium          141 M      0 M 
      869           300  1000 1729954 chromium          139 M      0 M 
      868           300  1000 1729861 chromium          108 M      0 M 
      868           300  1000 1729892 chromium          102 M      0 M 
      868           300  1000 1729938 chromium           86 M      0 M 
      868           300  1000 1730046 chromium           62 M      0 M 
      823           233  1000 1715829 Web Content        76 M      0 M 
      823           233  1000 1716255 Web Content        77 M      0 M 
      823           233  1000 1717467 Web Content        77 M      0 M 
      823           233  1000 1728777 Web Content        77 M      0 M 
      823           233  1000 1728929 Web Content        77 M      0 M 
      823           233  1000 1729553 Web Content        78 M      0 M 
      802           200  1000 1729806 chromium          102 M      0 M 
      802           200  1000 1729811 chromium          144 M      0 M 
      800           200  1000 1729816 chromium           42 M      0 M 
      794           167  1000 1715095 Isolated Web Co   756 M      0 M 
      786           167  1000 1714419 Isolated Web Co   372 M      0 M 
      783           167  1000 1724524 Isolated Web Co   235 M      0 M 
      783           167  1000 1724940 Isolated Web Co   258 M      0 M 
      782           167  1000 1714091 Isolated Web Co   195 M      0 M 
      782           167  1000 1714204 Isolated Web Co   233 M      0 M 
      782           167  1000 1723431 Isolated Web Co   219 M      0 M 
      781           167  1000 1714050 Privileged Cont   156 M      0 M 
      781           167  1000 1714177 Isolated Web Co   170 M      0 M 
      781           167  1000 1714378 Privileged Cont   156 M      0 M 
      781           167  1000 1714415 Isolated Web Co   161 M      0 M 
      781           167  1000 1714476 Isolated Web Co   175 M      0 M 
      780           167  1000 1714110 Isolated Web Co   125 M      0 M 
      780           167  1000 1714125 Isolated Web Co   124 M      0 M 
      780           167  1000 1714456 Isolated Web Co   119 M      0 M 
      780           167  1000 1715372 Isolated Web Co    97 M      0 M 
      780           167  1000 1716566 Isolated Web Co   124 M      0 M 
      780           167  1000 1719624 Isolated Web Co   126 M      0 M 
      780           167  1000 1719723 Isolated Web Co   111 M      0 M 
      780           167  1000 1723509 Isolated Web Co   100 M      0 M 
      780           167  1000 1723547 Isolated Web Co   109 M      0 M 
      780           167  1000 1724172 Isolated Web Co   102 M      0 M 
      745           100  1000 1714439 Isolated Web Co   530 M      0 M 
      738           100  1000 1714087 Isolated Web Co   210 M      0 M 
      737           100  1000 1714982 WebExtensions     178 M      0 M 
      736           100  1000 1714463 Isolated Web Co   123 M      0 M 
      736           100  1000 1714500 Isolated Web Co   135 M      0 M 
      736           100  1000 1714637 WebExtensions     158 M      0 M 
      692             0  1000 1713799 firefox-esr      1154 M      0 M 
      676             0  1000 1713972 firefox-esr       432 M      0 M 
      672             0  1000 1729757 chromium          254 M      0 M 
      669             0     0 1713139 Xorg              131 M      0 M 

[vinc17fr]

Perhaps it was OK to kill the "Web Content" processes if they were not important. Perhaps the bug is that pipewire-pulse has an oom_score_adj value 200 here while it is an important process: when it is killed, the audio stops.

That said, oom_score_adj seems broken, at least for earlyoom, whose goal is to kill the largest process. In my case, the culprit tab was taking 3257 MiB while pipewire-pulse was just taking 40 MiB. This is a huge difference, and oom_score_adj should have had a very little influence here.

[hakavlad]

Perhaps the bug is that pipewire-pulse has an oom_score_adj value 200 here while it is an important process

Yes, it's weird. That's not earlyoom's problem. There is no reason to set positive oom_core_adj for pipewire-pulse.

Maybe you should report here https://gitlab.freedesktop.org/pipewire/pipewire/-/issues

[hakavlad]

AI:

It’s a bug in the systemd service configuration for pipewire-pulse

By default, the pipewire-pulse unit ships with oom_score_adj=200, which actually increases its chance of being killed under OOM conditions instead of protecting it.
Why this is incorrect

• A positive oom_score_adj value raises the process’s kill priority when memory is low.
• Critical services like the audio subsystem should use zero or a negative value to avoid being targeted first.
Where to report the issue

In your distribution’s bug tracker
• Identify the package (e.g., pipewire or pipewire-pulse).
• Describe the misconfigured oom_score_adj=200 and recommend setting it to 0 or a negative value.
On the PipeWire upstream tracker
• GitLab: https://gitlab.freedesktop.org/pipewire/pipewire/-/issues
• Create a “New issue” and include the output of systemctl cat pipewire-pulse.service.
Via the PipeWire developers’ mailing list
• Email: [pipewire-devel@lists.freedesktop.org](mailto:pipewire-devel@lists.freedesktop.org)

Temporary workaround

Create a systemd override to reset the score:

sudo systemctl edit pipewire-pulse.service
# Add under [Service]:
#   oom_score_adj=0
sudo systemctl daemon-reload
sudo systemctl restart pipewire-pulse

[vinc17fr]

AI:

It’s a bug in the systemd service configuration for pipewire-pulse

By default, the pipewire-pulse unit ships with oom_score_adj=200, [...]

This seems to be incorrect:

qaa% grep -r oom_score_adj /usr/lib/systemd /etc/systemd
grep: /usr/lib/systemd/systemd: binary file matches
grep: /usr/lib/systemd/systemd-executor: binary file matches

And there's no oom_score_adj either in the pipewire Debian source package (from which pipewire-pulse is generated).

[rfjakob]

These two having the same oom_score is bizzare

Aug 04 23:41:28 qaa earlyoom[1308]: sending SIGTERM to process 2002 uid 1000 "pipewire-pulse": oom_score 800, VmRSS 40 MiB, cmdline "/usr/bin/pipewire-pulse"
Aug 04 23:41:35 qaa earlyoom[1308]: sending SIGTERM to process 5446 uid 1000 "Isolated Web Co": oom_score 800, VmRSS 3257 MiB, cmdline "/usr/lib/firefox/firefox -contentproc -isForBrowser -prefsHandle 0:34285 -prefMapHandle 1:283567 -jsInitHandle 2:242012 -parentBuildID 20250717180000 -sandboxReporter 3 -ipcHandle 4 -initialChannelId {4ac59450-5248-4a8c-83ff-bced872d4463} -parentPid 46 -"

I think that earlyoom should also output the oom_score_adj value

Ack. In the meantime:

1. Can you check what oom_score_adj is actually set to for the pipewire-pulse process? This should do it:

   cat /proc/$(pgrep pipewire-pulse)/oom_score_adj

2. Add --sort-by-rss to /etc/default/earlyoom

[vinc17fr]

1. Can you check what oom_score_adj is actually set to for the pipewire-pulse process? This should do it:
   cat /proc/$(pgrep pipewire-pulse)/oom_score_adj

qaa% cat /proc/$(pgrep pipewire-pulse)/oom_score_adj
200

FYI, all the following processes have a value of 200:

   200 dbus-daemon
   200 pipewire
   200 pipewire
   200 wireplumber
   200 mpris-proxy
   200 xdg-permission-
   200 gvfsd
   200 gvfsd-fuse
   200 gpg-agent
   200 at-spi-bus-laun
   200 dbus-daemon
   200 at-spi2-registr
   200 xdg-desktop-por
   200 xdg-document-po
   200 fusermount3
   200 xdg-desktop-por
   200 dconf-service
   200 notification-da
   200 pipewire-pulse

They are the descendants of the same process:

  vinc17 ├─> 1981  systemd --user --deserialize=34
  vinc17 │ ├─> 1983  (sd-pam)
  vinc17 │ ├─> 1997  dbus-daemon --session --address=systemd: --nofork --nopidfile --systemd-activation --syslog-only
  vinc17 │ ├─> 1998  pipewire
  vinc17 │ ├─> 1999  pipewire -c filter-chain.conf
  vinc17 │ ├─> 2001  wireplumber
  vinc17 │ ├─> 2003  mpris-proxy
  vinc17 │ ├─> 2126  xdg-permission-store
  vinc17 │ ├─> 3486  gvfsd
  vinc17 │ ├─> 3492  gvfsd-fuse gvfs -f
  vinc17 │ ├─> 3827  gpg-agent --supervised
  vinc17 │ ├─> 3833  at-spi-bus-launcher
  vinc17 │ │ └─> 3840  dbus-daemon --config-file=/usr/share/defaults/at-spi2/accessibility.conf --nofork --print-address 11 --address=unix:path=/run/user/1000/at-spi/bus_0
  vinc17 │ ├─> 3848  at-spi2-registryd --use-gnome-session
  vinc17 │ ├─> 92214  xdg-desktop-portal
  vinc17 │ ├─> 92243  xdg-document-portal
    root │ │ └─> 92250  fusermount3 -o rw,nosuid,nodev,fsname=portal,auto_unmount,subtype=portal -- doc
  vinc17 │ ├─> 92258  xdg-desktop-portal-gtk
  vinc17 │ ├─> 152596  dconf-service
  vinc17 │ ├─> 339845  notification-daemon
  vinc17 │ └─> 1028192  pipewire-pulse

So it could be a systemd issue.

2. Add --sort-by-rss to /etc/default/earlyoom

OK.

[vinc17fr]

The systemd --user --deserialize=34 process itself has an oom_score_adj value of 100. This could be due to the OOMScoreAdjust=100 line in /usr/lib/systemd/system/user@.service. But I did not find anything that could yield 200 for its descendants.

[vinc17fr]

The systemd-system.conf(5) man page says:

DefaultOOMScoreAdjust=
[...] This defaults to unset (meaning the forked off processes inherit the service manager's OOM score adjustment value), except if the service manager is run for an unprivileged user, in which case this defaults to the service manager's OOM adjustment value plus 100 (this makes service processes slightly more likely to be killed under memory pressure than the manager itself). [...]

It seems that this "plus 100" is the explanation.

But all these systemd DefaultOOMScoreAdjust settings makes no sense in a context where not all the processes are started by systemd.

[vinc17fr]

Concerning Firefox in particular, it was supposed to take into account the systemd settings (mentioning the 100+100), but this proposed solution assumes that the initial oom_score_adj value for Firefox has already been increased (while it is 0 here), so that this does not work. I've posted I comment to a related Firefox bug.

[rfjakob]

So firefox has 100 while pipewire-pulse has 200? Did i get this right?