Killed by the Kernel: What Really Happens When Your Server Runs Out of RAM

Somebody pings you: "the site is down." You SSH in and everything looks suspiciously normal. Uptime is 14 days. Load is 0.4. But MariaDB is not running, and nothing in the MariaDB log explains why - it just stops mid-sentence. You restart it, the site comes back, and you spend the next three weeks quietly afraid.

That is an OOM kill, and here is the thesis: the Linux OOM killer is deterministic, it leaves a detailed receipt, and it almost never kills the process that caused the problem. Once you can read the receipt and place hard caps on the guilty cgroup, this whole class of outage becomes boring.

Why the crash happens hours after the mistake

Linux overcommits memory. When your app calls malloc(500MB), the kernel says yes without reserving anything. Physical pages are only allocated when the process actually touches the memory. So the moment of the lie and the moment of the consequence are far apart, and by the time the kernel runs out of pages there is no allocation to fail gracefully - the kernel is already committed.

At that point the kernel has three options: reclaim page cache, swap anonymous pages to disk, or kill something. If there is no swap and the cache is already minimal, it kills. The policy that picks the victim is the OOM killer, and it optimises for "free the most memory by killing the fewest processes" - which is exactly why it picks your database.

Step 1: prove it was an OOM kill

sudo dmesg -T | grep -iE "killed process|out of memory|oom-kill"
journalctl -k --since "3 days ago" -g "Out of memory"
journalctl --since "2026-07-20 02:00" --until "2026-07-20 03:00" -p warning

A real kill record looks like this:

[Mon Jul 20 02:41:33 2026] node invoked oom-killer: gfp_mask=0x140cca(GFP_HIGHUSER_MOVABLE|__GFP_COMP), order=0, oom_score_adj=0
[Mon Jul 20 02:41:33 2026] oom-kill:constraint=CONSTRAINT_NONE,nodemask=(null),cpuset=/,mems_allowed=0,global_oom,task_memcg=/system.slice/billing.service,task=mariadbd,pid=812,uid=110
[Mon Jul 20 02:41:33 2026] Out of memory: Killed process 812 (mariadbd) total-vm:2318412kB, anon-rss:1186320kB, file-rss:0kB, shmem-rss:0kB, UID:110 pgtables:3204kB oom_score_adj:0

Read it carefully, because three separate facts are in there:

• Who triggered it: the first line - node invoked oom-killer. Node asked for memory that was not there. Node is the culprit.
• Who died: the last line - Killed process 812 (mariadbd). MariaDB had 1.1GB resident, so killing it frees the most. MariaDB is the victim, not the cause.
• What kind of OOM: constraint=CONSTRAINT_NONE plus global_oom means the whole machine ran out. If you see CONSTRAINT_MEMCG and a line reading Memory cgroup out of memory, only one cgroup hit its ceiling and the rest of the box was fine.

That distinction is the fork in the road. Global OOM means you need to reduce total demand or add RAM or swap. Cgroup OOM means one unit hit the limit you set, which is the system working as designed.

Step 2: find out where the memory actually went

free -h
ps -eo pid,user,rss,vsz,comm --sort=-rss | head -15
systemd-cgtop -m --order=memory
sudo smem -t -k -c "pid user command rss pss uss" -s uss 2>/dev/null | tail -20

In free -h, the only column that matters is available. People panic at a small "free" number, but on a healthy Linux box free memory is close to zero by design - the kernel uses spare RAM as page cache and hands it back instantly when something needs it. Cached memory is not a leak.

RSS in ps double-counts shared pages, so for anything with worker processes (PHP-FPM, Gunicorn, Frappe's RQ workers) use smem and read the USS column - that is memory that genuinely disappears if you kill the process.

Step 3: cgroups v2, where the truth lives

On Ubuntu 22.04 and 24.04 every systemd service is in a cgroup v2 with real accounting. This is the highest-signal data on the machine and almost nobody looks at it:

CG=/sys/fs/cgroup/system.slice/billing.service
cat $CG/memory.current      # bytes in use right now
cat $CG/memory.max          # hard ceiling, or "max"
cat $CG/memory.events       # counters
cat $CG/memory.pressure     # PSI for this service alone

memory.events gives you something like:

low 0
high 8412
max 41
oom 3
oom_kill 3

Three OOM kills inside that unit since boot. That is a leak, and you now have a number to track over time instead of a feeling. Verify a unit's effective limits with:

systemctl show billing.service -p MemoryMax -p MemoryHigh -p MemoryCurrent -p MemoryAccounting

Step 4: the earliest possible warning - PSI

Pressure Stall Information tells you how much time tasks spent stalled waiting for memory. It goes bad minutes before anything dies:

cat /proc/pressure/memory

some avg10=0.00 avg60=0.00 avg300=0.00 total=1044821
full avg10=0.00 avg60=0.00 avg300=0.00 total=402113

Zeroes are healthy. If some avg60 climbs past roughly 10 and stays there, the machine is thrashing and you have a few minutes to act. This is a far better alerting metric than "memory used percent", because it measures actual harm rather than a number that is supposed to be high.

The fixes, in the order you should apply them

1. Cap the guilty service, do not protect the victim

People's first instinct is to shield MariaDB with a negative oom_score_adj. That works, sort of - the OOM killer then picks the next biggest thing, which might be your app, or sshd. Better to put a ceiling on the process that leaks:

sudo systemctl edit billing.service

[Service]
MemoryHigh=640M
MemoryMax=768M
OOMPolicy=stop
Restart=on-failure
RestartSec=10s

Now when the leak recurs, the kernel kills inside that cgroup, systemd restarts just that service, and the rest of the box never notices. MemoryHigh applies throttling first, so a brief spike gets slowed down instead of killed.

2. Right-size your database, seriously

The classic 2GB VPS disaster is MariaDB with a default-ish innodb_buffer_pool_size plus 4 Gunicorn workers plus a Node build. Do the arithmetic before the kernel does it for you:

# /etc/mysql/mariadb.conf.d/60-tuning.cnf
[mysqld]
innodb_buffer_pool_size = 512M
innodb_log_file_size    = 128M
max_connections         = 60
performance_schema      = OFF

On a 2GB box, budget roughly: 512M database, 700M app, 200M Nginx plus Redis, 300M kernel and page cache. Anything left is your safety margin, and it should not be zero.

3. Add swap - it is not shameful

A 2GB VPS with zero swap is a machine with no shock absorber. Swap does not make you fast; it makes the difference between "slow for 40 seconds" and "MariaDB is dead".

sudo fallocate -l 2G /swapfile
sudo chmod 600 /swapfile
sudo mkswap /swapfile
sudo swapon /swapfile
echo '/swapfile none swap sw 0 0' | sudo tee -a /etc/fstab
swapon --show
free -h

Then tune how eagerly the kernel uses it. On a server with a database, low swappiness is right - you want anonymous pages to stay resident:

echo 'vm.swappiness=10' | sudo tee /etc/sysctl.d/99-swap.conf
sudo sysctl --system

If your VPS has slow disk or you pay for IOPS, use compressed RAM swap instead:

sudo apt install -y systemd-zram-generator
printf '[zram0]\nzram-size = ram / 2\ncompression-algorithm = zstd\n' | sudo tee /etc/systemd/zram-generator.conf
sudo systemctl daemon-reload
sudo systemctl start systemd-zram-setup@zram0.service

4. Kill early, in userspace

The kernel OOM killer only fires when the machine is already unusable - often after minutes of thrashing during which SSH itself will not respond. Userspace killers act on PSI while you can still log in.

systemctl is-active systemd-oomd    # present on modern Ubuntu, not always enabled on server
oomctl                              # shows what systemd-oomd is monitoring

To let systemd-oomd act on a specific slice, opt in explicitly:

# /etc/systemd/system/user.slice.d/oomd.conf
[Slice]
ManagedOOMMemoryPressure=kill
ManagedOOMMemoryPressureLimit=60%

The simpler alternative, and my usual choice on a small box, is earlyoom:

sudo apt install -y earlyoom
sudo systemctl enable --now earlyoom
journalctl -u earlyoom -n 20

Its defaults kill the highest oom_score process when free memory drops below 10 percent. Configure the preference list in /etc/default/earlyoom so it avoids your database.

5. Only then, change oom_score_adj

If you truly must, use the systemd knob rather than poking /proc, which resets on restart:

[Service]
OOMScoreAdjust=-500

Valid range is -1000 to 1000. The final score is roughly the proportion of memory used, scaled to 0-1000, plus oom_score_adj. Inspect any process live:

cat /proc/$(pgrep -f mariadbd | head -1)/oom_score
cat /proc/$(pgrep -f mariadbd | head -1)/oom_score_adj

Setting -1000 makes a process completely immune, which sounds great until the immune process is the one leaking and the kernel has to kill sshd instead. Use with care.

Two failure modes that look like OOM but are not

If your service dies with status=137 in Docker, that is 128 plus signal 9 - a SIGKILL. It might be OOM, or it might be a health check timeout. Check docker inspect --format '{{.State.OOMKilled}}' <container> before you go tuning memory.

And the one that catches ERPNext and Frappe teams every single time: bench build runs esbuild and Node with a large heap. On a 2GB production box it will happily consume 1.5GB, trigger a global OOM, and take MariaDB with it - mid-deploy, with the database mid-migration. Build assets on a separate machine or in CI, or at minimum cap the build:

sudo systemd-run --scope -p MemoryMax=800M -p MemoryHigh=700M \
  --uid=frappe --working-directory=/home/frappe/frappe-bench \
  bench build --production

A five-minute prevention checklist

1. Confirm swap exists: swapon --show. If it prints nothing, fix that today.
2. Put MemoryMax on every non-database service you wrote yourself.
3. Set innodb_buffer_pool_size explicitly rather than accepting whatever the package chose.
4. Alert on /proc/pressure/memory some avg60 above 10, not on percent-used.
5. Add dmesg -T | grep -i "killed process" to your weekly review, or wire it into a cron that emails you.

The OOM killer is not your enemy. It is a very reasonable actor making the best decision it can with terrible information. Give it better information - explicit limits per cgroup - and it will make better decisions.