The OOM Killer Took Down My App: Reading dmesg to Find the Real Memory Hog

Your application is gone. Not crashed with a stack trace, not logged an exception, just gone. The process is no longer running, the service shows as failed, and when you open the application log expecting an answer, the last line is a perfectly ordinary log message from thirty seconds before it died. Nothing. It is as if the app simply evaporated.

It did not crash. It was executed. Somewhere under memory pressure, the Linux kernel decided the machine was about to run out of RAM, picked a process, and killed it with SIGKILL to save the system. That is the OOM (out of memory) killer, and the reason your app log is silent is that the kill happened one level below your app, in the kernel, which does not write to your log file.

The mistake almost everyone makes next is to blame the victim. The app that got killed is frequently not the process that ate the memory. The kernel writes down exactly who the real hog was, but only if you read the whole report instead of the last line.

The exact log line, and exit code 137

When systemd reports the death, it shows the signal:

systemd[1]: myapp.service: Main process exited, code=killed, status=9/KILL
systemd[1]: myapp.service: Failed with result 'signal'.

Under Docker or a plain shell you see the numeric form instead, exit code 137, which is 128 + 9: terminated by signal 9. And in the kernel ring buffer, the smoking gun:

Out of memory: Killed process 24871 (python3) total-vm:2484556kB,
anon-rss:1932244kB, file-rss:0kB, shmem-rss:452kB, UID:1011
pgtables:4608kB oom_score_adj:0

That single line tells you who died. It does not tell you why, and it does not tell you who was really responsible. For that you have to read upward.

Why this happens: overcommit and the kernel's terrible choice

Linux lets processes allocate more virtual memory than physically exists, on the bet that most of it will never be touched at once. This is called overcommit, and it is usually a good bet. When the bet goes wrong and processes actually try to use memory that is not there, the kernel cannot give it out, and it cannot simply block forever. So it invokes the OOM killer to free memory by killing a process.

The kernel scores every process with an oom_score, driven mostly by how much memory the process is using, adjusted by oom_score_adj. It kills the highest score. The intent is to reclaim the most memory for the least number of kills, which is why a large, well-behaved process (your database, your app server) is an attractive target even when it is not the thing that caused the pressure. The actual cause is often a cron job, a backup, a runaway log-processing script, or an unbounded cache that spiked briefly, pushed the system over the edge, and then got away with it while your app took the bullet.

Step 1: Get the OOM report, in a form that survives a reboot

The raw dmesg ring buffer is wiped on reboot and can be overwritten under load, so prefer journalctl, which persists the kernel log via journald:

# Kernel messages, persisted, searchable by string
journalctl -k --grep "Out of memory"

# Or, for a live/recent box, human-readable timestamps
dmesg -T | grep -i -E 'oom|killed process'

Expected output includes one or more Out of memory: Killed process ... lines with timestamps. Note the exact time; you will want the block of log immediately before it.

Step 2: Read the whole report, especially the process table

This is the step almost everyone skips, and it is the entire game. Just above the "Killed process" line, the kernel dumps a table of every process it considered, with each one's RSS and oom_score_adj. Pull the full block:

journalctl -k --since "2026-07-24 02:00" --until "2026-07-24 02:05" -o short-precise

You are looking for the table with a header like [ pid ] uid tgid total_vm rss ... oom_score_adj name. Sort your attention by the rss column (resident memory, in pages). The row with the largest rss is the process that was actually holding the most physical memory at the moment of crisis. If that is a backup script, a log processor, or a Redis instance without a memory cap, you have found your real hog, even though the kernel killed something else.

Step 3: Fix the cause, not the victim

What you do depends on what the table revealed:

• A one-off spike (cron, backup): schedule it when the box is idle, stream instead of buffering the whole dataset in memory, or run it with a memory cap so it cannot starve everything else.
• An unbounded cache: a Redis instance with no maxmemory will grow until it triggers OOM. Set maxmemory and an eviction policy. I go into the trap here in Redis 'OOM command not allowed': why switching to LRU can cost you data.
• A genuine leak: the process's RSS grows monotonically over hours or days with steady load. That is a leak to profile and fix, not a sizing problem.
• An honestly undersized host: if every process is right-sized and the sum simply exceeds RAM, add RAM. Sometimes the answer really is a bigger VPS, and on a Nairobi budget that is a real cost decision, not a shrug.

Contain a runaway with systemd, do not let it starve the host

The cleanest structural fix is to cap the service that misbehaves so the kernel never has to choose between it and your database. systemd exposes this per unit:

[Service]
# Soft cap: reclaim aggressively above this
MemoryHigh=1.5G
# Hard cap: kill only THIS cgroup if it exceeds this
MemoryMax=2G

Now a runaway hits its own wall and gets killed inside its own cgroup, long before it can drag down the rest of the machine. Combine this with a robust unit so the service comes back cleanly, which I cover in a systemd unit file that actually keeps it running.

Verification

After a fix, confirm two things. First, that no new OOM events are firing:

journalctl -k --since "1 hour ago" --grep "Out of memory"
# expect: no output

Second, that your capped service is honouring its limit. With the process running under systemd:

systemctl show myapp.service -p MemoryCurrent -p MemoryMax
# expect MemoryCurrent to stay comfortably below MemoryMax under load

If you set an oom_score_adj on a critical process, verify it took:

cat /proc/$(pgrep -o mariadbd)/oom_score_adj

What people get wrong

"Just add swap." Swap does not create memory, it trades speed for space. On a box under real pressure, swap turns a clean, instant OOM kill into minutes of disk thrashing where every request crawls and the machine looks alive but serves nobody. For a short, genuine spike a little swap is reasonable insurance; as a fix for a leak or an undersized host it makes the failure slower and worse, not gone.

"Disable overcommit." Setting vm.overcommit_memory=2 blindly can cause allocations to fail across the board, including in processes that would never have touched the memory, breaking software that relies on optimistic allocation. It is a scalpel some workloads need, not a default to flip because a blog said so.

"Protect everything with oom_score_adj=-1000." If you make every important process unkillable, the kernel has nothing left to kill under pressure and can panic or kill something even more critical. Protection is a relative dial. Lower the score on the one or two processes that must survive, and let the kernel still have valid targets.

When it is still broken

• No OOM line in the log at all, but the process still dies with 137. A container runtime enforcing a memory limit kills at the cgroup level and may not print the classic kernel line. Check docker inspect for OOMKilled: true, or the Kubernetes pod's last state. I walk through that path in OOMKilled and nothing else: finding what really ate your pod.
• The victim keeps changing. That is a sign of broad, sustained pressure rather than one hog. Trend total memory over time with free -m on a timer or your metrics stack, and look for the slow climb.
• It only happens at a fixed time. Cross-reference the OOM timestamp with cron and timers (systemctl list-timers). A nightly job lining up exactly with the kill is your answer.

The kernel is not being capricious. It left you a full witness statement in dmesg. Read past the last line, and it will tell you exactly who to blame.