OOMKilled and Nothing Else: Finding What Really Ate Your Pod

The pod restarted at 02:14. You run kubectl describe pod, and the entire forensic record Kubernetes has kept for you is one word: OOMKilled. Exit code 137. No stack trace, no allocation profile, no hint about which of the three processes inside that container was holding the memory, and no record of what usage looked like in the ninety seconds before the kill. The container filesystem is gone. The cgroup is gone. The evidence deleted itself.

This is not an oversight you can configure away. Kubernetes never had the information. The kernel killed your process, the kubelet noticed afterwards that the container had exited, and it wrote down the only fact it could observe. Every "how to debug OOMKilled" article that tells you to raise the memory limit is telling you to stop investigating, because raising the limit is what you do when you have given up on knowing why. The data you need exists before the kill, on the node, in the cgroup. If you are not collecting it in advance, you will not have it afterwards.

The exact output you get, and what each field means

kubectl describe pod api-7c9f4d8b6-x2wql

    Last State:     Terminated
      Reason:       OOMKilled
      Exit Code:    137
      Started:      Wed, 22 Jul 2026 02:12:41 +0300
      Finished:     Wed, 22 Jul 2026 02:14:07 +0300
    Ready:          True
    Restart Count:  4

Exit code 137 is 128 + 9, the shell convention for "terminated by signal 9". Signal 9 is SIGKILL. Reason: OOMKilled is not something the kernel reported to Kubernetes; it is the container runtime's interpretation of a process that exited by SIGKILL inside a cgroup that had an OOM event.

One consequence matters more than it looks: only the container's main process produces this label. If your entrypoint is a shell script that starts a worker and the kernel kills the worker, PID 1 keeps running, the container never restarts, and Kubernetes reports nothing at all. You get silent partial degradation instead of a crash, which is far harder to notice than a restart loop.

Why Kubernetes cannot tell you the cause

Under cgroup v2, each container gets a cgroup with memory.max set to its limits.memory. When a charge would push memory.current above memory.max, the kernel first tries to reclaim: drop clean page cache, write back dirty pages, swap if swap exists (in most clusters it does not). If reclaim cannot free enough, the cgroup OOM killer runs, picks a victim inside that cgroup, and sends SIGKILL.

SIGKILL is not deliverable to a handler. There is no window in which your process gets told "you are about to be killed, please dump a heap profile". The kernel writes a line to the kernel ring buffer and the process is gone. This is the single most important fact in this article, and it is why the common advice to "dump a profile on SIGTERM" is useless here: SIGTERM is what happens on a pod deletion or a rollout, not on an OOM kill.

Meanwhile the kubelet is polling container state on an interval. It sees a container that is no longer running, asks the runtime why, and is told signal 9 plus an OOM event for that cgroup. That is the sum total of the information in the system, and Kubernetes faithfully reports all of it. There is no hidden verbose flag.

Step 1: Prove it was a limit kill, not a node eviction

These look identical on a dashboard and have opposite fixes. An OOM kill means the container exceeded its own limit. An eviction means the node ran short and the kubelet chose your pod to sacrifice, usually because it was using more than its requests.

kubectl get pod api-7c9f4d8b6-x2wql -o jsonpath='{range .status.containerStatuses[*]}{.name}{"\t"}{.lastState.terminated.reason}{"\t"}{.lastState.terminated.exitCode}{"\n"}{end}'

api	OOMKilled	137
otel-agent		

If instead the pod phase is Failed with Reason: Evicted and a message mentioning memory pressure, you have a node capacity or requests problem, not an application memory problem. Do not tune the wrong thing.

Step 2: Read the kernel log on the node

This is the step almost every article skips, and it is the one that actually names the process. Get onto the node (a debug pod, SSH, or kubectl debug node/<node> -it --image=busybox) and read the ring buffer:

dmesg -T | grep -i -A4 'memory cgroup out of memory'

A cgroup OOM kill produces a block that ends with a line naming the victim, its total virtual size, its anonymous RSS, its file-backed RSS and its OOM score adjustment. That line answers the question kubectl describe cannot: which process, and how big it was at the moment it died. If the victim is not the process you assumed, stop; you have been profiling the wrong thing.

Just above the kill line the kernel prints a per-task table listing every process in that cgroup with its RSS. On a container running a supervisor plus four workers, that table tells you instantly whether one worker ballooned or all of them grew evenly. One ballooning worker is a request-specific leak. Even growth is a sizing problem.

If dmesg has already rotated past the event, check the node's journal instead, where the same OOM lines are usually retained far longer. My general approach to that is in reading Linux logs when it is 3am.

Step 3: Read the cgroup counters while the container is still alive

You cannot read a dead cgroup, so this is a step you take on a pod that is climbing, not one that has already died. Exec into the container:

kubectl exec -it api-7c9f4d8b6-x2wql -c api -- sh -c \
  'cat /sys/fs/cgroup/memory.current /sys/fs/cgroup/memory.max; echo ---; cat /sys/fs/cgroup/memory.events'

1789562880
2147483648
---
low 0
high 0
max 1043
oom 2
oom_kill 2

Those counters distinguish situations that all look identical from outside. max counts how many times allocation hit the ceiling and forced reclaim: a high max with oom_kill 0 means you are living right at the limit and paying for it in reclaim CPU, but nothing has died yet. That is your warning shot. oom_kill is the number of processes actually killed in this cgroup; if it is non-zero while the container is still running, a child process was killed and Kubernetes never told you. On newer kernels, memory.peak in the same directory gives you the high-water mark without needing a metrics pipeline at all.

Then split the usage by type:

kubectl exec -it api-7c9f4d8b6-x2wql -c api -- \
  grep -E '^(anon|file|slab|sock|shmem|inactive_file|kernel_stack) ' /sys/fs/cgroup/memory.stat

This is where the diagnosis actually happens, and it is the step that separates four completely different bugs hiding behind one word. Kubernetes and cAdvisor report working set, which is roughly memory.current minus inactive_file, so a dashboard graphing raw memory.current will show alarming page cache that was never going to kill anything. The one that catches people out is shmem: anything written to /dev/shm or to an emptyDir with medium: Memory counts against your limit and is not reclaimable. Growth in slab or kernel_stack is kernel-side, usually thread or socket sprawl, and no heap profiler will ever show it to you.

Step 4: Capture the curve so next time you skip steps 2 and 3

The underlying problem is resolution. kubectl top pod reads metrics-server, which samples on a coarse interval and keeps no history. A container can go from steady to dead in under a second on one bad request, and all you will see is a flat line followed by a restart. Scrape cAdvisor through the kubelet at 5 to 10 second resolution and keep two series:

groups:
  - name: oom-forensics
    rules:
      - record: pod:mem_working_set_ratio
        expr: |
          container_memory_working_set_bytes{container!="",container!="POD"}
            /
          on(namespace,pod,container) kube_pod_container_resource_limits{resource="memory"}
      - alert: ContainerOOMKilled
        expr: increase(container_oom_events_total[10m]) > 0

container_oom_events_total is the metric that catches the silent case from step 3, because cAdvisor derives it from kernel OOM events rather than from container restarts. A child-process kill that never restarts the container still increments it.

Step 5: Get a profile out before the kill, not after

Since SIGKILL cannot be intercepted, the profile has to be produced while the process is still alive. Three approaches work, in decreasing order of how much I like them:

1. Continuous profiling. Ship heap profiles on a timer to somewhere outside the pod. When the kill happens you already have the last sample from twenty seconds before death. This is the only approach that reliably survives a kill, because it does not depend on the dying process doing anything.
2. A runtime ceiling below the cgroup ceiling. Make the language runtime hit its limit first so it raises a catchable error instead of the kernel sending SIGKILL. In Go, set GOMEMLIMIT to roughly 80-90% of the container limit so the collector works harder before the kernel gets involved. On the JVM, -XX:MaxRAMPercentage plus -XX:+HeapDumpOnOutOfMemoryError gives you a heap dump, but only for Java heap exhaustion; a cgroup kill caused by native memory or thread stacks bypasses it entirely. On Node.js, --max-old-space-size plus --heapsnapshot-near-heap-limit=1 writes a snapshot as V8 approaches its own limit.
3. A polling sidecar. A small sidecar that reads the cgroup and triggers your app's profiling endpoint once usage crosses a threshold. Crude, but it costs nothing and works for runtimes with no self-limiting story, Python included.

Note the asymmetry in option 2: the runtime limit must sit below the container limit with headroom for everything the runtime does not count. Off-heap buffers, thread stacks, allocator overhead and any page cache your process dirties all live inside the cgroup limit but outside the runtime's accounting. A JVM configured for 100% of the container limit is a JVM that gets OOMKilled while reporting a perfectly healthy heap.

Verification: proving you fixed the right thing

Do not verify by "it has not restarted since". Verify by measuring headroom.

# 1. The kill counter should stop moving. Check it after a full traffic cycle.
kubectl exec deploy/api -c api -- cat /sys/fs/cgroup/memory.events | grep oom_kill

# 2. Working set should plateau well below the limit, not creep.
kubectl top pod -l app=api --containers

# 3. Restart count should be stable across a peak period.
kubectl get pods -l app=api -o custom-columns=\
'NAME:.metadata.name,RESTARTS:.status.containerStatuses[*].restartCount'

The test that matters is sustained load at realistic concurrency for longer than one full garbage collection or cache-expiry cycle. A leak that takes six hours to kill a pod will happily pass a ten minute smoke test. And if memory.events shows max climbing steadily even with oom_kill at zero, you have not fixed it; you have moved the pod into permanent reclaim pressure, which resurfaces later as latency rather than crashes.

What people get wrong

Common advice	Why it fails
"Just raise the memory limit"	Correct only if you first proved the workload legitimately needs more. Against a leak it changes the crash interval from one hour to four and consumes cluster capacity that other pods needed. It is a schedule change disguised as a fix.
"Dump a heap profile on SIGTERM"	An OOM kill is SIGKILL. Your handler does not run. You will get profiles for graceful shutdowns and rollouts, and nothing at all for the event you care about.
"Remove the limit so it cannot be OOMKilled"	You have not removed the kill, you have promoted it to the node level. Now the node OOM killer or the kubelet's eviction logic chooses a victim, and it may not be your pod. One unbounded container takes down neighbours.
"Set requests equal to limits"	Good for scheduling stability, and Guaranteed QoS makes the pod the last thing evicted under node pressure. It does nothing about hitting your own limit. Guaranteed pods are OOMKilled exactly as readily as any other.
"Use kubectl top to find the cause"	Too coarse and completely historyless. By the time you run it, the pod that died has been replaced by a fresh one showing 200MB.

When it is still broken

1. Memory jumps in a single step rather than climbing. That is one allocation, usually a request that reads an entire file or result set into memory. Cap it at the boundary: a max request body size, a streaming parser, a LIMIT on the query. No amount of tuning fixes an unbounded input.
2. It only happens on one node pool. Compare kernel and cgroup versions across pools. Behaviour under memory pressure genuinely differs between cgroup v1 and v2, and a mixed fleet produces a bug that "only happens on some pods".
3. Usage grows with uptime but the heap does not. Look at native allocations, thread count and open file descriptors. Glibc's allocator holding freed arenas is a classic here; a musl-based image or an explicit arena limit often changes the picture entirely.
4. It is not memory at all. If the container keeps dying but the OOM counters stay at zero, something else is killing it. A failing liveness probe restarts a container without any OOM event. My CrashLoopBackOff with no logs walkthrough covers that path, and sharing one health endpoint between liveness and readiness is the usual culprit.

For the node-level version of the same mechanism, where the kernel chooses between whole processes rather than inside one cgroup, how the Linux OOM killer picks its victim is the companion piece. The cgroup killer uses the same scoring, just scoped.

The uncomfortable summary: Kubernetes will never tell you why a pod was OOMKilled, because by the time Kubernetes is involved the answer no longer exists. The fix is not a better kubectl command. It is deciding in advance to keep the cgroup counters and the working-set curve, so the next kill arrives with evidence attached.