OOMKilled Exit Code 137: Leak, Bad Limit, or JVM Off-Heap?

Your pod restarts. You look, and the last state reason is OOMKilled, exit code 137. So you bump the memory limit, redeploy, and it buys you a day. Then it happens again. You bump it again. Three weeks later the limit is 8Gi, the app still dies, and your cluster capacity is gone.

Raising the limit is not a diagnosis. Exit code 137 looks identical whether the app has a genuine memory leak, the limit was copy-pasted from an unrelated service, or a JVM is allocating memory the container limit never accounted for. Those three causes need three different fixes, and the kill message tells you nothing about which one you have. This article is about telling them apart before you waste another week guessing.

Confirm it is actually a cgroup OOM kill

Before diagnosing, prove it is the OOM killer and not an app crash that also exits non-zero:

kubectl describe pod my-pod

    Last State:     Terminated
      Reason:       OOMKilled
      Exit Code:    137
      Started:      ...
      Finished:     ...

Reason: OOMKilled with Exit Code: 137 is the kernel's cgroup OOM killer sending SIGKILL (128 + 9 = 137) because the container tried to use more memory than its cgroup limit. This is not a Go panic or an unhandled exception; the process was killed from outside. If the reason is Error with a different code, you are looking at an application crash instead, and this is the wrong article.

Why one exit code hides three different bugs

The cgroup OOM killer does exactly one thing: when the processes in a cgroup collectively exceed memory.limit_in_bytes, it kills one. It does not know or care why you were over the line. Whether your heap grew forever, or your limit was always too small, or your runtime allocated memory Kubernetes never sized for, the kernel's response and the resulting 137 are byte-for-byte identical. The information that distinguishes the three lives in the memory-over-time curve, not in the death certificate.

Cause 1: A real memory leak

The signature is a line that climbs steadily across the pod's entire lifetime and never falls back after garbage collection or request completion. Plot container memory over hours or days:

# quick look
kubectl top pod my-pod --containers

# the real answer: chart working_set over time in Prometheus
container_memory_working_set_bytes{pod="my-pod"}

If working_set ramps up monotonically regardless of traffic, you have a leak. Raising the limit only changes when the crash lands, not whether. The fix is in the application: profile it (pprof for Go, a heap dump for the JVM, tracemalloc for Python) and find what is retained. A larger limit here is money spent to fail slightly later.

Cause 2: The limit was set too low

The signature is a healthy sawtooth, memory rising with work and dropping after GC or between requests, that simply bumps into a ceiling well below what the app genuinely needs at peak. This is what you get when a limit was copied from an unrelated service or picked from a single quiet-hour observation.

The fix is to measure real usage across a representative window, including your busiest period, and set the limit above observed peak with headroom:

max_over_time(container_memory_working_set_bytes{pod=~"my-app.*"}[7d])

A limit derived from one kubectl top reading misses peak load by design. Watch it for days, not seconds.

Cause 3: JVM (or other runtime) off-heap memory

This is the sneaky one. The JVM sizes its heap as a percentage of the container's memory limit, but it also allocates on top of the heap: metaspace, thread stacks, the code cache, GC structures, and direct byte buffers. If you set the container limit to your desired heap size, the off-heap allocations push the total over the limit and the container is killed while the heap graph still looks calm and half-empty.

The fix is to leave room. Set the heap to a conservative fraction of the limit, not all of it:

env:
  - name: JAVA_TOOL_OPTIONS
    value: "-XX:MaxRAMPercentage=75.0 -XX:InitialRAMPercentage=75.0"
resources:
  limits:
    memory: "2Gi"
  requests:
    memory: "2Gi"

With a 2Gi limit and MaxRAMPercentage=75, the heap caps around 1.5Gi and roughly 512Mi is left for everything off-heap. If you instead pass -Xmx2g against a 2Gi limit, you have budgeted zero for off-heap and the kill is only a matter of load. The percentage flags are container-aware on any modern JVM; do not hardcode -Xmx equal to the limit.

Signature over pod lifetime	Cause	Correct fix
Linear climb, never drops	Memory leak	Profile and fix the app; do not raise the limit
Healthy sawtooth clipped low	Limit too low	Set limit above 7-day observed peak
Killed while heap looks fine	Off-heap not budgeted	Lower MaxRAMPercentage, leave off-heap headroom

Verification: prove the fix, do not just wait and hope

For the limit and JVM fixes, redeploy and watch working_set settle below the limit across a full traffic cycle, including peak:

watch -n 30 'kubectl top pod -l app=my-app --containers'

For a leak fix, the proof is different and stronger: the previously monotonic curve now returns to a stable baseline after GC. If the line still climbs, you fixed a symptom, not the leak.

What people get wrong

Raising the limit as a reflex. It is the single most common response and it only helps in exactly one of the three cases. For a leak it delays the crash and wastes cluster capacity; for a JVM off-heap problem it can help by accident, which teaches the wrong lesson. Diagnose the curve first.

Setting requests equal to limits everywhere because someone said Guaranteed QoS is better. Requests equal to limits gives the pod Guaranteed QoS, so it is evicted last under node pressure, but it also removes all burst headroom and can over-reserve the node. That is a deliberate tradeoff for latency-critical workloads, not a default to copy onto everything. Understand what you are buying.

Reading kubectl top once and calling it capacity planning. A single reading misses peak. Only a multi-day chart tells you the real ceiling.

When it is still broken

1. Confirm which container in the pod died. A multi-container pod reports OOM per container. kubectl describe pod shows the reason under each container; do not tune the wrong one.
2. Check node-level pressure vs container limit. A container killed at its own limit is different from a pod evicted because the node ran out of memory. The events section of kubectl describe distinguishes OOMKilling a cgroup from node MemoryPressure eviction.
3. Look for a burst you are not graphing. Startup, a big batch job, or a cache warm can spike memory for seconds and be missed by a 30s scrape interval. Lower the interval or catch it with container_memory_max_usage_bytes.
4. For the full walkthrough of locating the exact cause in Kubernetes, including events and the OOM score, see my companion piece on finding the real cause of an OOMKilled pod. And if you want the mechanism one layer down, the Linux OOM killer explained covers how the kernel picks a victim.