CrashLoopBackOff with no logs: a systematic way to find the real cause

The pod says CrashLoopBackOff, the restart count is climbing, and kubectl logs prints nothing at all. Not an error, not a stack trace, just an empty response and a cursor. The obvious next move, "check the logs", is exactly the advice that does not apply, because the logs you can see belong to a container that has already been replaced.

CrashLoopBackOff is not a diagnosis. It is the kubelet telling you it has given up restarting something quickly, and nothing more. The container might be exiting cleanly with status 0. It might be getting killed by the kernel for using too much memory. It might be perfectly healthy and being murdered by a liveness probe you wrote three months ago. These have nothing in common except the status string.

So the first move is not to fix anything. It is to work out which of four categories you are in, which takes two commands and about ninety seconds.

kubectl logs <pod> --previous
kubectl describe pod <pod> | grep -A5 "Last State"

The status you are looking at

NAME                        READY   STATUS             RESTARTS      AGE
api-7d9f8c6b45-x2klm        0/1     CrashLoopBackOff   7 (2m14s ago) 21m

The number in brackets is the time since the last restart, and it grows as the backoff grows. That growth is itself information: if it is already at several minutes, the pod has been failing for a while and you are late to it.

Why the logs are empty

Three separate reasons, and they need different responses.

You are reading the wrong container. By default kubectl logs reads the current container. In a crash loop, the current container was just created and has produced nothing yet, or is in the backoff wait and does not exist. --previous (or -p) reads the last terminated one. This alone solves a large share of "no logs" cases.

Your application buffers stdout and dies before flushing. This is brutal and extremely common with Python. When stdout is not a TTY, Python block-buffers it, so a process that prints a startup message and then crashes half a second later may never flush that message. Set PYTHONUNBUFFERED=1 in the container env, or run with python -u. Node, Go and the JVM write to stderr unbuffered in most configurations, but any framework logging to a file inside the container instead of stdout has the same effect: Kubernetes only collects stdout and stderr.

The container never actually started. If the image entrypoint cannot be executed at all, you usually get a different status, such as CreateContainerError, RunContainerError or CreateContainerConfigError, rather than CrashLoopBackOff. But a binary that exists and immediately fails on a missing shared library produces exit 127 or 126 with a one-line message that is easy to miss. Check the exit code before assuming your application ran at all.

The four categories

Exit code 0: the entrypoint finished

A Deployment has restartPolicy: Always, so a container that exits cleanly is restarted anyway, and after enough restarts you get CrashLoopBackOff with a completely healthy exit code. What this means is that no process stayed in the foreground.

The usual culprits: a command that starts a daemon and returns, a shell script ending in &, a service manager expecting to daemonise, or an image whose entrypoint runs a migration and exits. The container is doing exactly what you told it to.

# Exits immediately: nginx daemonises and the shell returns
command: ["nginx"]

# Stays in the foreground, which is what a container needs
command: ["nginx", "-g", "daemon off;"]

If the work genuinely is one-shot, it does not belong in a Deployment. Use a Job, or an initContainer if it has to run before the real container starts.

Exit code 137 with OOMKilled: a memory limit problem

137 is 128 plus 9, meaning the process received SIGKILL. When Reason: OOMKilled accompanies it, the container exceeded resources.limits.memory and the kernel's cgroup OOM killer terminated it. Your application did not crash; it was executed.

kubectl describe pod <pod> | grep -A2 "Last State"

    Last State:     Terminated
      Reason:       OOMKilled
      Exit Code:    137

The reflex is to double the limit. Sometimes that is right, particularly for a JVM or a Node process whose heap was sized for a bigger box. Often it just delays the same kill by an hour, because the real problem is a leak or an unbounded cache. Before raising the limit, look at whether memory grows steadily under normal traffic. The mechanics of the kernel side, and why the numbers rarely match what you expect, are the same as on a plain server, which I covered in how the Linux OOM killer decides.

A subtlety worth knowing: exit 137 without an OOMKilled reason means something else sent SIGKILL. A node draining, a stuck SIGTERM handler being force-killed after the grace period, or an external process manager.

Nonzero exit codes: a real crash

127 on an image that works on your laptop is very often an architecture mismatch: an arm64 image built on an Apple Silicon machine and scheduled onto amd64 nodes, or the reverse. Build multi-arch, or set the platform explicitly.

Probe-driven restarts: the app is fine

kubectl get events --field-selector involvedObject.name=<pod> --sort-by=.lastTimestamp

If you see repeated Liveness probe failed entries, the kubelet is killing a container that never crashed. The two usual causes are a probe pointing at a path that does not exist, and an application that takes longer to become ready than initialDelaySeconds allows.

The correct fix for slow startup is not a larger initialDelaySeconds on the liveness probe. It is a startupProbe, which suspends the liveness and readiness probes until the application has started once. That way you can allow three minutes for a cold JVM start while still detecting a hang in two seconds afterwards.

startupProbe:
  httpGet: { path: /healthz, port: 8080 }
  failureThreshold: 30
  periodSeconds: 5        # allows up to 150s to start
livenessProbe:
  httpGet: { path: /healthz, port: 8080 }
  periodSeconds: 10
  failureThreshold: 3

Getting a shell when the container will not stay up

kubectl exec is useless here, because there is nothing to exec into. Use an ephemeral debug container, which attaches to the running pod without restarting it:

kubectl debug -it <pod> --image=busybox:1.36 --target=<container-name>

--target shares the process namespace with the named container, so you can see its processes and, depending on your runtime, its filesystem through /proc. If the crash is too fast even for that, copy the pod and replace the command with something that does nothing:

kubectl debug <pod> --copy-to=api-debug --container=api -- sleep infinity
kubectl exec -it api-debug -- sh

That gives you the exact image, environment variables, mounted secrets and volumes, with a container that stays up. Now you can run the real entrypoint by hand and watch it fail in front of you, which is the fastest path to a config or permissions problem. Delete the copy when you are done.

The backoff timing, and why you cannot tune it away

The kubelet backs off exponentially: roughly 10s, 20s, 40s, 80s, 160s, then capped at 300 seconds. The counter resets after the container has run successfully for ten minutes.

This has been the subject of a long-running request, kubernetes/kubernetes#57291, precisely because five minutes between attempts is painful when you are iterating on a fix. The current state is worth knowing:

Both are node-level settings on the kubelet, not fields you can set in a Deployment, and on a managed cluster you may not be able to change them at all. So for practical purposes: while you are debugging, do not wait out the backoff. Delete the pod so the controller creates a fresh one with a reset counter, or scale the Deployment to zero and back.

kubectl delete pod <pod>   # controller recreates it, backoff resets

Verification

1. Watch the restart count stop moving.

   kubectl get pod <pod> -w

   Expected output: STATUS becomes Running, READY becomes 1/1, and RESTARTS stays fixed for at least ten minutes. Ten is the number that matters, because that is when the kubelet resets its backoff counter and declares the container stable.
2. Confirm the last state is now clean:

   kubectl get pod <pod> -o jsonpath='{.status.containerStatuses[0].lastState}{"\n"}'

   An empty result on a freshly created pod means it has never terminated.
3. Confirm no probe failures are still being recorded with the events command above. A pod can be Running and still be quietly failing readiness, which keeps it out of the Service endpoints while looking healthy in kubectl get pods.

What people get wrong

"Increase the memory limit." Correct roughly half the time, and a slow leak the other half. Raising a limit without knowing whether usage plateaus or grows is guessing with production memory. Watch usage over a full traffic cycle first.

"Remove the liveness probe." This makes the symptom disappear and removes your only automatic recovery from a hung process. If the probe is causing restarts, the probe is either pointing at the wrong thing or too aggressive. Add a startupProbe, widen the failure threshold, or make the health endpoint cheaper. Deleting it is not a fix, it is turning off the smoke alarm.

"Add sleep infinity to the command so it stays up." This appears in a distressing number of answers for the exit-code-0 case. It gives you a pod that is Ready and serving nothing, which is worse than a crash loop because now nobody gets paged.

"Delete and recreate the Deployment." If the pod spec is unchanged, the new pods will do exactly the same thing, and you have destroyed the terminated container whose logs you needed.

When it is still broken

1. Check the node, not the pod. kubectl describe node <node> shows memory and disk pressure conditions. A node under DiskPressure evicts and restarts pods for reasons that have nothing to do with your image.
2. Run the image locally with the same command. docker run --rm -it --entrypoint sh yourimage:tag then invoke the entrypoint by hand. If it fails there too, you have removed Kubernetes from the problem entirely, which is a much smaller haystack. The general approach is the same one I use for any container that will not start, covered in the practical Docker onramp.
3. Check what the container can actually read. A ConfigMap key that does not exist, a Secret mounted at a path the process cannot read as a non-root user, or a readOnlyRootFilesystem where the app wants to write a lock file. These fail instantly and often with unhelpful output.
4. Look for an init container failing silently. kubectl describe pod lists init container statuses separately, above the main containers, and people scroll straight past them.

The habit worth building from this: when a system gives you a status string, ask what mechanism produced it before you ask how to make it go away. CrashLoopBackOff describes the kubelet's retry policy, not your application. Every minute spent identifying which of the four categories you are in saves twenty spent reading the wrong logs.