'0/3 nodes are available: Insufficient memory' - requests, not usage

A pod goes out in a deploy and never leaves Pending. You open your dashboards, and the nodes are barely warm: memory graphs sitting at twenty, thirty percent. Then kubectl describe pod tells you, with a straight face, that there is not enough memory anywhere in the cluster. The monitoring and the scheduler cannot both be right, and the instinct is to trust the graph and go hunting for a scheduler bug.

The graph is not lying and neither is the scheduler. They are measuring two different things, and the entire fix depends on understanding which one the scheduler cares about. Worse, this exact message is also what you get for problems that have nothing to do with memory at all: an untolerated taint, a volume pinned to the wrong zone. Shrinking your resource requests until the pod schedules is the popular move, and it is often fixing the wrong thing.

0/3 nodes are available: 3 Insufficient memory. preemption: 0/3 nodes are available: 3 Preemption is not helpful for scheduling.

The exact message, and why it has two halves

Pull it from the pod's events:

kubectl describe pod api-7c9d-fk2lp | grep -A2 Events

Warning  FailedScheduling  default-scheduler  0/3 nodes are available:
  3 Insufficient memory.
  preemption: 0/3 nodes are available: 3 Preemption is not helpful for scheduling.

Read it as two sentences. The first, before preemption:, is the scheduling attempt: three nodes exist, all three were rejected, and the reason was insufficient memory. The second half is the preemption attempt: having failed to place the pod, the scheduler asked whether evicting lower-priority pods would help, and concluded it would not. When you see mixed reasons, they are counts, and they add up per node:

0/3 nodes are available: 1 Insufficient cpu, 2 Insufficient memory,
  1 node(s) had untolerated taint {node-role.kubernetes.io/control-plane: }.

That last line is the whole reason this problem is hard. The message is a summary of all failing checks across all nodes. If you read only "Insufficient memory" and start trimming requests, you will never touch the tainted node that was never going to accept your pod regardless of size.

Why "the node looks idle" is a false lead

Kubernetes schedules on requests, which are a reservation. When a pod declares resources.requests.memory: 2Gi, the scheduler subtracts 2Gi from that node's allocatable memory the instant the pod is placed, and keeps it subtracted for the pod's whole life, whether the process touches 2Gi or 2Mi. Live usage, the thing kubectl top and Grafana show, is irrelevant to placement. A node can therefore be 95 percent committed by requests while using 15 percent of its RAM.

These two numbers come from two different commands, and confusing them is the core mistake:

So the honest first question is not "is the node busy" but "is the node's request budget spent". kubectl describe node answers it directly.

The fix, in numbered steps

Step 1: Read every reason, not the first

kubectl describe pod <pod> | sed -n '/Events/,$p'

Write down every distinct reason in the message. If you see untolerated taint, volume node affinity conflict, or node(s) didn't match Pod's node affinity/selector anywhere in there, treat those as separate problems from the memory count. They are not solved by resizing anything.

Step 2: Check the request budget on the nodes

kubectl describe node node-2 | sed -n '/Allocated resources/,/Events/p'

Allocated resources:
  Resource           Requests      Limits
  cpu                1500m (75%)   2 (100%)
  memory             7600Mi (95%)  8Gi (100%)

That 95 percent under Requests is what the scheduler saw. If every node reads like this, the cluster genuinely has no room for another 2Gi reservation, no matter how idle kubectl top node looks.

Step 3: If it is a taint, add a toleration (do not tune requests)

Confirm the taint on the node the pod could otherwise use:

kubectl get node node-3 -o jsonpath='{.spec.taints}'
# [{"effect":"NoSchedule","key":"dedicated","value":"gpu"}]

Then let the pod tolerate exactly that taint, in the pod spec (or the Deployment's template):

tolerations:
  - key: "dedicated"
    operator: "Equal"
    value: "gpu"
    effect: "NoSchedule"

Control-plane, spot-instance and dedicated-hardware nodes are the usual sources. A toleration says the pod is allowed onto tainted nodes; it does not force it there.

Step 4: If it is genuine capacity, right-size or add a node

If the request was honestly too high, lower it to what the workload needs, informed by real usage over time, not guessed down until it fits:

resources:
  requests:
    memory: "512Mi"   # measured p95, not a number chosen to make it schedule
  limits:
    memory: "768Mi"

If the requests are already honest and every node is committed, the cluster needs more capacity: add a node, or if the autoscaler is running, confirm it is scaling for this pod (Step 4 of the next section).

Verification

Watch the pod move out of Pending:

kubectl get pod <pod> -w
# NAME          READY  STATUS    ...
# api-7c9d...   0/1    Pending
# api-7c9d...   0/1    ContainerCreating
# api-7c9d...   1/1    Running

Then confirm why it fit, so you know you fixed the real thing:

kubectl get pod <pod> -o jsonpath='{.spec.nodeName}{"\n"}'
kubectl describe node $(kubectl get pod <pod> -o jsonpath='{.spec.nodeName}') | grep -A6 'Allocated resources'

If it landed on the previously tainted node, the toleration was the fix. If it landed on a node whose request budget had room, capacity was.

What people get wrong

The most common bad move is shrinking requests.memory until the pod schedules. Sometimes the request really was inflated and this is correct. Often it is a workaround that lets the pod onto a node with no headroom, and it gets OOMKilled the first time it does real work. You have converted a visible Pending into an intermittent crash, which is strictly worse.

The second is assuming the cluster autoscaler will rescue any Pending pod. It only adds nodes for capacity shortfalls. A taint mismatch, a node-selector that matches nothing, or a volume node affinity conflict all leave the pod Pending, and a fresh node reproduces the same condition. The autoscaler will churn and give up.

The third is trusting the usage graph over the describe output. Live usage tells you whether a node is working hard. It tells you nothing about whether the scheduler will accept another reservation. For scheduling questions, kubectl describe node is the source of truth and kubectl top is a distraction.

When it is still broken

• The reason mentions volume node affinity. A PersistentVolume provisioned in zone-a can only be used by a pod scheduled to zone-a. If all your headroom is in zone-b, the pod is stuck. Check kubectl describe pv for the node affinity and either move the workload or use a topology-aware storage class.
• No node matches a nodeSelector or affinity. A typo in a label key produces "didn't match Pod's node affinity/selector". Compare kubectl get nodes --show-labels against the selector in your spec.
• Requests are unset and defaults are huge. A LimitRange in the namespace can inject a large default request you never wrote. kubectl describe limitrange -n <ns> shows it.
• The autoscaler is capped. If it should be adding nodes but is not, check its logs and the max size of the node group; a hit ceiling looks exactly like a stuck Pending pod. This is a close cousin of diagnosing a pod that never starts cleanly: read the machine's own explanation before you theorise.

The discipline that saves you here is boring: read the entire describe pod reason, then look at requests with describe node, before you change a single number. The scheduler always tells you exactly why it said no. The trick is believing it over the dashboard.