Kubernetes HPA Not Scaling: The Conditions You Are Not Reading

The load test is running, the pods are pinned at 95% CPU, and the replica count has not moved off 2 for eleven minutes. Someone restarts metrics-server. Someone else deletes and recreates the HPA. A third person suggests bumping maxReplicas, which is already 20.

Meanwhile kubectl get hpa has been showing <unknown>/70% the entire time, which is the HPA saying, in the only way it can, that it has no idea what your CPU usage is and is therefore not going to guess.

The Horizontal Pod Autoscaler is one of the more communicative controllers in Kubernetes. It maintains three conditions with human-readable reasons and it emits events. Almost every HPA incident I have been called into was already fully explained in the output of one command that nobody had run.

The exact errors

Here is what a broken HPA actually looks like, and the part everyone scrolls past:

Name:                                                  api
Namespace:                                             prod
Reference:                                             Deployment/api
Metrics:                                               ( current / target )
  resource cpu on pods  (as a percentage of request):  <unknown> / 70%
Min replicas:                                          2
Max replicas:                                          20
Deployment pods:                                       2 current / 2 desired
Conditions:
  Type           Status  Reason                   Message
  ----           ------  ------                   -------
  AbleToScale    True    SucceededGetScale        the HPA controller was able to get the target's current scale
  ScalingActive  False   FailedGetResourceMetric  the HPA was unable to compute the replica count: failed to get cpu utilization: unable to get metrics for resource cpu: no metrics returned from resource metrics API
Events:
  Type     Reason                   Age                 From                       Message
  ----     ------                   ----                ----                       -------
  Warning  FailedGetResourceMetric  3m2s (x12 over 6m)  horizontal-pod-autoscaler  failed to get cpu utilization: unable to get metrics for resource cpu: no metrics returned from resource metrics API

And the second most common variant, which is a completely different problem wearing the same condition:

  ScalingActive  False   FailedGetResourceMetric  the HPA was unable to compute the replica count: failed to get cpu utilization: missing request for cpu

no metrics returned from resource metrics API means the metrics pipeline is broken. missing request for cpu means the metrics pipeline is fine and your pod spec is incomplete. Restarting metrics-server fixes exactly one of those, and it is not the one people usually have.

What the three conditions mean

Condition	Status you want	When it goes wrong
AbleToScale	True	False means the controller cannot read or write the target's scale. Usually a wrong scaleTargetRef name or kind, a deleted Deployment, or missing RBAC. Nothing else will work until this is True.
ScalingActive	True, reason ValidMetricFound	False is the metrics problem. Read the message: it distinguishes an unreachable metrics API from a missing resource request from an invalid label selector. It is also False when the target is scaled to zero, since there is nothing to measure.
ScalingLimited	False, reason DesiredWithinRange	True with TooManyReplicas means it wants more than maxReplicas and is capped. True with TooFewReplicas means it wants fewer than minReplicas. Neither is a bug, but both explain "it stopped scaling".

Note the trap in that last row: ScalingLimited: True / TooManyReplicas looks like a failure and is actually the HPA working perfectly, telling you your ceiling is too low. That is a capacity conversation, not a debugging session.

Why a missing resource request stops everything

The HPA does not scale on raw CPU. For a Resource metric with type: Utilization, it computes usage as a percentage of what the container requested, then applies the standard formula:

desiredReplicas = ceil[ currentReplicas * ( currentMetricValue / desiredMetricValue ) ]

The request is the denominator. Without it there is no percentage, so there is no ratio, so there is no decision, so the HPA does nothing. It is not that scaling is inaccurate; it is that the arithmetic is undefined.

The version of this that costs people an afternoon: a service mesh sidecar. If your pod has an injected proxy container with no CPU request, the resource metric for the pod cannot be computed even though your own container is configured correctly. Either give the sidecar a request, or switch the metric to ContainerResource and name the container you care about:

metrics:
  - type: ContainerResource
    containerResource:
      name: cpu
      container: api
      target:
        type: Utilization
        averageUtilization: 70

The diagnostic sequence

Step 1: Read the conditions

kubectl describe hpa api -n prod

Do this first, always. If AbleToScale is False, fix that and stop; nothing downstream is meaningful. If ScalingActive is False, the message routes you to step 2 or step 3.

Step 2: Can anything read pod metrics?

kubectl top pods -n prod

If this errors, the HPA was never going to work and the problem is entirely in the metrics pipeline. Expect a table of pods with CPU and memory columns. Then confirm the API is registered and available:

kubectl get apiservices v1beta1.metrics.k8s.io

You want AVAILABLE True. If it says False, the message column names the cause, and on self-managed clusters it is very often TLS: metrics-server refuses the kubelet's self-signed serving certificate. The usual fix is on the metrics-server deployment:

args:
  - --kubelet-insecure-tls
  - --kubelet-preferred-address-types=InternalIP,Hostname,ExternalIP

--kubelet-insecure-tls disables verification of the kubelet certificate and is a compromise, not a solution. It is fine for a lab or a bootstrapped cluster; on anything production-shaped, provision proper kubelet serving certificates instead of leaving verification off permanently.

Step 3: Do the containers have requests?

kubectl get deploy api -n prod \
  -o jsonpath='{range .spec.template.spec.containers[*]}{.name}{"\t"}{.resources.requests.cpu}{"\n"}{end}'

Every container listed must show a value. An empty second column next to istio-proxy or linkerd-proxy is your answer.

Step 4: Check the boundaries and the behaviour

apiVersion: autoscaling/v2
kind: HorizontalPodAutoscaler
metadata:
  name: api
  namespace: prod
spec:
  scaleTargetRef:
    apiVersion: apps/v1
    kind: Deployment
    name: api
  minReplicas: 2
  maxReplicas: 20
  metrics:
    - type: Resource
      resource:
        name: cpu
        target:
          type: Utilization
          averageUtilization: 70
  behavior:
    scaleDown:
      stabilizationWindowSeconds: 300

minReplicas equal to maxReplicas is a surprisingly common copy-paste result and produces an HPA that reports healthy and never moves. And if the complaint is that it will not scale down, that is usually the stabilisation window doing its job: the controller deliberately waits before shrinking so a brief dip does not cause a flap. Five minutes of no downscaling is expected behaviour, not a fault.

Step 5: For custom metrics, check the other API

kubectl get apiservices v1beta1.custom.metrics.k8s.io
kubectl get --raw "/apis/custom.metrics.k8s.io/v1beta1" | head -c 400

A Prometheus adapter that is running but has not matched your metric name produces exactly the same silence as no adapter at all. Query the raw API and confirm the metric you referenced in the HPA actually appears in the list.

Verification

kubectl get hpa api -n prod -w

The TARGETS column must show a real number instead of <unknown>, for example 82%/70%. Within a sync interval or two the REPLICAS column should start climbing under load. Then confirm the conditions flipped:

kubectl describe hpa api -n prod | sed -n '/Conditions/,/Events/p'

You are looking for ScalingActive True ValidMetricFound and an event with reason SuccessfulRescale giving the old and new replica counts. If TARGETS is populated but replicas are static, check ScalingLimited: you may simply be at your ceiling.

What people get wrong

Restarting metrics-server as the first move. It is the reflex, and it fixes the minority case where metrics-server is genuinely wedged. When the message says missing request for cpu, restarting it changes nothing, and you have just discarded the pod that had the useful logs.

Deleting and recreating the HPA. The HPA object holds no state worth resetting. Recreating it clears the conditions and events, which is to say it deletes your evidence, and then rebuilds the identical broken state thirty seconds later.

Setting CPU limits and expecting the HPA to use them. Utilization is measured against requests. Limits affect throttling and eviction, not autoscaling arithmetic. A container with a limit and no request is invisible to the HPA.

Trusting kubectl top nodes. Node metrics can look healthy while pod metrics fail, and the HPA only ever consumes pod metrics. Test with kubectl top pods in the target namespace.

Fighting your GitOps controller. If your Deployment manifest specifies replicas: 2 and Argo CD or Flux keeps the cluster in sync with git, every scale-up is reverted within seconds. The HPA scales, the controller scales back, and describe hpa shows successful rescales that never stick. Remove replicas from the committed manifest, or configure the controller to ignore that field. This is the same class of mistake as two writers touching the same Terraform state, which I went through in when force-unlock is safe.

When it is still broken

1. The HPA scales but pods stay Pending. That is a cluster capacity problem, not an HPA problem. Check kubectl describe pod for Insufficient cpu and whether a cluster autoscaler exists and is allowed to add nodes. On a fixed-size cluster, maxReplicas above what the nodes can hold is a promise you cannot keep.
2. Metrics appear and then vanish. Newly started pods are excluded from the calculation for a short initialisation period, because their CPU readings are dominated by startup. On a workload that restarts frequently, the HPA can spend most of its time with too few usable samples. Fix the restarts first.
3. It scales up and immediately back down. Look at your behavior policies and target value. A target of 70% with an application whose CPU swings hard between requests will oscillate; raise the stabilisation window or scale on a smoother signal such as queue depth.
4. It scales and nothing gets faster. Then replicas were never the bottleneck. More application pods contending for the same database will make things worse, not better, and the investigation belongs at the data layer instead, which is where I would start with a query that is disk-bound rather than CPU-bound.

The habit worth building is simple: for any Kubernetes controller that is not doing what you expect, read its status conditions before you read anything else. The HPA writes down its reasoning every fifteen seconds. It is rude not to look.

Reference: the Kubernetes Horizontal Pod Autoscaling documentation and the metrics-server project.