Helm Upgrade Stuck in PENDING_UPGRADE After a Hook Timeout: The Cleanup Path

The migration usually takes ninety seconds. Tonight, against a table that has grown past forty million rows, it takes eleven minutes. Your helm upgrade was started with the default five minute timeout, so at minute five your terminal prints an error and your pipeline goes red.

You retry. Helm refuses:

Error: UPGRADE FAILED: another operation (install/upgrade/rollback) is in progress

You run helm history to see what it thinks is happening, and find two revisions both marked deployed. That is not supposed to be possible.

The thesis: --timeout is a client-side wait, not a cancel. When it expires, the helm binary on your laptop gives up and exits. The pre-upgrade hook Job it created is still running in the cluster, still holding whatever it was holding, and the release record in Kubernetes is frozen halfway through a state transition that nobody is going to finish.

The exact errors

The first failure, when the hook outlives the client's patience:

Error: UPGRADE FAILED: pre-upgrade hooks failed: timed out waiting for the condition

The second failure, on every retry after it:

Error: UPGRADE FAILED: another operation (install/upgrade/rollback) is in progress

And with --atomic set, which is better but still leaves you with questions:

Error: UPGRADE FAILED: release payments-api failed, and has been rolled back due to
atomic being set: pre-upgrade hooks failed: timed out waiting for the condition

Helm 3 prints statuses in lowercase, so helm history shows pending-upgrade. Helm 2 showed PENDING_UPGRADE, which is the spelling most of the older threads use, including the long-running upstream issue about multiple deployed revisions. Same failure, different casing.

Why the release history goes inconsistent

Helm 3 stores each revision of a release as a Kubernetes Secret in the release namespace, named sh.helm.release.v1.<release>.v<revision>, with labels including owner=helm, name, version and status. That Secret is the entire state machine. There is no server-side controller reconciling it, which is the fact that explains everything else.

A healthy upgrade goes like this:

1. Revision 41 exists with status deployed.
2. Helm creates revision 42 with status pending-upgrade. This is the lock: any other operation now refuses to start.
3. Helm applies the pre-upgrade hooks and waits for them, then applies the manifests.
4. Helm flips 41 to superseded and 42 to deployed. Exactly one revision is deployed.

Step 4 is performed by the client process. If that process exits at step 3 because the timeout expired, or because CI cancelled the job, or because someone pressed Ctrl+C, nobody performs step 4. Revision 42 stays pending-upgrade forever and revision 41 stays deployed. The lock is permanent because the thing that was meant to release it is gone.

The two-deployed-revisions case comes from what happens next. A subsequent operation that partially proceeds, or a rollback interleaved with a retry, can produce a second revision marked deployed without ever demoting the first. The Helm data model has no representation for that, so later upgrades and rollbacks pick whichever one they find and the results stop being predictable.

The cleanup path

Step 1: See what state the release is actually in

helm history payments-api -n payments

REVISION  UPDATED                   STATUS           CHART                APP VERSION  DESCRIPTION
40        Mon Feb  9 20:11:02 2026  superseded       payments-api-1.9.2   1.9.2        Upgrade complete
41        Tue Feb 10 21:03:44 2026  deployed         payments-api-1.10.0  1.10.0       Upgrade complete
42        Wed Feb 11 22:41:08 2026  pending-upgrade  payments-api-1.11.0  1.11.0       Preparing upgrade

Revision 42 in pending-upgrade is your lock. Revision 41 is what is actually running. If you see two rows saying deployed, you are in the corrupted case and the cleanup below still applies, you just have more to undo.

Step 2: Look at the hook Job independently

Helm will not tell you this, because the client that was watching it exited.

kubectl get jobs -n payments
kubectl logs -n payments job/payments-api-db-migrate --tail=50

NAME                      STATUS    COMPLETIONS   DURATION   AGE
payments-api-db-migrate   Running   0/1           23m        23m

This is the fork in the road. If the migration is still making progress, the least destructive move is to let it finish, then clean up the release state. If it is stuck, kill it. Killing a migration mid-transaction is safe on a database with transactional DDL and decidedly not safe on MySQL or MariaDB, so know which one you are on before you type anything.

Step 3: Remove the hook Job

kubectl delete job payments-api-db-migrate -n payments
kubectl get pods -n payments --selector=job-name=payments-api-db-migrate

The second command should print No resources found in payments namespace. A Pod still in Terminating holds its database connection and its locks, and will hang the next attempt exactly as the first one hung.

Step 4: Roll back, which is the supported way to clear the lock

helm rollback payments-api 41 -n payments

Expect Rollback was a success! Happy Helming! and a new revision 43 with status deployed, describing itself as a rollback to 41. This is the right first attempt: it goes through Helm's own state machine rather than around it.

Step 5: If rollback also refuses, edit the state directly

Sometimes rollback hits the same lock. Then you go to the Secrets:

kubectl get secrets -n payments -l "owner=helm,name=payments-api"

NAME                                    TYPE                 DATA   AGE
sh.helm.release.v1.payments-api.v40     helm.sh/release.v1   1      3d
sh.helm.release.v1.payments-api.v41     helm.sh/release.v1   1      2d
sh.helm.release.v1.payments-api.v42     helm.sh/release.v1   1      31m

Delete only the revision that is stuck in pending-upgrade. That revision never reached a deployed state, so removing it discards a record of something that did not happen:

kubectl delete secret sh.helm.release.v1.payments-api.v42 -n payments

Never delete the Secret for the revision that is deployed. That is the record of what is currently running, and without it Helm loses track of your live release entirely.

A lighter-touch alternative is patching the status label rather than deleting:

kubectl patch secret sh.helm.release.v1.payments-api.v42 -n payments \
  --type=merge -p '{"metadata":{"labels":{"status":"failed"}}}'

Be aware that Helm also keeps the release status inside the encoded payload in the Secret's data, not only in the label, so on some versions the label patch alone is not sufficient. Always confirm with helm history afterwards, and fall back to deleting the pending revision if the patch did not take.

Step 6: Fix the release so the next upgrade is single-revision clean

helm history payments-api -n payments
helm upgrade payments-api ./chart -n payments --atomic --wait --timeout 20m

Exactly one revision should read deployed before you run the upgrade. If two do, roll back to the older of them first so Helm re-establishes a single current revision.

Making it not happen again

Three changes, in order of how much they buy you.

1. Bound the hook in Kubernetes, not just in the helm client. Helm has no per-hook timeout. helm.sh/hook-weight is often described as one and it is not: it only controls the order in which hooks run, sorted ascending. The only real deadline available is on the Job itself:

apiVersion: batch/v1
kind: Job
metadata:
  name: {{ include "payments-api.fullname" . }}-db-migrate
  annotations:
    helm.sh/hook: pre-upgrade
    helm.sh/hook-weight: "-5"
    helm.sh/hook-delete-policy: before-hook-creation
spec:
  activeDeadlineSeconds: 600
  backoffLimit: 1
  template:
    spec:
      restartPolicy: Never
      containers:
        - name: migrate
          image: "{{ .Values.image.repository }}:{{ .Values.image.tag }}"
          command: ["./manage.py", "migrate"]

Set activeDeadlineSeconds comfortably below your --timeout. Then a hanging hook fails in Kubernetes first, helm observes a failed hook, and you get a clean failed upgrade instead of a hung client and a frozen release. Ten minutes of hook against twenty minutes of helm timeout is a sane starting ratio.

2. Use --atomic in CI, always. It implies --wait and rolls the release back automatically when the upgrade fails, which is precisely the step nobody performs manually at midnight. It does not solve everything, because the rollback itself can be interrupted, but it converts the common case from a corrupted release into a failed pipeline.

3. Keep before-hook-creation as the delete policy. It is the default, and it is what you want: the previous hook resource is removed before a new one is created, and a failed Job stays in the namespace so you can read its logs. hook-succeeded deletes the Job the moment it works, which is exactly when you want the audit trail most.

Verification

helm history payments-api -n payments | grep -c deployed

1

One. Not zero, not two. Then confirm the release status and that no orphaned hook Jobs remain:

helm status payments-api -n payments | head -5
kubectl get jobs -n payments

To prove the deadline works, deploy a hook with command: ["sleep", "900"] and activeDeadlineSeconds: 60 to a staging release, run the upgrade, and check that the Job fails on its own:

kubectl get job payments-api-db-migrate -n payments \
  -o jsonpath='{.status.conditions[*].reason}'

DeadlineExceeded

Helm should then report a failed pre-upgrade hook and, with --atomic, roll back cleanly. That is the behaviour you want on your worst night.

What people get wrong

"Just increase --timeout." This helps only when the hook genuinely finishes and simply needed longer. It does nothing for a hook that is stuck, and it makes the failure take longer to appear. A bigger timeout with no activeDeadlineSeconds is a longer fuse on the same bomb.

"helm uninstall and reinstall to get a clean slate." Uninstalling deletes the release's resources. On a stateful workload, or anything with a PersistentVolumeClaim whose reclaim policy you have not checked, this is how a stuck deployment becomes a data loss incident.

"helm upgrade --force will push through." --force makes Helm delete and recreate resources rather than patch them. It causes real downtime, and it does not clear a pending-upgrade lock.

"hook-weight sets a per-hook timeout." It sorts hooks. That is all it does. Helm has no per-hook timeout; the deadline has to come from the Job spec.

"Delete all the helm release secrets and start over." Delete the deployed one and Helm no longer knows what is running, so the next upgrade computes a diff against nothing. Only the pending revision is safe to remove.

When it is still broken

1. A GitOps controller is fighting you. If Flux or ArgoCD drives this release, the controller retries on its own schedule and can recreate the stuck state seconds after you clear it. Suspend reconciliation for the release before doing any manual cleanup, and resume afterwards.
2. The release Secret is too large. Very large charts can push a release Secret past what etcd will accept, and the failures look unrelated. Cap history with --history-max on upgrades so old revisions are pruned.
3. Check the storage backend. Helm 3 defaults to Secrets, but HELM_DRIVER can be set to configmap or sql. If the Secrets above do not exist, look at what the driver actually is before assuming the release is gone.
4. Stop putting long migrations in hooks. This is the honest fix. A migration that can run for eleven minutes does not belong inside a deploy step whose failure mode is a corrupted release record. Run it as its own pipeline stage, with its own monitoring and its own retry, and let the deploy be a deploy. That requires the migration to survive being run twice, which is covered in Your Migration Will Run Twice.

ArgoCD users hit the same wall from a different direction, with sync hooks that never report completion, which I have written up in ArgoCD stuck on waiting for completion of hook. Different tool, identical lesson: if a step in your deploy can block indefinitely, give it a deadline before it gives you a night.