ArgoCD Stuck on 'Waiting for Completion of Hook': Breaking the Deadlock

It is 22:40. A release that was supposed to take four minutes has been running for forty. The ArgoCD UI shows the Application spinning, and one line of status text that has not changed since the first minute:

waiting for completion of hook batch/Job/db-migrate

So you click Terminate. Then Sync. Same message. You click it again. Same message. Somewhere in a Slack channel someone is asking whether ArgoCD is broken.

ArgoCD is not broken. It is doing precisely what you configured it to do: create a Kubernetes Job and wait for that Job to finish. The Job is not finishing. ArgoCD has no opinion about how long a migration should take, and in most versions it will not impose a deadline on your behalf. The deadlock is yours to break, and clicking Terminate does not break it, because Terminate ends ArgoCD's operation, not the Job.

The exact message: "waiting for completion of hook batch/Job/..."

From the CLI, the stuck operation looks like this (output trimmed):

argocd app get payments-api --show-operation

Sync Status:        OutOfSync from HEAD (3f2c1a9)
Health Status:      Progressing

Operation:          Sync
Sync Revision:      3f2c1a9c4b1e0a6d2f8b91c07d4e5a3f6b2c8d10
Phase:              Running
Start:              2026-02-11 22:41:08 +0300 EAT
Finished:
Duration:           41m12s
Message:            waiting for completion of hook batch/Job/db-migrate

The important fields are Phase: Running with an empty Finished. ArgoCD is not confused or wedged, it is polling. Which means the thing to debug is not ArgoCD.

Why ArgoCD waits forever

A hook in ArgoCD is not a special construct. It is an ordinary manifest carrying the annotation argocd.argoproj.io/hook, with a value of PreSync, Sync, PostSync, SyncFail, Skip, PreDelete or PostDelete. When that manifest is a batch/v1 Job, ArgoCD derives the hook phase from the Job's status conditions: a Job with a Complete condition is Succeeded, a Job with a Failed condition is Failed, and anything else is Running.

There is no fourth state. A Job whose Pod is sitting on a SELECT ... FOR UPDATE that another connection holds, or whose Pod is in CreateContainerConfigError because a Secret key was renamed, or that cannot be scheduled because the node pool is full, has neither condition. So the hook is Running, so the sync is Running, and both stay that way until somebody intervenes.

Three distinct causes produce the same message, and you need to know which one you have:

Cause	How to tell	Fix
The Job genuinely is stuck	kubectl get jobs shows 0/1 with a long DURATION and a live Pod	Fix or kill the Pod, then delete the Job
The Job never started	Pod in Pending, ImagePullBackOff or CreateContainerConfigError	Fix the image tag, Secret or resource request
The Job finished but vanished	No Job exists at all, yet ArgoCD still waits	Raise ttlSecondsAfterFinished, upgrade ArgoCD

That third row surprises people. If a hook Job sets ttlSecondsAfterFinished: 0 or a very small value, the Kubernetes TTL controller can delete the Job the instant it completes, before ArgoCD's next reconcile reads the final status. ArgoCD loses the live object it was tracking and cannot resolve a phase, so the operation hangs on a Job that no longer exists. This was tracked as argo-cd issue 21055 and fixed in later releases, but plenty of clusters are still running versions that have it. If kubectl get jobs returns nothing and ArgoCD still says it is waiting, this is your bug.

Breaking the deadlock

Step 1: Find the hook resource, not the ArgoCD status

The message tells you the group, kind and name: batch/Job/db-migrate. The namespace is the Application's destination namespace.

kubectl get jobs -n payments

NAME         STATUS    COMPLETIONS   DURATION   AGE
db-migrate   Running   0/1           41m        41m

0/1 with a duration in the tens of minutes is your confirmation. If the Job is missing entirely, you have the TTL race, and the fix is in Step 6.

Step 2: Read the Pod, because that is where the truth is

kubectl get pods -n payments --selector=job-name=db-migrate
kubectl logs -n payments job/db-migrate --tail=50
kubectl describe job db-migrate -n payments

Three outcomes, and each points somewhere different:

• Pod Running, logs stop mid-migration. Your migration is blocked on a lock. Something else holds it, usually the previous release's application pods still writing to the table you are altering.
• Pod not Running. Describe the Pod. CreateContainerConfigError means a Secret or ConfigMap key is wrong. ImagePullBackOff means the tag does not exist in the registry yet, which happens constantly when the image build has not finished pushing.
• No Pod at all. Check the Job's events for a scheduling failure or an admission webhook rejection.

Step 3: Terminate the ArgoCD operation

argocd app terminate-op payments-api

The operation Phase in argocd app get moves to Terminating and then to Failed. Do this before deleting the Job, so ArgoCD is not mid-poll on a resource disappearing under it.

Step 4: Delete the Job yourself

This is the step everybody skips, and the reason retrying feels cursed.

kubectl delete job db-migrate -n payments
kubectl get pods -n payments --selector=job-name=db-migrate

The second command should print No resources found in payments namespace. Deleting a Job removes its Pods through the ownerReference, but confirm it, because a Pod stuck in Terminating is still holding your database lock and will hang the next attempt in exactly the same way.

Step 5: Fix the cause, then re-sync

argocd app sync payments-api

Note the ordering. If the migration was blocked on a lock held by the old ReplicaSet, scale the old deployment down first, or change the migration to acquire its lock with a timeout. Re-syncing into unchanged conditions buys you another forty minutes of the same.

Step 6: Give Kubernetes permission to kill the hook

The permanent fix is not an ArgoCD setting. It is a Job spec that cannot hang indefinitely:

apiVersion: batch/v1
kind: Job
metadata:
  name: db-migrate
  annotations:
    argocd.argoproj.io/hook: PreSync
    argocd.argoproj.io/hook-delete-policy: BeforeHookCreation
spec:
  activeDeadlineSeconds: 600
  backoffLimit: 2
  ttlSecondsAfterFinished: 900
  template:
    spec:
      restartPolicy: Never
      containers:
        - name: migrate
          image: registry.example.com/payments-api:1.42.0
          command: ["./manage.py", "migrate"]

The parts that matter:

• activeDeadlineSeconds: 600 is the line that ends this class of outage. When the deadline passes, Kubernetes terminates the Job's Pods and adds a Failed condition with reason DeadlineExceeded. ArgoCD now sees a terminal state, marks the hook failed, and fails the sync in ten minutes instead of never. A failed sync you see in Slack beats a hung sync you discover at breakfast.
• backoffLimit: 2 stops a crash-looping migration from retrying forever. Note that activeDeadlineSeconds takes precedence: once the deadline hits, the Job fails regardless of retries remaining.
• ttlSecondsAfterFinished: 900 cleans up finished Jobs without being aggressive enough to race the controller. Do not set this to 0.
• hook-delete-policy: BeforeHookCreation is ArgoCD's default when no policy is given, and it is the right one here: the previous Job is removed before a new one is created, and a failed Job stays around long enough for you to read its logs. HookSucceeded deletes the Job the moment it works, which is when you least need it gone and most want the audit trail.

Verification

Prove the sync completed:

argocd app get payments-api --show-operation

Sync Status:        Synced to HEAD (3f2c1a9)
Health Status:      Healthy

Operation:          Sync
Phase:              Succeeded
Message:            successfully synced (all tasks run)

Then prove the guard rail works, which is the part nobody tests. Deploy a hook Job that deliberately hangs, with command: ["sleep", "900"] and activeDeadlineSeconds: 60. Sync it, wait a minute, then run:

kubectl get job hang-test -n payments -o jsonpath='{.status.conditions[*].reason}'

DeadlineExceeded

ArgoCD's operation should move to Failed shortly after, with a message naming the hook. That is the whole point: the failure is now loud and bounded. Delete the test Job when you are done.

What people get wrong

"Just hit Terminate and Sync again." Terminate ends ArgoCD's operation record. It does not delete the hook Job and it does not kill the Pod. If the Pod holds a database lock, it still holds it after you terminate. The next sync either re-adopts the same Job or deletes and recreates it into identical conditions. This is the most repeated piece of advice on the topic, and it is why threads about it run to a hundred comments without a resolution.

"Raise the timeout on argocd app sync." The --timeout flag on the CLI controls how long your terminal waits for the operation. The operation continues server-side after your shell gives up. Raising it makes you wait longer for the same non-answer.

"Set ttlSecondsAfterFinished: 0 to keep the namespace tidy." This actively causes the stuck-hook variant where no Job exists to inspect. Tidiness is not worth a failure mode that hides its own evidence.

"Delete the Application and let it recreate." Depending on your prune settings and finalizers, this can take production workloads with it. It is not a debugging step, it is a coin flip with your uptime.

"Add argocd.argoproj.io/hook: Skip to the migration." Skip exists for hooks bundled in a third-party chart that you never want ArgoCD to run. Applying it to your own migration means the migration silently stops running, and you find out when the new code queries a column that does not exist.

When it is still broken

1. Read the application controller logs. kubectl logs -n argocd deploy/argocd-application-controller --tail=200 | grep -i hook shows what phase the controller believes the hook is in, and whether it can read the resource at all.
2. Check controller sharding. If you run more than one application controller replica, confirm the Application has not been reassigned to a different shard mid-operation. This has historically produced exactly this symptom, and temporarily running a single replica is a fast way to rule it in or out.
3. Check RBAC and namespace scope. If the controller's ServiceAccount cannot read Jobs in the destination namespace, it can create the hook and then never observe its completion. Test with kubectl auth can-i get jobs -n payments --as=system:serviceaccount:argocd:argocd-application-controller.
4. Ask whether the migration belongs in the sync at all. A schema migration bound to the deploy is a migration that can hold your release hostage. Running it as an independently monitored step, with its own retries and alerting, removes this failure mode entirely. If you go that way, the migration has to survive being run twice, which is a discipline of its own - see Your Migration Will Run Twice: Write It That Way.

Helm users hit an almost identical wall with pre-upgrade hooks, with the extra wrinkle that the release history itself can be left inconsistent afterwards. That is covered in Helm upgrade stuck in PENDING_UPGRADE. If you are building the pipeline that drives all of this on a small budget, Ship to Your Own VPS covers the cheap end of the same problem.

The lesson I keep relearning on client clusters from Nairobi to Amsterdam: any deployment step that can block indefinitely eventually will block indefinitely, at the worst possible hour. Put a deadline on it, or accept that you have chosen to be woken up.