Your kubectl edit Keeps Getting Reverted in GitOps: The Right Way to Hotfix

It is the middle of an incident. A pod is misconfigured, traffic is failing, and you do the fastest thing you know: kubectl edit deployment, fix the value, save. The errors stop. You breathe. Three minutes later the errors come back, identical, as if your fix never happened. You edit again. It reverts again. Now you are convinced the bug is fighting you, when in fact your own cluster is fighting you, on purpose, exactly as configured.

This is GitOps doing its one job. ArgoCD with selfHeal on, or Flux on its reconcile interval, treats the cluster's desired state as whatever is in Git. Anything you change by hand is, from its point of view, unauthorised drift to be corrected. Your emergency kubectl edit is drift. So it gets reverted, on a timer, and the incident looks like it recurred.

The thesis: in a GitOps cluster, the live object is a cache of Git, not the source of truth. Editing the cache does not change the truth, so the controller overwrites you. The fix is not to fight the controller and it is definitely not to switch GitOps off across the cluster. It is to pause reconciliation for the one resource you need to touch, make the change, and then make Git agree with you.

Why your change comes back exactly on a timer

ArgoCD's automated sync policy has two switches. automated means it applies Git changes to the cluster. selfHeal: true means it also reverts cluster changes that diverge from Git. With self-heal on, ArgoCD periodically diffs the live objects against the rendered Git manifests, and any difference is reconciled back toward Git. The default application reconciliation interval is about three minutes, plus a small jitter, which is why your edit survives for a couple of minutes and then vanishes. It is not random; it is the resync loop coming around.

Flux works the same way with different words. Each Kustomization (or HelmRelease) has an interval, and on every interval Flux re-applies the source of truth from Git. A manual edit lives until the next reconcile and then gets overwritten by the applied manifest.

The emergency runbook: pause, edit, commit, resume

The correct sequence has four steps and the order is not negotiable. Skip the pause and your fix gets reverted; skip the commit and the next resume reverts it.

Step 1: pause reconciliation for the one affected resource

ArgoCD, scoped to a single Application:

argocd app set my-api --sync-policy none
# Sync policy is now 'none' for my-api only. Self-heal no longer runs for it.

Flux, scoped to a single Kustomization (or HelmRelease):

flux suspend kustomization my-api
# my-api reconciliation suspended. Other kustomizations keep reconciling.

Both of these touch exactly one object. Every other application in the cluster keeps its GitOps guarantees. That is the entire point of scoping the pause.

Step 2: make your emergency change

kubectl edit deployment my-api
# or a targeted patch, which is easier to reproduce in the Git commit later:
kubectl patch deployment my-api --type merge \
  -p '{"spec":{"template":{"spec":{"containers":[{"name":"api","resources":{"limits":{"memory":"1Gi"}}}]}}}}'

This change now sticks, because the controller that would revert it is paused for this resource. Verify the incident is actually resolved before moving on; a change that does not fix the problem is not worth committing.

Step 3: commit the same change to Git

This is the step people skip under pressure, and it is the one that makes the fix permanent. Edit the manifest in your GitOps repository to match exactly what you did by hand, and commit it.

   resources:
     limits:
-      memory: 512Mi
+      memory: 1Gi

Now Git, the source of truth, agrees with the cluster. When you resume, there is nothing to revert because there is no drift.

Step 4: resume reconciliation

# ArgoCD: re-enable automated sync (add --self-heal if that was your policy)
argocd app set my-api --sync-policy automated --self-heal

# Flux
flux resume kustomization my-api

The controller reconciles, sees the live state already matches Git, and does nothing. Your hotfix is now the desired state, reviewed in a commit, and it will survive every future reconcile.

Revert versus oscillation: they are not the same bug

There is a nastier variant. Instead of your change surviving for a couple of minutes and then reverting once, the resource flips between Synced and OutOfSync continuously, every reconcile loop, even when you have not touched anything. That is oscillation, and it means two different writers own the same field.

The usual culprit: something in the cluster sets a field that is also declared in Git. An HPA writes spec.replicas while your Deployment manifest in Git also pins replicas. A mutating admission webhook injects a sidecar or a label. Another controller stamps an annotation. Each reconcile, Git reasserts its value, the other writer reasserts theirs, and ArgoCD forever sees a diff.

# ArgoCD: repeated Sync operations are the tell
argocd app history my-api        # many entries close together in time
kubectl describe application my-api -n argocd | grep -A5 Conditions

# Flux: repeated reconcile log lines for the same object
flux logs --kind Kustomization --name my-api --follow

The fix is to stop fighting over the field. If an HPA owns replicas, remove replicas from the Git manifest entirely so the two are not both writing it. If an admission controller injects a field, tell ArgoCD to ignore it with a diff-customisation (ignoreDifferences) rather than trying to sync it away every loop. Re-syncing harder never wins an oscillation; you have to remove the contested ownership. This is a cousin of the drift I covered in an ArgoCD StatefulSet that stays OutOfSync under server-side apply, where the field-ownership model is the whole story.

Verification

Prove the fix holds across at least one full reconcile interval, because the whole failure mode is time-delayed.

# ArgoCD: force a sync and confirm it lands Synced and stays there
argocd app get my-api
# Health Status:  Healthy
# Sync Status:    Synced to 

# Wait past one interval (3+ minutes) and check the value survived
kubectl get deployment my-api -o jsonpath='{.spec.template.spec.containers[0].resources.limits.memory}'
# 1Gi

For Flux, flux get kustomization my-api should show Ready: True with the latest applied revision matching your commit, and the value should still be present after the next interval. If it reverts, your Git commit did not match the live change exactly; diff them field by field.

What people get wrong

Disabling sync for the whole cluster. Under pressure, someone runs a broad disable or scales down the ArgoCD/Flux controller to stop the reverts. Now nothing enforces desired state anywhere, drift accumulates silently across every app, and you find out weeks later when an unrelated deploy behaves strangely. Scope the pause to one Application. That capability exists precisely so you never have to reach for the cluster-wide switch.

Editing without pausing and blaming the tool. The revert is not a bug, it is the feature you configured. If you keep editing and it keeps reverting, you have simply not paused the resource. Pause first.

Pausing but never committing. A suspended resource with an uncommitted manual change is a landmine. The next person to resume it, or the next automated resume, reverts your fix and reopens the incident, often with nobody realising why. Always land the commit.

Re-syncing to fix oscillation. If the object flips every loop, more syncing makes it flip faster. Find the second writer and remove the contested field from Git or from the diff.

When it is still broken

• It reverts even after you committed. The commit and the live change differ. A common miss: you patched a field the manifest renders differently (a default that Git omits), so ArgoCD still sees a diff. Compare argocd app diff my-api output directly.
• The pause did not take. Confirm you scoped it to the right object. In ArgoCD, a resource may belong to a different Application than you assumed, especially with app-of-apps; check argocd app resources.
• The pod restarts and loses the change anyway. If your edit was to a pod directly rather than its Deployment, the ReplicaSet recreates it from the template. Edit the controller (Deployment/StatefulSet), not the pod.
• A hook or job keeps the app OutOfSync. A stuck sync hook can look like a revert loop. That is a different failure with its own path; see ArgoCD stuck on Waiting for completion of hook.

The reference is short and worth reading before your next incident: ArgoCD's automated sync policy documentation. The one idea to carry into the next 2am page is that in GitOps the cluster is downstream of Git, so an emergency change is only real once Git knows about it.