ArgoCD StatefulSet Stuck OutOfSync With ServerSideApply: Fix the Diff, Not the Manifest

The Loki chart deployed fine. Every pod is running, every PVC is bound, Grafana is querying logs happily. And ArgoCD has been sitting on OutOfSync for eleven days, with the self-heal controller re-syncing it every three minutes, forever, achieving nothing.

You open the diff. It is a StatefulSet, and the only things ArgoCD is unhappy about are fields you never wrote: volumeMode: Filesystem, creationTimestamp: null, a status block on a volume claim template, and a readiness probe that has grown a failureThreshold out of nowhere. Nothing in Git changed. Nothing in the cluster changed.

This is not drift. This is ArgoCD guessing what the API server would have done and guessing wrong. The fix is to stop making it guess, and the fix is emphatically not to turn off ServerSideApply, which is what most of the advice on the internet tells you to do.

What the phantom StatefulSet diff actually looks like

Run the diff and read it carefully before you touch anything:

argocd app diff loki --refresh

On an affected StatefulSet you get something in this shape. Lines marked + exist only on the live object:

  spec:
    volumeClaimTemplates:
    - metadata:
+       creationTimestamp: null
        name: storage
+     apiVersion: v1
+     kind: PersistentVolumeClaim
      spec:
        accessModes:
        - ReadWriteOnce
        resources:
          requests:
            storage: 10Gi
+       volumeMode: Filesystem
+     status:
+       phase: Pending
    template:
      spec:
        containers:
        - name: loki
          readinessProbe:
+           failureThreshold: 3
+           periodSeconds: 10
+           successThreshold: 1
            httpGet:
              path: /ready
              port: http-metrics
+             scheme: HTTP

Every single added line is something the Kubernetes API server writes itself. volumeMode: Filesystem is the default. status.phase: Pending on an embedded PVC template is a defaulted status struct. The probe fields are the documented defaults for failureThreshold, periodSeconds, successThreshold and scheme.

That is your confirmation test. If every field in the diff is a value you did not write and could not have written, it is a false positive. If a value you actually set in Git appears on the wrong side, stop reading this article, because you have real drift.

Why ServerSideApply makes the diff lie

ArgoCD has two ways of answering the question "is the live object the same as Git?", and they are wildly different pieces of machinery.

The classic path is a client-side, three-way merge. ArgoCD takes your desired manifest, the live object, and the kubectl.kubernetes.io/last-applied-configuration annotation, and computes locally what the object would look like after an apply. Then it compares that prediction to the live object. Because the prediction is computed inside the ArgoCD controller, it only knows the defaulting rules that ArgoCD's normalizers were taught. Kubernetes has thousands of defaulting rules across every built-in type, and ArgoCD does not reimplement them all.

Server-side apply removes the last-applied-configuration crutch. Ownership now lives in metadata.managedFields and the API server decides the merge. ArgoCD keeps predicting the merge locally, but it now has less information to predict from, and the prediction diverges from reality precisely on defaulted fields.

volumeClaimTemplates is close to the worst case in the whole API surface, because it is a list of complete embedded PersistentVolumeClaim objects. Each one gets apiVersion, kind, a null creationTimestamp, a defaulted volumeMode and an entire status struct written into it. Your Git manifest has none of those, ArgoCD's predicted merge has none of those, the live object has all of them, and the diff engine dutifully reports six differences that no human created.

Why it never settles: the self-heal loop

A cosmetic diff on a Deployment is annoying. On a StatefulSet it is corrosive, because volumeClaimTemplates is immutable after creation. The loop goes like this:

1. Controller compares, sees six phantom differences, marks the app OutOfSync.
2. Auto-sync with selfHeal: true fires and applies the StatefulSet.
3. The API server accepts the apply but changes nothing, because the fields in question are either immutable or already equal.
4. The next comparison produces the same six differences.

Nothing errors. Nothing alerts. You just get a permanently amber application and a sync history full of identical no-op syncs. In practice teams end up disabling auto-sync on exactly the applications that most need it, which is how a GitOps setup quietly stops being GitOps. If your ArgoCD apps get stuck for other structural reasons, the hook deadlock case is worth reading next.

The fix, step by step

Step 1: Prove it is a false positive

Ask the API server directly what an apply would change, bypassing ArgoCD completely:

kubectl diff -f rendered-statefulset.yaml

If kubectl diff comes back empty while argocd app diff reports differences, the divergence is in ArgoCD's diff engine, not in your cluster. That is the whole diagnosis. Write it down before you start editing manifests, because it is the fact that stops you from "fixing" Git.

Step 2: Turn on server-side diff for the application

This is the actual fix. Annotate the Application so the controller asks the API server to compute the merge instead of predicting it:

apiVersion: argoproj.io/v1alpha1
kind: Application
metadata:
  name: loki
  annotations:
    argocd.argoproj.io/compare-options: ServerSideDiff=true
spec:
  syncPolicy:
    syncOptions:
      - ServerSideApply=true

Expect the phantom diff to disappear on the next refresh, within a minute or two. This is the pairing nobody tells you about: ServerSideApply=true without ServerSideDiff=true is a half-migration, and the second half is where the false positives live.

Step 3: Enable it globally once you trust it

If you have more than a handful of applications, flip it for the whole controller rather than annotating each one:

apiVersion: v1
kind: ConfigMap
metadata:
  name: argocd-cmd-params-cm
  namespace: argocd
data:
  controller.diff.server.side: "true"

kubectl -n argocd rollout restart statefulset argocd-application-controller

Individual applications can opt back out with argocd.argoproj.io/compare-options: ServerSideDiff=false if one of them misbehaves, so this is reversible per app rather than all-or-nothing.

Step 4: Suppress whatever survives with ignoreDifferences

Server-side diff removes the defaulting noise. If a controller other than ArgoCD is genuinely writing to your object, you still need to tell ArgoCD to leave those fields alone. Three mechanisms, and the choice matters:

Approach	Use when	Risk
jsonPointers	You know the exact path and it is stable	Silently hides real drift on that path
jqPathExpressions	The field lives inside a list, like a volume claim template	Same, plus the expression can quietly stop matching
managedFieldsManagers	Another controller legitimately owns fields	Adapts automatically, but has had interaction bugs across upgrades

spec:
  ignoreDifferences:
    - group: apps
      kind: StatefulSet
      name: loki
      jqPathExpressions:
        - .spec.volumeClaimTemplates[]?.status
        - .spec.volumeClaimTemplates[]?.metadata.creationTimestamp
  syncPolicy:
    syncOptions:
      - ServerSideApply=true
      - RespectIgnoreDifferences=true

RespectIgnoreDifferences=true is the part people forget. Without it, ignoreDifferences only hides the fields in the diff view while sync still sends them, so you keep the pointless writes and just stop seeing them. With it, the ignored fields are excluded from the applied payload too.

Verifying it actually worked

Three checks, in order:

argocd app diff loki --refresh
argocd app get loki -o json | jq '.status.sync.status, .status.health.status'

You want argocd app diff to print nothing at all, and the second command to return "Synced" and "Healthy". Then confirm the loop has stopped, which is the check that actually matters:

argocd app history loki | tail -5

Before the fix this list grows by one entry every few minutes with identical revisions. After the fix it should be static. Come back in an hour: if the newest entry is still the one from before you made the change, the loop is genuinely dead rather than temporarily quiet.

What people get wrong

"Just set ServerSideApply=false." This is the top-voted answer in most threads and it is a bad trade. You adopted server-side apply for reasons: escaping the size limit on the last-applied-configuration annotation that breaks large CRDs, and getting real field ownership so two controllers touching one object produce a conflict instead of silently clobbering each other. Turning it off to silence a cosmetic diff gives all of that back in exchange for a green tick.

Copying the defaulted fields into Git. Adding volumeMode: Filesystem and a status block to your StatefulSet so it "matches" the live object. It usually does not even work, because status is stripped on write, and where it appears to work you have polluted your source of truth with API server implementation details that will change in a future Kubernetes release.

Ignoring /spec wholesale. An ignoreDifferences entry with jsonPointers: [/spec] makes the warning disappear and makes the StatefulSet effectively unmanaged. The next time somebody changes the image tag in Git and it does not deploy, you will spend a day on it.

Deleting the StatefulSet with --cascade=orphan as a routine fix. This is a legitimate manoeuvre for an actually-immutable-field change, where you want to recreate the StatefulSet while keeping pods and PVCs in place. It is not a fix for a diff bug, and running it repeatedly is how you eventually orphan a set of pods that nothing owns.

Disabling auto-sync "temporarily". Nothing is more permanent. Six months later the cluster has drifted from Git in ways nobody can enumerate, which is the same class of problem as hand-copied secrets across namespaces: every manual step is a drift generator.

When it is still broken

• A mutating webhook is rewriting the object. Service meshes and policy engines inject sidecars and labels after apply. Server-side diff excludes mutation webhook output by default; add IncludeMutationWebhook=true to the same compare-options annotation to bring it into the comparison, or write an ignoreDifferences rule for the injected fields.
• managedFieldsManagers is being ignored. There are known cases where a managedFieldsManagers rule that worked on an older ArgoCD stopped matching after an upgrade. If yours is not taking effect, add the equivalent jsonPointers or jqPathExpressions alongside it rather than assuming the rule is written wrong.
• The diff is real and you did not notice. Re-read it for a value you actually set. A changed storage request in volumeClaimTemplates is a genuinely immutable-field change: no diff setting will apply it, and the fix is a controlled recreate. If PVCs are involved, unbound claims are the failure you hit next.
• Check the version before you build a workaround. This family of StatefulSet diff bugs has been fixed and re-broken across several ArgoCD releases. Reproduce on the current stable version before you commit a permanent ignoreDifferences block, because permanent workarounds outlive the bugs they were written for. The diff strategies page is the canonical reference for what your version supports.

The lesson generalises past ArgoCD: any tool that predicts what a server will do, rather than asking it, will eventually disagree with reality. When it does, move the computation to the thing that owns the answer.