x509 Certificate Signed by Unknown Authority on the ingress-nginx Admission Webhook: The Real Fix

You pushed a routine change: a new hostname on an existing Ingress, or a tweak to an annotation. kubectl apply hangs for a few seconds and then rejects it with an x509 trust error. You try a different Ingress, in a different namespace, owned by a different team. Same rejection. Nothing about Ingress works anymore, cluster-wide, and you never touched the ingress controller.

The blast radius is wildly out of proportion to the cause. What has happened is that the admission webhook that validates every Ingress can no longer be trusted by the API server, because the CA bundle the API server was told to verify it against no longer matches the certificate the webhook is actually serving. One certificate went stale, and because that webhook sits in front of every Ingress create and update in the cluster, everything that touches an Ingress is now blocked.

The tempting fix, the one that appears at the top of every rushed thread, is to delete the ValidatingWebhookConfiguration so deploys flow again. That works the way pulling a smoke detector's battery works. It removes the admission safety check for the whole cluster to silence one CA mismatch. The real fix is to make the CA bundle match the certificate again, and it is not much more work once you know which two things have to agree.

The exact error

Error from server (InternalError): error when creating "ingress.yaml":
Internal error occurred: failed calling webhook
"validate.nginx.ingress.kubernetes.io": failed to call webhook:
Post "https://ingress-nginx-controller-admission.ingress-nginx.svc:443/networking/v1/ingresses?timeout=10s":
x509: certificate signed by unknown authority

The cert-manager variant is the same shape with a different webhook name:

Internal error occurred: failed calling webhook "webhook.cert-manager.io":
failed to call webhook: Post "https://cert-manager-webhook.cert-manager.svc:443/validate?timeout=10s":
x509: certificate signed by unknown authority

Two details in that message are the whole diagnosis. It got far enough to make the HTTPS Post to the webhook Service, so the network path is open; this is not a connectivity failure. And it failed on x509: certificate signed by unknown authority, which means the API server received a certificate it could not verify against the CA it was told to trust. The connection happened; the trust did not.

Why one stale certificate blocks the entire cluster

An admission webhook is a gate. When you create or update an Ingress, the API server does not just write it. It first calls out over HTTPS to the ingress-nginx admission webhook and asks "is this Ingress valid?". Only if the webhook says yes does the object get persisted. This is a good thing: it is what catches a malformed Ingress before it takes down the controller for everyone.

For that call to happen securely, two pieces have to agree on a certificate:

1. The webhook Service serves TLS using a certificate stored in a Kubernetes Secret (for ingress-nginx, typically ingress-nginx-admission).
2. The ValidatingWebhookConfiguration carries a caBundle field: the CA certificate the API server should use to verify whatever the webhook serves.

When the webhook was installed, a job generated a self-signed CA, issued the serving certificate from it, stored the cert in the Secret, and wrote the CA into the caBundle. As long as those two stay in lockstep, every admission call verifies cleanly. They drift apart when something regenerates one without updating the other: a Helm upgrade that re-runs the cert-generation job and rotates the Secret but leaves an old caBundle, a cert-manager rotation, a manual Secret delete, or a partial reinstall. Now the API server is verifying a new certificate against an old CA, or vice versa, and every verification fails.

Because the ValidatingWebhookConfiguration is a cluster-scoped object and its rules match Ingresses in all namespaces, that single broken trust relationship rejects every Ingress operation in the cluster. One certificate, whole-cluster outage. That asymmetry is exactly why it feels like "nothing works" during an unrelated deploy.

Step 1: Confirm it is a cert mismatch, not a connectivity failure

The fixes for "wrong CA" and "cannot reach the webhook at all" are completely different, so decide which one you have before touching anything. The error text already tells you: x509: certificate signed by unknown authority is a trust problem, whereas a connectivity failure reads more like context deadline exceeded, connection refused, or no endpoints available. Verify the webhook is actually up and has endpoints:

kubectl -n ingress-nginx get pods -l app.kubernetes.io/component=controller
kubectl -n ingress-nginx get endpoints ingress-nginx-controller-admission

You want running controller pods and a non-empty endpoints list. If the endpoints are empty or the pods are not ready, you have a connectivity or scheduling problem, not a CA problem, and patching the caBundle will not help. In that case chase the pod first; the systematic approach in my CrashLoopBackOff with no logs guide is the right method. If the pods are healthy and endpoints are populated, it is the CA, and you continue here.

Step 2: Compare the CA the webhook serves against the caBundle

Pull the CA out of the webhook's Secret. This is the CA that actually signed the certificate the webhook is serving right now:

kubectl -n ingress-nginx get secret ingress-nginx-admission \
  -o jsonpath='{.data.ca}' | base64 -d | openssl x509 -noout -fingerprint -sha256

Now pull the CA that the API server has been told to trust, from the ValidatingWebhookConfiguration:

kubectl get validatingwebhookconfiguration ingress-nginx-admission \
  -o jsonpath='{.webhooks[0].clientConfig.caBundle}' | base64 -d | openssl x509 -noout -fingerprint -sha256

If the two fingerprints differ, that is your confirmed diagnosis: the caBundle is stale. If they are identical, the CA is fine and you are chasing the wrong thing; go back to connectivity, or check whether the serving cert has simply expired with openssl x509 -noout -dates.

Step 3: Patch the caBundle to match the served CA

Copy the current CA from the Secret straight into the webhook configuration's caBundle. This is the whole fix, and it is deliberately narrow: you are changing one field to match reality, not deleting anything.

CA=$(kubectl -n ingress-nginx get secret ingress-nginx-admission \
      -o jsonpath='{.data.ca}')

kubectl patch validatingwebhookconfiguration ingress-nginx-admission \
  --type='json' \
  -p="[{'op':'replace','path':'/webhooks/0/clientConfig/caBundle','value':'${CA}'}]"

The value is already base64-encoded in the Secret, and the caBundle field expects base64, so you pass it through unchanged. If your webhook configuration has more than one entry under webhooks, patch each index. Expect the command to report the configuration patched, and Ingress operations to start succeeding within seconds.

For a cleaner long-term fix on ingress-nginx installed by Helm, re-running the admission cert-generation job regenerates the Secret and the caBundle together so they cannot disagree. The simplest reliable version is to let the chart recreate them:

helm upgrade ingress-nginx ingress-nginx/ingress-nginx \
  -n ingress-nginx --reuse-values

If the chart's cert job is what broke things in the first place, prefer the direct patch above to get the cluster working, then investigate the chart values that caused the drift.

Verifying it actually worked

Do a dry-run create so you exercise the webhook without leaving a test object behind:

kubectl apply --dry-run=server -f your-ingress.yaml

--dry-run=server sends the object through admission, including the webhook, but does not persist it. If it returns configured or created (server dry run) with no x509 error, the trust is restored. Confirm the two fingerprints from step 2 now match, and check the API server audit or the webhook pod logs for a clean admission review rather than a TLS handshake failure:

kubectl -n ingress-nginx logs -l app.kubernetes.io/component=controller --tail=20 | grep -i admission

What people get wrong

Deleting the ValidatingWebhookConfiguration to unblock deploys. This is the single most common and most dangerous "fix". It makes deploys flow instantly, because now nothing validates Ingresses at all. You have removed a cluster-wide safety check to work around a certificate mismatch, and the check usually never gets recreated, because once deploys work nobody remembers to restore it. Months later a malformed Ingress that the webhook would have caught takes down the controller for everyone. Patch the caBundle; do not amputate the gate.

Setting failurePolicy: Ignore and calling it done. Flipping the webhook's failure policy from Fail to Ignore also unblocks deploys, by telling the API server to admit objects when the webhook call fails. Same problem: you have disabled validation, just more quietly. It is a legitimate deliberate choice in some setups, but reaching for it to dodge a cert fix means every future invalid Ingress sails straight through.

Regenerating the serving cert but not the caBundle. People delete the admission Secret so it gets recreated, which rotates the serving certificate, and then wonder why the error is now worse. Deleting the Secret without also refreshing the caBundle guarantees a mismatch. The two always move together. Either regenerate both (re-run the Helm cert job) or patch the caBundle to whatever the Secret currently holds.

Assuming a Helm upgrade is atomic across the cert and the config. The drift almost always appears right after a chart upgrade that rotated the certs but did not update every reference to the new CA. If this started immediately after an upgrade, that is your lead, and it is worth pinning the chart version and reading its changelog rather than assuming the cluster is corrupted.

When it is still broken

• The serving certificate has actually expired. A matching CA does not help if the cert itself is past its notAfter. Run openssl x509 -noout -dates on the Secret's tls.crt. If it is expired, re-run the cert-generation job or let cert-manager reissue it, which rotates both the cert and, if wired correctly, the caBundle.
• cert-manager's webhook is the one failing. The mechanism is identical: webhook.cert-manager.io has its own Secret and its own ValidatingWebhookConfiguration, and its caBundle can drift the same way. cert-manager also uses a cainjector component whose job is to keep that caBundle in sync, so if it is failing, check the cainjector pod is running before patching by hand.
• The API server genuinely cannot reach the Service. On some managed clusters the control plane sits on a network that reaches pods through specific paths. If endpoints are healthy but the API server still times out, you have a network-policy or firewall issue between the control plane and the webhook Service, which is a connectivity fix, not a CA fix.
• It comes back after every deploy. If the drift reappears on a schedule, something in your pipeline is regenerating one half. Find the job or GitOps sync that rotates the Secret and make sure the same process updates the caBundle. A recurring version of this is a drift problem, and the same discipline I describe for secrets that drift across namespaces applies: whatever writes one copy must write the other.

The lesson that outlives ingress-nginx: any time a client verifies a server against a separately-stored CA, those two stores are a pair, and a process that updates one without the other creates an outage that looks far larger than its cause. Keep the cert and its trust anchor moving together, and this whole class of "nothing works" incident goes away.