Kubernetes Has No Cross-Namespace Secrets: Why Manual Copies Always Drift

You rotated the registry credential on Tuesday. On Thursday a deployment in staging-b would not start, and the events said:

Failed to pull image "registry.example.com/api:1.9.2": failed to authorize:
  authentication required
Warning  Failed  kubelet  Error: ImagePullBackOff

The secret exists. It is just the old one, because it was copied into nine namespaces by hand eleven months ago and only six of them got updated. Nothing warned you. The three stale copies sat there being wrong until a pod happened to restart.

The instinct is to treat this as a discipline failure, someone forgot a step. It is not. Kubernetes has no way for a Pod to reference a Secret in another namespace, and no built-in mechanism to keep copies in sync. This is a genuine, long-standing gap in the platform, not a misconfiguration you can settle down and fix properly. Any solution is a controller you install or a secret store you point at, and choosing to do neither means choosing eventual drift.

The errors you will actually see

There is no error that says "your secret is stale". You get one of these instead, hours or weeks after the rotation:

Error: secret "regcred" not found
MountVolume.SetUp failed for volume "tls" : secret "wildcard-tls" not found
Error: couldn't find key SMTP_PASSWORD in Secret app/mail-credentials
Failed to pull image: failed to authorize: authentication required

The third one is the worst of the set, because a Secret with the right name and the wrong contents produces an application-level authentication failure that looks like a problem with the upstream service.

Why this happens

Namespaces are the unit of authorisation in Kubernetes. RBAC Roles are namespace-scoped, and a Pod's service account can only read Secrets in its own namespace. If a Pod in team-b could mount a Secret from team-a, then anyone who can create a Pod in team-b could read every Secret in the cluster by mounting it and printing it. Permission to create Pods would silently become permission to read all secrets everywhere.

So the restriction is not an oversight. It is the boundary doing its job. The gap is that Kubernetes provides the boundary and no supported way to deliberately cross it, which leaves everyone building the same workaround.

Choosing a mechanism

Approach	Use when	Watch out for
Reflector annotations	The value is produced in-cluster, typically a cert-manager certificate or a registry credential. Lowest effort by a distance.	Another controller to run and upgrade. Copies are real Secrets, so anyone with read access in the target namespace can read them.
Kyverno generate with synchronize: true	You already run Kyverno for policy, and you want new namespaces to receive the Secret automatically.	Policy engine in the critical path of namespace creation. Deleting the policy can delete the generated Secrets.
External Secrets Operator	The value's owner is outside the cluster: Vault, AWS Secrets Manager, GCP Secret Manager, Azure Key Vault.	More moving parts, and cluster access to the external store to configure. Rotation is genuinely solved, which is why it is worth it.
Manual kubectl apply per namespace	One namespace, one secret, a value that never changes.	Drifts the first time the value rotates. Every time.

Option A: a sync controller

Step 1: Annotate the source Secret

Reflector watches for its annotations and mirrors the Secret into the namespaces you name:

apiVersion: v1
kind: Secret
metadata:
  name: wildcard-tls
  namespace: platform
  annotations:
    reflector.v1.k8s.emberstack.com/reflection-allowed: "true"
    reflector.v1.k8s.emberstack.com/reflection-allowed-namespaces: "team-a,team-b,staging-.*"
    reflector.v1.k8s.emberstack.com/reflection-auto-enabled: "true"
    reflector.v1.k8s.emberstack.com/reflection-auto-namespaces: "team-a,team-b,staging-.*"
type: kubernetes.io/tls

reflection-allowed permits mirroring at all. reflection-auto-enabled makes the controller create the mirrors for you rather than waiting for you to declare each one, and the two namespace lists accept comma-separated names or regular expressions. Because the lists are patterns, a namespace created next month that matches staging-.* gets the Secret without anyone editing anything, which is the property that actually stops drift.

Step 2: Confirm the copies exist and match

kubectl get secret wildcard-tls -A

NAMESPACE    NAME           TYPE                DATA   AGE
platform     wildcard-tls   kubernetes.io/tls   2      41d
team-a       wildcard-tls   kubernetes.io/tls   2      41d
team-b       wildcard-tls   kubernetes.io/tls   2      41d
staging-b    wildcard-tls   kubernetes.io/tls   2      6d

Option B: External Secrets Operator

When the value belongs to something outside the cluster, do not copy it around inside the cluster at all. Define the store once, then one small object per namespace that needs it:

apiVersion: external-secrets.io/v1
kind: ExternalSecret
metadata:
  name: mail-credentials
  namespace: team-b
spec:
  refreshInterval: 1h
  secretStoreRef:
    name: vault-backend
    kind: ClusterSecretStore
  target:
    name: mail-credentials
  data:
    - secretKey: SMTP_PASSWORD
      remoteRef:
        key: platform/smtp
        property: password

The operator polls the store on refreshInterval and rewrites the Kubernetes Secret when the upstream value changes. Rotation becomes one write in Vault and a wait, rather than a list of namespaces someone has to work through. The ClusterSecretStore is cluster-scoped so it is defined once; the ExternalSecret is namespaced and belongs to the team that needs it, which keeps the RBAC boundary intact.

Note that pods do not pick up a changed Secret automatically in every case. Values mounted as files are updated in place after a delay, but values injected as environment variables are fixed at container start. If you use envFrom, plan a rollout: kubectl rollout restart deployment/api -n team-b.

The related problem: updating a Secret created from a file

This trips people up constantly. kubectl create secret generic --from-file fails if the Secret already exists, and there is no merge-friendly update command. The idempotent pattern is to render and apply:

kubectl create secret generic mail-credentials \
  --from-file=./smtp.conf \
  --namespace team-b \
  --dry-run=client -o yaml | kubectl apply -f -

secret/mail-credentials configured

--dry-run=client builds the object without sending it, and kubectl apply then does a proper update. To change a single key without touching the rest, patch it, remembering the value must be base64 encoded:

kubectl patch secret mail-credentials -n team-b \
  -p "{\"data\":{\"SMTP_PASSWORD\":\"$(printf %s 'new-value' | base64 -w0)\"}}"

Verification

The only verification that means anything is comparing contents across namespaces, not checking that objects exist. Hash them:

for ns in platform team-a team-b staging-b; do
  printf '%-12s %s\n' "$ns" "$(kubectl get secret wildcard-tls -n "$ns" \
    -o jsonpath='{.data.tls\.crt}' | sha256sum | cut -c1-16)"
done

platform     9f2c41ab77e0d3c5
team-a       9f2c41ab77e0d3c5
team-b       9f2c41ab77e0d3c5
staging-b    9f2c41ab77e0d3c5

Identical hashes mean the copies are genuinely in sync. Run this from a scheduled job after every rotation, or better, on a timer, and alert when the hashes diverge. That single check is what turns a silent failure into a noisy one, and it is worth more than any of the tooling choices above.

Then prove the rotation path end to end, on a value you can safely change: update the source, wait for the refresh interval, and re-run the hash comparison. A sync mechanism you have never watched propagate is a sync mechanism you do not know works.

What people get wrong

Copying the Secret YAML into each namespace's Git directory. It feels like GitOps, so it feels safe. It is nine copies of a value that must be changed in nine places, and reviewers will approve a pull request that updates six of them without noticing. If it must live in Git, it should live there once and be templated out by a controller.

Putting everything in one namespace to avoid the problem. This works right up until you need to give a team access to their own workloads, at which point they can read every Secret you own. You have traded a synchronisation problem for an authorisation problem, and the second one is much harder to walk back.

Assuming the copies are as protected as the original. Every copy is a real Secret readable by anyone with get access in that namespace. Reflecting a database superuser credential into all namespaces because one workload needed it is how a limited credential becomes a cluster-wide one. Reflect the narrowest thing that works, and reflect it only where it is needed.

Believing Sealed Secrets solves this. Sealed Secrets solves storing secrets in Git, which is a different problem. Its encrypted values are scoped to a namespace and name by default, so it does not give you cross-namespace sharing.

Rotating the source and calling it done. Pods with values injected via envFrom keep the old value until they restart. Rotation is not complete when the Secret is updated, it is complete when the workloads have picked it up.

When it is still broken

• The copy exists but the pod still fails. Check the type as well as the name. An image pull secret must be kubernetes.io/dockerconfigjson and referenced under imagePullSecrets, not envFrom. A TLS secret must be kubernetes.io/tls with tls.crt and tls.key keys.
• New namespaces do not receive the Secret. Your namespace list is literal rather than a pattern. Switch to a regular expression that matches the naming convention, and confirm the convention is actually followed.
• The controller has no permission. Sync controllers need cluster-wide read on the source and write in every target namespace. If some namespaces sync and others do not, check the controller's ClusterRoleBinding and any admission policy restricting what it may create.
• You cannot deploy the fix. If the controller is installed by Helm and the release is wedged, you will hit another operation is in progress before you get anywhere near the Secret. Clear that first, then come back.
• Values look right but the application disagrees. Check for a trailing newline. --from-file preserves the file exactly, and an editor that adds a final newline to an API key produces an authentication failure that is invisible in kubectl get -o yaml. Decode and pipe through xxd | tail -1 to see it. If you are handling secrets outside the cluster too, my notes on a SOPS and age workflow for small teams cover keeping the source of truth in one place.

The honest summary is that this is a gap you work around, not a problem you solve. Pick one mechanism, apply it to every shared secret rather than the one that bit you, and put the hash comparison on a timer. The failure mode you are defending against is not a loud one, and that is exactly why it needs a check that runs whether or not anyone is thinking about it.