Kubernetes Secrets Are base64, Not Encrypted: What Actually Protects Them

Someone on your team just found a Slack thread where a colleague pasted a database password. The response is predictable: "put it in a Kubernetes Secret." So it goes into a Secret, the pull request gets approved, and everyone feels better. The word "Secret" did its job. It sounds like a vault.

It is not a vault. A stock Kubernetes Secret is base64-encoded plaintext sitting in etcd. Anyone who can run kubectl get secret, anyone with an old etcd snapshot on a backup disk, anyone with an RBAC role that happens to include get on secrets, has your password in one command. base64 is not a lock. It is an envelope you can hold up to the light.

This is the single most common false-confidence bug I see in production Kubernetes clusters, and it is dangerous precisely because nothing errors out. The Secret works. The app reads it. The illusion holds until an auditor, or an attacker, reads it too.

Prove it to yourself: base64 -d recovers the plaintext instantly

Create a Secret the normal way and then read it back. No special access, no exploit, just the standard kubectl a developer already has.

kubectl create secret generic db-creds \
  --from-literal=password='S3cr3t-P@ss'

# Now "decode" it
kubectl get secret db-creds -o jsonpath='{.data.password}' | base64 -d
# S3cr3t-P@ss

That is the entire "attack". The data field of every Secret is base64, and base64 is a two-way encoding designed so binary values survive being stored as text. It has a decode function. It is supposed to. Compare it to the same value dumped raw:

kubectl get secret db-creds -o yaml
# data:
#   password: UzNjcjN0LVBAc3M=      <-- this is not ciphertext, it is base64

If you can read the Secret object, you have the password. Full stop. The base64 layer stops the value from being splattered in plaintext across your terminal history and logs, which is a real if modest benefit over a ConfigMap. It provides exactly zero confidentiality.

Why this happens: "Secret" is a storage type, not a security guarantee

The Kubernetes Secret resource was designed to be a distinct object type so the API server can treat it differently from a ConfigMap: it can avoid printing values, restrict them with their own RBAC verbs, and mount them as tmpfs rather than writing them to disk on the node. Those are meaningful properties. None of them is encryption.

By default, when you create a Secret, the API server serialises it and hands it to etcd, which writes it to disk. Unless you have explicitly configured encryption at rest, that on-disk representation is the base64 value. So the real exposure surface is larger than most people picture:

The fix, in the order that actually reduces risk

Step 1: Lock down RBAC first, because it is the real boundary

Encryption at rest protects a stolen disk. It does nothing against a live API call from an over-privileged ServiceAccount. In practice the over-privileged ServiceAccount is the far more likely breach, so start here. Audit who can read secrets:

# Who can get secrets in this namespace?
kubectl auth can-i get secrets --as=system:serviceaccount:app:worker -n app

# Find every Role/ClusterRole that grants secret read
kubectl get clusterroles -o json | \
  jq -r '.items[] | select(.rules[]? | (.resources[]? == "secrets") and (.verbs[]? | . == "get" or . == "list" or . == "*")) | .metadata.name'

The dangerous grant is a wildcard: a Role with resources: ["*"] and verbs: ["*"], or any binding to the built-in cluster-admin handed to a CI ServiceAccount "to make the pipeline work". Scope reads to the exact secret names an app needs with resourceNames:

apiVersion: rbac.authorization.k8s.io/v1
kind: Role
metadata:
  name: read-db-creds
  namespace: app
rules:
  - apiGroups: [""]
    resources: ["secrets"]
    resourceNames: ["db-creds"]   # this secret only, not all secrets
    verbs: ["get"]

Step 2: Turn on encryption at rest in etcd

On a self-managed cluster this is off unless you configured it. It protects the on-disk etcd data and, critically, etcd backups. Prefer a KMS provider so the encryption key is not itself sitting on the control-plane node; the local aescbc provider is better than nothing but keeps the key next to the data. A KMS-based config looks like this:

# /etc/kubernetes/enc/encryption-config.yaml
apiVersion: apiserver.config.k8s.io/v1
kind: EncryptionConfiguration
resources:
  - resources:
      - secrets
    providers:
      - kms:
          apiVersion: v2
          name: myKmsProvider
          endpoint: unix:///var/run/kmsplugin/socket.sock
      - identity: {}   # fallback so existing plaintext reads still work

Point the API server at it with --encryption-provider-config, restart, then rewrite existing Secrets so they are encrypted on disk (new writes are encrypted automatically, old ones are not until touched):

kubectl get secrets --all-namespaces -o json | kubectl replace -f -

On managed platforms you usually do not edit the API server flags. EKS, GKE and AKS each expose envelope encryption for Secrets as a cluster setting backed by their KMS; enable that instead of hand-rolling a config.

Step 3: For anything genuinely sensitive, do not store it in a core Secret at all

Encryption at rest plus tight RBAC makes a Secret acceptable for a config value. For a production database master password or a payment provider key, keep the real secret in a dedicated manager and sync a short-lived copy in. The External Secrets Operator pulls from Vault or a cloud secrets manager and materialises a Kubernetes Secret your pods read normally:

apiVersion: external-secrets.io/v1beta1
kind: ExternalSecret
metadata:
  name: db-creds
  namespace: app
spec:
  refreshInterval: 1h
  secretStoreRef:
    name: vault-backend
    kind: SecretStore
  target:
    name: db-creds          # the k8s Secret it creates/updates
  data:
    - secretKey: password
      remoteRef:
        key: secret/data/prod/db
        property: password

The source of truth now has real access logs, rotation and revocation. The Kubernetes Secret becomes a disposable cache you can rotate by rotating upstream. The Secrets Store CSI driver is the alternative if you want values mounted straight into the pod filesystem without a Secret object at all.

Verification: confirm encryption at rest actually took effect

Do not trust the config file, trust the bytes on disk. Read the raw etcd value for a Secret and check it is not base64 plaintext:

ETCDCTL_API=3 etcdctl \
  --cacert=/etc/kubernetes/pki/etcd/ca.crt \
  --cert=/etc/kubernetes/pki/etcd/server.crt \
  --key=/etc/kubernetes/pki/etcd/server.key \
  get /registry/secrets/app/db-creds | hexdump -C | head

Encrypted, you will see a provider prefix such as k8s:enc:kms:v2: or k8s:enc:aescbc:v1: followed by ciphertext. Unencrypted, you will see the readable base64 and the string db-creds in the clear. That prefix is the proof; the config file being present is not.

What people get wrong

"We use Secrets, so credentials are encrypted." This is the whole bug. Secrets are base64 until you enable encryption at rest, and encryption at rest still does not stop a live kubectl get. Two different controls, two different threats.

"We base64-encode the value ourselves before putting it in, so it is double-encoded and safe." Encoding twice is still encoding. base64 -d twice recovers it. This adds nothing but confusion.

"Sealed Secrets means it is encrypted." Sealed Secrets encrypts the manifest so it is safe to commit to Git. Once the controller decrypts it into the cluster, it becomes an ordinary base64 Secret in etcd. Useful for GitOps, irrelevant to at-rest and RBAC exposure inside the cluster.

"RBAC on the namespace is enough." A namespace-scoped Role still grants read to every secret in that namespace unless you use resourceNames. And a single ClusterRoleBinding to a shared ServiceAccount can re-open the door across all namespaces. Audit the bindings, not just the roles. This is the same failure mode I wrote about in cross-namespace secret drift.

When it is still broken

If a value is leaking and you have done all three steps, check these in order:

• Inlined env vars. If a pod sets env with a literal value instead of valueFrom.secretKeyRef, the plaintext is in the pod spec and visible to anyone with get pods. Grep your manifests.
• Exec access. Anyone with create pods/exec can shell into a running pod and read the mounted secret file or printenv. Encryption and RBAC on the Secret object do not touch this path; restrict exec.
• Old etcd backups. Enabling encryption today does not retroactively encrypt yesterday's snapshot sitting on a backup bucket. Rotate the credential and re-take backups.
• Audit logs and CI artifacts. A pipeline that echoes a secret, or an app that logs its own config on boot, defeats every cluster control. Check application logs and CI job output.

The mental model that keeps you out of trouble: a Kubernetes Secret is a labelled envelope, not a safe. Treat it as "keep casual eyes off this and let RBAC gate it", and put anything you would genuinely fear losing behind a real secrets manager. For smaller teams not on Kubernetes, the same principle drives my write-up on managing secrets with SOPS and age.