kubectl Unauthorized on EKS: Check the IAM Identity, Not RBAC

A new engineer joins. You send them the onboarding doc, they run aws eks update-kubeconfig, they run kubectl get pods, and they get Unauthorized. You run the same command from your laptop and it works perfectly. So they run update-kubeconfig again. And again with --region spelled out. And again after deleting ~/.kube/config. Two hours disappear.

Meanwhile you are inside the cluster running kubectl get clusterrolebindings looking for the RBAC object that is denying them, and you will not find it, because it does not exist. On EKS, the mapping between an AWS identity and a Kubernetes identity happens before Kubernetes RBAC is consulted at all, and for the identity that created the cluster it is not represented as any object you can list.

The single most useful thing you can do when someone reports this is stop looking at Kubernetes and run one AWS command.

The exact errors

error: You must be logged in to the server (Unauthorized)

The AWS documentation groups this with two siblings, and the one you get tells you a lot:

could not get token: AccessDenied: Access denied
error: You must be logged in to the server (Unauthorized)
error: the server doesn't have a resource type "svc"

What you see	What it actually means	Where to fix it
could not get token: AccessDenied	The AWS CLI could not even produce a token. Your AWS credentials are missing, expired, or lack eks:DescribeCluster.	AWS credentials / IAM policy
You must be logged in to the server (Unauthorized)	The token was valid, but the IAM principal is not mapped to any Kubernetes user.	Access entries or aws-auth
Error from server (Forbidden): ... cannot list resource "pods"	You are authenticated. Kubernetes RBAC denied the action.	Role / ClusterRoleBinding
the server doesn't have a resource type "svc"	Authenticated as someone with no discovery permissions, so kubectl cannot even read the API surface.	Access policy or RBAC

Learn the 401 versus 403 distinction and you will never waste time on this again. 401 is "I do not know who you are". 403 is "I know exactly who you are and no".

Why this happens: authentication lives outside the cluster

When you run a kubectl command against EKS, your kubeconfig does not contain a password or a certificate. It contains an exec block that shells out to the AWS CLI:

users:
- name: arn:aws:eks:eu-west-1:111122223333:cluster/prod
  user:
    exec:
      apiVersion: client.authentication.k8s.io/v1beta1
      command: aws
      args:
      - --region
      - eu-west-1
      - eks
      - get-token
      - --cluster-name
      - prod

That command produces a short-lived bearer token derived from your current AWS credentials. The EKS control plane validates it against AWS STS, learns which IAM principal you are, and then has to answer a second question: which Kubernetes username and groups does this IAM principal correspond to? That mapping comes from one of two places:

• EKS access entries, stored in the EKS API, managed with aws eks create-access-entry. This is the current approach.
• The aws-auth ConfigMap in the kube-system namespace. The legacy approach, still in use on plenty of clusters.

If the principal appears in neither, there is no Kubernetes username to authorise. The request is rejected as unauthenticated, which is why no amount of ClusterRoleBinding archaeology helps.

The creator-identity trap

When a cluster is created, the IAM principal that created it is granted administrative access to the Kubernetes API. On older clusters this grant appears nowhere: not in aws-auth, not in any RBAC object. So the person who ran eksctl create cluster has permanent working access and cannot reproduce anyone else's problem.

It gets worse when the cluster was created by automation. If Terraform ran under a CI role, then the CI role is the admin and no human being is mapped. Every engineer on the team hits Unauthorized, the cluster looks broken, and the only identity that can fix it is one that only exists inside a pipeline.

The fix, step by step

Step 1: Find out who you actually are

aws sts get-caller-identity

Expected output:

{
    "UserId": "AROAEXAMPLEID:karani",
    "Account": "111122223333",
    "Arn": "arn:aws:sts::111122223333:assumed-role/DeveloperRole/karani"
}

Read that ARN carefully. It says sts and assumed-role. The aws-auth ConfigMap does not want this string. It wants the underlying role ARN:

arn:aws:iam::111122223333:role/DeveloperRole

Pasting the assumed-role ARN into aws-auth is silent. Nothing validates it, nothing warns you, and the mapping simply never matches. I have watched people do this three times in a row while getting steadily more convinced that EKS is broken.

Step 2: Check which credentials kubectl is really using

kubectl config view --minify --raw

Look at the exec block. If it has an env section setting AWS_PROFILE, that overrides whatever profile your shell has exported, and your get-caller-identity output is not the identity kubectl is using. Also check apiVersion: it should be client.authentication.k8s.io/v1beta1. A kubeconfig generated years ago may still say v1alpha1, which modern kubectl refuses to use, and the resulting error mentions ExecCredential rather than authorisation.

Step 3: Find out which mapping mechanism the cluster uses

aws eks describe-cluster --name prod --query 'cluster.accessConfig'

{
    "authenticationMode": "API_AND_CONFIG_MAP"
}

API or API_AND_CONFIG_MAP means access entries are available. CONFIG_MAP means you are on the legacy path only. If you want to move, the switch is one command, and it is one-way: you can add the EKS API, but you cannot later remove it.

aws eks update-cluster-config --name prod \
  --access-config authenticationMode=API_AND_CONFIG_MAP

Step 4a: Map the identity with an access entry (preferred)

# what already exists
aws eks list-access-entries --cluster-name prod

# add the role
aws eks create-access-entry --cluster-name prod \
  --principal-arn arn:aws:iam::111122223333:role/DeveloperRole \
  --type STANDARD

# give it permissions
aws eks associate-access-policy --cluster-name prod \
  --principal-arn arn:aws:iam::111122223333:role/DeveloperRole \
  --access-scope type=namespace,namespaces=team-a \
  --policy-arn arn:aws:eks::aws:cluster-access-policy/AmazonEKSEditPolicy

Use type=cluster instead of type=namespace,... for cluster-wide scope, and AmazonEKSClusterAdminPolicy for full admin. Resist the urge to hand out cluster admin to make the ticket go away; scoping to a namespace costs you one extra flag, and it is the same least-privilege reasoning you would apply when hardening a production Linux box. Note that access entry changes are eventually consistent and can take a few seconds to take effect, so do not conclude it failed after one retry.

Access entries also accept role ARNs that contain a path, such as arn:aws:iam::111122223333:role/development/apps/my-role. The aws-auth ConfigMap does not, which is a real migration reason if your organisation uses IAM paths.

Step 4b: Map the identity via aws-auth (legacy clusters)

kubectl -n kube-system get configmap aws-auth -o yaml

Do not hand-edit this with kubectl edit unless you enjoy risk. It is a YAML string inside a ConfigMap, indentation errors are accepted without complaint, and a broken aws-auth locks out everyone whose access depends on it. Use eksctl, which validates the shape:

eksctl create iamidentitymapping --cluster prod --region eu-west-1 \
  --arn arn:aws:iam::111122223333:role/DeveloperRole \
  --group system:masters --username admin-user

eksctl get iamidentitymapping --cluster prod

Keep a second terminal open with a working session before you touch this file. Break-glass access is exactly like a backup: it is worth nothing until you have proven it works, which is the same argument as restoring first rather than trusting the backup job.

Step 5: Regenerate the kubeconfig for the right identity

aws eks update-kubeconfig --region eu-west-1 --name prod \
  --role-arn arn:aws:iam::111122223333:role/DeveloperRole

--role-arn makes the exec plugin assume that role explicitly, which removes the ambiguity about which profile is in play.

Verification

1. Confirm the identity the cluster sees. On a recent kubectl against a cluster that supports the SelfSubjectReview API:

kubectl auth whoami

This is the single best confirmation available, because it reports the Kubernetes username and groups the API server resolved for you rather than what you think you configured.

2. Confirm the mapping exists on the cluster side.

aws eks list-associated-access-policies --cluster-name prod \
  --principal-arn arn:aws:iam::111122223333:role/DeveloperRole

3. Confirm you can do something specific.

kubectl auth can-i get pods --namespace team-a

One caveat worth knowing: permissions granted through EKS access policies do not show up in kubectl auth can-i --list. Only permissions granted through Kubernetes RBAC objects appear there. So an empty-looking list is not proof that access is broken. Test the actual verb.

What people get wrong

The move	Why it does not work
Running aws eks update-kubeconfig repeatedly	It rewrites a file that was already correct. The problem is on the cluster side, not in the kubeconfig.
Creating a cluster-admin ClusterRoleBinding for the user	RBAC binds a Kubernetes username. You do not have one yet. This changes nothing.
Pasting the assumed-role ARN into aws-auth	Nothing validates it and it never matches. Use the arn:aws:iam::...:role/Name form.
Installing aws-iam-authenticator because a 2019 answer said so	Modern AWS CLI has aws eks get-token built in. Mixing an old external authenticator with a current kubeconfig is a way to create new problems.
kubectl edit configmap aws-auth on a live cluster	An indentation slip is accepted silently and can lock out every mapped identity, including yours, with no undo path.
Assuming the cluster is broken because kubectl works for you	You are almost certainly the cluster creator or in an admin group. Your working session proves nothing about theirs.

When it is still broken

1. Wrong region or wrong account. Two clusters called prod in different regions is common. Check that the --region in the kubeconfig exec args matches the cluster you mean, and that get-caller-identity shows the account you expect.
2. Expired SSO session. With IAM Identity Center, an expired session produces a token failure that reads like an authorisation problem. Run aws sso login --profile your-profile and try again.
3. The API endpoint is private. If the cluster has private-only endpoint access, requests from outside the VPC fail. That usually surfaces as a timeout rather than Unauthorized, but a proxy in between can turn it into something less obvious.
4. The IAM role was deleted and recreated with the same name. EKS stores the principal's internal ID alongside the ARN. A recreated role has the same ARN but a different ID, so the old access entry no longer matches. Delete the access entry and create it again.
5. Nobody has admin any more. If the creating identity is gone and no mapping exists, you are not locked out permanently on a cluster with access entries enabled: an AWS principal with the right EKS API permissions can create an access entry without touching the Kubernetes API at all. That recovery path is the strongest argument for switching off the ConfigMap-only mode before you need it.

For the full reference on entry types, access policies and migration from aws-auth, the EKS access entries documentation is genuinely good and worth reading once end to end rather than in fragments from search results.