kubectl 'did you specify the right host or port?': read the address first

A deploy job that has run every day for eight months fails at the first kubectl line. The message is one sentence long, ends with a question, and tells you nothing:

The connection to the server localhost:8080 was refused - did you specify the right host or port?

People read that as "the cluster is down". It is almost never the cluster. The single most useful thing in that sentence is the address, and localhost:8080 is a very specific address: it is what client-go falls back to when it could not find a kubeconfig at all. Nobody configured your cluster to run on port 8080. kubectl invented it.

So before you touch anything on the cluster, read the address in the error and decide which of three completely different problems you have. That decision takes ten seconds and saves an hour.

The exact messages, and what each one rules out

These come from the same Go networking layer with the same hint appended, so they look interchangeable. They are not.

The connection to the server localhost:8080 was refused - did you specify the right host or port?

The connection to the server 10.0.14.22:6443 was refused - did you specify the right host or port?

Unable to connect to the server: dial tcp 10.0.14.22:6443: i/o timeout

Unable to connect to the server: dial tcp: lookup ABC123.gr7.eu-west-1.eks.amazonaws.com on 127.0.0.53:53: no such host

What the message names	What it means	Where to look
localhost:8080 refused	No kubeconfig found or parsed	Your machine, your env, your CI job
Real host, refused	Reached the host, nothing listening on that port	API server process, or wrong port in the config
Real host, i/o timeout	Packets dropped in transit	VPN, security group, firewall, private endpoint
Real host, no such host	DNS did not resolve the endpoint	Split-horizon DNS, missing VPN DNS, wrong region
Unauthorized / Forbidden	Network is fine, identity is the problem	Not this article, see the note at the end

Why kubectl invents localhost:8080

kubectl resolves its cluster configuration in a fixed order, and stops at the first hit:

1. The --kubeconfig flag, if present.
2. The KUBECONFIG environment variable. This is a list, colon-separated on Linux and macOS, and the files are merged, first-wins per key.
3. $HOME/.kube/config.

If none of those yields a usable cluster entry, client-go does not error out. It falls back to a default cluster whose server is http://localhost:8080, a leftover from the era when you ran an insecure API server on your own machine. Then it tries to connect there, finds nothing, and prints the message everyone misreads.

That fallback is why the error is so misleading in three specific situations:

• sudo kubectl. sudo does not carry your $HOME by default, so kubectl looks in /root/.kube/config, finds nothing, and falls back. Your unprivileged kubectl works fine two seconds later and you conclude the cluster is flapping.
• A CI job. The runner has no home directory config. If the step that writes the kubeconfig is skipped, fails silently, or runs in a different job than the deploy, you get localhost:8080.
• A stale KUBECONFIG. Pointing at a file that was deleted, or at a path that exists but contains an empty or malformed config, also lands you on the fallback. An export line in .bashrc from a cluster you decommissioned last year is a classic.

The fix, step by step

Step 1: Ask kubectl what server it is using

kubectl config current-context
kubectl config view --minify

--minify strips everything not used by the current context, so you get exactly the cluster, user and namespace in play. Expect something like:

apiVersion: v1
clusters:
- cluster:
    certificate-authority-data: DATA+OMITTED
    server: https://ABC123.gr7.eu-west-1.eks.amazonaws.com
  name: prod
contexts:
- context:
    cluster: prod
    user: prod-admin
  name: prod
current-context: prod

If instead current-context prints an error, or view --minify comes back empty, you have confirmed the localhost:8080 diagnosis in one command. To pull just the URL for a script:

kubectl config view --minify -o jsonpath='{.clusters[0].cluster.server}'; echo

Step 2: Find which files kubectl is reading

echo "KUBECONFIG=[$KUBECONFIG]"
ls -l ~/.kube/config
id -un

Three things go wrong here. The variable is empty when you expected it to be set. The variable is set to a path that no longer exists. Or the file exists but is owned by root because you copied it with sudo cp and never fixed ownership, so your user cannot read it. That last one produces the fallback too, because an unreadable file is an absent file as far as the loader is concerned.

sudo chown "$(id -u):$(id -g)" ~/.kube/config
chmod 600 ~/.kube/config

Step 3: Test the network path without kubectl in the way

This is the step that ends most arguments. Take the server URL from Step 1 and hit it directly:

curl -k -sS -o /dev/null -w '%{http_code}\n' https://ABC123.gr7.eu-west-1.eks.amazonaws.com/version

A healthy, reachable API server that does not know who you are answers 401 or 403. Either of those is good news: DNS, routing, firewall and TLS all worked, and your problem is credentials, not connectivity. If curl hangs and times out, the network path is blocked. If curl says connection refused, nothing is listening.

To separate DNS from routing:

getent hosts ABC123.gr7.eu-west-1.eks.amazonaws.com
nc -vz ABC123.gr7.eu-west-1.eks.amazonaws.com 443

On a private EKS or GKE endpoint the name resolves to a VPC-internal address, which only works from inside the VPC, over a VPN, or through a bastion. If getent returns a 10.x address and you are on hotel wifi in Nairobi, the cluster is fine and your tunnel is not up.

Step 4: Regenerate the kubeconfig from the source of truth

Do not hand-edit a kubeconfig. Ask the provider for a fresh one:

# EKS
aws eks update-kubeconfig --name prod --region eu-west-1

# GKE
gcloud container clusters get-credentials prod --region europe-west1

# kubeadm control plane node, for that node's admin only
mkdir -p "$HOME/.kube"
sudo cp -i /etc/kubernetes/admin.conf "$HOME/.kube/config"
sudo chown "$(id -u):$(id -g)" "$HOME/.kube/config"

These commands write or merge a context into ~/.kube/config and set it current. Run Step 1 again afterwards to confirm the server URL changed.

Step 5: In CI, prove the config exists in the same step that uses it

CI is where this error is most often misdiagnosed, because the kubeconfig is written by one step and consumed by another, and the shell does not persist between steps in most runners. Put the check in the same script block as the deploy:

set -euo pipefail
test -s "$KUBECONFIG" || { echo "kubeconfig missing or empty at $KUBECONFIG"; exit 1; }
kubectl config view --minify -o jsonpath='{.clusters[0].cluster.server}'
kubectl version -o yaml | head -n 20
kubectl apply -f k8s/

Also check expiry. Short-lived service account tokens and OIDC credentials baked into a pipeline secret months ago will produce authentication errors, not this one, but a job that writes an empty file when the secret is missing produces exactly this one. test -s catches it.

Step 6: Only now, check whether the API server is actually down

If, and only if, Step 3 gave you connection refused against a real address, go to the control plane. On a kubeadm cluster the API server runs as a static pod managed by the kubelet:

systemctl status kubelet
sudo ss -lntp | grep 6443
sudo crictl ps -a --name kube-apiserver
sudo journalctl -u kubelet -n 200 --no-pager

An API server that starts and immediately exits usually tells you why in its own container logs, and the two causes I see most are expired certificates and a full disk on etcd. For the first:

sudo kubeadm certs check-expiration

Verification

Two commands. The first proves the client can talk to the server and negotiate a version:

kubectl version -o yaml

You should see both clientVersion and serverVersion blocks. A client version alone means you are still not reaching the API server.

The second proves the control plane considers itself ready, component by component:

kubectl get --raw='/readyz?verbose'

[+]ping ok
[+]log ok
[+]etcd ok
[+]poststarthook/start-kube-apiserver-admission-initializer ok
...
readyz check passed

If that returns readyz check passed, the connection problem is over. Anything failing in that list is a real control plane issue and now you have the component name.

What people get wrong

"Copy /etc/kubernetes/admin.conf and you are done." This is the top answer everywhere and it is correct only on a kubeadm control plane node, for that node's operator. admin.conf is a cluster-admin credential with a client certificate you cannot revoke individually. Copying it to developer laptops as a way of "fixing kubectl" hands out permanent root on the cluster to everyone who ever hit this error. On managed clusters it does not exist at all.

Restarting kubelet, docker, or the whole node. When the error says localhost:8080, nothing on any node is involved. Restarting a healthy control plane to fix a missing file in your home directory is how a five-minute problem becomes an incident.

chmod 777 ~/.kube/config. The file often contains a bearer token or a client key. Making it world-readable on a shared build host is a credential leak. If it is a permissions problem, the fix is chown to your user and chmod 600.

Assuming it is RBAC. RBAC failures say Error from server (Forbidden) and name the resource and verb. Authentication failures say error: You must be logged in to the server (Unauthorized). Neither of them mentions a host or a port. If you are seeing Unauthorized on EKS specifically, that is an identity mapping problem and I wrote it up separately in kubectl Unauthorized on EKS.

Reinstalling the cluster. I have watched someone tear down a working three-node cluster over an unset KUBECONFIG in a tmux pane. Run Step 1 first, always.

When it is still broken

• A proxy is eating the request. kubectl honours HTTP_PROXY, HTTPS_PROXY and NO_PROXY. On a corporate laptop or a locked-down CI image, a proxy that cannot reach a private VPC endpoint gives you timeouts. Add the API server host or the VPC CIDR to NO_PROXY and retry, and check that no_proxy in lowercase is set too, since some tooling only reads one case.
• The context is right but the namespace is not. Not this error, but it produces the next hour of confusion. kubectl config set-context --current --namespace=prod.
• Multiple merged kubeconfigs disagreeing. If KUBECONFIG is a list, keys from the first file win. kubectl config view --raw shows the merged result, including which cluster entry survived.
• Certificate errors dressed as connection errors. x509: certificate has expired or is not yet valid can be clock skew on the client, not an expired certificate. Check timedatectl before regenerating anything.
• It connects, then commands hang. That is a different failure mode, usually DNS or conntrack inside the cluster rather than at its edge. Kubernetes DNS timeouts that come and go covers it, and CrashLoopBackOff with no logs covers the workload side.

The full flag reference for the command in Step 1 is at kubernetes.io. Make kubectl config view --minify the first thing you type when a cluster "goes down". Most of the time it comes back up in your shell.