Kubernetes DNS Timeouts That Come and Go: ndots, conntrack and the Five Second Stall

The alert fires at 20:41. Checkout is throwing 502s, but only sometimes. You restart the CoreDNS pods, the errors stop, everyone goes back to dinner. Four days later it happens again, and this time restarting CoreDNS does nothing for eleven minutes and then it clears on its own. Nobody changed anything. The dashboards for CPU, memory and node pressure are all boring.

Here is the thing that took me too long to learn: "intermittent Kubernetes DNS failure" is not one bug, it is at least three, and they have completely different fixes. Scaling CoreDNS fixes exactly one of them. Restarting CoreDNS fixes none of them, it just resets a counter you were not watching.

The fact that a restart appears to help is itself a diagnostic signal, and it points away from CoreDNS configuration, not towards it. A misconfigured Corefile fails identically before and after a restart. Something that heals on restart is a resource or a race.

The exact error: i/o timeout on a DNS lookup

In a Go service this is the shape you get:

dial tcp: lookup payments.example.com on 10.96.0.10:53: read udp 10.244.2.31:41277->10.96.0.10:53: i/o timeout

The same underlying problem in a Python service:

socket.gaierror: [Errno -3] Temporary failure in name resolution

And from a shell inside the pod:

curl: (6) Could not resolve host: payments.example.com

The important detail is not the wording, it is the timing. Instrument the call. If the failures cluster at almost exactly 5.00 seconds or 10.00 seconds, you have the conntrack race. If they are scattered across arbitrary latencies and you see SERVFAIL in the CoreDNS log, you have a capacity problem. Those two do not share a fix.

Why this happens

Failure mode 1: ndots:5 multiplies every external lookup by eight

Kubelet writes /etc/resolv.conf into every pod. Look at it:

kubectl exec deploy/checkout -- cat /etc/resolv.conf

nameserver 10.96.0.10
search default.svc.cluster.local svc.cluster.local cluster.local
options ndots:5

ndots:5 means: if the name you are resolving contains fewer than 5 dots, treat it as relative and try every entry in the search list before trying the name as written. payments.example.com has two dots. So it is relative. So the resolver walks the list.

A service that opens connections to three external APIs per request, with no DNS caching in the client library, generates twenty-four UDP DNS packets per request. Multiply by your request rate. That is the real load on CoreDNS, and it is why the problem only appears under traffic.

Failure mode 2: the conntrack insert race and the five second stall

This one is not a Kubernetes bug and not a CoreDNS bug. It is in the Linux kernel's connection tracking layer, and Kubernetes just happens to trigger it constantly.

When glibc resolves a name it sends the A query and the AAAA query in parallel from the same UDP socket, which means the same source port. Both packets leave the pod destined for the cluster DNS service IP, and that service IP is not real. It is a DNAT rule in iptables or IPVS that rewrites the destination to a CoreDNS pod IP.

So: two packets, identical source IP, identical source port, identical destination, arriving at the netfilter DNAT hook at effectively the same moment on different CPUs. Both create a new conntrack entry, both try to confirm it, and only one can win because the entries collide on the same tuple. The loser's packet is dropped. Not rejected, not logged, dropped. The application sees nothing at all, because glibc is now sitting in poll() waiting for a reply that was destroyed inside the kernel, and its default resolver timeout is five seconds.

Parts of this race have been addressed in mainline netfilter over the years, so modern kernels clash less often than they used to. But plenty of managed clusters still run node images where it reproduces, and the amplification above means "less often" is not "never". If your latencies cluster hard at 5.00 seconds, do not assume your kernel is too new for this.

Failure mode 3: CoreDNS is genuinely too small

This is the one that actually is CoreDNS's fault, and it looks different. You get SERVFAIL rather than silence, latencies are all over the place rather than pinned to five seconds, and kubectl get pods -n kube-system shows restart counts creeping up. A CoreDNS pod that is being killed by the OOM killer and restarted looks exactly like flaky DNS from inside your application.

The fix, in order

Step 1: Establish which failure mode you actually have

Check the conntrack insert failures on a node running the affected pods. This counter is the single most useful number in this entire investigation:

sudo conntrack -S | grep -o 'insert_failed=[0-9]*' | cut -d= -f2 | paste -sd+ | bc

Sample it, wait sixty seconds under load, sample again. If that number is climbing while your app is throwing DNS timeouts, you have the race. If it is flat, stop reading about conntrack and go look at CoreDNS.

Then check CoreDNS health directly:

kubectl -n kube-system get pods -l k8s-app=kube-dns
kubectl -n kube-system top pods -l k8s-app=kube-dns
kubectl -n kube-system logs -l k8s-app=kube-dns --tail=200 | grep -c SERVFAIL

A non-zero and rising RESTARTS column, or memory sitting at the limit, is failure mode 3. If you scrape CoreDNS metrics, coredns_dns_responses_total carries an rcode label, so the SERVFAIL and NXDOMAIN rates are directly graphable, and the ratio between them tells you the amplification story too.

Finally, prove the amplification. Temporarily add the log plugin to the Corefile with kubectl -n kube-system edit configmap coredns, then watch one hostname with kubectl -n kube-system logs -l k8s-app=kube-dns -f. Three NXDOMAIN lines for suffixed variants before the one that succeeds is failure mode 1, confirmed on your own cluster with your own search list. Remove the log plugin afterwards, it is not free at volume.

Step 2: Cut the query amplification with ndots

Set it per workload, on the pod template. Do not change it cluster-wide as a first move:

apiVersion: apps/v1
kind: Deployment
metadata:
  name: checkout
spec:
  template:
    spec:
      dnsPolicy: ClusterFirst
      dnsConfig:
        options:
          - name: ndots
            value: "1"
      containers:
        - name: checkout
          image: registry.example.com/checkout:1.14

The options in dnsConfig are merged over whatever dnsPolicy produced, so ndots:1 replaces ndots:5 and the search list stays intact. Bare service names like postgres have zero dots, still fall under the threshold, and still resolve. Cross-namespace forms like postgres.billing now cost one wasted absolute query before falling back, a fair trade against saving three on every external call.

The alternative is a trailing dot on the hostname, which marks it absolute and skips search expansion entirely. That works cleanly in database DSNs and raw resolver calls. Be careful with HTTP clients: some TLS stacks pass the trailing dot straight into SNI and the certificate hostname check, and the handshake then fails. Test it against the real endpoint first.

Step 3: Stop the A and AAAA queries sharing a source port

For glibc-based images, one more resolver option removes the race at the source:

      dnsConfig:
        options:
          - name: ndots
            value: "1"
          - name: single-request-reopen

single-request-reopen makes the resolver close the socket after the A query and open a fresh one for the AAAA query, so the two packets carry different source ports and cannot collide on the same conntrack tuple. Note that the option takes no value, which is legal here because value is optional in a DNS config option.

Read the next sentence twice, because it is where most people lose an evening. musl libc ignores this option. If your image is based on Alpine, adding single-request-reopen changes nothing at all, and musl will keep sending both queries in parallel from one socket. Your options for Alpine images are to move to a glibc base image such as a Debian slim variant, or to go to Step 4, which fixes it regardless of libc.

Step 4: Deploy NodeLocal DNSCache

This is the structural fix, and it is what I reach for on any cluster serious enough to have a paging rotation. It runs a CoreDNS instance as a DaemonSet on every node, listening on a link-local address, and pods talk to that instead of the cluster DNS service IP.

That single change removes the DNAT step, which removes the conntrack entry, which removes the race. It also caches on the node, and it upgrades the node-to-CoreDNS hop from UDP to TCP, so those conntrack entries close on connection close instead of lingering for the 30 second UDP timeout and filling the table.

Take the nodelocaldns.yaml sample manifest from the Kubernetes addons tree and substitute the placeholders: __PILLAR__DNS__SERVER__ with your kube-dns service IP, __PILLAR__LOCAL__DNS__ with your chosen link-local address, and __PILLAR__DNS__DOMAIN__ with your cluster domain. The NodeLocal DNSCache task page covers the kube-proxy iptables and IPVS variants separately, so read the half that matches your cluster. This is a DaemonSet in the data path of every DNS query you make, so roll it to one node pool first and watch it for a day.

Step 5: Size CoreDNS for the load you actually generate

Once Steps 2 to 4 have cut your query volume, revisit sizing rather than guessing at it. Give CoreDNS a memory limit with real headroom, because the cache grows with your service and endpoint count, and use the cluster-proportional autoscaler if your node count moves around. Then check the forwarder: CoreDNS sends anything outside the cluster domain to the node's upstream resolver, so on a cluster sitting behind a slow or overloaded upstream, your in-cluster DNS is only ever as good as that link.

Verification

Do not verify by watching the error rate for an hour. Measure the thing directly. From inside an affected pod, time a large batch of lookups and look at the distribution:

kubectl exec -it deploy/checkout -- sh -c \
  'i=0; while [ $i -lt 200 ]; do \
     s=$(date +%s%N); getent hosts payments.example.com >/dev/null; \
     e=$(date +%s%N); echo $(( (e-s)/1000000 )); i=$((i+1)); \
   done' | sort -n | uniq -c | tail -20

Before the fix the result is bimodal: most values in single-digit milliseconds, and a second cluster sitting near 5000 and 10000. After the fix that tail should be gone entirely.

Confirm the resolver config actually landed, since a typo in dnsConfig fails silently rather than rejecting the pod:

kubectl exec deploy/checkout -- cat /etc/resolv.conf

nameserver 10.96.0.10
search default.svc.cluster.local svc.cluster.local cluster.local
options ndots:1 single-request-reopen

Then re-run the conntrack counter from Step 1 across the same window of load. It should be flat. That is the proof, and unlike an error rate it does not require you to wait for the next incident.

What people get wrong

Common advice	Why it fails
Restart the CoreDNS pods	It resets nothing that was broken. It appears to work because the race is probabilistic and traffic dipped while the pods rolled. This is the single most expensive habit in this list, because it stops the investigation.
Scale CoreDNS to 15 replicas	Only helps failure mode 3. If the packet is dropped by the kernel before it reaches any CoreDNS pod, adding pods changes nothing, and you now pay for fifteen caches with fifteen separate warm-up periods.
Set dnsPolicy: Default to bypass cluster DNS	Breaks every in-cluster service name resolution in that pod. It "fixes" external lookups by removing the ability to reach your own services.
Set ndots:1 globally in the kubelet config	Every cross-namespace and headless-service short name in the cluster starts paying an extra failed query at once, with no obvious cause. Do it per workload.
Pin the external hostnames in hostAliases	Works until the provider rotates an IP and you spend a day debugging calls to a decommissioned load balancer. That is not caching, it is a stale copy with no expiry.
Raise the client HTTP timeout above five seconds	Converts a visible failure into invisible latency. You still burn five seconds per affected request, you have just stopped being told about it.

When it is still broken

1. Capture the packets. Run tcpdump -i any -n port 53 on the affected node during a failure window. If you can see the query leave and no reply arrive, and CoreDNS's own log has no record of that query, the packet died in the kernel and you are back to conntrack. If CoreDNS logged it and answered, the problem is on the return path or in the client.
2. Check the conntrack table size. sysctl net.netfilter.nf_conntrack_max against conntrack -C. A table at capacity drops new flows indiscriminately, and UDP DNS entries sitting around for 30 seconds each are very good at filling it on a busy node.
3. Test the upstream directly from the node, not from a pod: dig @<upstream-ip> payments.example.com in a loop. If the upstream is flaky, no amount of cluster-side work helps.
4. Look at the client library. Some HTTP clients resolve on every request and cache nothing; others cache forever and ignore TTLs. A connection pool that reuses sockets removes most of this load for free. Worth knowing before you blame the platform.
5. Read the node logs properly rather than skimming them. journalctl with the right filters during the failure window puts kubelet, kube-proxy and kernel messages in one ordered stream, which is usually where the correlation becomes obvious.

When a system heals on restart, the restart is a clue about the class of bug, not a fix. Find the counter that proves it instead.