PacketSensei

The VPC DNS Limit: 1,024 Packets per Second per ENI

PacketSensei

The Amazon-provided DNS resolver in a VPC enforces a hard limit of 1,024 packets per second per elastic network interface. Queries beyond that are dropped, not throttled with an error. There is no service quota to raise, no support case that changes it, and no CloudWatch metric that reports it directly. It is per-ENI, not per-instance and not per-VPC, which is the detail that determines every workaround.

What hitting it looks like

The failure mode is what makes this hard to find. Dropped packets are not refusals, so the resolver library waits for its timeout — typically five seconds — then retries. From inside the application you see:

  • Intermittent SERVFAIL or Temporary failure in name resolution, with no pattern anyone can pin down.
  • Latency histograms with a hard cluster at exactly 5 seconds, and another at 10.
  • Health checks that fail under load and pass immediately when you test them by hand.
  • Errors that scale with traffic and vanish entirely in staging.
  • Nothing at all in VPC flow logs, because flow logs do not record traffic to the VPC resolver.

The 5-second cluster is the tell. Nothing in a healthy path takes exactly five seconds; resolv.conf’s default timeout does.

Why it hits Kubernetes hardest

An EC2 instance running one application resolving a handful of hostnames will never approach 1,024 queries per second. A Kubernetes node will, for two reasons that compound.

Every pod’s DNS traffic exits through the node. CoreDNS forwards anything it cannot answer to the VPC resolver, and all of that leaves through the node’s primary ENI. A node with 60 pods concentrates 60 pods’ worth of resolution into one interface’s budget.

The ndots:5 search path multiplies every query. Kubernetes writes options ndots:5 into every pod’s /etc/resolv.conf, along with a search list like:

search default.svc.cluster.local svc.cluster.local cluster.local eu-west-1.compute.internal
options ndots:5

Any name with fewer than five dots is tried against each search domain first. So a pod resolving api.example.com — three dots — generates:

  1. api.example.com.default.svc.cluster.local → NXDOMAIN
  2. api.example.com.svc.cluster.local → NXDOMAIN
  3. api.example.com.cluster.local → NXDOMAIN
  4. api.example.com.eu-west-1.compute.internal → NXDOMAIN
  5. api.example.com → the answer

Five lookups for one name. With IPv4 and IPv6 lookups issued in parallel by glibc, that is ten packets. One application-level resolution becomes ten packets against a 1,024-per-second budget, so the real ceiling is closer to 100 useful resolutions per second per node.

Confirming it

There is no metric, so measure it on the node:

# Count DNS packets leaving for the VPC resolver, per second.
sudo tcpdump -i any -n 'udp port 53' -c 10000 -tt 2>/dev/null \
  | awk '{print int($1)}' | uniq -c | sort -rn | head

# Same idea, aggregated by queried name, to find what is generating them.
sudo tcpdump -i any -n -l 'udp port 53' \
  | grep -oP '(?<=A\? )[^ ]+' | sort | uniq -c | sort -rn | head -20

If the per-second count is pushing four figures, or if the top names are dominated by *.svc.cluster.local variants of external hostnames, this is your problem.

On an EKS node, CoreDNS metrics give you the same picture more cheaply — coredns_forward_requests_total and coredns_forward_responses_total diverging means forwarded queries are not coming back.

The fixes, roughly in order of effort

1. Add a trailing dot to external hostnames

The cheapest fix available. A fully qualified name ending in a dot — api.example.com. — skips the search path entirely. One packet instead of five. If your application configuration lets you write the trailing dot, this removes 80% of the traffic for the names it applies to.

2. Lower ndots per pod

Set it in the pod spec, where it applies only to workloads that talk to external services rather than cluster-internal ones:

spec:
  dnsConfig:
    options:
      - name: ndots
        value: "2"

With ndots:2, api.example.com is tried as an absolute name first. In-cluster short names like my-service still traverse the search path, so cluster DNS keeps working.

3. Run NodeLocal DNSCache

This is the structural fix, and the one AWS and the Kubernetes project both recommend at scale. A DaemonSet places a caching resolver on every node, listening on a link-local address. Pods query the local cache; only cache misses go to CoreDNS and then to the VPC resolver.

The cache also switches upstream queries to TCP, which sidesteps the UDP-oriented failure mode entirely and gives connection-level error signals instead of silent drops. Cache hit rates of 70–90% are typical, which takes a node from the edge of the limit to nowhere near it.

4. Spread the traffic across more ENIs

Because the limit is per-ENI rather than per-instance, more interfaces means more budget. Attaching secondary ENIs raises the ceiling, and so does running more, smaller nodes rather than fewer large ones. Both are blunt instruments compared to caching, but they help when the query volume is genuinely irreducible.

5. Route conditionally through Route 53 Resolver endpoints

An outbound Resolver endpoint has its own capacity and its own scaling behaviour. Forwarding specific domains there moves that share of the traffic off the per-ENI budget. It costs money per endpoint per hour and adds a component to operate, so it is a fit for high-volume forwarding to on-premises or partner domains rather than a general fix.

What does not work

  • Raising a quota. There is no quota for it. The limit is enforced in the VPC data plane.
  • Using 169.254.169.253 instead of the .2 address. Both reach the same resolver and share the same per-ENI budget.
  • Bigger instances. The limit is per-ENI. A c5.24xlarge gets the same 1,024 packets per second on its primary interface as a t3.micro.
  • More CoreDNS replicas. They help CoreDNS’s own CPU, but every forwarded query still exits through some node’s ENI. If your CoreDNS pods are concentrated on a few nodes, more replicas on those same nodes makes the ENI concentration worse, not better.

The short version

If an application inside a VPC shows intermittent DNS failures that correlate with load and cluster at exactly five seconds, assume the per-ENI resolver limit until proven otherwise. On EKS, install NodeLocal DNSCache and set ndots appropriately; the combination usually removes the problem in an afternoon.