PacketSensei

100.64.0.0/10 in AWS: Using CGNAT Space to Fix EKS IP Exhaustion

PacketSenseiUpdated

100.64.0.0/10 is carrier-grade NAT space, defined by RFC 6598 for service providers to use between their subscribers and their public NAT infrastructure. It is not RFC 1918 private space and it is not internet-routable, which puts it in a useful third category: addresses you can use internally without owning them, and without colliding with the 10.0.0.0/8 allocations your organisation already spent.

In AWS that makes it the standard answer to a specific, common problem — an EKS cluster out of pod addresses in a VPC with no RFC 1918 room left to grow.

The range

Property Value
CIDR 100.64.0.0/10
Range 100.64.0.0 – 100.127.255.255
Addresses 4,194,304
Defined by RFC 6598 (2012)
Purpose Shared address space for carrier-grade NAT

Four million addresses, sitting in a block most enterprise address plans never touched because it did not exist when they were written.

The problem it solves

The AWS VPC CNI gives every Kubernetes pod a real VPC IP address from the subnet its node sits in. That is what makes EKS networking simple — pods are first-class citizens on the VPC network, reachable directly, visible in flow logs, controllable with security groups. It is also what makes addresses run out.

The arithmetic is unforgiving. Each node holds a number of ENIs, each ENI holds a number of secondary addresses, and both are fixed per instance type. An m5.large supports 3 ENIs × 10 addresses, giving 29 usable pod addresses per node. Thirty of those nodes consume 870 subnet addresses — more than three /24s — before you count the nodes themselves, the load balancer ENIs, or anything else in the subnet.

Teams meet this as pods stuck in ContainerCreating with:

failed to assign an IP address to container

and a subnet whose AvailableIpAddressCount has reached zero.

The natural fix is a bigger subnet. But a subnet’s CIDR cannot be changed after creation, and by the time this happens the VPC’s own /16 is usually fully carved up. Enlarging the VPC means a secondary CIDR block — and AWS requires a secondary CIDR to come from the same RFC 1918 block as the primary, unless you use CGNAT space, which has no such restriction.

That exception is why 100.64.0.0/10 shows up in so many EKS runbooks.

What the fix looks like

Add the secondary CIDR to the VPC:

aws ec2 associate-vpc-cidr-block \
  --vpc-id vpc-0123456789abcdef0 \
  --cidr-block 100.64.0.0/16

Create subnets from it — one per availability zone, sized generously, because you are not going to get another chance to resize them either:

aws ec2 create-subnet --vpc-id vpc-0123456789abcdef0 \
  --cidr-block 100.64.0.0/19 --availability-zone eu-west-1a
aws ec2 create-subnet --vpc-id vpc-0123456789abcdef0 \
  --cidr-block 100.64.32.0/19 --availability-zone eu-west-1b
aws ec2 create-subnet --vpc-id vpc-0123456789abcdef0 \
  --cidr-block 100.64.64.0/19 --availability-zone eu-west-1c

Then tell the VPC CNI to place pods there while leaving nodes on the original subnets. This is custom networking mode: nodes keep their primary ENI on the original subnet, and pods get addresses from an ENIConfig that points at the new ones.

kubectl set env daemonset aws-node -n kube-system \
  AWS_VPC_K8S_CNI_CUSTOM_NETWORK_CFG=true
kubectl set env daemonset aws-node -n kube-system \
  ENI_CONFIG_LABEL_DEF=topology.kubernetes.io/zone
apiVersion: crd.k8s.amazonaws.com/v1alpha1
kind: ENIConfig
metadata:
  name: eu-west-1a
spec:
  subnet: subnet-0aaa111bbb222ccc3
  securityGroups:
    - sg-0123456789abcdef0

One ENIConfig per zone, named after the zone so the label selector matches. Existing nodes keep their current addressing until they are replaced, so the rollout is a node rotation rather than a cutover.

What it breaks

This is not free, and the costs are all at boundaries.

Pods cannot reach the internet without NAT. CGNAT addresses are not routable. Any pod in 100.64.0.0/10 needs a NAT gateway for outbound traffic. That is usually already true for private subnets, but it is worth confirming rather than assuming — and confirming that the new subnets’ route tables actually point at one.

On-premises networks may already be using it. Some enterprises adopted 100.64.0.0/10 years ago for exactly the same reason you are about to. Some ISPs use it for customer CPE, which means a VPN user at home might be inside 100.64.0.0/10 themselves. Check before allocating, and prefer a specific /16 out of the /10 rather than claiming the whole thing — it leaves room to negotiate later.

Direct Connect and Transit Gateway need explicit routes. CGNAT ranges are not implicitly reachable from anywhere. Every route table, propagation, and firewall object that needs to reach these pods has to name the range.

Some security tooling treats it as public. Tools that classify addresses as “internal” by checking RFC 1918 membership will label 100.64.x.x external. Expect alerts, and expect to update allow-lists in monitoring and SIEM rules.

Load balancer targets need thought. An Application Load Balancer in IP target mode registering pods on CGNAT addresses works, provided the load balancer subnets have a route. It is the kind of thing that works in one environment and fails in another because a route table was copied and the new range was not added.

The alternatives worth weighing first

100.64.0.0/10 is the right answer often, but not always.

Prefix delegation is usually the better first move. With ENABLE_PREFIX_DELEGATION=true, the CNI assigns /28 prefixes to ENIs instead of individual addresses. On Nitro instances that raises pod density sharply — an m5.large goes from 29 pods to 110, at which point the kubelet’s maxPods is the binding constraint rather than the address count. It uses more subnet address space in aggregate but far more efficiently, and it needs no new CIDR block at all. If you have not enabled it, do that before adding CGNAT space.

A second VPC is cleaner where the cluster is genuinely a separate concern. It costs peering or Transit Gateway attachments, but it avoids the boundary problems above.

An alternative CNI — Calico or Cilium in overlay mode — takes pods off the VPC network entirely, so pod addressing stops consuming VPC addresses. You give up the direct reachability and security-group-per-pod integration that make the AWS CNI attractive in the first place, which is a large trade for a subnet problem.

In short

100.64.0.0/10 is legitimate, well-defined, non-routable space that AWS explicitly permits as a secondary VPC CIDR when RFC 1918 room has run out. Reach for prefix delegation first; reach for CGNAT space when the addresses genuinely are not there, allocate a /16 rather than the whole /10, and audit every boundary — NAT, Direct Connect, monitoring, load balancers — before rolling nodes onto it.