PacketSensei

EC2 MTU: 1500 vs 9001, and Why Jumbo Frames Break at the VPC Edge

PacketSenseiUpdated

EC2 instances on the Nitro system use a 9001-byte MTU — jumbo frames — for traffic that stays inside the VPC, and fall back to 1500 bytes for traffic that leaves it. The instance’s interface is configured at 9001 either way; the VPC network handles the reduction for you on paths that require it, provided Path MTU Discovery works. When it does not, you get the signature failure: small requests succeed, large transfers hang forever, and ping reports a perfectly healthy path.

Where 9001 applies and where it does not

Jumbo frames survive:

  • Between instances in the same VPC.
  • Between instances in peered VPCs in the same region.
  • Over AWS Direct Connect.
  • Through a Transit Gateway, at up to 8500 bytes — a lower ceiling than the 9001 on the instance itself.
  • Inside a cluster placement group.

Traffic drops to 1500 bytes when it:

  • Goes to or from the internet through an internet gateway.
  • Passes through a NAT gateway.
  • Crosses an inter-region VPC peering connection.
  • Traverses a Site-to-Site VPN, where the effective MTU is lower still — around 1436 bytes after IPsec overhead.
  • Reaches a VPC endpoint, which caps at 8500 bytes.

The Transit Gateway number is the one that catches people. An 8500-byte ceiling in the middle of a path where both endpoints are configured for 9001 means the largest packets are too big for the middle hop.

This used to be a silent failure: Transit Gateway simply dropped oversized packets on VPC attachments, which is why migrating from VPC peering to Transit Gateway had a reputation for breaking large transfers, and why the standard advice was to manually lower instance MTU below 8500 before migrating. Transit Gateway now supports Path MTU Discovery — when it meets a packet it cannot forward, it returns an ICMP Fragmentation Needed (or Packet Too Big for IPv6) to the sender, which then reduces its segment size on its own. Instances can stay at 9001 and the manual reconfiguration is no longer required.

That is a real improvement, and it makes the rest of this page more important rather than less: the whole mechanism now depends on that ICMP message reaching the sender. Block it and you are back to the silent drop, with the added confusion that it used to work.

How the failure actually presents

Path MTU Discovery is how a sender learns it must send smaller packets. The sender emits a full-size packet with the Don’t Fragment bit set; a router that cannot forward it replies with ICMP type 3, code 4 — “fragmentation needed and DF set” — carrying the MTU it can handle; the sender reduces its segment size and retries.

The entire mechanism depends on that ICMP message getting back. Security groups are stateful and allow the reply to a flow they permitted, so they are rarely the problem. Network ACLs are stateless, and an ACL that permits TCP but not ICMP silently discards it. So do many corporate firewalls, configured years ago by someone who decided ICMP was dangerous.

With the ICMP blocked, the sender never learns. It keeps retransmitting a packet that is silently dropped somewhere in the middle. The result is the classic PMTUD black hole:

  • ping works — 64-byte packets fit anywhere.
  • SSH connects and interactive typing is fine — small packets.
  • The connection hangs the moment something large moves: scp of a big file, a TLS handshake with a long certificate chain, an HTTP response over a few kilobytes, a database result set.
  • The hang is indefinite rather than an error, because TCP retransmits patiently.

A TLS handshake failing while plain TCP connects is a particularly strong indicator, because the certificate chain is often the first packet in the conversation big enough to exceed the path MTU.

Diagnosing it in one command

Send a DF-marked packet of a specific size and see whether it arrives:

# Linux: 8972 = 9001 - 28 bytes of IP and ICMP header
ping -M do -s 8972 -c 3 <target>

# If that fails, bisect down
ping -M do -s 1472 -c 3 <target>   # standard 1500 MTU
ping -M do -s 1400 -c 3 <target>   # typical VPN path
# Windows
ping -f -l 8972 <target>
# macOS
ping -D -s 8972 <target>

If -s 1472 succeeds and -s 8972 fails with “message too long” or simply times out, the path MTU is 1500 and something is not telling your sender about it.

tracepath <target> on Linux walks the path and reports the MTU at each step, which is the fastest way to find where it drops.

To see what the interface believes:

ip link show dev eth0 | grep -o 'mtu [0-9]*'
# and what the kernel has cached for a specific destination
ip route get <target>

Fixing it

Allow ICMP type 3 code 4. This is the correct fix. In a network ACL:

aws ec2 create-network-acl-entry \
  --network-acl-id acl-0123456789abcdef0 \
  --rule-number 120 --protocol 1 \
  --icmp-type-code Type=3,Code=4 \
  --rule-action allow --ingress \
  --cidr-block 0.0.0.0/0

Blocking all ICMP is a habit rather than a security control. Type 3 code 4 is load-bearing for the internet to work correctly, and blocking it breaks large transfers in a way that is very hard to attribute back to the firewall rule that caused it.

Set the interface MTU to 1500 where the workload’s traffic mostly leaves the VPC. A web server behind an internet gateway gains nothing from 9001 and loses nothing by dropping to 1500:

sudo ip link set dev eth0 mtu 1500

Make it persistent — on Amazon Linux 2023 and other NetworkManager systems:

sudo nmcli connection modify "System eth0" 802-3-ethernet.mtu 1500
sudo nmcli connection up "System eth0"

Setting it in user data at launch is more reliable than editing a running instance, because a reboot or an ENI replacement resets it otherwise.

Clamp MSS on the tunnel for VPN paths. Where you control the router, --clamp-mss-to-pmtu or an explicit MSS of 1387 for a VPN path stops the endpoints from ever generating an oversized segment:

sudo iptables -t mangle -A FORWARD -p tcp --tcp-flags SYN,RST SYN \
  -j TCPMSS --clamp-mss-to-pmtu

This is the pragmatic fix when the ICMP block is somewhere you do not administer.

Does 9001 actually help

Less than the number suggests. Jumbo frames reduce per-packet overhead — fewer headers, fewer interrupts, fewer trips through the stack — which matters for sustained bulk transfer between instances: large data loads, HPC traffic, storage replication, backup streams. Benchmarks in that category commonly show single-digit to low-double-digit percentage throughput gains.

For request/response traffic they do essentially nothing, because the packets were never near 1500 bytes to begin with. An API server exchanging 2 KB payloads gets no benefit from being able to send 9001-byte frames.

Modern offloads muddy the comparison further. TSO and GRO already let the stack hand large buffers to the NIC and receive coalesced segments, capturing much of the per-packet saving without any MTU change at all.

The practical position: leave the default 9001 alone for instance-to-instance traffic, set 1500 explicitly on anything whose traffic mainly leaves the VPC, and make sure ICMP type 3 code 4 is allowed everywhere. That last one prevents more incidents than either MTU choice.