PacketSensei

Ephemeral Ports and the Network ACL Rule Everyone Forgets

PacketSensei

When a client opens a connection, it picks a source port from a high range — the ephemeral range — and the server replies to that port. Security groups handle this for you because they are stateful. Network ACLs do not, because they are not. A network ACL that allows inbound TCP 443 but has no outbound rule for 1024–65535 will let the connection begin and then silently discard the reply, producing a hang that looks like nothing in the rule set explains.

What an ephemeral port is

Every TCP connection has four parts: source address, source port, destination address, destination port. The destination port is the well-known one — 443 for HTTPS, 5432 for PostgreSQL. The source port is picked by the client’s operating system, usually at random, from a range reserved for the purpose. When the server replies, it sends to that source port.

So a connection to a web server is not just “port 443 traffic”. It is:

client 10.0.1.50:54321  →  server 10.0.2.10:443     (request)
server 10.0.2.10:443    →  client 10.0.1.50:54321   (reply)

The reply arrives on 54321, not 443. Any stateless filter in the path has to be told to allow it.

The ranges differ by platform

There is no single ephemeral range, which is why “allow 1024–65535” is the standard advice — it is the superset:

Platform Range
Linux (modern kernels) 32768–60999
Windows (Vista and later) 49152–65535
Windows (Server 2003 and earlier) 1025–5000
macOS / FreeBSD 49152–65535
AWS NAT gateway 1024–65535
AWS Network Load Balancer 1024–65535
Elastic Load Balancing 1024–65535

The AWS-managed components sit at the bottom of the range, which is why narrowing a NACL to “Linux only, 32768–60999” breaks the moment traffic passes through a NAT gateway. Use 1024–65535 unless you have a specific reason not to.

Check the local range on Linux:

cat /proc/sys/net/ipv4/ip_local_port_range
# 32768	60999

Why security groups do not need this

Security groups are stateful connection trackers. When a rule permits an inbound connection, the return traffic is allowed automatically regardless of port, and vice versa. That is why a working security group set-up contains only the rule you would intuitively write — inbound 443 — and nothing about ephemeral ports.

Network ACLs evaluate each packet independently against numbered rules, in order, with no memory of what came before. Inbound and outbound are separate rule sets, and both have to permit their half of the conversation.

The default network ACL allows all traffic in both directions, which is why most VPCs never encounter this. The problem appears the moment someone creates a custom NACL — usually to satisfy a compliance requirement asking for “defence in depth at the subnet layer” — and writes the intuitive rules.

The failure

A custom NACL like this:

Direction Rule Protocol Port Source/Dest Action
Inbound 100 TCP 443 0.0.0.0/0 ALLOW
Inbound * all all 0.0.0.0/0 DENY
Outbound 100 TCP 443 0.0.0.0/0 ALLOW
Outbound * all all 0.0.0.0/0 DENY

…looks complete and is not. It allows inbound requests to 443 and outbound requests to 443, but nothing outbound to the ephemeral range, so:

  • Inbound connections from clients arrive and the server’s reply is dropped.
  • Outbound connections from instances in this subnet succeed on the SYN and then the SYN-ACK is dropped on the way in — there is no inbound ephemeral rule either.

Symptoms:

  • TCP connections that establish and then hang with no error.
  • curl sitting at “Connected to…” indefinitely.
  • Health checks timing out while the application is demonstrably running.
  • TLS handshakes failing partway rather than being refused.
  • Everything working when the custom NACL is detached, which people discover by accident and then re-attach because “the security team requires it”.

The tell is the asymmetry: the connection starts. A blocked security group produces a connection that never starts at all.

The correct rule set

Inbound
  100  TCP  443           0.0.0.0/0    ALLOW
  110  TCP  1024-65535    0.0.0.0/0    ALLOW    ← replies to outbound connections
  *    all  all           0.0.0.0/0    DENY

Outbound
  100  TCP  443           0.0.0.0/0    ALLOW
  110  TCP  1024-65535    0.0.0.0/0    ALLOW    ← replies to inbound connections
  *    all  all           0.0.0.0/0    DENY

Both ephemeral rules are needed, and for opposite reasons. The inbound one carries replies to connections your instances initiate; the outbound one carries replies to connections they receive.

Rules are evaluated in ascending number order and the first match wins, so leave gaps between numbers — 100, 110, 120 — and remember that the * rule is always a DENY you cannot remove.

Do not forget ICMP

The same rule set that blocks ephemeral ports usually blocks ICMP, which breaks Path MTU Discovery. Allow ICMP type 3, code 4 in both directions, or large transfers will hang while small ones succeed — a second, independent silent failure that looks almost identical from the application’s side.

What NACLs are actually good for

Given all this, it is worth being honest about the value. Network ACLs are a blunt, stateless instrument, and the ephemeral-range requirement means a “restrictive” NACL usually ends up allowing 1024–65535 in both directions anyway — which is most of the port space.

They earn their place in two situations:

  1. Blocking specific sources at the subnet edge. A DENY rule for a hostile CIDR, evaluated before any ALLOW, stops that traffic for every instance in the subnet at once. Security groups cannot express deny at all.
  2. A guardrail against security group mistakes. A NACL that permits only the protocols a subnet should ever see limits the damage from an over-permissive security group added later.

For everything else, security groups are the better tool: stateful, referenceable by group rather than CIDR, and far harder to get subtly wrong.

Checking it quickly

aws ec2 describe-network-acls \
  --filters Name=association.subnet-id,Values=subnet-0123456789abcdef0 \
  --query 'NetworkAcls[].Entries[?Egress==`true`].[RuleNumber,Protocol,PortRange,CidrBlock,RuleAction]' \
  --output table

Or skip the reading and let AWS trace it — VPC Reachability Analyzer names the exact rule that blocks a path:

aws ec2 create-network-insights-path \
  --source <source-eni> --destination <destination-eni> \
  --protocol tcp --destination-port 443