PacketSensei

The Hidden EKS Costs: Storage, Networking, and Orphaned Resources

Written by

packetadmin

in

While EC2 instance hours dominate EKS bills, storage, networking, and forgotten resources quietly accumulate charges that can represent 20-40% of total spend. These costs are easy to overlook because they appear as small line items spread across multiple services—but they compound quickly in multi-cluster, multi-team environments.

Key Takeaways

  • Orphaned EBS volumes from deleted StatefulSets can cost hundreds of dollars monthly
  • Unused Application Load Balancers charge hourly fees plus data processing costs
  • Cross-AZ data transfer adds $0.01/GB—significant for chatty microservices
  • NAT gateway charges $0.045/GB for outbound traffic that could use VPC endpoints
  • EBS gp2 volumes cost ~20% more than gp3 for equivalent performance

Orphaned EBS Volumes: The Silent Cost Multiplier

Every StatefulSet with a PersistentVolumeClaim creates an EBS volume. When you delete the StatefulSet, Kubernetes deletes the pod—but the PersistentVolume and underlying EBS volume persist by default.

Over months, these orphaned volumes accumulate. A 100GB gp3 volume costs $8/month. Across 50 forgotten volumes, that’s $400/month for storage you’re not using.

Finding Orphaned Volumes

List all PersistentVolumes in Available state (not bound to a pod):

kubectl get pv --all-namespaces -o wide | grep Available

Cross-reference with AWS to find unattached EBS volumes:

aws ec2 describe-volumes \ --filters Name=status,Values=available \ --region us-east-1 \ --query 'Volumes[*].[VolumeId,Size,CreateTime,Tags]' \ --output table

Look for volumes created months ago with no recent attachment history. These are prime candidates for deletion.

Safe Deletion Process

Before deleting any volume:

  1. Verify it’s truly orphaned (no PVC reference, no running pods)
  2. Create a snapshot for safety: aws ec2 create-snapshot --volume-id vol-xxxxx --description "backup before deletion"
  3. Wait 7-30 days to confirm no one needs it
  4. Delete the snapshot after the grace period to avoid ongoing snapshot storage costs

Delete the volume:

aws ec2 delete-volume --volume-id vol-xxxxx

Preventing Future Orphans

Set reclaimPolicy: Delete in your StorageClass so PVs are automatically removed when PVCs are deleted:

apiVersion: storage.k8s.io/v1 kind: StorageClass metadata: name: gp3-delete provisioner: ebs.csi.aws.com parameters: type: gp3 reclaimPolicy: Delete volumeBindingMode: WaitForFirstConsumer

Warning: This makes deletions permanent. For production databases, use reclaimPolicy: Retain and manage cleanup manually with snapshot policies.

Unused Load Balancers

Every Kubernetes Service of type LoadBalancer provisions an AWS Application Load Balancer or Network Load Balancer. ALBs cost $0.0225/hour (~$16/month) plus $0.008/LCU for data processing.

When you delete a Service, the AWS Load Balancer Controller should delete the corresponding ALB—but if the controller isn’t running or the Service has finalizer issues, load balancers can persist indefinitely.

Finding Orphaned Load Balancers

List all Kubernetes Services with external load balancers:

kubectl get svc -A -o json | jq '.items[] | select(.spec.type=="LoadBalancer") | {namespace: .metadata.namespace, name: .metadata.name, ingress: .status.loadBalancer.ingress}'

List all AWS load balancers:

aws elbv2 describe-load-balancers \ --region us-east-1 \ --query 'LoadBalancers[*].[LoadBalancerName,DNSName,CreatedTime,State.Code]' \ --output table

Compare the two lists. Any AWS load balancer without a matching Kubernetes Service is a candidate for deletion.

Consolidating with Ingress

Instead of creating one ALB per Service, use an Ingress controller to share a single ALB across multiple services:

apiVersion: networking.k8s.io/v1 kind: Ingress metadata: name: shared-ingress annotations: alb.ingress.kubernetes.io/scheme: internet-facing spec: ingressClassName: alb rules: - host: api.example.com http: paths: - path: / pathType: Prefix backend: service: name: api-service port: number: 80 - host: www.example.com http: paths: - path: / pathType: Prefix backend: service: name: web-service port: number: 80

This provisions a single ALB with host-based routing rules—saving $16/month per consolidated Service.

Cross-AZ Data Transfer Charges

AWS charges $0.01/GB for traffic between availability zones. For a microservice architecture with 10 services each sending 50GB/day to other services, that’s $150/month in transfer fees—just for internal communication.

Measuring Cross-AZ Traffic

Use AWS Cost Explorer to identify cross-AZ charges:

aws ce get-cost-and-usage \ --time-period Start=2025-01-01,End=2025-01-31 \ --granularity DAILY \ --metrics UnblendedCost \ --filter file://filter.json

Filter JSON for cross-AZ usage types:

{ "Dimensions": { "Key": "USAGE_TYPE", "Values": ["DataTransfer-Regional-Bytes"] } }

Reducing Cross-AZ Traffic

See the “5 EKS Networking Tweaks” post for detailed configuration, but the quick fixes are:

  • Use internalTrafficPolicy: Local on high-traffic Services
  • Enable topology-aware routing with trafficDistribution: PreferClose
  • Deploy per-AZ node groups to keep pod-to-pod traffic local
  • Use topology spread constraints to ensure even pod distribution

NAT Gateway Egress Costs

Every byte leaving your VPC through a NAT gateway costs $0.045/GB. For clusters that frequently pull container images from ECR or make API calls to AWS services, this adds up fast.

Measuring NAT Gateway Spend

aws ce get-cost-and-usage \ --time-period Start=2025-01-01,End=2025-01-31 \ --granularity DAILY \ --metrics UnblendedCost \ --filter '{ "Dimensions": { "Key": "USAGE_TYPE", "Values": ["NatGateway-Bytes"] } }'

Eliminating NAT Costs with VPC Endpoints

For AWS services like ECR, S3, and DynamoDB, create VPC endpoints to route traffic privately:

aws ec2 create-vpc-endpoint \ --vpc-id vpc-xxxxx \ --service-name com.amazonaws.us-east-1.ecr.api \ --subnet-ids subnet-xxxxx subnet-yyyyy \ --security-group-ids sg-xxxxx aws ec2 create-vpc-endpoint \ --vpc-id vpc-xxxxx \ --service-name com.amazonaws.us-east-1.ecr.dkr \ --subnet-ids subnet-xxxxx subnet-yyyyy \ --security-group-ids sg-xxxxx aws ec2 create-vpc-endpoint \ --vpc-id vpc-xxxxx \ --service-name com.amazonaws.us-east-1.s3 \ --route-table-ids rtb-xxxxx

Interface endpoints cost ~$7/month per AZ, but eliminate NAT charges. Break-even is around 60GB/month of traffic per AZ.

Storage Type Optimization

EBS gp2 volumes were the default for years, but gp3 offers better price-performance:

  • gp2: $0.10/GB-month, 3 IOPS per GB baseline
  • gp3: $0.08/GB-month, 3000 IOPS baseline (configurable up to 16,000)

For a 1TB volume, switching from gp2 to gp3 saves $20/month—with better performance.

Migrating to gp3

Update your StorageClass:

apiVersion: storage.k8s.io/v1 kind: StorageClass metadata: name: gp3 provisioner: ebs.csi.aws.com parameters: type: gp3 iops: "3000" throughput: "125" volumeBindingMode: WaitForFirstConsumer allowVolumeExpansion: true

For existing volumes, modify them in place:

aws ec2 modify-volume --volume-id vol-xxxxx --volume-type gp3

Monitor the modification:

aws ec2 describe-volumes-modifications --volume-id vol-xxxxx

Snapshot and AMI Cleanup

Old EBS snapshots and unused AMIs accumulate over time—especially from CI/CD pipelines that create custom images.

Finding Old Snapshots

aws ec2 describe-snapshots \ --owner-ids self \ --query 'Snapshots[?StartTime<=`2024-01-01`].[SnapshotId,VolumeSize,StartTime,Description]' \ --output table

Implement a lifecycle policy to automatically delete snapshots older than 90 days (adjust retention based on compliance needs).

Automated Cleanup Workflows

Manual cleanup doesn’t scale. Automate with scheduled scripts or tools:

Lambda Function for Volume Cleanup

Create a Lambda function that runs weekly to:

  1. Query available EBS volumes older than 30 days
  2. Check for associated snapshots
  3. Create final snapshot
  4. Delete volume after 7-day grace period
  5. Send summary to Slack/email

Cost Anomaly Detection

Enable AWS Cost Anomaly Detection to alert on unexpected spikes:

aws ce create-anomaly-monitor \ --anomaly-monitor Name=eks-cost-monitor,MonitorType=DIMENSIONAL,MonitorDimension=SERVICE aws ce create-anomaly-subscription \ --anomaly-subscription file://subscription.json

This automatically emails you when EBS, EC2, or data transfer costs exceed historical patterns.

Measuring Impact

After implementing cleanup and optimization:

  • Track monthly EBS volume count and total GB
  • Monitor load balancer count in Cost Explorer
  • Measure cross-AZ and NAT gateway data transfer trends
  • Document cost per cleanup action (e.g., “deleted 47 orphaned volumes, saved $376/month”)

Typical results from storage and networking cleanup:

  • 10-30% reduction in EBS costs (orphaned volumes + gp3 migration)
  • 5-15% reduction in data transfer (networking optimizations)
  • $15-50/month saved per consolidated load balancer

Conclusion

Hidden costs in EKS often go unnoticed because they appear as small line items—but across dozens of orphaned volumes, unused load balancers, and unoptimized network paths, they compound to significant waste. Implement automated discovery for orphaned EBS volumes and load balancers, migrate to gp3 storage, deploy VPC endpoints for AWS service access, and use topology-aware routing to reduce cross-AZ traffic. These fixes require minimal engineering effort but deliver sustained monthly savings that scale with your cluster growth. Start with a quarterly audit, automate cleanup with Lambda or scheduled jobs, and make cost hygiene part of your regular operational reviews.

More posts