Invisible on the Console, Only on the Invoice — AWS's Hidden Costs

In the bill-shock article I wrote that cloud cost comes in two flavors: the sudden kind, like a hack, and the quiet kind that leaks every month. This is the quiet kind. A hack stands out because it costs ₩100M a day. Invisible costs leak for years. The architecture works, the graphs look calm, and the invoice just gets a little heavier each month.

This article covers six. Fix complexity and payback depend on traffic volume, AZ count, and the change procedure.

1. Missing VPC endpoints — S3/ECR traffic leaks through NAT

The most common and most invisible case. If a private subnet's default route points to a NAT Gateway and it has no S3 or ECR endpoints, traffic to those services also traverses the NAT gateway. NAT charges an hourly fee plus a per-GB data processing charge on everything that passes through — in Seoul, $0.059 per hour and $0.059 per GB.¹ The catch: much of this traffic is headed inside AWS, not to the internet.

VPC endpoints send that traffic straight to the AWS backbone, no NAT. There are two kinds, and they behave differently.

S3 has a free gateway endpoint. You add one route to the route table; the cost is zero.² It bypasses NAT and the data processing charge disappears. The reason a free thing goes unbuilt is simple: the console VPC wizard doesn't attach it automatically. So log archiving, backups, and data pipelines — anything chatty with S3 — get billed wholesale through NAT.

ECR takes a little more. EKS/ECS nodes download image layers on every pull, and autoscaling or CI deploys repeat that traffic. Running ECR privately needs three endpoints:³

• ecr.api — auth and describe APIs (interface, paid)
• ecr.dkr — Docker registry API, push/pull (interface, paid)
• s3 — required because the layers themselves live in S3 (gateway, free)

Image layers come from S3, so in workloads where image pulls dominate traffic, the free S3 gateway endpoint can remove a large share of ECR-related NAT bytes. The two interface endpoints put authentication and metadata traffic on a separate path.

CI image builds and ECS pull frequency determine ECR traffic. Our GitHub Actions and Blacksmith comparison with measured ECR image sizes examines the cache boundary and four compressed final images.

Moving 1 TB (1,000 GB) of S3 traffic off a Seoul-region NAT gateway and onto a gateway endpoint avoids about $59 in NAT processing charges. NAT hourly fees and cross-AZ transfer are separate; actual savings depend on the route and AZ topology.¹²

S3 and DynamoDB gateway endpoints have no hourly or data-processing charge, so they are worth evaluating when private traffic reaches those services. Interface endpoints cost $0.013 per AZ-hour plus $0.010 per GB for the first PB.⁴ Against NAT processing alone, one endpoint in one AZ recovers its fixed cost at about 194 GB per month. ECR needs both ecr.api and ecr.dkr in every participating AZ, so calculate the full topology.

2. Cross-AZ data transfer — the charge that isn't on the diagram

For common paths between EC2 instances and NLBs or ENIs, 1 GB crossing an AZ boundary once incurs $0.010 on the sending side and $0.010 on the receiving side, or $0.020 total. A same-size 1 GB response crossing back adds another $0.020. RDS, ELB, and other services have pricing exceptions, so verify their pricing pages.⁵ It does not appear on an architecture diagram. Application servers spread across AZs talking constantly to a database, cache, or each other cross the boundary again and again; the traffic is normal, the charge accrues anyway. Keep data-intensive components in the same AZ, or route with AZ-awareness to cut boundary crossings.

3. Public IPv4, always billed — you pay for idle IPs too

Since 2024, AWS-provided public IPv4 addresses associated with VPC resources or idle in the account cost $0.005 per hour.⁶ It used to be only unattached Elastic IPs; now an address attached to a running instance is billed too. The usual leaks are unused EIPs and public IPs left on forgotten experiments. One address is about $3.60 a month — small alone, but dozens or hundreds left lying around is real fixed cost. Keep only what you need and move the rest behind private subnets and endpoints.

4. CloudWatch Logs kept forever — never deleted, so never expires

If a log group has no retention set, the default is "never expire."⁷ Debug and access logs pile up forever and the storage bill runs forever with them. You already paid to ingest them; now you pay every month to store logs you'll never read again. Set a retention period on every log group, and export to cheaper S3 tiers when long-term retention is genuinely required.

5. Orphaned storage — the things you thought you deleted

• Orphaned EBS snapshots: the instance and volume are gone, but the separately managed snapshot remains and bills storage.⁸
• Incomplete multipart uploads: parts from failed S3 uploads remain billable until the upload completes or is aborted. Clean them with a lifecycle rule.⁹
• gp2 not moved to gp3: gp3 has a 20% lower base price per GiB than gp2 and lets you provision performance independently of capacity. Compare any additional IOPS and throughput charges before migrating.¹⁰

6. Non-prod running 24/7 — the nights and weekends nobody uses

Dev and staging environments run through nights and weekends. Running schedulable compute only for the 8 hours a day and 5 days a week it is needed can theoretically cut uptime by about 76%. Costs that remain while stopped, such as EBS and some managed databases, need a separate calculation.

How to find it

• Cost Explorer, grouped by Usage Type. A large NatGateway-Bytes or intra-region transfer (DataTransfer-Regional-Bytes) is a signal to inspect network costs. EC2-Other mixes NAT, EBS, and other charges, so break it down again by Usage Type.
• VPC Flow Logs to break down where NAT-bound traffic is headed. If the destinations cluster on S3, ECR, and other AWS services, those are your endpoint candidates.
• Tagging discipline first. Without tags you can't attribute spend to a team or service.

Visibility comes first. The bill-shock article uses runtime metrics and cost controls for sudden abuse; structural waste likewise requires visibility into Usage Types and traffic paths.

Many invisible costs can be reduced through configuration changes, but payback must include traffic volume and change risk. Browse related diagnostics in Cloud / AWS in practice, or see our network and storage cost work at Cloud & Infrastructure.

References