What AWS actually is
A set of primitives with an identity system in front of everything, deployed into failure domains you choose. Almost every service is a managed version of something you could run yourself.
It helps to stop thinking of AWS as two hundred products and start thinking of it as a small number of ideas. Compute, storage, networking, and managed versions of software you would otherwise operate. What is genuinely AWS-shaped — the parts with no direct equivalent elsewhere — is IAM, which gates every single API call, and the region and availability zone structure, which is how you choose what you can survive losing.
Everything else is a variation. RDS is Postgres you do not patch. SQS is a queue you do not run. EKS is Kubernetes whose control plane you do not upgrade. Recognising that means you can reason about a service you have never used by asking what it is a managed version of, and what AWS took responsibility for.
The organising question
The most useful lens for architecture on AWS is which decisions are irreversible. Four are, and they are the subject of the first half of this book: the account and identity boundary, the network layout, the data store per workload, and the failure domain strategy. Almost everything else — compute choice, deployment tooling, observability vendor — is reversible in a quarter.
Most teams spend their architecture debate on the reversible choices and an afternoon on the irreversible ones. Leading with what cannot be walked back is what makes an architecture conversation useful - and what makes it sound senior.
Regions and availability zones
A region is a geographic area; an availability zone is an independent failure domain inside it. Inter-AZ latency is about a millisecond, and that single number shapes almost every design.
A region is a geographic area with its own set of AZs. Most services are regional, data does not leave a region unless you move it, and pricing differs between them. An availability zone is one or more discrete datacentres with independent power, cooling and physical security, connected to the other zones by dedicated low-latency links.
inter-AZ round trip ~1 ms -> synchronous replication is FINE
cross-region 10-150 ms -> synchronous replication is NOT
Therefore the default posture:
MULTI-AZ ALWAYS.
Multi-region only with a stated RTO and RPO, or a residency requirement.
And: AZ NAMES are per-account aliases.
your us-east-1a is not necessarily another account's us-east-1a.
Use the AZ ID (use1-az1) when you need to correlate across accounts.The shared responsibility model divides the stack: AWS secures the cloud itself — facilities, hardware, the hypervisor, managed service internals — and you secure what you put in it: IAM, network rules, encryption choices, patching anything you run, and your data. The line moves with the service, which is worth being precise about: on EC2 you patch the OS; on RDS AWS patches the engine but you still own network placement, credentials and backups; on S3 you own only access control and encryption configuration, which is exactly where the well-known public-bucket incidents came from.
Two AZs is the minimum for availability; three is the production standard, because losing one costs a third of your capacity rather than half - and because quorum systems need an odd number.
Accounts as the boundary
An AWS account is the only hard blast-radius, quota and IAM boundary. Everything else — tags, naming conventions, permission prefixes — is a convention one mistaken policy can cross.
This is the decision that is hardest to change later, because every resource ARN, every policy, every cost report and every log destination is built on it. A single account holding development and production means they share an IAM boundary, share service quotas, and share the consequences of any mistake.
- Blast radius. A misconfigured policy in dev cannot touch production data if they are different accounts.
- Quotas. Service limits are per-account, so a load test cannot exhaust production's Lambda concurrency or EC2 capacity.
- IAM clarity. Policies become comprehensible when this account is production rather than when every statement needs a name-prefix condition.
- Cost attribution. The account is the dimension — no tagging discipline required to answer what production costs.
- Compliance scope. An auditor can be given a boundary rather than a filter.
A TYPICAL STRUCTURE
ou-security
log-archive CloudTrail + Config, Object Lock, nobody can delete
security-tooling GuardDuty, Security Hub, Access Analyzer (delegated admin)
ou-shared
build ECR - built once, pulled cross-account by every environment
ou-workloads
payments-dev
payments-staging
payments-prod <- SCPs applied here
ou-data
data-platform different consumers, different access patterns
management account: Organizations and billing ONLY. No workloads. No data.The caution to raise unprompted: more accounts means more to automate. Account creation, baseline guardrails, networking, logging and identity assignment must be automated from day one, or the structure becomes a burden and people route around it. A boundary is only valuable if a correct account is easy to create.
Organizations and service control policies
SCPs are guardrails nobody can escape — not an administrator, not the root user. They grant nothing; they only subtract, so use them for invariants rather than for permissions.
An Organization groups accounts into organizational units and applies policies to them. A service control policy caps what any principal in that OU may do, and it applies above IAM: even an account administrator who grants themselves AdministratorAccess is still bounded by it.
{ "Statement": [
{ "Sid": "RegionContainment", "Effect": "Deny",
"NotAction": ["iam:*","organizations:*","cloudfront:*","route53:*","support:*"],
"Resource": "*",
"Condition": { "StringNotEquals": { "aws:RequestedRegion":
["eu-west-1","eu-west-2"] } } },
{ "Sid": "ProtectAudit", "Effect": "Deny",
"Action": ["cloudtrail:StopLogging","cloudtrail:DeleteTrail",
"config:DeleteConfigurationRecorder","guardduty:DeleteDetector"],
"Resource": "*" },
{ "Sid": "NoLongLivedCredentials", "Effect": "Deny",
"Action": ["iam:CreateUser","iam:CreateAccessKey"],
"Resource": "*" } ] }Note what those three have in common: they are rules that should hold no matter who is asking, and they almost never fire during normal work. That is the correct use. Using SCPs for fine-grained permission is the wrong tool — it makes them fire constantly, produces confusing denials, and is better expressed in identity policies.
Permission boundaries solve a related problem: letting developers create their own roles without letting them create an administrator. A boundary attached to a role caps its effective permissions, and an IAM policy can require the boundary on iam:CreateRole — so self-service role creation becomes safe.
An SCP denial does not appear in the IAM policy simulator, which only evaluates identity and resource policies. CloudTrail's error message names it explicitly - 'with an explicit deny in a service control policy' - and that is the tell.
IAM policies
JSON documents granting or denying actions on resources, optionally with conditions. Conditions are where least privilege actually gets expressed.
{ "Version": "2012-10-17",
"Statement": [
{ "Sid": "ReadSettlementFiles",
"Effect": "Allow",
"Action": ["s3:GetObject"],
"Resource": "arn:aws:s3:::settlements-prod/incoming/*", // PREFIX, not bucket
"Condition": {
"StringEquals": { "aws:SourceVpce": "vpce-0a1b2c3d" }, // only via our endpoint
"Bool": { "aws:SecureTransport": "true" } } } ] }A policy allowing s3:GetObject on an entire bucket is not least privilege even though it names one resource. Scope to a key prefix, and add conditions: restrict by source VPC endpoint, by tag matching, by MFA presence, by requested region, by time. Conditions are the difference between a policy that is technically scoped and one that is genuinely constrained.
Identity versus resource policies
An identity policy is attached to a role or user and says what that principal may do. A resource policy is attached to the resource — a bucket, a queue, a KMS key — and says who may use it. Resource policies are required for cross-account access and for principals you do not control, such as a service principal or a partner's account.
Write policies from CloudTrail history rather than from imagination. IAM Access Analyzer's policy generation reads what a role ACTUALLY used and produces a policy containing exactly those actions. Broad-then-tighten with a deadline beats a perfect policy nobody can write.
Users, roles and STS
A user has permanent credentials; a role has none and is assumed, yielding short-lived tokens. Applications and humans should both use roles.
A user is a long-lived identity with permanent credentials — a password, or an access key and secret. A role is an identity with no credentials attached that a principal assumes; STS then issues temporary credentials that expire, typically in an hour.
That expiry is the entire point. A leaked access key is valid until a human notices and revokes it, and credential-scanning bots find keys committed to public repositories within minutes. Temporary credentials expire on their own, are rotated automatically by the SDK, are never written to disk, and cannot be committed.
HOW A WORKLOAD GETS CREDENTIALS - same shape everywhere:
something in the environment vouches for its identity, and STS exchanges
that for temporary credentials.
EC2 instance profile, served by the metadata service
ECS task role, from a link-local endpoint
EKS IRSA / Pod Identity - the pod's ServiceAccount token exchanged
via an OIDC provider for a role scoped to that ServiceAccount
GitHub CI an OIDC token presented to AssumeRoleWithWebIdentity
On-prem IAM Roles Anywhere, using an X.509 certificate
# CI trust policy - note the CONDITION
{ "Action": "sts:AssumeRoleWithWebIdentity",
"Condition": { "StringLike": { "token.actions.githubusercontent.com:sub":
"repo:acme/payments-api:ref:refs/heads/main" } } }
# omit that condition and ANY GitHub repository can assume the roleWhen is an IAM user still legitimate? Rarely, and always with a note: a third-party SaaS that genuinely cannot do OIDC or cross-account assumption. Then a dedicated user, a single narrow policy, scheduled rotation, and CloudTrail alerting on use from unexpected sources. Never a shared user.
The policy evaluation chain
Eight layers, all of which must permit and none of which may deny. Reciting them in order is also the debugging algorithm.
1 SCP organization-level cap. Denies here end it.
2 Resource control newer org-wide cap applied to resources
policy (RCP)
3 Identity policy does the ROLE have the action on that resource?
4 Resource policy does the BUCKET / QUEUE / KEY allow this principal?
5 Permission boundary if attached, caps the role's effective permissions
6 Session policy if passed at AssumeRole time
7 KMS KEY POLICY if the resource is encrypted with a CMK
8 VPC endpoint policy if the request travels through one
RULES
an explicit DENY anywhere wins immediately and cannot be overridden
otherwise an explicit ALLOW is required; the default is deny
SAME account: an allow in EITHER identity or resource policy suffices
CROSS-account: you need an allow on BOTH sidesWhere it actually breaks
Most often at step seven. A KMS-encrypted bucket requires the caller to have kms:GenerateDataKey on the key, and the key policy in the owning account must grant it — because a key policy is the root of trust for that key, and IAM alone cannot grant access to a key whose policy does not permit it. The symptom is an opaque AccessDenied with a demonstrably correct bucket policy, and it costs people hours.
# Debugging, in this order:
aws cloudtrail lookup-events --lookup-attributes ... # errorMessage names the layer
aws iam simulate-principal-policy ... # identity side only
# then check, in order: SCP, KMS key policy, endpoint policy
# (the simulator sees NONE of those three)The confused deputy problem belongs here: when a service assumes a role on your behalf, pin aws:SourceAccount and aws:SourceArn in the trust policy so it cannot be tricked into acting for a different customer. A service-principal trust without those conditions should be flagged in review.
Identity in practice
Humans federate through single sign-on; machines assume roles. No IAM users, no long-lived keys, and joiners and leavers handled in one place.
IAM Identity Center is the mechanism for human access: permission sets mapped to identity-provider groups, assigned per account. A person's access follows their group membership, so a leaver is removed everywhere by one change in the directory rather than by remembering which accounts they had.
IdP group -> permission set -> accounts
payments-engineers -> PaymentsDeveloper -> dev, staging
payments-engineers -> ReadOnly -> prod
payments-oncall -> BreakGlassProd -> prod (approval + alert)
security -> SecurityAudit -> all
# Note the shape: the SAME group gets different permission sets per account.
# Production is read-only by default; write access is a break-glass path.Break-glass
Build the emergency path deliberately: a separate role, assumed with approval, time-limited, alerting to an incident channel on assumption, every action in the audit log, and an automatic ticket to review afterwards. If the path does not exist officially, people improvise one under pressure with less oversight — which is strictly worse.
- No IAM users. Enforce it with an SCP denying
iam:CreateUserandiam:CreateAccessKeyin production. - MFA everywhere, ideally hardware-backed for anything privileged.
- Access Analyzer unused-access findings to catch roles, permissions and credentials nobody has used in ninety days.
- Quarterly review of who has production access and why.
The workflow that actually achieves least privilege: deploy with a broad policy in a non-production environment, run real traffic for a week or two, generate a tightened policy from CloudTrail, review it, apply. Teams that try to write the perfect policy up front either block delivery or attach FullAccess and never revisit it.
The VPC
A private IP space you choose. You cannot shrink a CIDR and overlapping ranges cannot be peered, which makes the address plan one of the four irreversible decisions.
A VPC is a logically isolated network with a CIDR block you pick — say 10.20.0.0/16, giving 65,536 addresses. Everything you run lives inside it, and its addressing constrains what you can connect to later.
- You cannot shrink a CIDR. You can add secondary blocks, which is awkward but possible.
- Overlapping ranges cannot be routed between. If your VPC overlaps your on-premises network, or another VPC you need to peer with, you cannot connect them without NAT gymnastics.
- Someone must own IP allocation across the organisation. In a company with many accounts this is a real job, and skipping it produces a peering problem three years later that nobody can fix cheaply.
SIZING - be generous, it costs nothing
/16 65,536 addresses a comfortable default per VPC per region
/20 4,096 a tier across three AZs
/24 251 usable AWS reserves 5 in every subnet
AND ON EKS: every POD takes a subnet address.
a /24 app subnet holds ~251 PODS, not 251 nodes.
IP exhaustion presents as pods stuck Pending with no resource shortage.Ask about the on-premises ranges before choosing anything. The single most expensive VPC mistake is picking 10.0.0.0/16 because it is the default, and discovering later that the corporate network already uses it - which permanently forecloses Direct Connect and peering.
Subnets and routing
A subnet lives in exactly one availability zone, and public versus private is a routing decision rather than a property of the subnet.
Subnets carve the VPC up, and each is pinned to one AZ. That is the constraint that shapes every layout: a three-tier application across three zones needs nine subnets, not three.
A subnet is public if its route table sends 0.0.0.0/0 to an internet gateway. It is private if it sends 0.0.0.0/0 to a NAT gateway — outbound only, because NAT forwards return traffic for connections initiated from inside but nothing inbound. A subnet with no default route at all is isolated, which is the right choice for a database tier.
VPC 10.20.0.0/16 (eu-west-1)
AZ a AZ b AZ c
public 10.20.0.0/24 10.20.1.0/24 10.20.2.0/24 -> IGW
(ALB, NAT-a) (ALB, NAT-b) (ALB, NAT-c)
app 10.20.16.0/20 10.20.32.0/20 10.20.48.0/20 -> NAT in SAME AZ
(4,091 usable - sized for PODS, not instances)
data 10.20.200.0/24 10.20.201.0/24 10.20.202.0/24 -> NO default route
(RDS subnet group spans all three)
Route tables:
rtb-public 0.0.0.0/0 -> igw-xxxx
rtb-app-a 0.0.0.0/0 -> nat-a <- per-AZ, avoids cross-AZ charges
rtb-data local only <- structurally cannot reach the internetSeparating tiers into their own subnets makes 'the database cannot reach the internet' a STRUCTURAL guarantee expressed in routing, rather than a security group rule someone can loosen. That is why it is worth nine subnets instead of three.
Internet and NAT gateways
An internet gateway is free and bidirectional; a NAT gateway is outbound-only, zonal, and billed per gigabyte processed on top of its hourly rate.
An internet gateway attaches to the VPC and provides a route to and from the internet for resources with public addresses. It is horizontally scaled, redundant, and free.
A NAT gateway lets private resources reach the internet outbound without being reachable inbound. It is a managed, zonal resource — so one per AZ for availability, and the route table in each zone should point at its own NAT, or outbound traffic crosses a zone boundary and pays cross-AZ transfer on top of everything else.
NAT GATEWAY COST HAS TWO PARTS
hourly rate per gateway, per AZ
PER-GB PROCESSED on every byte, in addition to any transfer charge
WHAT GOES THROUGH IT (and should not)
container image pulls on every cold node
S3 reads and writes
Secrets Manager calls
CloudWatch Logs
third-party API calls <- the only one that genuinely needs it
At any scale this becomes a large line item, and it hides in the bill under
"EC2-Other" rather than under a service name - so it is invisible in a
service-level cost breakdown.For development, a single NAT gateway shared across zones is a legitimate cost saving — just be explicit that it is a deliberate trade rather than an oversight, since it is both a single point of failure and a source of cross-AZ charges.
The fix for most NAT traffic is VPC endpoints, covered shortly. Gateway endpoints for S3 and DynamoDB are FREE and remove the largest component immediately. If you take one action on an unfamiliar AWS estate, make it that one.
Security groups and network ACLs
Security groups are stateful, attach to an interface, allow-only, and can reference other groups. NACLs are stateless, attach to a subnet, and support deny. Default to security groups for essentially everything.
SECURITY GROUP NETWORK ACL
stateful (return traffic auto) STATELESS (needs explicit ephemeral rules)
attaches to an ENI attaches to a SUBNET
allow only allow AND deny
evaluated as a set, no order numbered rules, FIRST MATCH WINS
can reference another SG CIDR only
applies to SAME-SUBNET traffic only evaluates traffic CROSSING the subnetThe feature that decides it is group references. A rule saying allow 5432 from sg-app stays correct forever as instances scale, are replaced and change addresses. It expresses the relationship rather than a snapshot of addresses — and once you have used it you stop writing CIDR rules for internal traffic.
sg-alb in: 443 from 0.0.0.0/0
out: 8080 to sg-app
sg-app in: 8080 from sg-alb <- a GROUP, not a CIDR
out: 5432 to sg-data, 443 to endpoints
sg-data in: 5432 from sg-app
out: (none) <- the database initiates nothing
Scale from 2 to 200 instances: no rule changes.
Replace every instance: no rule changes.
Compare "allow 5432 from 10.20.16.0/20": correct until someone adds a subnet,
and it lets ANY pod in that range reach the database.The NACL trap is statelessness: an inbound allow on 443 also needs an outbound allow on the ephemeral port range, or responses are silently dropped. And because NACLs only evaluate traffic crossing a subnet boundary, they do nothing about lateral movement within a subnet — which is why security groups, not NACLs, are your lateral-movement control.
Security groups are allow-only, so blocking a specific attacking address requires either a NACL deny or - better - a WAF rule at the load balancer, where you also get rate limiting and logging. NACL is the blunt instrument for when you need it immediately.
VPC endpoints
Reach AWS services without traversing the internet or NAT. Gateway endpoints for S3 and DynamoDB are free; interface endpoints cost an hourly rate and are far cheaper than NAT at volume.
GATEWAY ENDPOINT S3, DynamoDB only
works by adding a ROUTE to the subnet's route table
FREE. There is no reason not to have these.
INTERFACE ENDPOINT almost everything else (PrivateLink)
an ENI in your subnet with a private DNS name
hourly per endpoint PER AZ, plus a small per-GB rate
THE ECR TRIO - the omission that makes people think endpoints do not work
com.amazonaws.<region>.ecr.api interface (authentication, manifests)
com.amazonaws.<region>.ecr.dkr interface (the Docker protocol)
com.amazonaws.<region>.s3 GATEWAY <- ECR stores LAYERS in S3
# configure only the first two, watch pulls still traverse NAT, and be baffledBeyond cost, endpoints are a security improvement: traffic never leaves the AWS network, and an endpoint policy can restrict what may be reached through it. Combined with a bucket policy condition on aws:SourceVpce, you can make a bucket reachable only from your VPC — which is a much stronger statement than a set of IAM policies.
# On the bucket: only reachable through our endpoint
{ "Effect": "Deny", "Principal": "*", "Action": "s3:*",
"Resource": ["arn:aws:s3:::settlements-prod",
"arn:aws:s3:::settlements-prod/*"],
"Condition": { "StringNotEquals": { "aws:SourceVpce": "vpce-0a1b2c3d" } } }Do the arithmetic before adding interface endpoints everywhere: they cost per endpoint PER AZ, so in a small development account with three zones and ten endpoints the fixed cost can exceed the NAT charge they replace. Above a few hundred gigabytes a month through NAT, they win comfortably.
Data transfer costs
Inbound free, outbound charged, cross-AZ charged in BOTH directions, and NAT charged per gigabyte on top. Two of those are invisible in a service-level bill.
inbound from internet FREE
outbound to internet the headline egress rate
CROSS-AZ billed BOTH directions (send AND receive)
cross-region more, both directions
via NAT gateway hourly + PER-GB PROCESSED, ON TOP of the above
within one AZ, private IPs freeThe two that hide
NAT processing, which appears under EC2-Other rather than under a service name. And cross-AZ chatter between services: twelve microservices spread across three zones for availability means roughly two thirds of internal calls cross a zone and are billed twice, appearing as a regional transfer line with no indication of which services caused it.
FIND IT: Cost Explorer, group by USAGE TYPE - not by service
EU-NatGateway-Bytes <- NAT processing
EU-DataTransfer-Regional-Bytes <- cross-AZ
EU-DataTransfer-Out-Bytes <- internet egress
Then VPC flow logs to attribute cross-AZ traffic to specific services.
DESIGN IT OUT - configuration, not redesign
S3 + DynamoDB -> gateway endpoints (FREE)
ECR, Secrets, Logs, STS -> interface endpoints
chatty service pairs -> topology-aware routing: prefer a SAME-ZONE
endpoint while keeping replicas spread
high internet egress -> CloudFront (lower egress rate + caching)Reading Cost Explorer by usage type rather than by service is the first thing to do on an unfamiliar bill. It is where NAT processing and cross-AZ transfer become visible, and both are routinely 10-20% of a bill that nobody has attributed.
DNS and Route 53
Seven routing policies, health checks, and one hard limitation: DNS is cached by resolvers and by the JVM, so failover is slower than its TTL.
simple one target, no logic
weighted split by percentage - migrations, cross-region canaries
latency each client to the region with the lowest measured latency
geolocation by country or continent - DATA RESIDENCY
geoproximity by distance, with an adjustable bias
failover primary + health-checked secondary
multi-value several healthy records, simple client-side spreadingAlias records point at AWS targets (a load balancer, CloudFront, an S3 website) and have two advantages over a CNAME: they work at the zone apex, and alias queries are not billed. Use alias for AWS targets and CNAME only for external hostnames.
Why failover is slower than the TTL
You set a 60-second TTL. Recursive resolvers may honour it or not; corporate resolvers frequently extend it; browsers keep their own cache. And for a Java client it is worse and specific: the JVM caches DNS resolutions according to a security property, and many applications never change it. Combined with a connection pool holding open sockets, the client stays pinned to a dead endpoint long after DNS has moved.
-Dsun.net.inetaddr.ttl=5 # or in java.security:
networkaddress.cache.ttl=5
networkaddress.cache.negative.ttl=1
spring.datasource.hikari.max-lifetime: 600000 # recycle connections
# TEST THIS with a real client, not with dig. It is frequently the difference
# between a 2-minute failover and a 30-minute one.
# For seconds-level failover, DNS is the wrong tool entirely:
# Global Accelerator anycast IPs, failover happens IN THE NETWORK
# client-side retry the SDK tries endpoint A then BHealth checks must hit a DEEP endpoint that verifies the target can actually serve, and they come from multiple global locations so a single bad network path does not trigger a failover. A health check against a static file will happily fail you over into a region whose database is unreachable.
CloudFront and the edge
A CDN, and also a way to terminate TLS close to the user and carry the rest over the AWS backbone — which helps even at a zero percent cache hit rate.
For a globally distributed audience, CloudFront helps regardless of caching. A TLS handshake is multiple round trips, and doing those at 15ms to a nearby edge rather than 180ms to a distant region is a large, measurable improvement. Connection reuse from edge to origin compounds it.
Be honest about the converse: for a single-region audience hitting a single-region origin, CloudFront mostly adds a hop. Say so rather than recommending it reflexively.
The cache key is where deployments fail
Including every header, cookie and query string in the key fragments the cache so badly that the hit rate approaches zero — every request is unique, nothing is reused, and you have added a hop for nothing. Use a cache policy that includes only what genuinely changes the response, and a separate origin request policy for things the origin needs but that must not be part of the key.
CachePolicy:
HeadersConfig: { whitelist: [Accept-Encoding, Accept-Language] }
CookiesConfig: { none }
QueryStringsConfig: { whitelist: [page, size] }
OriginRequestPolicy:
HeadersConfig: { whitelist: [Authorization, X-Request-Id] }
# ^ forwarded to the origin, but NOT part of the cache key
# Personalised responses are never shareable
Cache-Control: private, no-store
# caching per-user data at a shared edge is a LEAK
# Invalidation is slow and metered - do not design around it
/assets/app.9f2c4a.js versioned path: a new deploy is a new URL
create-invalidation "/*" emergency onlyOrigin Shield adds a regional caching layer so many edges do not all miss through to your origin at once - useful protection against a cache-clear stampede. And CloudFront Functions handle header rewrites and cache-key normalisation far more cheaply than Lambda@Edge.
EC2 and instance families
Families encode a resource ratio, not a quality tier. Match the ratio to your workload, then size — and for JVM services, Graviton is the default recommendation.
c compute-optimised ~2 GB RAM per vCPU
m general purpose ~4 GB
r memory-optimised ~8 GB
i/d + fast local NVMe
g/p + GPUs
t BURSTABLE with CPU credits
SUFFIXES
g Graviton (ARM) ~20% cheaper, equal or better performance
i Intel
a AMD
n enhanced networking
d local NVMe
m7g.xlarge = general purpose, 7th generation, Graviton, 4 vCPU / 16 GBGraviton for JVM workloads is the recommendation worth making explicitly. Java bytecode is architecture-neutral, so the migration is usually building a multi-arch image and checking native dependencies — JNI libraries, some compression and crypto libraries, older agents. Roughly 20% cheaper per hour at equal or better performance is the largest easy saving available.
Avoid t-family for sustained production load
Burstable instances earn CPU credits when idle and spend them under load. Exhaust the balance and you are throttled to a baseline that can be 10 or 20% of a vCPU. The symptom is a service that performs fine for twenty minutes after deploy and then degrades catastrophically — a genuinely confusing incident if you do not know the mechanism.
Right-size from observed p95, not from the shape someone picked at launch. Right-sizing plus Graviton is typically 20-40% on compute for a few days of work, and it is the highest ratio of saving to effort available on AWS.
Auto Scaling groups
Not primarily a scaling tool. It gives you replacement, AZ distribution, and then scaling — and the health check type determines whether replacement actually works.
- Replacement. It maintains a desired count, so a failed instance is terminated and replaced.
- AZ distribution. Given subnets in several zones it balances across them and rebalances after a zone recovers.
- Scaling policies. Target tracking on a metric, step scaling, or scheduled.
HealthCheckType: ELB # <-- NOT the default 'EC2'
HealthCheckGracePeriod: 300 # JVM start + warmup
# The default EC2 check only asks whether the HYPERVISOR thinks the instance
# is alive. A JVM that has deadlocked, OOMed, or is returning 500s to every
# request passes it forever.
# ELB uses the load balancer's application-level check, so a broken instance
# is actually replaced.
# And the grace period must cover startup, or the ASG kills instances during
# boot in an endless loop.MixedInstancesPolicy:
Overrides: [m7g.large, m7g.xlarge, m6g.large, c7g.xlarge] # diversify
InstancesDistribution:
OnDemandBaseCapacity: 3 # baseline never on spot
OnDemandPercentageAboveBaseCapacity: 20 # burst is 80% spot
TargetTrackingPolicy:
Metric: ALBRequestCountPerTarget # NOT CPU for an IO-bound service
TargetValue: 800
# Warm-up matters for a JVM: a fresh instance is slow until JIT compiles.
# Use target group slow start, or it takes full traffic while interpreting.Launch configurations are deprecated - immutable, unversioned, no mixed instance types. If you see them in a scenario, flagging them as legacy and naming launch templates is a small, easy point.
Load balancers
ALB is layer 7 and terminates the connection; NLB is layer 4, preserves the source IP and supports static addresses. Choose on protocol and on whether a partner must allowlist an address.
ALB (L7) NLB (L4) GWLB
protocol HTTP/HTTPS/gRPC TCP/UDP/TLS GENEVE
routing host/path/header port only n/a
client source IP X-Forwarded-For PRESERVED n/a
static/Elastic IP no YES n/a
WAF integration yes no n/a
latency added ~ms ~microseconds n/a
CHOOSE
ALB normal web APIs, path routing, WAF, TLS termination
NLB a partner must allowlist a fixed IP; non-HTTP protocols; extreme
connection counts; fronting your own proxy layer
GWLB inserting third-party firewalls or IDS transparentlyA useful hybrid worth knowing: an ALB behind an NLB gives you the static addresses a partner allowlists plus layer 7 routing. The alternative — an ALB with Global Accelerator in front — also provides static anycast addresses and improves global latency, and is often the better answer when fixed IPs are the only reason NLB was considered.
# The keep-alive race: a very common and very confusing 502
ALB idle timeout 60s
app server idle timeout 60s <- SAME. Race condition.
-> the ALB reuses a connection the target is closing
-> intermittent 502, correlated with connection age rather than load,
and NOTHING in the application logs
FIX (one line):
server.tomcat.keep-alive-timeout: 70s # MUST exceed the ALB'sALB access logs are the only source with per-request detail including which target served it. Enable them on day one - they are cheap, they go to S3, and their absence is only ever felt during an incident.
Target groups and health checks
A shallow health check, a deregistration delay above your longest request, and slow start for a JVM. Getting the first one wrong turns a dependency blip into a total outage.
TargetGroupAttributes:
deregistration_delay.timeout_seconds: 60 # > longest request, or deploys
# cut connections mid-flight
slow_start.duration_seconds: 60 # JVM warms up before full traffic
load_balancing.algorithm.type: least_outstanding_requests
# better than round_robin when
# request cost varies
stickiness.enabled: false # unless you genuinely need it
HealthCheck:
Path: /actuator/health/readiness # SHALLOW - local only
Port: 9090 # management port
Interval: 10
HealthyThreshold: 2
UnhealthyThreshold: 3Why the health check must be shallow
A health check that verifies the database means that when the database blips, every target fails simultaneously, the load balancer removes all of them, and you get 503 with no healthy targets. A twenty-second degradation becomes a complete outage. The check should answer can this instance serve, not is the whole system well.
Deregistration delay is the drain window. Set it too low and every deploy cuts in-flight requests, which shows as a small burst of 502s nobody explains. Set it absurdly high and deploys crawl.
# 5xx with healthy targets - split by code, they mean different things
502 target returned something unparseable, or closed the connection
(usually the keep-alive race; check target_status_code = "-" in the logs)
503 no healthy targets at that instant
(check HealthyHostCount over time - a dip to zero is definitive)
504 target did not respond within the idle timeout
(a downstream slower than your timeout; nest your timeout budgets)least_outstanding_requests is a free improvement to tail latency when request cost varies: round robin keeps sending work to a target stuck on a slow query, while least-outstanding naturally routes around it.
ECS
AWS-native orchestration: task definitions, services, and direct integration with ALB, IAM and CloudWatch. Far less to operate than Kubernetes, and correspondingly less expressive.
TASK DEFINITION the blueprint: image, cpu, memory, ports, env, secrets,
log configuration, and the two IAM roles
SERVICE keeps N tasks running, registers them with a target group,
performs rolling or blue/green deployments
CLUSTER a logical grouping; with Fargate it holds no infrastructure
TWO ROLES, ROUTINELY CONFUSED
executionRoleArn used by the ECS AGENT: pull the image from ECR,
fetch secrets, write logs
taskRoleArn used by YOUR CODE: the application's AWS permissions{ "family": "payments-api",
"requiresCompatibilities": ["FARGATE"],
"cpu": "1024", "memory": "2048",
"runtimePlatform": { "cpuArchitecture": "ARM64" }, # Graviton
"executionRoleArn": "arn:aws:iam::1234:role/ecsTaskExecution",
"taskRoleArn": "arn:aws:iam::1234:role/payments-api",
"containerDefinitions": [{
"name": "api",
"image": "1234.dkr.ecr.eu-west-1.amazonaws.com/payments-api@sha256:...",
"portMappings": [{ "containerPort": 8080 }],
"secrets": [{ "name": "DB_PASSWORD",
"valueFrom": "arn:aws:secretsmanager:...:prod/db:password::" }],
"healthCheck": { "command": ["CMD-SHELL","curl -f localhost:9090/health || exit 1"],
"interval": 10, "retries": 3, "startPeriod": 90 },
"stopTimeout": 60,
"logConfiguration": { "logDriver": "awslogs", "options": {...} } }] }Service Connect provides service discovery and a managed proxy for retries and connection management — the ECS equivalent of a Service plus a light mesh. It gives you name resolution and basic resilience without the operational surface of a full mesh.
stopTimeout is the ECS equivalent of terminationGracePeriodSeconds, and the deregistration delay on the target group is the equivalent of a preStop sleep. The same graceful-shutdown discipline applies; only the field names change.
EKS
Standard Kubernetes with a managed control plane. You get the entire ecosystem and portability; you take on upgrades, add-ons and a platform team's worth of operational surface.
AWS runs the API server and etcd across multiple zones and you do not see them. Everything else is yours: node groups or Karpenter, the CNI, CoreDNS, kube-proxy, CSI drivers, the autoscaler, ingress, policy, observability — each with its own version compatibility matrix.
WHAT YOU TAKE ON
control plane version upgrades, roughly annual, with removed-API breakage
add-on lifecycle: VPC CNI, CoreDNS, kube-proxy, EBS CSI, autoscaler
IRSA or Pod Identity setup
admission policy, network policy enforcement
an observability stack
EKS-SPECIFIC THINGS TO KNOW
VPC CNI gives every POD a real VPC address
-> subnets sized for pods, not nodes
-> pod density capped by ENIs per instance type
-> ENABLE_PREFIX_DELEGATION raises that ceiling substantially
NetworkPolicy is NOT enforced unless explicitly enabled (or Calico/Cilium)
IMDSv2 with hop limit 1 - the single highest-value hardening step
access entries (or the older aws-auth ConfigMap) map IAM to cluster identityEKS Auto Mode has moved the calculus: AWS manages node provisioning, scaling and core add-ons, which removes a real slice of the burden. It does not remove control plane upgrades, deprecated-API breakage, or the need to understand Kubernetes when something breaks at 3am — but acknowledging that the answer is closer than it was two years ago demonstrates current knowledge.
The control plane charge is negligible. The real cost of EKS is nodes and ENGINEERING TIME - which is why the ECS-versus-EKS decision is a staffing decision as much as a technical one.
Fargate, and choosing a platform
Fargate is a launch mode, not a competitor. The real axis is ECS versus EKS, and for a small team without a platform engineer, ECS on Fargate is usually the honest answer.
Fargate removes node management from either ECS or EKS at a per-vCPU and per-GB premium of roughly 15–25%, with some capability loss: no DaemonSets in the usual sense, no privileged containers, no GPUs, and smaller maximum task shapes. Underneath it runs microVMs, which is why the isolation is stronger than shared-kernel containers.
ECS + Fargate EKS + Fargate EKS + EC2
time to first deploy ~1 day 1-2 weeks 2-4 weeks
ongoing ops burden near zero moderate HIGH
ecosystem AWS only full k8s full k8s
portability none real real
compute premium ~15-25% ~15-25% 0
8 engineers, 12 Java services, no platform team
-> ECS on Fargate. The premium is far cheaper than the fraction of an
engineer that operating EKS actually consumes.
CHANGE THE ANSWER IF
the team already runs Kubernetes competently
you need the ecosystem specifically (operators, Argo Rollouts, KEDA, a mesh)
portability is a REQUIREMENT, not a slide
you are heading to dozens of services across several teamsMigrating from ECS to EKS later is real but bounded work: the container images and twelve-factor discipline transfer unchanged, and what you rewrite is the orchestration layer. So starting on ECS is not a trap, provided application code stays free of platform-specific assumptions.
Choose the least infrastructure that meets the requirement. If nobody owns the platform, you have not chosen Kubernetes - you have chosen an unmaintained cluster, which is worse than ECS by every measure.
Lambda
One execution environment per concurrent request, reused between invocations. Cold starts, concurrency and connection pools are what break when a JVM service is lifted into it unchanged.
Lambda creates an execution environment per concurrent request. Within one environment invocations are serial. Code outside the handler runs once per environment and is reused — so clients, connection pools and anything expensive to construct belong there. Creating an S3 client per invocation is the most common Lambda performance bug.
// WRONG - per invocation
public Response handleRequest(Event e, Context ctx) {
var s3 = S3Client.create(); // ~200ms, EVERY time
var conn = DriverManager.getConnection(URL); // a new DB connection each time
}
// RIGHT - init once per environment
private static final S3Client S3 = S3Client.create();
private static final DataSource DS = pool(2); // SMALL pool
static { warmUp(); }WHAT BREAKS ON A NAIVE LIFT-AND-SHIFT
COLD STARTS a Spring Boot context is 5-10 SECONDS
fixes: SnapStart (snapshot the initialised JVM), provisioned
concurrency (paying for idle - which defeats the point),
a leaner framework, or not using Lambda for this workload
CONCURRENCY per-account and per-region, shared across every function
300 concurrent x 5 connections = 1,500
RDS max_connections = 200 -> INSTANT OUTAGE for every other
service on that database
fixes: reserved concurrency as a CAP, a tiny pool per
environment, and RDS Proxy to multiplex
HARD LIMITS 15-minute timeout | 6 MB sync payload | 10 GB memory
CPU is proportional to MEMORY - a CPU-bound function gets
faster AND often cheaper by being given more memoryError handling depends entirely on invocation type, which trips people up. Synchronous: no retry, the error returns to the caller. Asynchronous: two retries then a destination or DLQ if configured, otherwise SILENT LOSS. Poll-based: the event source's semantics apply. Always configure a failure destination for async.
S3 fundamentals
Eleven nines of durability, lower availability, and strong read-after-write consistency since 2020. The cost trap is requests and object count, not bytes.
Durability — the probability an object survives — is designed at eleven nines by replicating across at least three availability zones. Availability — the probability you can reach it right now — is a much lower 99.99%. So S3 essentially never loses data and can be briefly unreachable, which means your client needs retries.
Since December 2020 S3 is strongly read-after-write consistent for PUTs of new objects, overwrites and deletes, at no cost and with no configuration. A generation of blog posts and interview answers is built on the old eventually-consistent behaviour, so knowing it changed is a small current-knowledge signal. Bucket configuration changes remain eventually consistent.
STORAGE CLASSES - trading storage price against RETRIEVAL price and latency
Standard no retrieval fee, ms access
Standard-IA cheaper storage, retrieval fee, 30-day minimum,
128KB minimum billable size
One Zone-IA cheaper again, single AZ - only for RE-DERIVABLE data
Glacier Instant cheap, ms access, 90-day minimum
Glacier Flexible minutes to hours retrieval
Deep Archive cheapest, hours, 180-day minimum
Intelligent-Tiering moves objects automatically, small monitoring fee
-> the default when access patterns are unknownThe expensive mistake is not storage - it is REQUESTS. Millions of small PUTs cost more in requests than in bytes, and lifecycle transitions are billed PER OBJECT, so moving ten million tiny files to Glacier can cost more than leaving them in Standard forever.
S3 in production
Presigned URLs so your service never proxies bytes, event notifications to SQS rather than Lambda, and a lifecycle policy that aborts incomplete multipart uploads.
Presigned URLs
For uploads of any size, your service authorises and signs while the client uploads directly to S3 — so bandwidth and memory never touch your instances. Constrain the signature narrowly: short expiry, an exact key chosen server-side, a content-length range, a pinned content type. A presigned URL is a bearer token, so scope is the only control.
var key = "uploads/%s/%s".formatted(tenantId, UUID.randomUUID()); // SERVER chosen
var presign = s3Presigner.presignPutObject(b -> b
.signatureDuration(Duration.ofMinutes(10))
.putObjectRequest(r -> r
.bucket("merchant-uploads-prod")
.key(key) // exact object
.contentType("application/pdf")
.contentLength(declaredSize) // client cannot exceed
.serverSideEncryption(ServerSideEncryption.AWS_KMS)));
uploadRepo.record(tenantId, key, declaredSize, PENDING); // we know what to expectLIFECYCLE POLICY EVERY BUCKET SHOULD HAVE
{ "Rules": [
{ "Id": "abort-orphaned-multipart", // THE ONE EVERYONE FORGETS
"Status": "Enabled",
"AbortIncompleteMultipartUpload": { "DaysAfterInitiation": 7 } },
// incomplete parts are billed INDEFINITELY and do not appear in a listing
{ "Id": "tier-large-objects-only",
"Filter": { "And": { "Prefix": "settlements/",
"ObjectSizeGreaterThan": 131072 } }, // 128KB
"Transitions": [ { "Days": 30, "StorageClass": "STANDARD_IA" },
{ "Days": 180, "StorageClass": "GLACIER_IR" } ] },
{ "Id": "expire-old-versions", // versioning without this = never delete
"NoncurrentVersionExpiration": { "NoncurrentDays": 90 } } ] }Trigger processing with an event notification to SQS, not directly to Lambda: the queue gives you buffering, per-message retry, a dead-letter queue, and controlled concurrency so a bulk upload does not instantly spawn hundreds of Lambdas and exhaust your database connections.
Making a bucket genuinely private: Block Public Access at account AND bucket level (it overrides any policy anyone later writes), a bucket policy denying requests without aws:SecureTransport, access via VPC endpoints with an endpoint condition, and Access Analyzer reporting anything reachable from outside.
EBS and instance storage
EBS is network-attached, AZ-scoped and persists independently of the instance. Instance store is local NVMe, extremely fast, and gone when the instance stops.
gp3 general purpose SSD. Baseline 3,000 IOPS and 125 MB/s INDEPENDENT of
size - which is the improvement over gp2, where performance scaled
with capacity. Provision more IOPS and throughput separately.
io2 provisioned IOPS, higher durability, for demanding databases
st1 throughput-optimised HDD, for large sequential reads
INSTANCE STORE local NVMe, hundreds of thousands of IOPS,
EPHEMERAL: lost on stop, terminate or host failureThe property that shapes architecture is that an EBS volume lives in one availability zone and cannot be attached from another. So an instance with an EBS volume is pinned to its zone, and recovering from a zone failure means restoring from a snapshot into another zone — not moving the volume.
Snapshots are incremental, stored in S3, and regional rather than zonal — which is how you move a volume between zones or regions. They are also crash-consistent by default: a snapshot of a running database captures the disk as it was at an instant, which for a mid-write database is a torn state. Application-consistent snapshots need the filesystem quiesced or the database's own mechanism.
Instance store is right for caches, scratch space, and anything you can rebuild - and never for a primary copy. Its speed is genuine and its ephemerality is absolute: a STOP loses it, not only a termination.
Unattached volumes and orphaned snapshots are a recurring silent cost. They survive the instance, they survive a StatefulSet deletion in Kubernetes, and nothing reports them by default - so a scheduled check for volumes in the 'available' state is worth having.
RDS
Multi-AZ is availability with a non-readable standby; read replicas are scale with lag. They solve different problems and production usually needs both.
Multi-AZ maintains a synchronous standby in another zone: a commit is not acknowledged until it is durable on both. That is why it roughly doubles cost and slightly raises write latency. The standby is not readable — it exists purely so that when the primary fails, AWS promotes it and updates the DNS endpoint. Failover takes typically 60–120 seconds and every open connection is dropped.
Read replicas replicate asynchronously and are readable with their own endpoints. They scale reads, can live in another region, and can be promoted manually. But they lag — usually milliseconds, occasionally seconds under write bursts — so reading your own write through one is a real and common bug.
MULTI-AZ synchronous, NOT readable, ~60-120s failover
-> AVAILABILITY
READ REPLICA asynchronous, readable, lags
-> READ SCALE, and read-your-own-writes is NOT guaranteed
ROUTE READS CORRECTLY
writer endpoint all writes, plus reads inside a read-your-writes window
reader endpoint reporting, dashboards, exports, analytics
# Alarm on ReplicaLag, and check it in the application for anything
# correctness-sensitive - a health check can refuse to serve stale reads.Other things RDS gives you: automated backups with point-in-time recovery to any second in the retention window, minor version patching in a maintenance window, parameter groups, Performance Insights for query-level diagnosis, and encryption at rest with KMS (which must be enabled at creation — you cannot encrypt an existing instance in place).
Multi-AZ DB CLUSTER deployments are a newer option with two READABLE standbys and much faster failover. If someone says they offload reporting to their Multi-AZ standby, they mean either that or a read replica - classic Multi-AZ has no readable standby at all.
Aurora
Storage decoupled from compute and replicated six ways across three zones. Replicas read the same storage rather than replaying a log, which changes both lag and failover time.
Aurora separates the database engine from a distributed storage layer that replicates every write six ways across three availability zones, with quorum-based durability. The engine writes log records rather than pages, which is why write amplification is much lower than a traditional engine over network storage.
CONSEQUENCES OF THE SHARED STORAGE LAYER
replicas read the SAME storage -> lag typically well under 100ms,
not a log replay
up to 15 replicas
failover in SECONDS, not a minute - there is no data to move, a reader
is simply promoted
storage grows automatically to 128 TiB; you pay for what you use
fast cloning - a copy-on-write clone of a production database for testing,
created in minutes regardless of size
backtrack (MySQL) - rewind the cluster in place, without a restore
ENDPOINTS
cluster endpoint always the writer
reader endpoint load-balances across replicas
instance endpoints a specific instanceAurora Serverless v2 scales capacity in fine-grained increments and is genuinely useful for variable or unpredictable workloads. Aurora Global Database replicates to another region with typical lag well under a second and promotion in about a minute — which is what makes a five-second RPO achievable across regions.
Aurora is not automatically the right answer. It costs more per instance-hour than RDS, and for a small, steady, single-region workload plain RDS Postgres with Multi-AZ is simpler and cheaper. Aurora earns its cost with read scale, fast failover, or a cross-region requirement.
Failover, and what actually breaks
AWS promotes a standby in 60–120 seconds. Your Java service is often down for much longer, because the JVM cached the DNS resolution and the pool holds sockets to a machine that no longer exists.
This is the gap that matters and the one people do not test: the difference between AWS recovered and we recovered. Failover repoints a DNS name. If your client does not re-resolve it, nothing improves.
WHAT AWS DOES ~60-120s
detect failure -> promote the standby -> repoint the DNS CNAME
WHAT AN UNCONFIGURED JVM DOES
java.security caches the OLD address (aggressively, by default in some setups)
HikariCP holds 20 sockets to an instance that no longer exists
every borrow blocks until the OS TCP timeout - which can be MINUTES
=> "the database failed over in 90 seconds but we were down for 10 minutes"
FIX IN THREE PLACES
# 1. JVM
-Dsun.net.inetaddr.ttl=5 # or networkaddress.cache.ttl=5
# 2. Pool
spring.datasource.hikari:
connection-timeout: 3000 # fail fast, do not block a request thread
validation-timeout: 1000
keepalive-time: 30000
max-lifetime: 600000 # recycle well below any idle timeout
# 3. Driver - failover-aware, handles topology changes directly
spring.datasource.url: jdbc:aws-wrapper:postgresql://cluster.cluster-abc...:5432/app
spring.datasource.driver-class-name: software.amazon.jdbc.DriverTest it deliberately with a forced failover — AWS Fault Injection Service can trigger one — and measure the application's recovery time, not the database's. That number is the most valuable single output of a chaos exercise, because it is almost always worse than people assume and it is entirely fixable with configuration.
The same discipline applies to caches and queues after any failover: expect a COLD CACHE in the new topology, and make sure the resulting miss storm cannot overwhelm the newly promoted database. That cascading failure is a very common way a technically successful failover still causes an outage.
DynamoDB: the key is the query plan
Design the access patterns first, then the key. In a relational database you model the data and query it later; here the key IS the query plan, and getting it wrong is not a slow query but an access pattern you cannot serve.
The partition key is hashed to choose a physical partition, so it determines placement and how load spreads. The sort key orders items within a partition and gives you range queries, prefix matching and reverse ordering — all cheap, because they are a contiguous read of one partition.
ACCESS PATTERNS FIRST - write them down before the schema
1. transactions for an account, date range, newest first <- primary
2. one transaction by id
3. transactions for a merchant, date range <- needs a GSI
4. pending transactions needing settlement <- sparse GSI
TABLE: transactions
PK ACCOUNT#{accountId}
SK TXN#{iso8601}#{txnId} # range + uniqueness
# 1. one Query, one partition, sorted, paginated
Query PK = "ACCOUNT#123"
SK between "TXN#2024-03-01" and "TXN#2024-04-01"
ScanIndexForward = false # newest firstHot partitions
A single key taking disproportionate traffic hits physical per-partition ceilings — roughly 3,000 read units and 1,000 write units per second — so one large merchant can throttle while the table as a whole is far under capacity. Adaptive capacity absorbs moderate skew; it does not absorb an unbounded hotspot.
FIXES
shard suffix PK = ACCOUNT#123#{hash(txnId) % 10}
reads fan out across shards and merge
time bucket PK = ACCOUNT#123#2026-08
also bounds partition size
THE PATHOLOGICAL CASE
a monotonically increasing partition key (a raw timestamp)
-> every write lands on the same partition
-> you have built a queue with one laneThe SPARSE INDEX is the most useful DynamoDB trick: write the indexed attribute ONLY when status is PENDING. Items without it are absent from the index, so the index stays tiny and scanning it is cheap - and deleting the attribute on settlement removes the item from the index. Ideal for work queues.
DynamoDB in production
Conditional writes are the idempotency primitive. GSIs are separate tables with their own capacity. Streams give you a change log you did not have to build.
// IDEMPOTENCY - a CONDITION, not a read-then-write. No race, no lock.
try {
ddb.putItem(r -> r.tableName("processed_events")
.item(Map.of("pk", s("EVT#" + eventId),
"ttl", n(Instant.now().plus(30, DAYS).getEpochSecond())))
.conditionExpression("attribute_not_exists(pk)")); // <- the whole trick
process(event); // first time only
} catch (ConditionalCheckFailedException dup) {
log.debug("duplicate delivery of {}", eventId); // safe no-op
}
// OPTIMISTIC CONCURRENCY - no SELECT FOR UPDATE, no lock table
ddb.updateItem(r -> r.tableName("accounts")
.key(Map.of("pk", s("ACCOUNT#123")))
.updateExpression("SET balance = :new, version = :nv")
.conditionExpression("version = :cur") // fails if someone else won
...);- GSI — effectively a separate replicated table with its own key and its own capacity, eventually consistent. Project only the attributes you read, because projection drives storage and write cost. And an under-provisioned GSI throttles writes to the base table, which is a genuinely surprising failure.
- LSI — shares the partition key with an alternative sort key, must be created with the table, and caps that partition at 10GB. GSIs are used far more often.
- Transactions — ACID across up to 100 items at roughly twice the cost, with condition expressions per item.
- Streams — an ordered change log per partition key, retained 24 hours, driving projections, search indexing or event fan-out. Because it derives from the transaction log, it is effectively a built-in outbox.
ON-DEMAND per request, absorbs spikes instantly, scales to zero cost
right for unpredictable or new workloads
PROVISIONED reserved capacity with auto scaling, meaningfully cheaper at
steady high volume - but auto scaling reacts in MINUTES,
so it does not protect you from a sudden spike
Rule of thumb: peak-to-average above ~4:1 -> on-demand usually wins.
Start on-demand, measure for a month, then decide.Be able to argue single-table design BOTH ways. For: one Query returns related entities of different types in a single round trip. Against: the schema is opaque, hard to onboard onto, and a genuinely new access pattern can require a migration rather than a new index.
Choosing a datastore
Reason from access pattern, consistency requirement, loss tolerance and — the one people skip — what someone will ask of this data in a year that nobody has thought of yet.
SESSIONS LEDGER DOCUMENTS (40TB)
access pattern GET/PUT by key txn writes + PUT once,
AD-HOC SQL GET rarely
consistency eventual is fine STRICT, ACID n/a
loss tolerance inconvenience unacceptable unacceptable
query flexibility none HIGH (unknown metadata only
future questions)
latency sub-ms low ms seconds ok
------------------------------------------------------------------------
CHOICE ElastiCache Aurora PostgreSQL S3 + DynamoDB
(or DynamoDB TTL) Multi-AZ + readers (Intelligent-Tiering)The ledger is the one where reaching for DynamoDB would be the wrong instinct, and saying so explicitly is the point. You need multi-row ACID transactions, foreign keys, and ad-hoc queries during reconciliation and investigation that nobody designed for. Immutability of a ledger is achieved with an append-only schema and permissions — INSERT only, corrections as reversing entries — not by choosing a different database.
-- Immutability is schema and permissions, not a product choice
CREATE TABLE ledger_entry (...);
REVOKE UPDATE, DELETE ON ledger_entry FROM app_role; -- INSERT only
-- Cost sanity check on the documents
40 TB S3 Standard ~ $920 / month
40 TB Glacier Instant Retrieval ~ $160 / month
40 TB on RDS gp3 storage ~$4,600 / month <- and backups multiply it
-- blobs never belong in a relational databaseMigrating a datastore is a project; choosing one is a meeting. That asymmetry is why this is one of the four irreversible decisions - and why 'what will someone ask of this data in a year' deserves more weight than today's access pattern.
Schema migration without downtime
Expand, backfill, flip, contract — in separate releases. It is the only pattern where rollback remains possible at every step, which is what 'zero downtime' actually requires.
The failure people design around is downtime during the DDL. The failure that actually hurts is a deploy you cannot roll back, because the schema has already moved forward and the previous application version no longer works against it.
RELEASE 1 - EXPAND. Old code still works. Rollback is FREE.
SET lock_timeout = '3s'; -- do not wait behind a long read
ALTER TABLE settlement ADD COLUMN currency_code char(3); -- nullable: INSTANT
-- app v2: writes BOTH columns, reads the old one
RELEASE 1.5 - BACKFILL in bounded batches, watching replica lag
UPDATE ... WHERE id >= lo AND id < hi AND currency_code IS NULL; -- 5k rows
COMMIT; check replica lag; pause if > 5s; sleep 50ms; repeat
-- NEVER one UPDATE over 400M rows: locks held for the duration, enormous WAL,
-- table bloat, autovacuum overwhelmed, REPLICA LAG through the roof
RELEASE 2 - FLIP READS behind a flag. Old code STILL works.
RELEASE 3 - CONTRACT. Only once you will never roll back past release 2.
ALTER TABLE settlement DROP COLUMN currency;WHAT IS ACTUALLY ONLINE (PostgreSQL)
ADD COLUMN nullable instant, metadata only
ADD COLUMN NOT NULL DEFAULT const instant in PG 11+, FULL REWRITE before
ADD CONSTRAINT use NOT VALID, then VALIDATE separately
CREATE INDEX CONCURRENTLY, or you lock WRITES for the
whole build - minutes on 400M rows
DROP COLUMN instant (space reclaimed later)
THE OPERATIONAL DETAIL THAT CATCHES PEOPLE
a migration WAITING for a lock QUEUES every subsequent query behind it.
One long-running read turns an "instant" DDL into a full outage.
-> SET lock_timeout and RETRY. Never let a migration block indefinitely.The invariant: at EVERY point, version N-1 of the application works against the CURRENT schema. That makes rollback a code-only operation - fast, safe and boring. Without it, your rollback plan is a database restore measured in hours.
Run migrations as a separate step BEFORE the service deploy - a standalone task or a pre-sync hook, not an init container. With three replicas, three init containers race to run the same migration, which most tools serialise with an advisory lock and turn into a confusing slow start.
Why asynchronous
Because it converts a timeout into a queue. A synchronous API under overload loses work; an asynchronous one grows a metric you can alarm on and scale against.
Go asynchronous when the work exceeds a couple of seconds, when the downstream cannot absorb the arrival rate, or when the work must survive a client disconnect. The shape: validate, persist the request with an idempotency key, enqueue, and return 202 Accepted with a status URL.
SYNCHRONOUS under overload
connections pile up -> threads exhausted -> 504 from the load balancer
-> the work is LOST, and the client retries, adding more load
ASYNCHRONOUS under the same load
queue depth grows -> a metric you ALARM on and SCALE against
-> nothing is lost
client dies -> the work continues, the result is still there
client retries -> the idempotency key makes it a no-opPOST /settlements/reconcile
Idempotency-Key: 7f3a-... # client-supplied, required
-> 201 Accepted
Location: /jobs/abc123
{ "jobId": "abc123", "status": "PENDING" }
# server: conditional insert on the key, then enqueue.
# duplicate key -> return the EXISTING jobId, do not enqueue again
GET /jobs/abc123
{ "status": "RUNNING", "progress": 0.4 }
{ "status": "SUCCEEDED", "result": "/settlements/reports/2026-08" }
{ "status": "FAILED", "error": { "code": "PARTNER_FILE_MALFORMED" } }For clients that want a synchronous feel, use long polling with a deadline: accept the request, wait a bounded few seconds, and return the result if it completed or 202 with a status URL if it did not. The common fast case feels synchronous; the slow case is not a failure.
Return Retry-After on the status endpoint and prefer a push channel for anything long-running. Polling every 100ms across thousands of clients is a self-inflicted load problem, and the status endpoint is often the first thing to fall over in an otherwise well-designed async system.
SQS
Receiving hides a message for the visibility timeout; you delete it after success. That means at-least-once delivery, never exactly-once, and idempotency is the consumer's job.
SQS does not delete a message when you receive it — it makes it invisible for the visibility timeout. You process, then call DeleteMessage. If your consumer crashes, the timeout expires and the message reappears. That is the entire reliability model.
VISIBILITY TIMEOUT must exceed your p99 processing time
too short -> a slow-but-succeeding job is redelivered while still running
-> two consumers process concurrently -> double writes
too long -> a genuinely crashed consumer's message is stuck for that period
long jobs -> HEARTBEAT with ChangeMessageVisibility rather than a huge value
DEAD-LETTER QUEUE
RedrivePolicy: maxReceiveCount 4 -> the message moves to a DLQ
without one, a POISON MESSAGE is redelivered forever - and on a FIFO queue
it blocks its entire message group permanently
alarm on ANY non-zero DLQ depth, and document the redrive path
STANDARD vs FIFO
standard near-unlimited throughput, at-least-once, best-effort ordering
(so never rely on order)
FIFO strict ordering and dedup WITHIN a message group, 5-minute dedup
window, much lower throughput (higher with high-throughput mode)
MessageGroupId is BOTH your ordering guarantee AND your
parallelism unit@SqsListener("settlements")
public void handle(SettlementMessage msg, Acknowledgement ack) {
// idempotency key from CONTENT, never from the SQS messageId
// (which changes on redelivery)
String key = "SETTLE#" + msg.fileId() + "#" + msg.rowRange();
try {
ddb.putItem(r -> r.tableName("processed")
.item(Map.of("pk", s(key), "ttl", n(ttl30d())))
.conditionExpression("attribute_not_exists(pk)"));
} catch (ConditionalCheckFailedException dup) {
ack.acknowledge(); return; // already done - delete, do not redo
}
try (var hb = visibility.heartbeat(msg, ofSeconds(60))) { process(msg); }
ack.acknowledge(); // delete ONLY after success
}Exactly-once delivery does not exist, because processing and deletion are in two systems with no atomic commit between them. Exactly-once EFFECT is achieved by making the processing idempotent - a conditional write on a business key, not a try/catch.
SNS and EventBridge
SQS is a queue, SNS is fan-out filtered on attributes, EventBridge routes on the event body with archive and replay. For a domain event with several consumers, EventBridge targeting per-consumer queues is the pattern.
SQS a queue: one logical consumer, per-message retry, DLQ, buffering
SNS pub/sub fan-out, filtered on message ATTRIBUTES, no retention
EventBridge a bus: content-based routing on the whole event BODY,
schema registry, ARCHIVE and REPLAY, many native targets// Producer: one call. Knows nothing about consumers.
eventBridge.putEvents(r -> r.entries(e -> e
.eventBusName("payments")
.source("payments.settlement")
.detailType("PaymentSettled")
.detail(json(Map.of(
"eventId", UUID.randomUUID().toString(), // consumer idempotency key
"version", "1.0",
"settlementId", id, "merchantId", merchantId,
"amountMinor", 4250, "currency", "GBP")))));
// Each consumer owns a RULE and a QUEUE. The producer never changes.
Rule "treasury-large-gbp" -> SQS treasury-queue
{ "detail-type": ["PaymentSettled"],
"detail": { "currency": ["GBP"],
"amountMinor": [{ "numeric": [">", 1000000] }] } }
// ^ content-based on the BODY. SNS filters only on attributes.The queue in front of each consumer is the failure isolation boundary. A slow or broken consumer accumulates depth in its own queue and affects nobody — not the producer, not the other consumers. Each gets its own visibility timeout, retry policy, DLQ and consumption rate. Deliver straight to a Lambda and a downstream outage becomes retries, then loss, and a spike becomes a concurrency storm that exhausts a shared account limit.
The event schema is a public contract: version it, add fields additively, never remove or repurpose one. And publish enough that consumers do not need to call back to enrich - an event carrying only an id forces every consumer to make an API call, which recreates the coupling you were removing AND creates a load spike on the producer at the moment it published.
The dual-write problem and the outbox
Writing to your database and publishing an event are two systems with no atomic commit. The outbox makes the intent to publish part of the same local transaction.
Crash between the two and you get either a state change nobody heard about or an event for a change that rolled back. Ordering does not fix it: publish-then-commit gives phantom events, commit-then-publish gives lost events, and wrapping the publish in the transaction achieves nothing because a broker acknowledgement cannot be rolled back.
@Transactional
void settle(Settlement s) {
repo.save(s);
outbox.insert(new OutboxRow(
UUID.randomUUID(), // eventId - the consumer's idempotency key
"PaymentSettled",
s.getAccountId(), // ORDERING KEY
json(s), // FULL PAYLOAD, snapshotted at write time
Instant.now(), null)); // sentAt
} // ONE database, ONE commit, atomic and fully unit-testable
// RELAY (poller). SKIP LOCKED prevents two instances double-publishing.
@Scheduled(fixedDelay = 500)
void relay() {
var sql = "SELECT * FROM outbox WHERE sent_at IS NULL "
+ "ORDER BY created_at LIMIT 100 FOR UPDATE SKIP LOCKED";
for (var r : jdbc.query(sql, MAPPER)) {
eventBridge.putEvents(toEntry(r)); // may publish then crash...
jdbc.update("UPDATE outbox SET sent_at = now() WHERE id = ?", r.id());
} // ...so it republishes. Fine.
}Store the full payload in the row, not just an id. Otherwise the relay must re-read the entity and may see a later state, publishing an event that does not match the transaction it came from.
The guarantee is at-least-once publication corresponding exactly to committed state. NOT exactly-once - the relay can publish and crash before marking the row sent. So consumer idempotency is the contract, not a footnote; a design without it has moved the bug rather than fixed it.
DynamoDB Streams gives you this for free: the stream derives from the transaction log, so every record corresponds to a committed change, ordered per partition key. That is a genuinely good reason to reach for DynamoDB on event-heavy write paths.
Change data capture
Read the transaction log and emit every committed change. No application code, no dual-write problem — and events that are row changes rather than business intent.
CDC reads the database's write-ahead log (Postgres) or binlog (MySQL) and emits committed changes as events. On AWS that is DMS or Debezium, typically into MSK. Three properties follow directly: no application changes, no dual-write problem because the log is the record of committed state, and complete ordering per key.
THE TRADE
CDC gives: { table: "settlement", op: "U",
before: { status: "PENDING" }, after: { status: "SETTLED" } }
outbox gives: { type: "PaymentSettled", settlementId: ..., amountMinor: ... }
CDC couples consumers to your INTERNAL SCHEMA.
A column rename becomes a breaking change for every downstream team.
That is the main argument FOR an outbox with deliberately designed events.Where CDC is clearly right: integrating with systems you do not control, replicating to a data platform, building search indexes, and migrating databases — which is exactly what DMS is designed for. Where an outbox is right: publishing domain events that other teams will build against.
DMS is genuinely used for ongoing replication, not only migrations, and is the low-effort choice. Debezium is more capable - better transformation and schema handling - at the cost of running it yourself on MSK Connect or Kubernetes.
Streams: Kinesis and MSK
Choose a log over a queue when you need replay from an offset, multiple independent consumer groups, or per-key ordering. The tell: will anyone ever want to read this history again?
CHOOSE A LOG WHEN YOU NEED
replay from an offset a consumer had a bug -> re-read last week,
with no backfill project
multiple consumer GROUPS each at its own pace; adding a 5th does not
affect the other 4
strict ordering within a key guaranteed by partition
retention as a source of truth rebuild projections from scratch
CHOOSE A QUEUE WHEN
simple work distribution, per-message retry, a DLQ, no cluster to operate,
and you will never need to re-read.
QUEUES CONSUME. LOGS RETAIN.OPERATIONAL COST - where teams underestimate
MSK brokers, storage, and PARTITION COUNT planning (hard to change: more
partitions breaks key-to-partition mapping and therefore ordering)
CONSUMER GROUP REBALANCING pauses consumption, and a slow consumer
can trigger a rebalance that slows everyone and triggers another
CONSUMER LAG must be monitored and ALERTED - a lagging consumer is
invisible until the retention window expires and data is GENUINELY
LOST. This is the metric people forget.
KINESIS no brokers, but shards with per-shard limits (1 MB/s in, 2 MB/s out)
and resharding to manageConsumer parallelism within a group is capped by partition count, so size partitions from your expected peak consumer count with headroom. Over-provisioning partitions is cheap; under-provisioning is a migration.
Alert on consumer lag measured in TIME, not only in records, and alarm well before it approaches the retention window. Past that point the data is gone, and no amount of scaling the consumer recovers it.
Step Functions and sagas
Durable state you did not write. Reach for it when a process spans services and outlives a request — and when you need compensation, because there is no distributed rollback.
A database transaction gives atomicity across rows in one database. It gives you nothing across services: you cannot roll back an HTTP call to a partner, a message already published, or a refund already issued. And holding a transaction open across a network call is actively harmful, pinning a connection and holding locks for someone else's latency.
WHAT IT GIVES YOU THAT YOUR OWN CODE DOES NOT
durable state every transition persisted -> a crash RESUMES
per-step retry backoff and jitter, declared not coded
catch and fallback route specific errors to compensating branches
parallel and map fan-out with a concurrency limit
WAIT states days, without a scheduler you maintain
callbacks waitForTaskToken - pause until a human or a partner responds
visual execution history - genuinely the best debugging artefact in AWS{ "StartAt": "ReserveFunds",
"States": {
"ReserveFunds": {
"Type": "Task", "Resource": "arn:aws:states:::lambda:invoke",
"Retry": [{ "ErrorEquals": ["States.TaskFailed"], "IntervalSeconds": 2,
"MaxAttempts": 4, "BackoffRate": 2.0, "JitterStrategy": "FULL" }],
"Next": "NotifyPartner" },
"NotifyPartner": {
"Type": "Task", "Resource": "...",
"Catch": [{ "ErrorEquals": ["States.ALL"], "Next": "ReleaseFunds" }],
"Next": "AwaitConfirmation" },
"AwaitConfirmation": {
"Type": "Task", "Resource": "arn:aws:states:::lambda:invoke.waitForTaskToken",
"TimeoutSeconds": 259200, // 3 days, no scheduler needed
"Catch": [{ "ErrorEquals": ["States.Timeout"], "Next": "ReleaseFunds" }],
"Next": "Settle" },
"ReleaseFunds": { "Type": "Task", "Resource": "...", "Next": "Fail" },
"Settle": { "Type": "Task", "Resource": "...", "End": true },
"Fail": { "Type": "Fail" } } }Standard workflows are durable with exactly-once transitions, up to a year long, billed per transition, with full history. Express workflows are up to five minutes, at-least-once, and cheap enough for high-volume event processing.
Keep the state machine THIN and the logic in ordinary code it invokes. Business logic in a JSON definition is hard to unit-test; business logic in a Lambda the machine calls is normal Java. Let the orchestrator own sequencing and durability, nothing else.
Idempotency
The single most important property in a distributed system that handles money. Derive a business key from content, claim it conditionally, and make replay a safe no-op.
Every mechanism in this part delivers at-least-once: SQS redelivers, the outbox republishes, a client retries a timed-out request, a failover replays. So the question is never how do I avoid duplicates — it is how do I make a duplicate harmless.
THE KEY
derive it from CONTENT, deterministically:
fileId + rowRange | clientIdempotencyKey | eventId from the producer
NEVER from something that changes on redelivery:
the SQS messageId, a timestamp, a random UUID generated by the consumer
THE CLAIM - a conditional write, not a read-then-write
DynamoDB conditionExpression("attribute_not_exists(pk)")
Postgres a UNIQUE constraint, and catch the duplicate-key violation
(the constraint does the work; the catch is just control flow)
Both are atomic. A read-then-write has a race and will eventually lose it.// The complete pattern
String key = "SETTLE#" + msg.fileId() + "#" + msg.rowRange();
try {
ddb.putItem(r -> r.tableName("processed")
.item(Map.of("pk", s(key), "ttl", n(ttl30d()))) // TTL: bounded growth
.conditionExpression("attribute_not_exists(pk)"));
} catch (ConditionalCheckFailedException dup) {
return AlreadyProcessed.INSTANCE; // safe no-op
}
process(msg);
// TTL matters: without it the table grows forever. Set it longer than any
// plausible redelivery window - 30 days is generous and cheap.Claim BEFORE doing the work, not after. Claiming afterwards leaves a window where a crash between the work and the claim causes the work to happen twice. Claiming first means a crash after the claim leaves work undone - which is recoverable by a reconciliation job, and is the far safer failure.
For a payments API, require an Idempotency-Key header from clients and honour it: return the ORIGINAL response for a repeated key rather than reprocessing. That turns client retries - which are constant at any real volume - from a correctness risk into a non-event.
CloudWatch metrics
Numeric time series with dimensions, aggregated progressively as they age. The cost trap is high-cardinality custom metrics — every unique dimension combination is billed separately.
Metrics are one-minute standard resolution or one-second high resolution, and they are aggregated as they age: high-resolution data is kept briefly, one-minute data for fifteen days, five-minute for sixty-three days, one-hour for fifteen months. So you cannot go back three months and look at per-minute detail — anything you will need at fine granularity later must be captured another way.
EVERY UNIQUE DIMENSION COMBINATION IS A SEPARATE BILLED METRIC
Latency{Service, Operation} ~20 metrics fine
Latency{Service, Operation, MerchantId} ~20 x 5,000 = 100,000 metrics
-> costs more than the service
RULE: dimensions must be LOW cardinality.
Service, Operation, Region, StatusClass - yes.
MerchantId, UserId, RequestId, full URL path - never.
Put high-cardinality context in LOGS or TRACES, not in metric dimensions.Embedded Metric Format solves the tension elegantly: your application emits a specially-structured JSON log line, and CloudWatch extracts metrics from it automatically. You get low-cardinality metrics for dashboards and alarms and high-cardinality context in the same event, without maintaining a separate metric filter.
{ "_aws": { "Timestamp": 1756000000,
"CloudWatchMetrics": [{ "Namespace": "Payments",
"Dimensions": [["Service","Operation"]], // LOW cardinality
"Metrics": [{ "Name": "Latency", "Unit": "Milliseconds" }] }] },
"Service": "payments-api", "Operation": "settle",
"Latency": 142,
"merchantId": "ACME-GB-0042", // high cardinality: stays in the LOG
"txnId": "9f2c..." }Container Insights and Lambda Insights give you per-container and per-function metrics without instrumentation, which is a fast way to get baseline visibility on an unfamiliar estate. They are billed per observation, so enable them deliberately rather than everywhere.
CloudWatch Logs
Log groups default to NEVER EXPIRE. That single default is the most common silent cost accumulation on AWS.
Logs are streams grouped into log groups, each with a retention setting — and the default is indefinite. A log group created automatically by a Lambda or an EKS add-on accrues storage cost forever, invisibly, until someone audits the bill. Set retention on every group and enforce it with a Config rule so new ones cannot escape.
METRIC FILTER evaluated AT INGESTION
matches a pattern and emits a CloudWatch metric
-> ALARMABLE, cheap, real time
-> but only sees events arriving AFTER you create it. Useless for the past.
LOGS INSIGHTS queries stored logs AFTER the fact
a real query language
-> billed PER GIGABYTE SCANNED. Narrow the time range.
THE RULE
metric filters for anything you want to ALARM on
Logs Insights for INVESTIGATION
if you run the same Insights query on a schedule, it should be a filterfields @timestamp, txn.id, error.message
| filter event = "settlement_failed"
| stats count() by error.code
| sort count desc
| limit 20
# Structured JSON logging makes this work. Unstructured prose means regex
# parsing at query time, which is slower, more expensive and more fragile.
logging.structured.format.console: ecs # Spring Boot 3.4+Drop noise BEFORE ingestion wherever you can - at the log shipper, or by not emitting it. CloudWatch bills on ingestion as well as storage, so filtering at query time means you have already paid for everything you filtered out.
Alarms worth having
Alarm on symptoms with a bounded remediation. Set treatMissingData deliberately, and build a dead-man's switch — because the failure that produces no data produces no alarm.
MetricName: Api5xxRate
Period: 60
EvaluationPeriods: 5
DatapointsToAlarm: 3 # 3 of 5 - tolerates a single blip
Threshold: 0.005
TreatMissingData: breaching # DEFAULT is 'missing', which means an alarm on a
# metric that STOPS being published NEVER FIRES -
# exactly the silent failure you wanted to catch
# COMPOSITE alarms cut noise:
# page only if error rate is high AND the service is receiving trafficSymptoms, not causes
Alarm on what users experience — latency, error rate, saturation. High CPU with healthy latency needs no human; a deadlocked connection pool takes you down at 5% CPU and never trips a CPU alarm. So CPU alarms produce false positives and false negatives, which is the worst combination available. Cause metrics belong on the dashboard for diagnosis, not in the alert rules.
ALARMS THAT EARN THEIR PLACE
5xx rate above threshold, sustained users are failing
p99 latency above the SLO users are suffering
SQS DLQ depth > 0 work is being lost
SQS oldest message age work is backing up
consumer lag approaching retention data will be LOST
RDS free storage, connection count a cliff you can see coming
NO TRAFFIC for N minutes <- the dead-man's switch.
Zero errors because zero
requests. Nothing else fires.If you cannot write a bounded remediation for an alarm, it should not page. Writing the runbook entry is a useful forcing function on alert quality - an alarm whose runbook says 'investigate' is not actionable, and that review often removes a third of a team's pages.
Tracing
A span is one timed operation with a parent; a trace is the tree across services. It is the only signal that answers where the latency went — and propagation is the hard part.
A metric is an aggregate you alarm on, cheap and detail-free. A log is an event with context, expensive at volume and searchable. A span has a parent, and the tree of spans is the only one of the three that crosses service boundaries coherently.
WHERE CONTEXT IS LOST - the traceparent header must survive every hop
thread pools a plain ExecutorService loses the thread-local
-> wrap it (ContextExecutorService)
async boundaries CompletableFuture, reactive schedulers
MESSAGE QUEUES no HTTP headers - inject into SQS message attributes
or Kafka headers, and extract on the consumer
(without this, every async workflow is a set of
ORPHAN traces)
scheduled jobs no incoming context - create a root span explicitly
Miss one hop and the trace FRAGMENTS into disconnected trees - which is worse
than no tracing, because it looks complete and hides the boundary you most
need to see.Instrument with OpenTelemetry and export where you like — X-Ray today, something else later, without re-instrumenting. The OTel Java agent instruments HTTP, JDBC, messaging and Spring at class-load time with no code change, injected by JAVA_TOOL_OPTIONS or by an operator, so twelve services get tracing from a configuration change.
# The collector is where cost is controlled
processors:
attributes/scrub:
actions: [{ key: db.statement, action: delete }] # SQL can contain PII
tail_sampling:
decision_wait: 10s
policies:
- { name: errors, type: status_code, status_code: { status_codes: [ERROR] } }
- { name: slow, type: latency, latency: { threshold_ms: 2000 } }
- { name: base, type: probabilistic, probabilistic: { sampling_percentage: 1 } }
# Head sampling at 1% is cheap and throws away exactly the traces you want.
# Tail sampling decides AFTER the trace completes.Put the trace id in every log line. Then a slow span leads to exactly the log lines emitted during it, and an error log leads to the full trace. One field, and it is worth more than either signal alone.
SLOs and error budgets
Pick a user-facing SLI, set a target, and derive alerting from the burn rate rather than from thresholds. The budget is what turns reliability into an engineering trade-off with a number.
SLI proportion of requests returning 2xx/3xx in under 300ms,
measured AT THE LOAD BALANCER
(closest to what the client experienced - it includes queueing and
connection time your application metrics never see)
SLO 99.9% over a rolling 30 days
-> ERROR BUDGET = 0.1% x 30 days = ~43 minutes of failure per monthBURN RATE - how fast you are consuming the budget
1x exactly on budget, exhausted at day 30
14.4x 2% of the budget in one hour -> the whole month gone in ~50 hours -> PAGE
6x 10% in six hours -> TICKET
MULTI-WINDOW, MULTI-BURN-RATE (the short window makes it RESOLVE quickly too)
- alert: FastBurn
expr: (error_ratio_1h > 14.4 * 0.001) and (error_ratio_5m > 14.4 * 0.001)
severity: page
- alert: SlowBurn
expr: (error_ratio_6h > 6 * 0.001) and (error_ratio_30m > 6 * 0.001)
severity: ticket
# Low-traffic caveat: ratios are noisy at low volume. Add a minimum-volume
# guard, or you page on three failed health checks at 4am.The budget's real value is the conversation it enables: budget remaining means ship features and take risks; budget exhausted means freeze and spend the sprint on reliability. That converts an argument between two people with opinions into a policy agreed in advance — and it only works if leadership agreed to it beforehand.
Pick the target from what users need and what you can afford, not from how many nines sound impressive. Measure current performance for a month first, and check it against your dependencies - an SLO tighter than what RDS or a partner API offers is arithmetically unachievable.
Investigating latency
Move edge inward and bisect by dimension — target, AZ, tenant, route, time. Guessing at causes is how an investigation takes a day; halving the space takes twenty minutes.
Start at the ALB access logs, because they are the only source with per-request detail including which target served it. Group by target and by availability zone first, because slow for some users most often means one bad instance or one impaired zone — and that is one query away.
-- 1. is it a subset of TARGETS or AZs?
SELECT target_az, target,
approx_percentile(target_processing_time, 0.99) AS p99, count(*)
FROM alb_logs WHERE day = '2026/08/29'
GROUP BY target_az, target ORDER BY p99 DESC;
-- one target at 4.2s and the rest at 0.09s -> ONE BAD INSTANCE. Done.
-- all targets in one AZ slow -> IMPAIRED ZONE. Evacuate.
-- 2. a subset of ROUTES? GROUP BY request_url
-- 3. a subset of TENANTS? requires tenant on the trace/log
-- 4. correlate with TIME deploy? scaling event? RDS failover?
p99 up while p50 is FLAT -> a TAIL, i.e. a SUBSET. Not everything is slow.THE USUAL CULPRITS AT THE BOTTOM
DynamoDB ThrottledRequests > 0 with spare capacity -> HOT PARTITION
(CloudWatch Contributor Insights -> top partition keys)
RDS Performance Insights -> top SQL by load, and the wait events
a query fine at 1k rows and catastrophic at 1M affects only
tenants who crossed the threshold
Cache hit rate dropped for one key space -> cold cache for that tenant
Network cross-AZ hop added by a scaling eventGray failures are harder than clean ones because the impaired component reports ITSELF healthy - the check runs inside the failure domain. Detect from OUTSIDE with client-side and cross-zone metrics, then evacuate deliberately with a zonal shift rather than waiting for automation to converge.
Runbooks and on-call
Written for someone who did not build the service and is being woken at 3am. Mitigation before diagnosis, causes ranked by frequency, and an explicit list of what not to do.
# RUNBOOK: settlement-api
## WHAT IT DOES, AND WHAT BREAKS FOR USERS
Serves merchant settlement views and drives nightly reconciliation.
IF DOWN: merchants cannot view or reconcile payouts. Funds movement is NOT
affected - no money is at risk. Sev-2 unless combined with a ledger alert.
## DASHBOARD + THE THREE METRICS THAT MATTER
1. 5xx rate 2. p99 latency 3. SQS queue depth
## ALERT: Api5xxHigh
Likely causes, MOST COMMON FIRST:
1. bad deploy (~60%) -> deployed in the last 30 min? ROLL BACK FIRST.
aws deploy stop-deployment --auto-rollback-enabled
2. RDS failover / connection exhaustion (~25%)
-> DatabaseConnections near max? rollout restart
(recycles the pool; raise a ticket for the leak)
3. partner API degraded (~10%)
-> flag settlement.partner_notify=off (degrades safely)
4. AZ impairment (~5%) -> zonal shift
## KNOWN FALSE ALARMS
02:00-02:06 nightly batch trips p99. Expected. Do not investigate.
## DO NOT
- do NOT restart the RDS instance (failover is 60-120s of full outage)
- do NOT purge the DLQ to clear the alarm - those are unprocessed settlements
- do NOT scale consumers above 40 - it exhausts the RDS pool and takes down
every other service on that database
## ESCALATE database #db-oncall | partner #payments-partners | comms J. Patel
## LAST TESTED 2026-08-12 game day (followed cold, 2 corrections made)Two things make it useful rather than decorative. The mitigation is a reversible operational action — roll back, scale, fail over, flip a flag — not a code fix, because nobody should be writing code at 3am. And the do-not section, which is the part most runbooks omit and the part that prevents the worst outcomes.
Test it by having someone who did not build the service follow it during a game day, and put the last-tested date at the top. An untested runbook is fiction - the commands have drifted, the dashboard was renamed, the escalation contact left.
KMS and encryption
Envelope encryption: KMS issues a data key, you encrypt locally, and only the small key transits KMS. The key policy is the root of trust — IAM alone cannot grant access to a key.
ENCRYPT
GenerateDataKey -> returns the data key in TWO forms: plaintext + encrypted
encrypt your data LOCALLY with the plaintext key
SCRUB the plaintext from memory
store the ENCRYPTED data key beside the ciphertext
DECRYPT
send the encrypted data key back -> get the plaintext -> decrypt locally
CONSEQUENCES
large payloads never transit KMS - cost scales with OPERATIONS, not volume
the CMK never leaves the HSM - the material cannot be exfiltrated
revoking the key policy makes every ciphertext UNDECRYPTABLE
-> a real control, not a theoretical oneReach for a customer-managed key when you need your own key policy, cross-account grants, your own rotation schedule, separation of duties between key administrator and data administrator, or — most importantly in a regulated context — the ability to deny access by changing the policy. That is the real control when data sits somewhere you do not fully trust: you cannot delete their copy, but you can make it undecryptable.
{ "Sid": "AllowSettlementService",
"Effect": "Allow",
"Principal": { "AWS": "arn:aws:iam::1234:role/settlement-worker" },
"Action": ["kms:GenerateDataKey","kms:Decrypt"],
"Resource": "*",
"Condition": {
"StringEquals": { "kms:EncryptionContext:purpose": "settlement" },
"Bool": { "aws:SecureTransport": "true" } } }
# ENCRYPTION CONTEXT is authenticated data bound to the ciphertext: decryption
# fails unless the same context is supplied. It also appears in CloudTrail,
# so key-usage logs carry business meaning. Free, and underused.- Request quotas are real: a high-volume service calling
Decryptper request will throttle. Data key caching (bounded by age, message count and bytes) trades a little key hygiene for a large reduction in calls. - Multi-region keys are required if a DR region must decrypt — and it is not fixable during an incident.
- Rotation creates new material while retaining the old, so nothing needs re-encrypting. It does not re-encrypt existing data, which is worth knowing before an auditor asks.
Secrets
Secrets Manager or Parameter Store, fetched at runtime through a role. No long-lived credential in an image, an environment variable set at build time, or a repository.
SECRETS MANAGER rotation (built-in for RDS, custom via Lambda),
cross-region replication, resource policies,
versioning with staging labels. Costs per secret.
PARAMETER STORE free standard tier, SecureString with KMS,
hierarchical paths. No built-in rotation.
RULE: anything with a rotation requirement or cross-account sharing ->
Secrets Manager. Configuration that happens to be sensitive ->
Parameter Store.// Fetch at runtime through the task/pod role - never bake it in
var secret = secretsClient.getSecretValue(r -> r.secretId("prod/payments/db"));
// ECS: injected by the AGENT (execution role), never visible in the task def
"secrets": [{ "name": "DB_PASSWORD",
"valueFrom": "arn:aws:secretsmanager:...:prod/db:password::" }]
// EKS: Secrets Store CSI driver - mounted as files, refreshed on rotation,
// and NOTHING is ever written to a Kubernetes Secret objectRotation that nothing consumes has not happened. A connection pool holding credentials from startup keeps using the old password until it recycles, so pair rotation with a bounded maxLifetime on the pool, or with a rollout. And for RDS, the dual-user rotation strategy exists precisely so there is always a valid credential during the change.
The leak that actually happens is not KMS or IAM - it is a Spring Boot /actuator/env endpoint exposed without authentication, printing the entire configuration including credentials. Keep actuator on a separate management port and disable the sensitive endpoints.
The edge
WAF stops undirected exploit traffic and scripted abuse. It knows nothing about your users, so it does not stop the authorisation defects that cause most real incidents.
CONTROL STOPS DOES NOT STOP
CloudFront volumetric noise; improves targeted logic abuse
global TLS latency
WAF managed rules SQLi, XSS, known-bad inputs, an authenticated user
scanners enumerating IDs
WAF rate-based rule credential stuffing, scraping low-and-slow abuse
Shield Advanced volumetric DDoS + COST application-layer logic
protection
TLS 1.2 + ACM + HSTS downgrade, expired certs anything above TLS
JWT authorizer unauthenticated access a valid token used for
someone else's data
API GW usage plans one partner starving others a partner abusing
their own quotaShield Standard is on by default. Shield Advanced adds enhanced detection, a response team, and — the part that matters commercially — cost protection, so a volumetric attack does not also arrive as a bill for the scaling it triggered.
Rate limiting belongs in layers, because each catches something different: WAF for crude per-IP abuse at the edge before it costs you anything, API Gateway usage plans for per-client fairness, and application-level limits keyed on the authenticated user or tenant — which is the only layer that can see this user is requesting an unusual number of other people's records.
Be honest about the ranking: for a payments API, AUTHORISATION DEFECTS cause far more real incidents than network attacks. WAF and Shield handle the noise; per-request authorisation, idempotency and audit prevent the incident that actually damages you.
Detection
GuardDuty finds behaviour no perimeter rule would; Security Hub aggregates; Access Analyzer answers what is reachable from outside. All three are delegated to a separate security account.
- GuardDuty — behavioural detection across CloudTrail, VPC flow logs, DNS logs, S3 data events and EKS audit and runtime activity. It finds credential exfiltration, reconnaissance, crypto mining and anomalous API calls — things no static rule would catch.
- Security Hub — aggregates findings from GuardDuty, Config, Inspector, Macie and Access Analyzer into a single posture view with standards-based scoring.
- Inspector — continuous vulnerability scanning of EC2, ECR images and Lambda, including rescanning already-published images as new CVEs appear.
- Macie — discovers sensitive data in S3, which is how you find the bucket nobody remembered contains card data.
- IAM Access Analyzer — external access findings (anything reachable from outside your organisation) and unused access findings (roles, permissions and credentials unused for ninety days).
ALERT ON BEHAVIOUR THAT SHOULD NEVER OCCUR, not on volume:
a workload role suddenly calling ListSecrets or ListBuckets
root account usage of any kind
CloudTrail or GuardDuty being disabled
an IAM user or access key being created in production
console login without MFA
STS assume-role from an unexpected source
a bucket policy change making something public
# a competent attacker makes very few, very targeted calls -
# volume-based alerting misses them entirelyDelegate GuardDuty and Security Hub to a dedicated SECURITY account rather than running them from the management account. The management account can modify SCPs and create accounts, so its compromise is total - and you do not want your detection to live inside the blast radius it is watching.
Evidence
CloudTrail records change and attribution; Config records state over time. Proving a control was in place CONTINUOUSLY is a Config question, and the evidence must survive compromise of what it records.
QUESTION SERVICE
"who disabled encryption on this bucket?" CloudTrail (change + attribution)
"was it encrypted continuously all year?" CONFIG (configuration timeline)
"what is our overall PCI posture?" Security Hub
"is anything reachable from outside?" Access Analyzer
CloudTrail alone cannot prove continuity: it shows CHANGES, so the absence of
a change is only evidence if you also know the starting state AND that logging
was never interrupted.
Config maintains a TIMELINE per resource - every state it has been in, with
timestamps, plus continuous rule evaluation. That is the artefact an auditor
accepts.THE CHAIN MUST SURVIVE COMPROMISE OF WHAT IT RECORDS
organization CloudTrail -> s3://audit-logs in a SEPARATE log account
versioning: enabled
Object Lock: COMPLIANCE mode, 7 years # not deletable, not even by root
bucket policy: production has s3:PutObject ONLY - no Delete, no Get
Config aggregator -> the same log archive account
SCP on every workload OU:
Deny cloudtrail:StopLogging | DeleteTrail
Deny config:DeleteConfigurationRecorder | StopConfigurationRecorder
# Without the separate account, a sufficiently privileged attacker deletes
# their own trail. That is not an audit trail; it is a convenience.Conformance packs bundle Config rules mapped to frameworks — PCI DSS, CIS, SOC 2 — so evidence lines up with the control an auditor is asking about rather than needing translation. And Config remediation via SSM Automation turns a finding into a self-healing control.
Config bills per configuration item recorded AND per rule evaluation, which gets expensive in a high-churn environment - autoscaling groups cycling instances, ephemeral EKS nodes. Scope the recorder to the resource types you actually need evidence for rather than recording everything by default.
Least privilege at scale
Three layers: SCPs as inescapable guardrails, permission boundaries for safe delegation, and generated policies from CloudTrail rather than hand-written guesses.
The cultural point first, because it determines whether any of this works: start from a working broad policy and tighten from real usage data. Teams that try to write a perfect least-privilege policy up front either block delivery for days or, far more commonly, attach a managed FullAccess policy that is never revisited.
THE WORKFLOW THAT WORKS
1. deploy with a deliberately broad policy in a NON-PRODUCTION environment
2. run realistic traffic for one to two weeks
3. GENERATE the tightened policy from CloudTrail history
aws accessanalyzer start-policy-generation \
--policy-generation-details principalArn=arn:aws:iam::1234:role/worker \
--cloud-trail-details 'trails=[{cloudTrailArn=...}],startTime=...'
4. review it, apply it
5. UNUSED-ACCESS findings keep it tight thereafter
aws accessanalyzer create-analyzer --type ACCOUNT_UNUSED_ACCESS \
--configuration '{"unusedAccessConfiguration":{"unusedAccessAge":90}}'
THE WORKFLOW THAT DOES NOT
"write least privilege first" -> blocked for days -> attach *FullAccess
-> never revisitedPERMISSION BOUNDARY - safe self-service role creation
# Developers may create roles, ONLY with this boundary attached:
{ "Effect": "Allow", "Action": "iam:CreateRole", "Resource": "*",
"Condition": { "StringEquals": {
"iam:PermissionsBoundary": "arn:aws:iam::1234:policy/dev-boundary" } } }
# The boundary caps whatever they write:
{ "Effect": "Allow",
"Action": ["s3:*","dynamodb:*","sqs:*","logs:*","secretsmanager:GetSecretValue"],
"Resource": "*" } # no iam:*, no kms key admin, no ec2For a role that legitimately needs broad permissions, do not pretend it can be narrow - constrain it another way. Scope by resource and condition (specific ARNs, a source VPC endpoint, MFA, session tags), and monitor its use. A broad role that is watched is far safer than a narrow one nobody understands.
RTO, RPO and the DR patterns
How long you may be down, and how much data you may lose. Independent numbers, derived from the business, and they select the pattern between them.
PATTERN RTO RPO COST PROVEN BY
backup + restore hours backup lowest a restore test
interval
pilot light 10-45 min seconds low partly - the COMPUTE path
(promote + is UNPROVEN until the day
scale up) you need it
warm standby minutes seconds moderate CONTINUOUSLY - it is running
active-active ~0 ~0 ~2x + continuously
real
complexityFor a payments API, warm standby is usually the right answer, and the argument is about proof rather than speed: a scaled-down but functioning copy is continuously exercised, so you know it works. Pilot light's compute path is a hypothesis until the incident.
A WARM STANDBY THAT MEETS RTO 15 MIN / RPO 5 SEC
data Aurora Global Database, writer eu-west-1, secondary eu-west-2
replication lag typically <1s - MEASURE IT, alarm above 3s
unplanned promotion ~60s
compute same IaC both regions, standby service at 20-30% capacity, WARM
routing Route 53 failover, deep health checks from >=3 regions
+ client-side retry (DNS caching makes the OBSERVED RTO longer)
secrets Secrets Manager cross-region replication
KEYS KMS MULTI-REGION KEYS - without these the standby decrypts NOTHING,
and it is not fixable during the incident
images ECR cross-region replicationDerive the numbers from the business, in money and obligation - what an hour of downtime costs, what a regulator requires, what the contract commits to. Engineers picking them unilaterally is how you end up over-engineering or under-protecting.
Multi-region, honestly
Stateless is easy; the data is hard. Synchronous cross-region replication is not viable, so you are choosing between async with a lag, last-writer-wins, or regional partitioning.
EASY - capacity and configuration
images, IaC, compute, secrets, DNS routing
HARD - the data. Cross-region sync replication adds 10-150ms to EVERY write.
OPTION CONSISTENCY SAFE FOR
active-passive single writer, ledgers, anything consistent
(Aurora Global DB) async replica
active-active LAST WRITER WINS sessions, caches, preferences
(DynamoDB Global Tables) (conflicts silently NOT ledgers - it silently
discard a write) discards a financial write
regional partitioning single writer PER anything, and it aligns with
(home region per record) record data residencyWhat does not fail over
- SQS queues are regional — messages in the dead region are stranded until it returns.
- ElastiCache is regional — you fail over into a cold cache, and the resulting miss storm can overwhelm the newly promoted database. A very common way a technically successful failover still causes an outage.
- EventBridge schedules exist per region — both firing is a double-execution bug, neither firing is a silent gap. Take a cross-region lock rather than relying on configuration.
- In-flight Step Functions executions do not migrate.
- WebSocket and streaming connections all drop.
AND STATE WHAT IS LOST ON UNPLANNED FAILOVER
transactions committed in the primary but not yet replicated are GONE.
Sub-second window, but not zero. In payments that is not an abstraction.
THEREFORE the design must make REPLAY SAFE:
idempotency key on every write
an outbox that can be re-driven
reconciliation against the scheme's or partner's records after failoverComplexity has its own failure rate. A badly-run multi-region system is frequently LESS available than a well-run single-region one, because you have doubled the surface of deploys, migrations, secrets and incidents. Recommend single-region multi-AZ plus a rehearsed warm standby unless a regulator, a contract, or a global user base says otherwise.
Backups you can restore
A snapshot shares the blast radius of what it protects. A backup lives under separate control — different account, immutable — and its recovery time is a measurement, not an estimate.
SNAPSHOT same account, often same region. Fast. SHARES THE BLAST RADIUS -
an attacker or a mistaken admin deletes the resource AND its
snapshots in the same breath.
BACKUP different ACCOUNT, ideally different region, independent
permissions, IMMUTABLE.
AWS BACKUP - plan + tag-based selection, so a NEW resource tagged
Backup=daily is covered automatically, not by someone remembering
CopyActions -> cross-region AND cross-account
VAULT LOCK in COMPLIANCE mode -> nobody can delete early. Not an admin.
Not root. Not AWS. Irreversible after a 3-day grace period.
This is the specific control against ransomware.PROVING RESTORE - a backup never restored has an UNKNOWN success rate
and an UNKNOWN RTO. You have a hypothesis, not a backup.
RestoreTestingPlan:
ScheduleExpression: "cron(0 4 ? * SUN *)" # weekly
# restores into an ISOLATED account, validates, records duration, tears down
VALIDATION MUST BE REAL
row counts within the replication window
checksum of a sampled key range
a canary query returning a KNOWN value
the APPLICATION can connect and read (not just: the instance exists)
RECORD THE ELAPSED TIME. That is your MEASURED RTO.
Re-run often enough that it reflects CURRENT data volume.Test the restore of the thing you would actually NEED. Restoring a database is common practice; restoring the IAM roles, KMS keys, parameter values and networking it needs in order to be USABLE is what people discover is missing during the real event.
Chaos and game days
Controlled failure experiments with a stop condition. The stop condition is what separates chaos engineering from an outage.
AWS Fault Injection Service runs experiments against real infrastructure — terminating instances, stopping tasks, injecting CPU pressure, adding network latency, throttling API calls, forcing an RDS failover — scoped by IAM and targeted by tag. A CloudWatch alarm as a stop condition aborts the experiment automatically if real users are affected.
ORDER, each with a HYPOTHESIS STATED BEFORE RUNNING
1 kill one task / pod H: zero client errors
-> fails? a graceful shutdown or deregistration-delay problem.
Every later experiment is meaningless until this passes.
2 force an RDS failover H: application recovers within 120s
-> THE MOST VALUABLE NUMBER: the gap between "AWS recovered" and
"WE recovered". This is where JVM DNS caching and a stale connection
pool turn a 90-second failover into a ten-minute outage.
3 inject 200ms downstream latency H: the circuit breaker opens BEFORE the
caller times out
-> looking for a badly NESTED timeout budget: caller waits 10s, you wait
30s, so you burn threads producing responses nobody will read
4 throttle an AWS API H: retries back off with jitter and a cap
-> unjittered retries turn a brief throttle into a sustained one
5 evacuate an AZ H: serve at reduced capacity, no errors
-> did topology spread ACTUALLY spread you? is n-1 capacity real?Always start with a hypothesis you EXPECT TO HOLD. The purpose is building evidence that the system behaves as designed; when it does not, you have found something real. An experiment you already expect to fail is not an experiment - fix the known problem first.
Announce it, run it in business hours, and treat a real incident during a game day as a SUCCESS - you found a genuine failure under controlled conditions with the right people watching. The only bad outcome is not having a stop condition.
Cost
Five engineering levers, in order of size — and Savings Plans come last, because they discount whatever you are running.
LEVER TYPICAL SAVING EFFORT
1 right-size + Graviton (arm64) 20-40% compute low
2 spot for stateless and batch 70-90% on that medium
3 storage lifecycle + debris often large low
4 DATA TRANSFER architecture 10-20% of TOTAL low <- most missed
5 observability hygiene varies low
THEN, and only then: Savings Plans and Reserved Instances.
They discount whatever you are running, so committing BEFORE right-sizing
locks in the waste for one to three years.LEVER 4 IN DETAIL - invisible until you read the bill BY USAGE TYPE
NAT Gateway: hourly + PER-GB PROCESSED
every ECR pull, S3 write and Secrets call from a private subnet
FIX: gateway endpoints for S3 + DynamoDB (FREE), interface endpoints for
ECR (api, dkr, AND the S3 gateway), Secrets Manager, Logs, STS
Cross-AZ: billed BOTH directions
12 services x 3 AZs -> ~2/3 of internal calls cross a zone
FIX: topology-aware routing (prefer same-zone), replicas still spread
FIND IT
Cost Explorer -> group by USAGE TYPE, not by service
EU-NatGateway-Bytes | EU-DataTransfer-Regional-Bytes | TimedStorage-ByteHrs
LEVER 3 - the ones that accrue silently
unattached EBS volumes and orphaned snapshots
CloudWatch log groups with NO retention (the default is never expire)
incomplete multipart uploads (billed forever, invisible in a listing)Estimate from units rather than from a price list — requests per day, tokens or gigabytes, instance-hours — because units are stable, they survive price changes, and they tell you which lever matters. Then sanity-check the total as cost per user per month, which is the number a director will actually ask for.
Make cost visible to the teams who create it: mandatory tagging enforced at admission, a monthly per-team report, and a budget they own. Attribution changes behaviour far more reliably than a policy does, and far more reliably than finance raising it quarterly in aggregate.