An AWS handbook · accounts and IAM to multi-region

AWS Ground Up

AWS as a set of primitives with an identity system in front of everything, deployed into failure domains you choose. Ordered so the irreversible decisions — accounts, network, data store, failure domain — come first, because those are the ones you cannot walk back.

I. Foundations
Chapter 01

What AWS actually is

A set of primitives with an identity system in front of everything, deployed into failure domains you choose. Almost every service is a managed version of something you could run yourself.

It helps to stop thinking of AWS as two hundred products and start thinking of it as a small number of ideas. Compute, storage, networking, and managed versions of software you would otherwise operate. What is genuinely AWS-shaped — the parts with no direct equivalent elsewhere — is IAM, which gates every single API call, and the region and availability zone structure, which is how you choose what you can survive losing.

Everything else is a variation. RDS is Postgres you do not patch. SQS is a queue you do not run. EKS is Kubernetes whose control plane you do not upgrade. Recognising that means you can reason about a service you have never used by asking what it is a managed version of, and what AWS took responsibility for.

The organising question

The most useful lens for architecture on AWS is which decisions are irreversible. Four are, and they are the subject of the first half of this book: the account and identity boundary, the network layout, the data store per workload, and the failure domain strategy. Almost everything else — compute choice, deployment tooling, observability vendor — is reversible in a quarter.

Principle

Most teams spend their architecture debate on the reversible choices and an afternoon on the irreversible ones. Leading with what cannot be walked back is what makes an architecture conversation useful - and what makes it sound senior.

Chapter 02

Regions and availability zones

A region is a geographic area; an availability zone is an independent failure domain inside it. Inter-AZ latency is about a millisecond, and that single number shapes almost every design.

A region is a geographic area with its own set of AZs. Most services are regional, data does not leave a region unless you move it, and pricing differs between them. An availability zone is one or more discrete datacentres with independent power, cooling and physical security, connected to the other zones by dedicated low-latency links.

The number that matters
inter-AZ round trip     ~1 ms      -> synchronous replication is FINE
cross-region            10-150 ms  -> synchronous replication is NOT

Therefore the default posture:
  MULTI-AZ ALWAYS.
  Multi-region only with a stated RTO and RPO, or a residency requirement.

And: AZ NAMES are per-account aliases.
  your us-east-1a is not necessarily another account's us-east-1a.
  Use the AZ ID (use1-az1) when you need to correlate across accounts.

The shared responsibility model divides the stack: AWS secures the cloud itself — facilities, hardware, the hypervisor, managed service internals — and you secure what you put in it: IAM, network rules, encryption choices, patching anything you run, and your data. The line moves with the service, which is worth being precise about: on EC2 you patch the OS; on RDS AWS patches the engine but you still own network placement, credentials and backups; on S3 you own only access control and encryption configuration, which is exactly where the well-known public-bucket incidents came from.

Principle

Two AZs is the minimum for availability; three is the production standard, because losing one costs a third of your capacity rather than half - and because quorum systems need an odd number.

Chapter 03

Accounts as the boundary

An AWS account is the only hard blast-radius, quota and IAM boundary. Everything else — tags, naming conventions, permission prefixes — is a convention one mistaken policy can cross.

This is the decision that is hardest to change later, because every resource ARN, every policy, every cost report and every log destination is built on it. A single account holding development and production means they share an IAM boundary, share service quotas, and share the consequences of any mistake.

  • Blast radius. A misconfigured policy in dev cannot touch production data if they are different accounts.
  • Quotas. Service limits are per-account, so a load test cannot exhaust production's Lambda concurrency or EC2 capacity.
  • IAM clarity. Policies become comprehensible when this account is production rather than when every statement needs a name-prefix condition.
  • Cost attribution. The account is the dimension — no tagging discipline required to answer what production costs.
  • Compliance scope. An auditor can be given a boundary rather than a filter.
Landing zone
A TYPICAL STRUCTURE

ou-security
  log-archive        CloudTrail + Config, Object Lock, nobody can delete
  security-tooling   GuardDuty, Security Hub, Access Analyzer (delegated admin)
ou-shared
  build              ECR - built once, pulled cross-account by every environment
ou-workloads
  payments-dev
  payments-staging
  payments-prod      <- SCPs applied here
ou-data
  data-platform      different consumers, different access patterns

management account: Organizations and billing ONLY. No workloads. No data.
Interview angle

The caution to raise unprompted: more accounts means more to automate. Account creation, baseline guardrails, networking, logging and identity assignment must be automated from day one, or the structure becomes a burden and people route around it. A boundary is only valuable if a correct account is easy to create.

Chapter 04

Organizations and service control policies

SCPs are guardrails nobody can escape — not an administrator, not the root user. They grant nothing; they only subtract, so use them for invariants rather than for permissions.

An Organization groups accounts into organizational units and applies policies to them. A service control policy caps what any principal in that OU may do, and it applies above IAM: even an account administrator who grants themselves AdministratorAccess is still bounded by it.

Invariants worth enforcing
{ "Statement": [
  { "Sid": "RegionContainment", "Effect": "Deny",
    "NotAction": ["iam:*","organizations:*","cloudfront:*","route53:*","support:*"],
    "Resource": "*",
    "Condition": { "StringNotEquals": { "aws:RequestedRegion":
                     ["eu-west-1","eu-west-2"] } } },

  { "Sid": "ProtectAudit", "Effect": "Deny",
    "Action": ["cloudtrail:StopLogging","cloudtrail:DeleteTrail",
               "config:DeleteConfigurationRecorder","guardduty:DeleteDetector"],
    "Resource": "*" },

  { "Sid": "NoLongLivedCredentials", "Effect": "Deny",
    "Action": ["iam:CreateUser","iam:CreateAccessKey"],
    "Resource": "*" } ] }

Note what those three have in common: they are rules that should hold no matter who is asking, and they almost never fire during normal work. That is the correct use. Using SCPs for fine-grained permission is the wrong tool — it makes them fire constantly, produces confusing denials, and is better expressed in identity policies.

Permission boundaries solve a related problem: letting developers create their own roles without letting them create an administrator. A boundary attached to a role caps its effective permissions, and an IAM policy can require the boundary on iam:CreateRole — so self-service role creation becomes safe.

Principle

An SCP denial does not appear in the IAM policy simulator, which only evaluates identity and resource policies. CloudTrail's error message names it explicitly - 'with an explicit deny in a service control policy' - and that is the tell.

Chapter 05

IAM policies

JSON documents granting or denying actions on resources, optionally with conditions. Conditions are where least privilege actually gets expressed.

Anatomy
{ "Version": "2012-10-17",
  "Statement": [
    { "Sid": "ReadSettlementFiles",
      "Effect": "Allow",
      "Action": ["s3:GetObject"],
      "Resource": "arn:aws:s3:::settlements-prod/incoming/*",   // PREFIX, not bucket
      "Condition": {
        "StringEquals": { "aws:SourceVpce": "vpce-0a1b2c3d" },  // only via our endpoint
        "Bool": { "aws:SecureTransport": "true" } } } ] }

A policy allowing s3:GetObject on an entire bucket is not least privilege even though it names one resource. Scope to a key prefix, and add conditions: restrict by source VPC endpoint, by tag matching, by MFA presence, by requested region, by time. Conditions are the difference between a policy that is technically scoped and one that is genuinely constrained.

Identity versus resource policies

An identity policy is attached to a role or user and says what that principal may do. A resource policy is attached to the resource — a bucket, a queue, a KMS key — and says who may use it. Resource policies are required for cross-account access and for principals you do not control, such as a service principal or a partner's account.

Principle

Write policies from CloudTrail history rather than from imagination. IAM Access Analyzer's policy generation reads what a role ACTUALLY used and produces a policy containing exactly those actions. Broad-then-tighten with a deadline beats a perfect policy nobody can write.

Chapter 06

Users, roles and STS

A user has permanent credentials; a role has none and is assumed, yielding short-lived tokens. Applications and humans should both use roles.

A user is a long-lived identity with permanent credentials — a password, or an access key and secret. A role is an identity with no credentials attached that a principal assumes; STS then issues temporary credentials that expire, typically in an hour.

That expiry is the entire point. A leaked access key is valid until a human notices and revokes it, and credential-scanning bots find keys committed to public repositories within minutes. Temporary credentials expire on their own, are rotated automatically by the SDK, are never written to disk, and cannot be committed.

Machine identity
HOW A WORKLOAD GETS CREDENTIALS - same shape everywhere:
something in the environment vouches for its identity, and STS exchanges
that for temporary credentials.

  EC2          instance profile, served by the metadata service
  ECS          task role, from a link-local endpoint
  EKS          IRSA / Pod Identity - the pod's ServiceAccount token exchanged
               via an OIDC provider for a role scoped to that ServiceAccount
  GitHub CI    an OIDC token presented to AssumeRoleWithWebIdentity
  On-prem      IAM Roles Anywhere, using an X.509 certificate

# CI trust policy - note the CONDITION
{ "Action": "sts:AssumeRoleWithWebIdentity",
  "Condition": { "StringLike": { "token.actions.githubusercontent.com:sub":
      "repo:acme/payments-api:ref:refs/heads/main" } } }
# omit that condition and ANY GitHub repository can assume the role
Interview angle

When is an IAM user still legitimate? Rarely, and always with a note: a third-party SaaS that genuinely cannot do OIDC or cross-account assumption. Then a dedicated user, a single narrow policy, scheduled rotation, and CloudTrail alerting on use from unexpected sources. Never a shared user.

Chapter 07

The policy evaluation chain

Eight layers, all of which must permit and none of which may deny. Reciting them in order is also the debugging algorithm.

The chain
1  SCP                  organization-level cap. Denies here end it.
2  Resource control      newer org-wide cap applied to resources
   policy (RCP)
3  Identity policy       does the ROLE have the action on that resource?
4  Resource policy       does the BUCKET / QUEUE / KEY allow this principal?
5  Permission boundary   if attached, caps the role's effective permissions
6  Session policy        if passed at AssumeRole time
7  KMS KEY POLICY        if the resource is encrypted with a CMK
8  VPC endpoint policy   if the request travels through one

RULES
  an explicit DENY anywhere wins immediately and cannot be overridden
  otherwise an explicit ALLOW is required; the default is deny
  SAME account:  an allow in EITHER identity or resource policy suffices
  CROSS-account: you need an allow on BOTH sides

Where it actually breaks

Most often at step seven. A KMS-encrypted bucket requires the caller to have kms:GenerateDataKey on the key, and the key policy in the owning account must grant it — because a key policy is the root of trust for that key, and IAM alone cannot grant access to a key whose policy does not permit it. The symptom is an opaque AccessDenied with a demonstrably correct bucket policy, and it costs people hours.

Diagnosis
# Debugging, in this order:
aws cloudtrail lookup-events --lookup-attributes ...   # errorMessage names the layer
aws iam simulate-principal-policy ...                  # identity side only
# then check, in order: SCP, KMS key policy, endpoint policy
#   (the simulator sees NONE of those three)
Interview angle

The confused deputy problem belongs here: when a service assumes a role on your behalf, pin aws:SourceAccount and aws:SourceArn in the trust policy so it cannot be tricked into acting for a different customer. A service-principal trust without those conditions should be flagged in review.

Chapter 08

Identity in practice

Humans federate through single sign-on; machines assume roles. No IAM users, no long-lived keys, and joiners and leavers handled in one place.

IAM Identity Center is the mechanism for human access: permission sets mapped to identity-provider groups, assigned per account. A person's access follows their group membership, so a leaver is removed everywhere by one change in the directory rather than by remembering which accounts they had.

Assignment
IdP group                -> permission set        -> accounts
  payments-engineers     -> PaymentsDeveloper      -> dev, staging
  payments-engineers     -> ReadOnly               -> prod
  payments-oncall        -> BreakGlassProd         -> prod (approval + alert)
  security               -> SecurityAudit          -> all

# Note the shape: the SAME group gets different permission sets per account.
# Production is read-only by default; write access is a break-glass path.

Break-glass

Build the emergency path deliberately: a separate role, assumed with approval, time-limited, alerting to an incident channel on assumption, every action in the audit log, and an automatic ticket to review afterwards. If the path does not exist officially, people improvise one under pressure with less oversight — which is strictly worse.

  • No IAM users. Enforce it with an SCP denying iam:CreateUser and iam:CreateAccessKey in production.
  • MFA everywhere, ideally hardware-backed for anything privileged.
  • Access Analyzer unused-access findings to catch roles, permissions and credentials nobody has used in ninety days.
  • Quarterly review of who has production access and why.
Principle

The workflow that actually achieves least privilege: deploy with a broad policy in a non-production environment, run real traffic for a week or two, generate a tightened policy from CloudTrail, review it, apply. Teams that try to write the perfect policy up front either block delivery or attach FullAccess and never revisit it.

II. The Network
Chapter 09

The VPC

A private IP space you choose. You cannot shrink a CIDR and overlapping ranges cannot be peered, which makes the address plan one of the four irreversible decisions.

A VPC is a logically isolated network with a CIDR block you pick — say 10.20.0.0/16, giving 65,536 addresses. Everything you run lives inside it, and its addressing constrains what you can connect to later.

  • You cannot shrink a CIDR. You can add secondary blocks, which is awkward but possible.
  • Overlapping ranges cannot be routed between. If your VPC overlaps your on-premises network, or another VPC you need to peer with, you cannot connect them without NAT gymnastics.
  • Someone must own IP allocation across the organisation. In a company with many accounts this is a real job, and skipping it produces a peering problem three years later that nobody can fix cheaply.
Address planning
SIZING - be generous, it costs nothing
  /16   65,536 addresses    a comfortable default per VPC per region
  /20    4,096              a tier across three AZs
  /24      251 usable       AWS reserves 5 in every subnet

AND ON EKS: every POD takes a subnet address.
  a /24 app subnet holds ~251 PODS, not 251 nodes.
  IP exhaustion presents as pods stuck Pending with no resource shortage.
Interview angle

Ask about the on-premises ranges before choosing anything. The single most expensive VPC mistake is picking 10.0.0.0/16 because it is the default, and discovering later that the corporate network already uses it - which permanently forecloses Direct Connect and peering.

Chapter 10

Subnets and routing

A subnet lives in exactly one availability zone, and public versus private is a routing decision rather than a property of the subnet.

Subnets carve the VPC up, and each is pinned to one AZ. That is the constraint that shapes every layout: a three-tier application across three zones needs nine subnets, not three.

A subnet is public if its route table sends 0.0.0.0/0 to an internet gateway. It is private if it sends 0.0.0.0/0 to a NAT gateway — outbound only, because NAT forwards return traffic for connections initiated from inside but nothing inbound. A subnet with no default route at all is isolated, which is the right choice for a database tier.

Three tiers, three zones
VPC 10.20.0.0/16  (eu-west-1)

              AZ a              AZ b              AZ c
public    10.20.0.0/24      10.20.1.0/24      10.20.2.0/24     -> IGW
          (ALB, NAT-a)      (ALB, NAT-b)      (ALB, NAT-c)

app       10.20.16.0/20     10.20.32.0/20     10.20.48.0/20    -> NAT in SAME AZ
          (4,091 usable - sized for PODS, not instances)

data      10.20.200.0/24    10.20.201.0/24    10.20.202.0/24   -> NO default route
          (RDS subnet group spans all three)

Route tables:
  rtb-public   0.0.0.0/0 -> igw-xxxx
  rtb-app-a    0.0.0.0/0 -> nat-a       <- per-AZ, avoids cross-AZ charges
  rtb-data     local only               <- structurally cannot reach the internet
Principle

Separating tiers into their own subnets makes 'the database cannot reach the internet' a STRUCTURAL guarantee expressed in routing, rather than a security group rule someone can loosen. That is why it is worth nine subnets instead of three.

Chapter 11

Internet and NAT gateways

An internet gateway is free and bidirectional; a NAT gateway is outbound-only, zonal, and billed per gigabyte processed on top of its hourly rate.

An internet gateway attaches to the VPC and provides a route to and from the internet for resources with public addresses. It is horizontally scaled, redundant, and free.

A NAT gateway lets private resources reach the internet outbound without being reachable inbound. It is a managed, zonal resource — so one per AZ for availability, and the route table in each zone should point at its own NAT, or outbound traffic crosses a zone boundary and pays cross-AZ transfer on top of everything else.

The cost
NAT GATEWAY COST HAS TWO PARTS
  hourly rate           per gateway, per AZ
  PER-GB PROCESSED      on every byte, in addition to any transfer charge

WHAT GOES THROUGH IT (and should not)
  container image pulls on every cold node
  S3 reads and writes
  Secrets Manager calls
  CloudWatch Logs
  third-party API calls           <- the only one that genuinely needs it

At any scale this becomes a large line item, and it hides in the bill under
"EC2-Other" rather than under a service name - so it is invisible in a
service-level cost breakdown.

For development, a single NAT gateway shared across zones is a legitimate cost saving — just be explicit that it is a deliberate trade rather than an oversight, since it is both a single point of failure and a source of cross-AZ charges.

Principle

The fix for most NAT traffic is VPC endpoints, covered shortly. Gateway endpoints for S3 and DynamoDB are FREE and remove the largest component immediately. If you take one action on an unfamiliar AWS estate, make it that one.

Chapter 12

Security groups and network ACLs

Security groups are stateful, attach to an interface, allow-only, and can reference other groups. NACLs are stateless, attach to a subnet, and support deny. Default to security groups for essentially everything.

The comparison
SECURITY GROUP                    NETWORK ACL
stateful (return traffic auto)    STATELESS (needs explicit ephemeral rules)
attaches to an ENI                attaches to a SUBNET
allow only                        allow AND deny
evaluated as a set, no order      numbered rules, FIRST MATCH WINS
can reference another SG          CIDR only
applies to SAME-SUBNET traffic    only evaluates traffic CROSSING the subnet

The feature that decides it is group references. A rule saying allow 5432 from sg-app stays correct forever as instances scale, are replaced and change addresses. It expresses the relationship rather than a snapshot of addresses — and once you have used it you stop writing CIDR rules for internal traffic.

Chaining
sg-alb    in:  443 from 0.0.0.0/0
          out: 8080 to sg-app
sg-app    in:  8080 from sg-alb           <- a GROUP, not a CIDR
          out: 5432 to sg-data, 443 to endpoints
sg-data   in:  5432 from sg-app
          out: (none)                     <- the database initiates nothing

Scale from 2 to 200 instances: no rule changes.
Replace every instance: no rule changes.

Compare "allow 5432 from 10.20.16.0/20": correct until someone adds a subnet,
and it lets ANY pod in that range reach the database.

The NACL trap is statelessness: an inbound allow on 443 also needs an outbound allow on the ephemeral port range, or responses are silently dropped. And because NACLs only evaluate traffic crossing a subnet boundary, they do nothing about lateral movement within a subnet — which is why security groups, not NACLs, are your lateral-movement control.

Interview angle

Security groups are allow-only, so blocking a specific attacking address requires either a NACL deny or - better - a WAF rule at the load balancer, where you also get rate limiting and logging. NACL is the blunt instrument for when you need it immediately.

Chapter 13

VPC endpoints

Reach AWS services without traversing the internet or NAT. Gateway endpoints for S3 and DynamoDB are free; interface endpoints cost an hourly rate and are far cheaper than NAT at volume.

Two kinds
GATEWAY ENDPOINT     S3, DynamoDB only
                     works by adding a ROUTE to the subnet's route table
                     FREE. There is no reason not to have these.

INTERFACE ENDPOINT   almost everything else (PrivateLink)
                     an ENI in your subnet with a private DNS name
                     hourly per endpoint PER AZ, plus a small per-GB rate

THE ECR TRIO - the omission that makes people think endpoints do not work
  com.amazonaws.<region>.ecr.api      interface   (authentication, manifests)
  com.amazonaws.<region>.ecr.dkr      interface   (the Docker protocol)
  com.amazonaws.<region>.s3           GATEWAY     <- ECR stores LAYERS in S3
# configure only the first two, watch pulls still traverse NAT, and be baffled

Beyond cost, endpoints are a security improvement: traffic never leaves the AWS network, and an endpoint policy can restrict what may be reached through it. Combined with a bucket policy condition on aws:SourceVpce, you can make a bucket reachable only from your VPC — which is a much stronger statement than a set of IAM policies.

Endpoint policy
# On the bucket: only reachable through our endpoint
{ "Effect": "Deny", "Principal": "*", "Action": "s3:*",
  "Resource": ["arn:aws:s3:::settlements-prod",
               "arn:aws:s3:::settlements-prod/*"],
  "Condition": { "StringNotEquals": { "aws:SourceVpce": "vpce-0a1b2c3d" } } }
Principle

Do the arithmetic before adding interface endpoints everywhere: they cost per endpoint PER AZ, so in a small development account with three zones and ten endpoints the fixed cost can exceed the NAT charge they replace. Above a few hundred gigabytes a month through NAT, they win comfortably.

Chapter 14

Data transfer costs

Inbound free, outbound charged, cross-AZ charged in BOTH directions, and NAT charged per gigabyte on top. Two of those are invisible in a service-level bill.

The rules
inbound from internet          FREE
outbound to internet           the headline egress rate
CROSS-AZ                       billed BOTH directions (send AND receive)
cross-region                   more, both directions
via NAT gateway                hourly + PER-GB PROCESSED, ON TOP of the above
within one AZ, private IPs     free

The two that hide

NAT processing, which appears under EC2-Other rather than under a service name. And cross-AZ chatter between services: twelve microservices spread across three zones for availability means roughly two thirds of internal calls cross a zone and are billed twice, appearing as a regional transfer line with no indication of which services caused it.

Finding and fixing
FIND IT: Cost Explorer, group by USAGE TYPE - not by service
  EU-NatGateway-Bytes                <- NAT processing
  EU-DataTransfer-Regional-Bytes     <- cross-AZ
  EU-DataTransfer-Out-Bytes          <- internet egress
Then VPC flow logs to attribute cross-AZ traffic to specific services.

DESIGN IT OUT - configuration, not redesign
  S3 + DynamoDB              -> gateway endpoints (FREE)
  ECR, Secrets, Logs, STS    -> interface endpoints
  chatty service pairs       -> topology-aware routing: prefer a SAME-ZONE
                                endpoint while keeping replicas spread
  high internet egress       -> CloudFront (lower egress rate + caching)
Interview angle

Reading Cost Explorer by usage type rather than by service is the first thing to do on an unfamiliar bill. It is where NAT processing and cross-AZ transfer become visible, and both are routinely 10-20% of a bill that nobody has attributed.

Chapter 15

DNS and Route 53

Seven routing policies, health checks, and one hard limitation: DNS is cached by resolvers and by the JVM, so failover is slower than its TTL.

Policies
simple          one target, no logic
weighted        split by percentage - migrations, cross-region canaries
latency         each client to the region with the lowest measured latency
geolocation     by country or continent - DATA RESIDENCY
geoproximity    by distance, with an adjustable bias
failover        primary + health-checked secondary
multi-value     several healthy records, simple client-side spreading

Alias records point at AWS targets (a load balancer, CloudFront, an S3 website) and have two advantages over a CNAME: they work at the zone apex, and alias queries are not billed. Use alias for AWS targets and CNAME only for external hostnames.

Why failover is slower than the TTL

You set a 60-second TTL. Recursive resolvers may honour it or not; corporate resolvers frequently extend it; browsers keep their own cache. And for a Java client it is worse and specific: the JVM caches DNS resolutions according to a security property, and many applications never change it. Combined with a connection pool holding open sockets, the client stays pinned to a dead endpoint long after DNS has moved.

The JVM caveat
-Dsun.net.inetaddr.ttl=5           # or in java.security:
networkaddress.cache.ttl=5
networkaddress.cache.negative.ttl=1
spring.datasource.hikari.max-lifetime: 600000     # recycle connections

# TEST THIS with a real client, not with dig. It is frequently the difference
# between a 2-minute failover and a 30-minute one.

# For seconds-level failover, DNS is the wrong tool entirely:
#   Global Accelerator  anycast IPs, failover happens IN THE NETWORK
#   client-side retry   the SDK tries endpoint A then B
Principle

Health checks must hit a DEEP endpoint that verifies the target can actually serve, and they come from multiple global locations so a single bad network path does not trigger a failover. A health check against a static file will happily fail you over into a region whose database is unreachable.

Chapter 16

CloudFront and the edge

A CDN, and also a way to terminate TLS close to the user and carry the rest over the AWS backbone — which helps even at a zero percent cache hit rate.

For a globally distributed audience, CloudFront helps regardless of caching. A TLS handshake is multiple round trips, and doing those at 15ms to a nearby edge rather than 180ms to a distant region is a large, measurable improvement. Connection reuse from edge to origin compounds it.

Be honest about the converse: for a single-region audience hitting a single-region origin, CloudFront mostly adds a hop. Say so rather than recommending it reflexively.

The cache key is where deployments fail

Including every header, cookie and query string in the key fragments the cache so badly that the hit rate approaches zero — every request is unique, nothing is reused, and you have added a hop for nothing. Use a cache policy that includes only what genuinely changes the response, and a separate origin request policy for things the origin needs but that must not be part of the key.

Policies
CachePolicy:
  HeadersConfig:      { whitelist: [Accept-Encoding, Accept-Language] }
  CookiesConfig:      { none }
  QueryStringsConfig: { whitelist: [page, size] }

OriginRequestPolicy:
  HeadersConfig:      { whitelist: [Authorization, X-Request-Id] }
#   ^ forwarded to the origin, but NOT part of the cache key

# Personalised responses are never shareable
Cache-Control: private, no-store
#   caching per-user data at a shared edge is a LEAK

# Invalidation is slow and metered - do not design around it
  /assets/app.9f2c4a.js      versioned path: a new deploy is a new URL
  create-invalidation "/*"   emergency only
Interview angle

Origin Shield adds a regional caching layer so many edges do not all miss through to your origin at once - useful protection against a cache-clear stampede. And CloudFront Functions handle header rewrites and cache-key normalisation far more cheaply than Lambda@Edge.

III. Compute
Chapter 17

EC2 and instance families

Families encode a resource ratio, not a quality tier. Match the ratio to your workload, then size — and for JVM services, Graviton is the default recommendation.

Reading the name
c   compute-optimised     ~2 GB RAM per vCPU
m   general purpose        ~4 GB
r   memory-optimised       ~8 GB
i/d + fast local NVMe
g/p + GPUs
t   BURSTABLE with CPU credits

SUFFIXES
  g   Graviton (ARM)   ~20% cheaper, equal or better performance
  i   Intel
  a   AMD
  n   enhanced networking
  d   local NVMe

m7g.xlarge = general purpose, 7th generation, Graviton, 4 vCPU / 16 GB

Graviton for JVM workloads is the recommendation worth making explicitly. Java bytecode is architecture-neutral, so the migration is usually building a multi-arch image and checking native dependencies — JNI libraries, some compression and crypto libraries, older agents. Roughly 20% cheaper per hour at equal or better performance is the largest easy saving available.

Avoid t-family for sustained production load

Burstable instances earn CPU credits when idle and spend them under load. Exhaust the balance and you are throttled to a baseline that can be 10 or 20% of a vCPU. The symptom is a service that performs fine for twenty minutes after deploy and then degrades catastrophically — a genuinely confusing incident if you do not know the mechanism.

Principle

Right-size from observed p95, not from the shape someone picked at launch. Right-sizing plus Graviton is typically 20-40% on compute for a few days of work, and it is the highest ratio of saving to effort available on AWS.

Chapter 18

Auto Scaling groups

Not primarily a scaling tool. It gives you replacement, AZ distribution, and then scaling — and the health check type determines whether replacement actually works.

  • Replacement. It maintains a desired count, so a failed instance is terminated and replaced.
  • AZ distribution. Given subnets in several zones it balances across them and rebalances after a zone recovers.
  • Scaling policies. Target tracking on a metric, step scaling, or scheduled.
The setting that matters
HealthCheckType: ELB               # <-- NOT the default 'EC2'
HealthCheckGracePeriod: 300        # JVM start + warmup

# The default EC2 check only asks whether the HYPERVISOR thinks the instance
# is alive. A JVM that has deadlocked, OOMed, or is returning 500s to every
# request passes it forever.
# ELB uses the load balancer's application-level check, so a broken instance
# is actually replaced.
# And the grace period must cover startup, or the ASG kills instances during
# boot in an endless loop.
Production shape
MixedInstancesPolicy:
  Overrides: [m7g.large, m7g.xlarge, m6g.large, c7g.xlarge]   # diversify
  InstancesDistribution:
    OnDemandBaseCapacity: 3                    # baseline never on spot
    OnDemandPercentageAboveBaseCapacity: 20    # burst is 80% spot

TargetTrackingPolicy:
  Metric: ALBRequestCountPerTarget             # NOT CPU for an IO-bound service
  TargetValue: 800

# Warm-up matters for a JVM: a fresh instance is slow until JIT compiles.
# Use target group slow start, or it takes full traffic while interpreting.
Interview angle

Launch configurations are deprecated - immutable, unversioned, no mixed instance types. If you see them in a scenario, flagging them as legacy and naming launch templates is a small, easy point.

Chapter 19

Load balancers

ALB is layer 7 and terminates the connection; NLB is layer 4, preserves the source IP and supports static addresses. Choose on protocol and on whether a partner must allowlist an address.

The three
                    ALB (L7)          NLB (L4)          GWLB
protocol            HTTP/HTTPS/gRPC   TCP/UDP/TLS       GENEVE
routing             host/path/header  port only         n/a
client source IP    X-Forwarded-For   PRESERVED         n/a
static/Elastic IP   no                YES               n/a
WAF integration     yes               no                n/a
latency added       ~ms               ~microseconds     n/a

CHOOSE
  ALB   normal web APIs, path routing, WAF, TLS termination
  NLB   a partner must allowlist a fixed IP; non-HTTP protocols; extreme
        connection counts; fronting your own proxy layer
  GWLB  inserting third-party firewalls or IDS transparently

A useful hybrid worth knowing: an ALB behind an NLB gives you the static addresses a partner allowlists plus layer 7 routing. The alternative — an ALB with Global Accelerator in front — also provides static anycast addresses and improves global latency, and is often the better answer when fixed IPs are the only reason NLB was considered.

The one to memorise
# The keep-alive race: a very common and very confusing 502
  ALB idle timeout        60s
  app server idle timeout 60s      <- SAME. Race condition.
  -> the ALB reuses a connection the target is closing
  -> intermittent 502, correlated with connection age rather than load,
     and NOTHING in the application logs

FIX (one line):
  server.tomcat.keep-alive-timeout: 70s     # MUST exceed the ALB's
Principle

ALB access logs are the only source with per-request detail including which target served it. Enable them on day one - they are cheap, they go to S3, and their absence is only ever felt during an incident.

Chapter 20

Target groups and health checks

A shallow health check, a deregistration delay above your longest request, and slow start for a JVM. Getting the first one wrong turns a dependency blip into a total outage.

Configuration
TargetGroupAttributes:
  deregistration_delay.timeout_seconds: 60     # > longest request, or deploys
                                               #   cut connections mid-flight
  slow_start.duration_seconds: 60              # JVM warms up before full traffic
  load_balancing.algorithm.type: least_outstanding_requests
                                               # better than round_robin when
                                               # request cost varies
  stickiness.enabled: false                    # unless you genuinely need it

HealthCheck:
  Path: /actuator/health/readiness   # SHALLOW - local only
  Port: 9090                         # management port
  Interval: 10
  HealthyThreshold: 2
  UnhealthyThreshold: 3

Why the health check must be shallow

A health check that verifies the database means that when the database blips, every target fails simultaneously, the load balancer removes all of them, and you get 503 with no healthy targets. A twenty-second degradation becomes a complete outage. The check should answer can this instance serve, not is the whole system well.

Deregistration delay is the drain window. Set it too low and every deploy cuts in-flight requests, which shows as a small burst of 502s nobody explains. Set it absurdly high and deploys crawl.

Diagnosing
# 5xx with healthy targets - split by code, they mean different things
502  target returned something unparseable, or closed the connection
     (usually the keep-alive race; check target_status_code = "-" in the logs)
503  no healthy targets at that instant
     (check HealthyHostCount over time - a dip to zero is definitive)
504  target did not respond within the idle timeout
     (a downstream slower than your timeout; nest your timeout budgets)
Interview angle

least_outstanding_requests is a free improvement to tail latency when request cost varies: round robin keeps sending work to a target stuck on a slow query, while least-outstanding naturally routes around it.

Chapter 21

ECS

AWS-native orchestration: task definitions, services, and direct integration with ALB, IAM and CloudWatch. Far less to operate than Kubernetes, and correspondingly less expressive.

The objects
TASK DEFINITION    the blueprint: image, cpu, memory, ports, env, secrets,
                   log configuration, and the two IAM roles
SERVICE            keeps N tasks running, registers them with a target group,
                   performs rolling or blue/green deployments
CLUSTER            a logical grouping; with Fargate it holds no infrastructure

TWO ROLES, ROUTINELY CONFUSED
  executionRoleArn   used by the ECS AGENT: pull the image from ECR,
                     fetch secrets, write logs
  taskRoleArn        used by YOUR CODE: the application's AWS permissions
A task definition
{ "family": "payments-api",
  "requiresCompatibilities": ["FARGATE"],
  "cpu": "1024", "memory": "2048",
  "runtimePlatform": { "cpuArchitecture": "ARM64" },      # Graviton
  "executionRoleArn": "arn:aws:iam::1234:role/ecsTaskExecution",
  "taskRoleArn":      "arn:aws:iam::1234:role/payments-api",
  "containerDefinitions": [{
    "name": "api",
    "image": "1234.dkr.ecr.eu-west-1.amazonaws.com/payments-api@sha256:...",
    "portMappings": [{ "containerPort": 8080 }],
    "secrets": [{ "name": "DB_PASSWORD",
                  "valueFrom": "arn:aws:secretsmanager:...:prod/db:password::" }],
    "healthCheck": { "command": ["CMD-SHELL","curl -f localhost:9090/health || exit 1"],
                     "interval": 10, "retries": 3, "startPeriod": 90 },
    "stopTimeout": 60,
    "logConfiguration": { "logDriver": "awslogs", "options": {...} } }] }

Service Connect provides service discovery and a managed proxy for retries and connection management — the ECS equivalent of a Service plus a light mesh. It gives you name resolution and basic resilience without the operational surface of a full mesh.

Principle

stopTimeout is the ECS equivalent of terminationGracePeriodSeconds, and the deregistration delay on the target group is the equivalent of a preStop sleep. The same graceful-shutdown discipline applies; only the field names change.

Chapter 22

EKS

Standard Kubernetes with a managed control plane. You get the entire ecosystem and portability; you take on upgrades, add-ons and a platform team's worth of operational surface.

AWS runs the API server and etcd across multiple zones and you do not see them. Everything else is yours: node groups or Karpenter, the CNI, CoreDNS, kube-proxy, CSI drivers, the autoscaler, ingress, policy, observability — each with its own version compatibility matrix.

The real work
WHAT YOU TAKE ON
  control plane version upgrades, roughly annual, with removed-API breakage
  add-on lifecycle: VPC CNI, CoreDNS, kube-proxy, EBS CSI, autoscaler
  IRSA or Pod Identity setup
  admission policy, network policy enforcement
  an observability stack

EKS-SPECIFIC THINGS TO KNOW
  VPC CNI gives every POD a real VPC address
    -> subnets sized for pods, not nodes
    -> pod density capped by ENIs per instance type
    -> ENABLE_PREFIX_DELEGATION raises that ceiling substantially
  NetworkPolicy is NOT enforced unless explicitly enabled (or Calico/Cilium)
  IMDSv2 with hop limit 1 - the single highest-value hardening step
  access entries (or the older aws-auth ConfigMap) map IAM to cluster identity

EKS Auto Mode has moved the calculus: AWS manages node provisioning, scaling and core add-ons, which removes a real slice of the burden. It does not remove control plane upgrades, deprecated-API breakage, or the need to understand Kubernetes when something breaks at 3am — but acknowledging that the answer is closer than it was two years ago demonstrates current knowledge.

Interview angle

The control plane charge is negligible. The real cost of EKS is nodes and ENGINEERING TIME - which is why the ECS-versus-EKS decision is a staffing decision as much as a technical one.

Chapter 23

Fargate, and choosing a platform

Fargate is a launch mode, not a competitor. The real axis is ECS versus EKS, and for a small team without a platform engineer, ECS on Fargate is usually the honest answer.

Fargate removes node management from either ECS or EKS at a per-vCPU and per-GB premium of roughly 15–25%, with some capability loss: no DaemonSets in the usual sense, no privileged containers, no GPUs, and smaller maximum task shapes. Underneath it runs microVMs, which is why the isolation is stronger than shared-kernel containers.

The decision
                        ECS + Fargate      EKS + Fargate    EKS + EC2
time to first deploy    ~1 day             1-2 weeks        2-4 weeks
ongoing ops burden      near zero          moderate         HIGH
ecosystem               AWS only           full k8s         full k8s
portability             none               real             real
compute premium         ~15-25%            ~15-25%          0

8 engineers, 12 Java services, no platform team
  -> ECS on Fargate. The premium is far cheaper than the fraction of an
     engineer that operating EKS actually consumes.

CHANGE THE ANSWER IF
  the team already runs Kubernetes competently
  you need the ecosystem specifically (operators, Argo Rollouts, KEDA, a mesh)
  portability is a REQUIREMENT, not a slide
  you are heading to dozens of services across several teams

Migrating from ECS to EKS later is real but bounded work: the container images and twelve-factor discipline transfer unchanged, and what you rewrite is the orchestration layer. So starting on ECS is not a trap, provided application code stays free of platform-specific assumptions.

Principle

Choose the least infrastructure that meets the requirement. If nobody owns the platform, you have not chosen Kubernetes - you have chosen an unmaintained cluster, which is worse than ECS by every measure.

Chapter 24

Lambda

One execution environment per concurrent request, reused between invocations. Cold starts, concurrency and connection pools are what break when a JVM service is lifted into it unchanged.

Lambda creates an execution environment per concurrent request. Within one environment invocations are serial. Code outside the handler runs once per environment and is reused — so clients, connection pools and anything expensive to construct belong there. Creating an S3 client per invocation is the most common Lambda performance bug.

Init versus invoke
// WRONG - per invocation
public Response handleRequest(Event e, Context ctx) {
    var s3 = S3Client.create();                  // ~200ms, EVERY time
    var conn = DriverManager.getConnection(URL); // a new DB connection each time
}

// RIGHT - init once per environment
private static final S3Client S3 = S3Client.create();
private static final DataSource DS = pool(2);    // SMALL pool
static { warmUp(); }
The three
WHAT BREAKS ON A NAIVE LIFT-AND-SHIFT

COLD STARTS      a Spring Boot context is 5-10 SECONDS
                 fixes: SnapStart (snapshot the initialised JVM), provisioned
                 concurrency (paying for idle - which defeats the point),
                 a leaner framework, or not using Lambda for this workload

CONCURRENCY      per-account and per-region, shared across every function
                 300 concurrent x 5 connections = 1,500
                 RDS max_connections = 200  -> INSTANT OUTAGE for every other
                 service on that database
                 fixes: reserved concurrency as a CAP, a tiny pool per
                 environment, and RDS Proxy to multiplex

HARD LIMITS      15-minute timeout | 6 MB sync payload | 10 GB memory
                 CPU is proportional to MEMORY - a CPU-bound function gets
                 faster AND often cheaper by being given more memory
Interview angle

Error handling depends entirely on invocation type, which trips people up. Synchronous: no retry, the error returns to the caller. Asynchronous: two retries then a destination or DLQ if configured, otherwise SILENT LOSS. Poll-based: the event source's semantics apply. Always configure a failure destination for async.

IV. Storage and Data
Chapter 25

S3 fundamentals

Eleven nines of durability, lower availability, and strong read-after-write consistency since 2020. The cost trap is requests and object count, not bytes.

Durability — the probability an object survives — is designed at eleven nines by replicating across at least three availability zones. Availability — the probability you can reach it right now — is a much lower 99.99%. So S3 essentially never loses data and can be briefly unreachable, which means your client needs retries.

Since December 2020 S3 is strongly read-after-write consistent for PUTs of new objects, overwrites and deletes, at no cost and with no configuration. A generation of blog posts and interview answers is built on the old eventually-consistent behaviour, so knowing it changed is a small current-knowledge signal. Bucket configuration changes remain eventually consistent.

Classes
STORAGE CLASSES - trading storage price against RETRIEVAL price and latency
  Standard              no retrieval fee, ms access
  Standard-IA           cheaper storage, retrieval fee, 30-day minimum,
                        128KB minimum billable size
  One Zone-IA           cheaper again, single AZ - only for RE-DERIVABLE data
  Glacier Instant       cheap, ms access, 90-day minimum
  Glacier Flexible      minutes to hours retrieval
  Deep Archive          cheapest, hours, 180-day minimum
  Intelligent-Tiering   moves objects automatically, small monitoring fee
                        -> the default when access patterns are unknown
Principle

The expensive mistake is not storage - it is REQUESTS. Millions of small PUTs cost more in requests than in bytes, and lifecycle transitions are billed PER OBJECT, so moving ten million tiny files to Glacier can cost more than leaving them in Standard forever.

Chapter 26

S3 in production

Presigned URLs so your service never proxies bytes, event notifications to SQS rather than Lambda, and a lifecycle policy that aborts incomplete multipart uploads.

Presigned URLs

For uploads of any size, your service authorises and signs while the client uploads directly to S3 — so bandwidth and memory never touch your instances. Constrain the signature narrowly: short expiry, an exact key chosen server-side, a content-length range, a pinned content type. A presigned URL is a bearer token, so scope is the only control.

Java
var key = "uploads/%s/%s".formatted(tenantId, UUID.randomUUID());  // SERVER chosen

var presign = s3Presigner.presignPutObject(b -> b
    .signatureDuration(Duration.ofMinutes(10))
    .putObjectRequest(r -> r
        .bucket("merchant-uploads-prod")
        .key(key)                                  // exact object
        .contentType("application/pdf")
        .contentLength(declaredSize)               // client cannot exceed
        .serverSideEncryption(ServerSideEncryption.AWS_KMS)));

uploadRepo.record(tenantId, key, declaredSize, PENDING);   // we know what to expect
Lifecycle
LIFECYCLE POLICY EVERY BUCKET SHOULD HAVE
{ "Rules": [
  { "Id": "abort-orphaned-multipart",             // THE ONE EVERYONE FORGETS
    "Status": "Enabled",
    "AbortIncompleteMultipartUpload": { "DaysAfterInitiation": 7 } },
    // incomplete parts are billed INDEFINITELY and do not appear in a listing

  { "Id": "tier-large-objects-only",
    "Filter": { "And": { "Prefix": "settlements/",
                         "ObjectSizeGreaterThan": 131072 } },   // 128KB
    "Transitions": [ { "Days": 30,  "StorageClass": "STANDARD_IA" },
                     { "Days": 180, "StorageClass": "GLACIER_IR" } ] },

  { "Id": "expire-old-versions",     // versioning without this = never delete
    "NoncurrentVersionExpiration": { "NoncurrentDays": 90 } } ] }

Trigger processing with an event notification to SQS, not directly to Lambda: the queue gives you buffering, per-message retry, a dead-letter queue, and controlled concurrency so a bulk upload does not instantly spawn hundreds of Lambdas and exhaust your database connections.

Principle

Making a bucket genuinely private: Block Public Access at account AND bucket level (it overrides any policy anyone later writes), a bucket policy denying requests without aws:SecureTransport, access via VPC endpoints with an endpoint condition, and Access Analyzer reporting anything reachable from outside.

Chapter 27

EBS and instance storage

EBS is network-attached, AZ-scoped and persists independently of the instance. Instance store is local NVMe, extremely fast, and gone when the instance stops.

Types
gp3    general purpose SSD. Baseline 3,000 IOPS and 125 MB/s INDEPENDENT of
       size - which is the improvement over gp2, where performance scaled
       with capacity. Provision more IOPS and throughput separately.
io2    provisioned IOPS, higher durability, for demanding databases
st1    throughput-optimised HDD, for large sequential reads
INSTANCE STORE  local NVMe, hundreds of thousands of IOPS,
                EPHEMERAL: lost on stop, terminate or host failure

The property that shapes architecture is that an EBS volume lives in one availability zone and cannot be attached from another. So an instance with an EBS volume is pinned to its zone, and recovering from a zone failure means restoring from a snapshot into another zone — not moving the volume.

Snapshots are incremental, stored in S3, and regional rather than zonal — which is how you move a volume between zones or regions. They are also crash-consistent by default: a snapshot of a running database captures the disk as it was at an instant, which for a mid-write database is a torn state. Application-consistent snapshots need the filesystem quiesced or the database's own mechanism.

Principle

Instance store is right for caches, scratch space, and anything you can rebuild - and never for a primary copy. Its speed is genuine and its ephemerality is absolute: a STOP loses it, not only a termination.

Interview angle

Unattached volumes and orphaned snapshots are a recurring silent cost. They survive the instance, they survive a StatefulSet deletion in Kubernetes, and nothing reports them by default - so a scheduled check for volumes in the 'available' state is worth having.

Chapter 28

RDS

Multi-AZ is availability with a non-readable standby; read replicas are scale with lag. They solve different problems and production usually needs both.

Multi-AZ maintains a synchronous standby in another zone: a commit is not acknowledged until it is durable on both. That is why it roughly doubles cost and slightly raises write latency. The standby is not readable — it exists purely so that when the primary fails, AWS promotes it and updates the DNS endpoint. Failover takes typically 60–120 seconds and every open connection is dropped.

Read replicas replicate asynchronously and are readable with their own endpoints. They scale reads, can live in another region, and can be promoted manually. But they lag — usually milliseconds, occasionally seconds under write bursts — so reading your own write through one is a real and common bug.

The distinction
MULTI-AZ            synchronous, NOT readable, ~60-120s failover
                    -> AVAILABILITY
READ REPLICA        asynchronous, readable, lags
                    -> READ SCALE, and read-your-own-writes is NOT guaranteed

ROUTE READS CORRECTLY
  writer endpoint   all writes, plus reads inside a read-your-writes window
  reader endpoint   reporting, dashboards, exports, analytics

# Alarm on ReplicaLag, and check it in the application for anything
# correctness-sensitive - a health check can refuse to serve stale reads.

Other things RDS gives you: automated backups with point-in-time recovery to any second in the retention window, minor version patching in a maintenance window, parameter groups, Performance Insights for query-level diagnosis, and encryption at rest with KMS (which must be enabled at creation — you cannot encrypt an existing instance in place).

Interview angle

Multi-AZ DB CLUSTER deployments are a newer option with two READABLE standbys and much faster failover. If someone says they offload reporting to their Multi-AZ standby, they mean either that or a read replica - classic Multi-AZ has no readable standby at all.

Chapter 29

Aurora

Storage decoupled from compute and replicated six ways across three zones. Replicas read the same storage rather than replaying a log, which changes both lag and failover time.

Aurora separates the database engine from a distributed storage layer that replicates every write six ways across three availability zones, with quorum-based durability. The engine writes log records rather than pages, which is why write amplification is much lower than a traditional engine over network storage.

What it buys
CONSEQUENCES OF THE SHARED STORAGE LAYER
  replicas read the SAME storage -> lag typically well under 100ms,
                                    not a log replay
  up to 15 replicas
  failover in SECONDS, not a minute - there is no data to move, a reader
    is simply promoted
  storage grows automatically to 128 TiB; you pay for what you use
  fast cloning - a copy-on-write clone of a production database for testing,
    created in minutes regardless of size
  backtrack (MySQL) - rewind the cluster in place, without a restore

ENDPOINTS
  cluster endpoint      always the writer
  reader endpoint       load-balances across replicas
  instance endpoints    a specific instance

Aurora Serverless v2 scales capacity in fine-grained increments and is genuinely useful for variable or unpredictable workloads. Aurora Global Database replicates to another region with typical lag well under a second and promotion in about a minute — which is what makes a five-second RPO achievable across regions.

Principle

Aurora is not automatically the right answer. It costs more per instance-hour than RDS, and for a small, steady, single-region workload plain RDS Postgres with Multi-AZ is simpler and cheaper. Aurora earns its cost with read scale, fast failover, or a cross-region requirement.

Chapter 30

Failover, and what actually breaks

AWS promotes a standby in 60–120 seconds. Your Java service is often down for much longer, because the JVM cached the DNS resolution and the pool holds sockets to a machine that no longer exists.

This is the gap that matters and the one people do not test: the difference between AWS recovered and we recovered. Failover repoints a DNS name. If your client does not re-resolve it, nothing improves.

The real timeline
WHAT AWS DOES                                    ~60-120s
  detect failure -> promote the standby -> repoint the DNS CNAME

WHAT AN UNCONFIGURED JVM DOES
  java.security caches the OLD address (aggressively, by default in some setups)
  HikariCP holds 20 sockets to an instance that no longer exists
  every borrow blocks until the OS TCP timeout - which can be MINUTES
  => "the database failed over in 90 seconds but we were down for 10 minutes"

FIX IN THREE PLACES
# 1. JVM
-Dsun.net.inetaddr.ttl=5            # or networkaddress.cache.ttl=5

# 2. Pool
spring.datasource.hikari:
  connection-timeout: 3000          # fail fast, do not block a request thread
  validation-timeout: 1000
  keepalive-time: 30000
  max-lifetime: 600000              # recycle well below any idle timeout

# 3. Driver - failover-aware, handles topology changes directly
spring.datasource.url: jdbc:aws-wrapper:postgresql://cluster.cluster-abc...:5432/app
spring.datasource.driver-class-name: software.amazon.jdbc.Driver

Test it deliberately with a forced failover — AWS Fault Injection Service can trigger one — and measure the application's recovery time, not the database's. That number is the most valuable single output of a chaos exercise, because it is almost always worse than people assume and it is entirely fixable with configuration.

Principle

The same discipline applies to caches and queues after any failover: expect a COLD CACHE in the new topology, and make sure the resulting miss storm cannot overwhelm the newly promoted database. That cascading failure is a very common way a technically successful failover still causes an outage.

Chapter 31

DynamoDB: the key is the query plan

Design the access patterns first, then the key. In a relational database you model the data and query it later; here the key IS the query plan, and getting it wrong is not a slow query but an access pattern you cannot serve.

The partition key is hashed to choose a physical partition, so it determines placement and how load spreads. The sort key orders items within a partition and gives you range queries, prefix matching and reverse ordering — all cheap, because they are a contiguous read of one partition.

Design order
ACCESS PATTERNS FIRST - write them down before the schema
  1. transactions for an account, date range, newest first    <- primary
  2. one transaction by id
  3. transactions for a merchant, date range                  <- needs a GSI
  4. pending transactions needing settlement                  <- sparse GSI

TABLE: transactions
  PK  ACCOUNT#{accountId}
  SK  TXN#{iso8601}#{txnId}        # range + uniqueness

# 1. one Query, one partition, sorted, paginated
Query  PK = "ACCOUNT#123"
       SK between "TXN#2024-03-01" and "TXN#2024-04-01"
       ScanIndexForward = false                  # newest first

Hot partitions

A single key taking disproportionate traffic hits physical per-partition ceilings — roughly 3,000 read units and 1,000 write units per second — so one large merchant can throttle while the table as a whole is far under capacity. Adaptive capacity absorbs moderate skew; it does not absorb an unbounded hotspot.

Spreading load
FIXES
  shard suffix     PK = ACCOUNT#123#{hash(txnId) % 10}
                   reads fan out across shards and merge
  time bucket      PK = ACCOUNT#123#2026-08
                   also bounds partition size

THE PATHOLOGICAL CASE
  a monotonically increasing partition key (a raw timestamp)
  -> every write lands on the same partition
  -> you have built a queue with one lane
Interview angle

The SPARSE INDEX is the most useful DynamoDB trick: write the indexed attribute ONLY when status is PENDING. Items without it are absent from the index, so the index stays tiny and scanning it is cheap - and deleting the attribute on settlement removes the item from the index. Ideal for work queues.

Chapter 32

DynamoDB in production

Conditional writes are the idempotency primitive. GSIs are separate tables with their own capacity. Streams give you a change log you did not have to build.

Conditional writes
// IDEMPOTENCY - a CONDITION, not a read-then-write. No race, no lock.
try {
  ddb.putItem(r -> r.tableName("processed_events")
      .item(Map.of("pk", s("EVT#" + eventId),
                   "ttl", n(Instant.now().plus(30, DAYS).getEpochSecond())))
      .conditionExpression("attribute_not_exists(pk)"));      // <- the whole trick
  process(event);                                             // first time only
} catch (ConditionalCheckFailedException dup) {
  log.debug("duplicate delivery of {}", eventId);              // safe no-op
}

// OPTIMISTIC CONCURRENCY - no SELECT FOR UPDATE, no lock table
ddb.updateItem(r -> r.tableName("accounts")
    .key(Map.of("pk", s("ACCOUNT#123")))
    .updateExpression("SET balance = :new, version = :nv")
    .conditionExpression("version = :cur")          // fails if someone else won
    ...);
  • GSI — effectively a separate replicated table with its own key and its own capacity, eventually consistent. Project only the attributes you read, because projection drives storage and write cost. And an under-provisioned GSI throttles writes to the base table, which is a genuinely surprising failure.
  • LSI — shares the partition key with an alternative sort key, must be created with the table, and caps that partition at 10GB. GSIs are used far more often.
  • Transactions — ACID across up to 100 items at roughly twice the cost, with condition expressions per item.
  • Streams — an ordered change log per partition key, retained 24 hours, driving projections, search indexing or event fan-out. Because it derives from the transaction log, it is effectively a built-in outbox.
Capacity
ON-DEMAND       per request, absorbs spikes instantly, scales to zero cost
                right for unpredictable or new workloads
PROVISIONED     reserved capacity with auto scaling, meaningfully cheaper at
                steady high volume - but auto scaling reacts in MINUTES,
                so it does not protect you from a sudden spike

Rule of thumb: peak-to-average above ~4:1 -> on-demand usually wins.
Start on-demand, measure for a month, then decide.
Principle

Be able to argue single-table design BOTH ways. For: one Query returns related entities of different types in a single round trip. Against: the schema is opaque, hard to onboard onto, and a genuinely new access pattern can require a migration rather than a new index.

Chapter 33

Choosing a datastore

Reason from access pattern, consistency requirement, loss tolerance and — the one people skip — what someone will ask of this data in a year that nobody has thought of yet.

Three workloads
                     SESSIONS          LEDGER              DOCUMENTS (40TB)
access pattern       GET/PUT by key    txn writes +        PUT once,
                                       AD-HOC SQL          GET rarely
consistency          eventual is fine  STRICT, ACID        n/a
loss tolerance       inconvenience     unacceptable        unacceptable
query flexibility    none              HIGH (unknown       metadata only
                                       future questions)
latency              sub-ms            low ms              seconds ok
------------------------------------------------------------------------
CHOICE               ElastiCache       Aurora PostgreSQL   S3 + DynamoDB
                     (or DynamoDB TTL) Multi-AZ + readers  (Intelligent-Tiering)

The ledger is the one where reaching for DynamoDB would be the wrong instinct, and saying so explicitly is the point. You need multi-row ACID transactions, foreign keys, and ad-hoc queries during reconciliation and investigation that nobody designed for. Immutability of a ledger is achieved with an append-only schema and permissions — INSERT only, corrections as reversing entries — not by choosing a different database.

Two details
-- Immutability is schema and permissions, not a product choice
CREATE TABLE ledger_entry (...);
REVOKE UPDATE, DELETE ON ledger_entry FROM app_role;   -- INSERT only

-- Cost sanity check on the documents
  40 TB S3 Standard                ~  $920 / month
  40 TB Glacier Instant Retrieval  ~  $160 / month
  40 TB on RDS gp3 storage         ~$4,600 / month   <- and backups multiply it
-- blobs never belong in a relational database
Principle

Migrating a datastore is a project; choosing one is a meeting. That asymmetry is why this is one of the four irreversible decisions - and why 'what will someone ask of this data in a year' deserves more weight than today's access pattern.

Chapter 34

Schema migration without downtime

Expand, backfill, flip, contract — in separate releases. It is the only pattern where rollback remains possible at every step, which is what 'zero downtime' actually requires.

The failure people design around is downtime during the DDL. The failure that actually hurts is a deploy you cannot roll back, because the schema has already moved forward and the previous application version no longer works against it.

The four steps
RELEASE 1 - EXPAND. Old code still works. Rollback is FREE.
  SET lock_timeout = '3s';                      -- do not wait behind a long read
  ALTER TABLE settlement ADD COLUMN currency_code char(3);   -- nullable: INSTANT
  -- app v2: writes BOTH columns, reads the old one

RELEASE 1.5 - BACKFILL in bounded batches, watching replica lag
  UPDATE ... WHERE id >= lo AND id < hi AND currency_code IS NULL;  -- 5k rows
  COMMIT; check replica lag; pause if > 5s; sleep 50ms; repeat
  -- NEVER one UPDATE over 400M rows: locks held for the duration, enormous WAL,
  --   table bloat, autovacuum overwhelmed, REPLICA LAG through the roof

RELEASE 2 - FLIP READS behind a flag. Old code STILL works.

RELEASE 3 - CONTRACT. Only once you will never roll back past release 2.
  ALTER TABLE settlement DROP COLUMN currency;
DDL
WHAT IS ACTUALLY ONLINE (PostgreSQL)
  ADD COLUMN nullable                instant, metadata only
  ADD COLUMN NOT NULL DEFAULT const  instant in PG 11+, FULL REWRITE before
  ADD CONSTRAINT                     use NOT VALID, then VALIDATE separately
  CREATE INDEX                       CONCURRENTLY, or you lock WRITES for the
                                     whole build - minutes on 400M rows
  DROP COLUMN                        instant (space reclaimed later)

THE OPERATIONAL DETAIL THAT CATCHES PEOPLE
  a migration WAITING for a lock QUEUES every subsequent query behind it.
  One long-running read turns an "instant" DDL into a full outage.
  -> SET lock_timeout and RETRY. Never let a migration block indefinitely.
Principle

The invariant: at EVERY point, version N-1 of the application works against the CURRENT schema. That makes rollback a code-only operation - fast, safe and boring. Without it, your rollback plan is a database restore measured in hours.

Interview angle

Run migrations as a separate step BEFORE the service deploy - a standalone task or a pre-sync hook, not an init container. With three replicas, three init containers race to run the same migration, which most tools serialise with an advisory lock and turn into a confusing slow start.

V. Messaging and Asynchronous Work
Chapter 35

Why asynchronous

Because it converts a timeout into a queue. A synchronous API under overload loses work; an asynchronous one grows a metric you can alarm on and scale against.

Go asynchronous when the work exceeds a couple of seconds, when the downstream cannot absorb the arrival rate, or when the work must survive a client disconnect. The shape: validate, persist the request with an idempotency key, enqueue, and return 202 Accepted with a status URL.

The difference
SYNCHRONOUS under overload
  connections pile up -> threads exhausted -> 504 from the load balancer
  -> the work is LOST, and the client retries, adding more load

ASYNCHRONOUS under the same load
  queue depth grows -> a metric you ALARM on and SCALE against
  -> nothing is lost
  client dies       -> the work continues, the result is still there
  client retries    -> the idempotency key makes it a no-op
The async API shape
POST /settlements/reconcile
Idempotency-Key: 7f3a-...                # client-supplied, required
  -> 201 Accepted
     Location: /jobs/abc123
     { "jobId": "abc123", "status": "PENDING" }
  # server: conditional insert on the key, then enqueue.
  #         duplicate key -> return the EXISTING jobId, do not enqueue again

GET /jobs/abc123
  { "status": "RUNNING", "progress": 0.4 }
  { "status": "SUCCEEDED", "result": "/settlements/reports/2026-08" }
  { "status": "FAILED", "error": { "code": "PARTNER_FILE_MALFORMED" } }

For clients that want a synchronous feel, use long polling with a deadline: accept the request, wait a bounded few seconds, and return the result if it completed or 202 with a status URL if it did not. The common fast case feels synchronous; the slow case is not a failure.

Principle

Return Retry-After on the status endpoint and prefer a push channel for anything long-running. Polling every 100ms across thousands of clients is a self-inflicted load problem, and the status endpoint is often the first thing to fall over in an otherwise well-designed async system.

Chapter 36

SQS

Receiving hides a message for the visibility timeout; you delete it after success. That means at-least-once delivery, never exactly-once, and idempotency is the consumer's job.

SQS does not delete a message when you receive it — it makes it invisible for the visibility timeout. You process, then call DeleteMessage. If your consumer crashes, the timeout expires and the message reappears. That is the entire reliability model.

The knobs
VISIBILITY TIMEOUT must exceed your p99 processing time
  too short -> a slow-but-succeeding job is redelivered while still running
               -> two consumers process concurrently -> double writes
  too long  -> a genuinely crashed consumer's message is stuck for that period
  long jobs -> HEARTBEAT with ChangeMessageVisibility rather than a huge value

DEAD-LETTER QUEUE
  RedrivePolicy: maxReceiveCount 4 -> the message moves to a DLQ
  without one, a POISON MESSAGE is redelivered forever - and on a FIFO queue
  it blocks its entire message group permanently
  alarm on ANY non-zero DLQ depth, and document the redrive path

STANDARD vs FIFO
  standard  near-unlimited throughput, at-least-once, best-effort ordering
            (so never rely on order)
  FIFO      strict ordering and dedup WITHIN a message group, 5-minute dedup
            window, much lower throughput (higher with high-throughput mode)
            MessageGroupId is BOTH your ordering guarantee AND your
            parallelism unit
A safe consumer
@SqsListener("settlements")
public void handle(SettlementMessage msg, Acknowledgement ack) {
    // idempotency key from CONTENT, never from the SQS messageId
    // (which changes on redelivery)
    String key = "SETTLE#" + msg.fileId() + "#" + msg.rowRange();
    try {
        ddb.putItem(r -> r.tableName("processed")
            .item(Map.of("pk", s(key), "ttl", n(ttl30d())))
            .conditionExpression("attribute_not_exists(pk)"));
    } catch (ConditionalCheckFailedException dup) {
        ack.acknowledge(); return;            // already done - delete, do not redo
    }
    try (var hb = visibility.heartbeat(msg, ofSeconds(60))) { process(msg); }
    ack.acknowledge();                        // delete ONLY after success
}
Principle

Exactly-once delivery does not exist, because processing and deletion are in two systems with no atomic commit between them. Exactly-once EFFECT is achieved by making the processing idempotent - a conditional write on a business key, not a try/catch.

Chapter 37

SNS and EventBridge

SQS is a queue, SNS is fan-out filtered on attributes, EventBridge routes on the event body with archive and replay. For a domain event with several consumers, EventBridge targeting per-consumer queues is the pattern.

Three shapes
SQS           a queue: one logical consumer, per-message retry, DLQ, buffering
SNS           pub/sub fan-out, filtered on message ATTRIBUTES, no retention
EventBridge   a bus: content-based routing on the whole event BODY,
              schema registry, ARCHIVE and REPLAY, many native targets
Fan out to queues
// Producer: one call. Knows nothing about consumers.
eventBridge.putEvents(r -> r.entries(e -> e
    .eventBusName("payments")
    .source("payments.settlement")
    .detailType("PaymentSettled")
    .detail(json(Map.of(
        "eventId", UUID.randomUUID().toString(),   // consumer idempotency key
        "version", "1.0",
        "settlementId", id, "merchantId", merchantId,
        "amountMinor", 4250, "currency", "GBP")))));

// Each consumer owns a RULE and a QUEUE. The producer never changes.
Rule "treasury-large-gbp"  -> SQS treasury-queue
  { "detail-type": ["PaymentSettled"],
    "detail": { "currency": ["GBP"],
                "amountMinor": [{ "numeric": [">", 1000000] }] } }
    //          ^ content-based on the BODY. SNS filters only on attributes.

The queue in front of each consumer is the failure isolation boundary. A slow or broken consumer accumulates depth in its own queue and affects nobody — not the producer, not the other consumers. Each gets its own visibility timeout, retry policy, DLQ and consumption rate. Deliver straight to a Lambda and a downstream outage becomes retries, then loss, and a spike becomes a concurrency storm that exhausts a shared account limit.

Principle

The event schema is a public contract: version it, add fields additively, never remove or repurpose one. And publish enough that consumers do not need to call back to enrich - an event carrying only an id forces every consumer to make an API call, which recreates the coupling you were removing AND creates a load spike on the producer at the moment it published.

Chapter 38

The dual-write problem and the outbox

Writing to your database and publishing an event are two systems with no atomic commit. The outbox makes the intent to publish part of the same local transaction.

Crash between the two and you get either a state change nobody heard about or an event for a change that rolled back. Ordering does not fix it: publish-then-commit gives phantom events, commit-then-publish gives lost events, and wrapping the publish in the transaction achieves nothing because a broker acknowledgement cannot be rolled back.

Java
@Transactional
void settle(Settlement s) {
    repo.save(s);
    outbox.insert(new OutboxRow(
        UUID.randomUUID(),        // eventId - the consumer's idempotency key
        "PaymentSettled",
        s.getAccountId(),         // ORDERING KEY
        json(s),                  // FULL PAYLOAD, snapshotted at write time
        Instant.now(), null));    // sentAt
}   // ONE database, ONE commit, atomic and fully unit-testable

// RELAY (poller). SKIP LOCKED prevents two instances double-publishing.
@Scheduled(fixedDelay = 500)
void relay() {
    var sql = "SELECT * FROM outbox WHERE sent_at IS NULL "
            + "ORDER BY created_at LIMIT 100 FOR UPDATE SKIP LOCKED";
    for (var r : jdbc.query(sql, MAPPER)) {
        eventBridge.putEvents(toEntry(r));       // may publish then crash...
        jdbc.update("UPDATE outbox SET sent_at = now() WHERE id = ?", r.id());
    }                                             // ...so it republishes. Fine.
}

Store the full payload in the row, not just an id. Otherwise the relay must re-read the entity and may see a later state, publishing an event that does not match the transaction it came from.

Principle

The guarantee is at-least-once publication corresponding exactly to committed state. NOT exactly-once - the relay can publish and crash before marking the row sent. So consumer idempotency is the contract, not a footnote; a design without it has moved the bug rather than fixed it.

Interview angle

DynamoDB Streams gives you this for free: the stream derives from the transaction log, so every record corresponds to a committed change, ordered per partition key. That is a genuinely good reason to reach for DynamoDB on event-heavy write paths.

Chapter 39

Change data capture

Read the transaction log and emit every committed change. No application code, no dual-write problem — and events that are row changes rather than business intent.

CDC reads the database's write-ahead log (Postgres) or binlog (MySQL) and emits committed changes as events. On AWS that is DMS or Debezium, typically into MSK. Three properties follow directly: no application changes, no dual-write problem because the log is the record of committed state, and complete ordering per key.

Row changes versus intent
THE TRADE
  CDC gives:    { table: "settlement", op: "U",
                  before: { status: "PENDING" }, after: { status: "SETTLED" } }
  outbox gives: { type: "PaymentSettled", settlementId: ..., amountMinor: ... }

CDC couples consumers to your INTERNAL SCHEMA.
A column rename becomes a breaking change for every downstream team.
That is the main argument FOR an outbox with deliberately designed events.

Where CDC is clearly right: integrating with systems you do not control, replicating to a data platform, building search indexes, and migrating databases — which is exactly what DMS is designed for. Where an outbox is right: publishing domain events that other teams will build against.

Interview angle

DMS is genuinely used for ongoing replication, not only migrations, and is the low-effort choice. Debezium is more capable - better transformation and schema handling - at the cost of running it yourself on MSK Connect or Kubernetes.

Chapter 40

Streams: Kinesis and MSK

Choose a log over a queue when you need replay from an offset, multiple independent consumer groups, or per-key ordering. The tell: will anyone ever want to read this history again?

The decision
CHOOSE A LOG WHEN YOU NEED
  replay from an offset          a consumer had a bug -> re-read last week,
                                 with no backfill project
  multiple consumer GROUPS       each at its own pace; adding a 5th does not
                                 affect the other 4
  strict ordering within a key   guaranteed by partition
  retention as a source of truth rebuild projections from scratch

CHOOSE A QUEUE WHEN
  simple work distribution, per-message retry, a DLQ, no cluster to operate,
  and you will never need to re-read.

QUEUES CONSUME. LOGS RETAIN.
What you take on
OPERATIONAL COST - where teams underestimate

MSK       brokers, storage, and PARTITION COUNT planning (hard to change: more
          partitions breaks key-to-partition mapping and therefore ordering)
          CONSUMER GROUP REBALANCING pauses consumption, and a slow consumer
          can trigger a rebalance that slows everyone and triggers another
          CONSUMER LAG must be monitored and ALERTED - a lagging consumer is
          invisible until the retention window expires and data is GENUINELY
          LOST. This is the metric people forget.

KINESIS   no brokers, but shards with per-shard limits (1 MB/s in, 2 MB/s out)
          and resharding to manage

Consumer parallelism within a group is capped by partition count, so size partitions from your expected peak consumer count with headroom. Over-provisioning partitions is cheap; under-provisioning is a migration.

Principle

Alert on consumer lag measured in TIME, not only in records, and alarm well before it approaches the retention window. Past that point the data is gone, and no amount of scaling the consumer recovers it.

Chapter 41

Step Functions and sagas

Durable state you did not write. Reach for it when a process spans services and outlives a request — and when you need compensation, because there is no distributed rollback.

A database transaction gives atomicity across rows in one database. It gives you nothing across services: you cannot roll back an HTTP call to a partner, a message already published, or a refund already issued. And holding a transaction open across a network call is actively harmful, pinning a connection and holding locks for someone else's latency.

The capabilities
WHAT IT GIVES YOU THAT YOUR OWN CODE DOES NOT
  durable state          every transition persisted -> a crash RESUMES
  per-step retry         backoff and jitter, declared not coded
  catch and fallback     route specific errors to compensating branches
  parallel and map       fan-out with a concurrency limit
  WAIT states            days, without a scheduler you maintain
  callbacks              waitForTaskToken - pause until a human or a partner responds
  visual execution history - genuinely the best debugging artefact in AWS
A saga
{ "StartAt": "ReserveFunds",
  "States": {
    "ReserveFunds": {
      "Type": "Task", "Resource": "arn:aws:states:::lambda:invoke",
      "Retry": [{ "ErrorEquals": ["States.TaskFailed"], "IntervalSeconds": 2,
                  "MaxAttempts": 4, "BackoffRate": 2.0, "JitterStrategy": "FULL" }],
      "Next": "NotifyPartner" },
    "NotifyPartner": {
      "Type": "Task", "Resource": "...",
      "Catch": [{ "ErrorEquals": ["States.ALL"], "Next": "ReleaseFunds" }],
      "Next": "AwaitConfirmation" },
    "AwaitConfirmation": {
      "Type": "Task", "Resource": "arn:aws:states:::lambda:invoke.waitForTaskToken",
      "TimeoutSeconds": 259200,                 // 3 days, no scheduler needed
      "Catch": [{ "ErrorEquals": ["States.Timeout"], "Next": "ReleaseFunds" }],
      "Next": "Settle" },
    "ReleaseFunds": { "Type": "Task", "Resource": "...", "Next": "Fail" },
    "Settle": { "Type": "Task", "Resource": "...", "End": true },
    "Fail": { "Type": "Fail" } } }

Standard workflows are durable with exactly-once transitions, up to a year long, billed per transition, with full history. Express workflows are up to five minutes, at-least-once, and cheap enough for high-volume event processing.

Principle

Keep the state machine THIN and the logic in ordinary code it invokes. Business logic in a JSON definition is hard to unit-test; business logic in a Lambda the machine calls is normal Java. Let the orchestrator own sequencing and durability, nothing else.

Chapter 42

Idempotency

The single most important property in a distributed system that handles money. Derive a business key from content, claim it conditionally, and make replay a safe no-op.

Every mechanism in this part delivers at-least-once: SQS redelivers, the outbox republishes, a client retries a timed-out request, a failover replays. So the question is never how do I avoid duplicates — it is how do I make a duplicate harmless.

Two decisions
THE KEY
  derive it from CONTENT, deterministically:
    fileId + rowRange | clientIdempotencyKey | eventId from the producer
  NEVER from something that changes on redelivery:
    the SQS messageId, a timestamp, a random UUID generated by the consumer

THE CLAIM - a conditional write, not a read-then-write
  DynamoDB   conditionExpression("attribute_not_exists(pk)")
  Postgres   a UNIQUE constraint, and catch the duplicate-key violation
             (the constraint does the work; the catch is just control flow)

  Both are atomic. A read-then-write has a race and will eventually lose it.
Java
// The complete pattern
String key = "SETTLE#" + msg.fileId() + "#" + msg.rowRange();
try {
    ddb.putItem(r -> r.tableName("processed")
        .item(Map.of("pk", s(key), "ttl", n(ttl30d())))     // TTL: bounded growth
        .conditionExpression("attribute_not_exists(pk)"));
} catch (ConditionalCheckFailedException dup) {
    return AlreadyProcessed.INSTANCE;                        // safe no-op
}
process(msg);

// TTL matters: without it the table grows forever. Set it longer than any
// plausible redelivery window - 30 days is generous and cheap.
Principle

Claim BEFORE doing the work, not after. Claiming afterwards leaves a window where a crash between the work and the claim causes the work to happen twice. Claiming first means a crash after the claim leaves work undone - which is recoverable by a reconciliation job, and is the far safer failure.

Interview angle

For a payments API, require an Idempotency-Key header from clients and honour it: return the ORIGINAL response for a repeated key rather than reprocessing. That turns client retries - which are constant at any real volume - from a correctness risk into a non-event.

VI. Observability and Operations
Chapter 43

CloudWatch metrics

Numeric time series with dimensions, aggregated progressively as they age. The cost trap is high-cardinality custom metrics — every unique dimension combination is billed separately.

Metrics are one-minute standard resolution or one-second high resolution, and they are aggregated as they age: high-resolution data is kept briefly, one-minute data for fifteen days, five-minute for sixty-three days, one-hour for fifteen months. So you cannot go back three months and look at per-minute detail — anything you will need at fine granularity later must be captured another way.

The cost trap
EVERY UNIQUE DIMENSION COMBINATION IS A SEPARATE BILLED METRIC

  Latency{Service, Operation}                  ~20 metrics    fine
  Latency{Service, Operation, MerchantId}      ~20 x 5,000    = 100,000 metrics
                                               -> costs more than the service

RULE: dimensions must be LOW cardinality.
      Service, Operation, Region, StatusClass - yes.
      MerchantId, UserId, RequestId, full URL path - never.

Put high-cardinality context in LOGS or TRACES, not in metric dimensions.

Embedded Metric Format solves the tension elegantly: your application emits a specially-structured JSON log line, and CloudWatch extracts metrics from it automatically. You get low-cardinality metrics for dashboards and alarms and high-cardinality context in the same event, without maintaining a separate metric filter.

EMF
{ "_aws": { "Timestamp": 1756000000,
            "CloudWatchMetrics": [{ "Namespace": "Payments",
                "Dimensions": [["Service","Operation"]],      // LOW cardinality
                "Metrics": [{ "Name": "Latency", "Unit": "Milliseconds" }] }] },
  "Service": "payments-api", "Operation": "settle",
  "Latency": 142,
  "merchantId": "ACME-GB-0042",     // high cardinality: stays in the LOG
  "txnId": "9f2c..." }
Interview angle

Container Insights and Lambda Insights give you per-container and per-function metrics without instrumentation, which is a fast way to get baseline visibility on an unfamiliar estate. They are billed per observation, so enable them deliberately rather than everywhere.

Chapter 44

CloudWatch Logs

Log groups default to NEVER EXPIRE. That single default is the most common silent cost accumulation on AWS.

Logs are streams grouped into log groups, each with a retention setting — and the default is indefinite. A log group created automatically by a Lambda or an EKS add-on accrues storage cost forever, invisibly, until someone audits the bill. Set retention on every group and enforce it with a Config rule so new ones cannot escape.

Two mechanisms
METRIC FILTER            evaluated AT INGESTION
  matches a pattern and emits a CloudWatch metric
  -> ALARMABLE, cheap, real time
  -> but only sees events arriving AFTER you create it. Useless for the past.

LOGS INSIGHTS            queries stored logs AFTER the fact
  a real query language
  -> billed PER GIGABYTE SCANNED. Narrow the time range.

THE RULE
  metric filters for anything you want to ALARM on
  Logs Insights for INVESTIGATION
  if you run the same Insights query on a schedule, it should be a filter
Insights
fields @timestamp, txn.id, error.message
| filter event = "settlement_failed"
| stats count() by error.code
| sort count desc
| limit 20

# Structured JSON logging makes this work. Unstructured prose means regex
# parsing at query time, which is slower, more expensive and more fragile.
logging.structured.format.console: ecs      # Spring Boot 3.4+
Principle

Drop noise BEFORE ingestion wherever you can - at the log shipper, or by not emitting it. CloudWatch bills on ingestion as well as storage, so filtering at query time means you have already paid for everything you filtered out.

Chapter 45

Alarms worth having

Alarm on symptoms with a bounded remediation. Set treatMissingData deliberately, and build a dead-man's switch — because the failure that produces no data produces no alarm.

Configuration
MetricName: Api5xxRate
Period: 60
EvaluationPeriods: 5
DatapointsToAlarm: 3          # 3 of 5 - tolerates a single blip
Threshold: 0.005
TreatMissingData: breaching   # DEFAULT is 'missing', which means an alarm on a
                              # metric that STOPS being published NEVER FIRES -
                              # exactly the silent failure you wanted to catch

# COMPOSITE alarms cut noise:
#   page only if error rate is high AND the service is receiving traffic

Symptoms, not causes

Alarm on what users experience — latency, error rate, saturation. High CPU with healthy latency needs no human; a deadlocked connection pool takes you down at 5% CPU and never trips a CPU alarm. So CPU alarms produce false positives and false negatives, which is the worst combination available. Cause metrics belong on the dashboard for diagnosis, not in the alert rules.

The set
ALARMS THAT EARN THEIR PLACE
  5xx rate above threshold, sustained          users are failing
  p99 latency above the SLO                    users are suffering
  SQS DLQ depth > 0                            work is being lost
  SQS oldest message age                       work is backing up
  consumer lag approaching retention           data will be LOST
  RDS free storage, connection count           a cliff you can see coming
  NO TRAFFIC for N minutes                     <- the dead-man's switch.
                                                  Zero errors because zero
                                                  requests. Nothing else fires.
Principle

If you cannot write a bounded remediation for an alarm, it should not page. Writing the runbook entry is a useful forcing function on alert quality - an alarm whose runbook says 'investigate' is not actionable, and that review often removes a third of a team's pages.

Chapter 46

Tracing

A span is one timed operation with a parent; a trace is the tree across services. It is the only signal that answers where the latency went — and propagation is the hard part.

A metric is an aggregate you alarm on, cheap and detail-free. A log is an event with context, expensive at volume and searchable. A span has a parent, and the tree of spans is the only one of the three that crosses service boundaries coherently.

Propagation
WHERE CONTEXT IS LOST - the traceparent header must survive every hop
  thread pools            a plain ExecutorService loses the thread-local
                          -> wrap it (ContextExecutorService)
  async boundaries        CompletableFuture, reactive schedulers
  MESSAGE QUEUES          no HTTP headers - inject into SQS message attributes
                          or Kafka headers, and extract on the consumer
                          (without this, every async workflow is a set of
                           ORPHAN traces)
  scheduled jobs          no incoming context - create a root span explicitly

Miss one hop and the trace FRAGMENTS into disconnected trees - which is worse
than no tracing, because it looks complete and hides the boundary you most
need to see.

Instrument with OpenTelemetry and export where you like — X-Ray today, something else later, without re-instrumenting. The OTel Java agent instruments HTTP, JDBC, messaging and Spring at class-load time with no code change, injected by JAVA_TOOL_OPTIONS or by an operator, so twelve services get tracing from a configuration change.

Sampling
# The collector is where cost is controlled
processors:
  attributes/scrub:
    actions: [{ key: db.statement, action: delete }]     # SQL can contain PII
  tail_sampling:
    decision_wait: 10s
    policies:
      - { name: errors, type: status_code, status_code: { status_codes: [ERROR] } }
      - { name: slow,   type: latency, latency: { threshold_ms: 2000 } }
      - { name: base,   type: probabilistic, probabilistic: { sampling_percentage: 1 } }

# Head sampling at 1% is cheap and throws away exactly the traces you want.
# Tail sampling decides AFTER the trace completes.
Principle

Put the trace id in every log line. Then a slow span leads to exactly the log lines emitted during it, and an error log leads to the full trace. One field, and it is worth more than either signal alone.

Chapter 47

SLOs and error budgets

Pick a user-facing SLI, set a target, and derive alerting from the burn rate rather than from thresholds. The budget is what turns reliability into an engineering trade-off with a number.

Definition
SLI   proportion of requests returning 2xx/3xx in under 300ms,
      measured AT THE LOAD BALANCER
      (closest to what the client experienced - it includes queueing and
       connection time your application metrics never see)

SLO   99.9% over a rolling 30 days
      -> ERROR BUDGET = 0.1% x 30 days = ~43 minutes of failure per month
Alerting
BURN RATE - how fast you are consuming the budget
  1x     exactly on budget, exhausted at day 30
  14.4x  2% of the budget in one hour -> the whole month gone in ~50 hours -> PAGE
  6x     10% in six hours -> TICKET

MULTI-WINDOW, MULTI-BURN-RATE  (the short window makes it RESOLVE quickly too)
- alert: FastBurn
  expr: (error_ratio_1h > 14.4 * 0.001) and (error_ratio_5m > 14.4 * 0.001)
  severity: page
- alert: SlowBurn
  expr: (error_ratio_6h > 6 * 0.001) and (error_ratio_30m > 6 * 0.001)
  severity: ticket

# Low-traffic caveat: ratios are noisy at low volume. Add a minimum-volume
# guard, or you page on three failed health checks at 4am.

The budget's real value is the conversation it enables: budget remaining means ship features and take risks; budget exhausted means freeze and spend the sprint on reliability. That converts an argument between two people with opinions into a policy agreed in advance — and it only works if leadership agreed to it beforehand.

Principle

Pick the target from what users need and what you can afford, not from how many nines sound impressive. Measure current performance for a month first, and check it against your dependencies - an SLO tighter than what RDS or a partner API offers is arithmetically unachievable.

Chapter 48

Investigating latency

Move edge inward and bisect by dimension — target, AZ, tenant, route, time. Guessing at causes is how an investigation takes a day; halving the space takes twenty minutes.

Start at the ALB access logs, because they are the only source with per-request detail including which target served it. Group by target and by availability zone first, because slow for some users most often means one bad instance or one impaired zone — and that is one query away.

Edge inward
-- 1. is it a subset of TARGETS or AZs?
SELECT target_az, target,
       approx_percentile(target_processing_time, 0.99) AS p99, count(*)
FROM alb_logs WHERE day = '2026/08/29'
GROUP BY target_az, target ORDER BY p99 DESC;
--  one target at 4.2s and the rest at 0.09s  -> ONE BAD INSTANCE. Done.
--  all targets in one AZ slow                -> IMPAIRED ZONE. Evacuate.

-- 2. a subset of ROUTES?     GROUP BY request_url
-- 3. a subset of TENANTS?    requires tenant on the trace/log
-- 4. correlate with TIME     deploy? scaling event? RDS failover?

p99 up while p50 is FLAT -> a TAIL, i.e. a SUBSET. Not everything is slow.
Common causes
THE USUAL CULPRITS AT THE BOTTOM
  DynamoDB   ThrottledRequests > 0 with spare capacity -> HOT PARTITION
             (CloudWatch Contributor Insights -> top partition keys)
  RDS        Performance Insights -> top SQL by load, and the wait events
             a query fine at 1k rows and catastrophic at 1M affects only
             tenants who crossed the threshold
  Cache      hit rate dropped for one key space -> cold cache for that tenant
  Network    cross-AZ hop added by a scaling event
Principle

Gray failures are harder than clean ones because the impaired component reports ITSELF healthy - the check runs inside the failure domain. Detect from OUTSIDE with client-side and cross-zone metrics, then evacuate deliberately with a zonal shift rather than waiting for automation to converge.

Chapter 49

Runbooks and on-call

Written for someone who did not build the service and is being woken at 3am. Mitigation before diagnosis, causes ranked by frequency, and an explicit list of what not to do.

The shape
# RUNBOOK: settlement-api

## WHAT IT DOES, AND WHAT BREAKS FOR USERS
Serves merchant settlement views and drives nightly reconciliation.
IF DOWN: merchants cannot view or reconcile payouts. Funds movement is NOT
affected - no money is at risk. Sev-2 unless combined with a ledger alert.

## DASHBOARD + THE THREE METRICS THAT MATTER
  1. 5xx rate   2. p99 latency   3. SQS queue depth

## ALERT: Api5xxHigh
Likely causes, MOST COMMON FIRST:
  1. bad deploy (~60%)   -> deployed in the last 30 min? ROLL BACK FIRST.
                            aws deploy stop-deployment --auto-rollback-enabled
  2. RDS failover / connection exhaustion (~25%)
                         -> DatabaseConnections near max? rollout restart
                            (recycles the pool; raise a ticket for the leak)
  3. partner API degraded (~10%)
                         -> flag settlement.partner_notify=off (degrades safely)
  4. AZ impairment (~5%) -> zonal shift

## KNOWN FALSE ALARMS
  02:00-02:06 nightly batch trips p99. Expected. Do not investigate.

## DO NOT
  - do NOT restart the RDS instance (failover is 60-120s of full outage)
  - do NOT purge the DLQ to clear the alarm - those are unprocessed settlements
  - do NOT scale consumers above 40 - it exhausts the RDS pool and takes down
    every other service on that database

## ESCALATE   database #db-oncall | partner #payments-partners | comms J. Patel
## LAST TESTED  2026-08-12 game day (followed cold, 2 corrections made)

Two things make it useful rather than decorative. The mitigation is a reversible operational action — roll back, scale, fail over, flip a flag — not a code fix, because nobody should be writing code at 3am. And the do-not section, which is the part most runbooks omit and the part that prevents the worst outcomes.

Principle

Test it by having someone who did not build the service follow it during a game day, and put the last-tested date at the top. An untested runbook is fiction - the commands have drifted, the dashboard was renamed, the escalation contact left.

VII. Security
Chapter 50

KMS and encryption

Envelope encryption: KMS issues a data key, you encrypt locally, and only the small key transits KMS. The key policy is the root of trust — IAM alone cannot grant access to a key.

Envelope encryption
ENCRYPT
  GenerateDataKey -> returns the data key in TWO forms: plaintext + encrypted
  encrypt your data LOCALLY with the plaintext key
  SCRUB the plaintext from memory
  store the ENCRYPTED data key beside the ciphertext

DECRYPT
  send the encrypted data key back -> get the plaintext -> decrypt locally

CONSEQUENCES
  large payloads never transit KMS - cost scales with OPERATIONS, not volume
  the CMK never leaves the HSM - the material cannot be exfiltrated
  revoking the key policy makes every ciphertext UNDECRYPTABLE
    -> a real control, not a theoretical one

Reach for a customer-managed key when you need your own key policy, cross-account grants, your own rotation schedule, separation of duties between key administrator and data administrator, or — most importantly in a regulated context — the ability to deny access by changing the policy. That is the real control when data sits somewhere you do not fully trust: you cannot delete their copy, but you can make it undecryptable.

Key policy
{ "Sid": "AllowSettlementService",
  "Effect": "Allow",
  "Principal": { "AWS": "arn:aws:iam::1234:role/settlement-worker" },
  "Action": ["kms:GenerateDataKey","kms:Decrypt"],
  "Resource": "*",
  "Condition": {
    "StringEquals": { "kms:EncryptionContext:purpose": "settlement" },
    "Bool": { "aws:SecureTransport": "true" } } }

# ENCRYPTION CONTEXT is authenticated data bound to the ciphertext: decryption
# fails unless the same context is supplied. It also appears in CloudTrail,
# so key-usage logs carry business meaning. Free, and underused.
  • Request quotas are real: a high-volume service calling Decrypt per request will throttle. Data key caching (bounded by age, message count and bytes) trades a little key hygiene for a large reduction in calls.
  • Multi-region keys are required if a DR region must decrypt — and it is not fixable during an incident.
  • Rotation creates new material while retaining the old, so nothing needs re-encrypting. It does not re-encrypt existing data, which is worth knowing before an auditor asks.
Chapter 51

Secrets

Secrets Manager or Parameter Store, fetched at runtime through a role. No long-lived credential in an image, an environment variable set at build time, or a repository.

Which one
SECRETS MANAGER      rotation (built-in for RDS, custom via Lambda),
                     cross-region replication, resource policies,
                     versioning with staging labels. Costs per secret.
PARAMETER STORE      free standard tier, SecureString with KMS,
                     hierarchical paths. No built-in rotation.

RULE: anything with a rotation requirement or cross-account sharing ->
      Secrets Manager. Configuration that happens to be sensitive ->
      Parameter Store.
Getting it into a workload
// Fetch at runtime through the task/pod role - never bake it in
var secret = secretsClient.getSecretValue(r -> r.secretId("prod/payments/db"));

// ECS: injected by the AGENT (execution role), never visible in the task def
"secrets": [{ "name": "DB_PASSWORD",
              "valueFrom": "arn:aws:secretsmanager:...:prod/db:password::" }]

// EKS: Secrets Store CSI driver - mounted as files, refreshed on rotation,
//      and NOTHING is ever written to a Kubernetes Secret object

Rotation that nothing consumes has not happened. A connection pool holding credentials from startup keeps using the old password until it recycles, so pair rotation with a bounded maxLifetime on the pool, or with a rollout. And for RDS, the dual-user rotation strategy exists precisely so there is always a valid credential during the change.

Principle

The leak that actually happens is not KMS or IAM - it is a Spring Boot /actuator/env endpoint exposed without authentication, printing the entire configuration including credentials. Keep actuator on a separate management port and disable the sensitive endpoints.

Chapter 52

The edge

WAF stops undirected exploit traffic and scripted abuse. It knows nothing about your users, so it does not stop the authorisation defects that cause most real incidents.

What each does
CONTROL                  STOPS                          DOES NOT STOP
CloudFront               volumetric noise; improves     targeted logic abuse
                         global TLS latency
WAF managed rules        SQLi, XSS, known-bad inputs,   an authenticated user
                         scanners                        enumerating IDs
WAF rate-based rule      credential stuffing, scraping  low-and-slow abuse
Shield Advanced          volumetric DDoS + COST         application-layer logic
                         protection
TLS 1.2 + ACM + HSTS     downgrade, expired certs       anything above TLS
JWT authorizer           unauthenticated access         a valid token used for
                                                         someone else's data
API GW usage plans       one partner starving others    a partner abusing
                                                         their own quota

Shield Standard is on by default. Shield Advanced adds enhanced detection, a response team, and — the part that matters commercially — cost protection, so a volumetric attack does not also arrive as a bill for the scaling it triggered.

Rate limiting belongs in layers, because each catches something different: WAF for crude per-IP abuse at the edge before it costs you anything, API Gateway usage plans for per-client fairness, and application-level limits keyed on the authenticated user or tenant — which is the only layer that can see this user is requesting an unusual number of other people's records.

Principle

Be honest about the ranking: for a payments API, AUTHORISATION DEFECTS cause far more real incidents than network attacks. WAF and Shield handle the noise; per-request authorisation, idempotency and audit prevent the incident that actually damages you.

Chapter 53

Detection

GuardDuty finds behaviour no perimeter rule would; Security Hub aggregates; Access Analyzer answers what is reachable from outside. All three are delegated to a separate security account.

  • GuardDuty — behavioural detection across CloudTrail, VPC flow logs, DNS logs, S3 data events and EKS audit and runtime activity. It finds credential exfiltration, reconnaissance, crypto mining and anomalous API calls — things no static rule would catch.
  • Security Hub — aggregates findings from GuardDuty, Config, Inspector, Macie and Access Analyzer into a single posture view with standards-based scoring.
  • Inspector — continuous vulnerability scanning of EC2, ECR images and Lambda, including rescanning already-published images as new CVEs appear.
  • Macie — discovers sensitive data in S3, which is how you find the bucket nobody remembered contains card data.
  • IAM Access Analyzer — external access findings (anything reachable from outside your organisation) and unused access findings (roles, permissions and credentials unused for ninety days).
What to alert on
ALERT ON BEHAVIOUR THAT SHOULD NEVER OCCUR, not on volume:
  a workload role suddenly calling ListSecrets or ListBuckets
  root account usage of any kind
  CloudTrail or GuardDuty being disabled
  an IAM user or access key being created in production
  console login without MFA
  STS assume-role from an unexpected source
  a bucket policy change making something public

# a competent attacker makes very few, very targeted calls -
# volume-based alerting misses them entirely
Interview angle

Delegate GuardDuty and Security Hub to a dedicated SECURITY account rather than running them from the management account. The management account can modify SCPs and create accounts, so its compromise is total - and you do not want your detection to live inside the blast radius it is watching.

Chapter 54

Evidence

CloudTrail records change and attribution; Config records state over time. Proving a control was in place CONTINUOUSLY is a Config question, and the evidence must survive compromise of what it records.

Which answers what
QUESTION                                      SERVICE
"who disabled encryption on this bucket?"     CloudTrail (change + attribution)
"was it encrypted continuously all year?"     CONFIG (configuration timeline)
"what is our overall PCI posture?"            Security Hub
"is anything reachable from outside?"         Access Analyzer

CloudTrail alone cannot prove continuity: it shows CHANGES, so the absence of
a change is only evidence if you also know the starting state AND that logging
was never interrupted.
Config maintains a TIMELINE per resource - every state it has been in, with
timestamps, plus continuous rule evaluation. That is the artefact an auditor
accepts.
Unfalsifiable
THE CHAIN MUST SURVIVE COMPROMISE OF WHAT IT RECORDS
  organization CloudTrail -> s3://audit-logs in a SEPARATE log account
    versioning: enabled
    Object Lock: COMPLIANCE mode, 7 years    # not deletable, not even by root
    bucket policy: production has s3:PutObject ONLY - no Delete, no Get
  Config aggregator -> the same log archive account
  SCP on every workload OU:
    Deny cloudtrail:StopLogging | DeleteTrail
    Deny config:DeleteConfigurationRecorder | StopConfigurationRecorder

# Without the separate account, a sufficiently privileged attacker deletes
# their own trail. That is not an audit trail; it is a convenience.

Conformance packs bundle Config rules mapped to frameworks — PCI DSS, CIS, SOC 2 — so evidence lines up with the control an auditor is asking about rather than needing translation. And Config remediation via SSM Automation turns a finding into a self-healing control.

Principle

Config bills per configuration item recorded AND per rule evaluation, which gets expensive in a high-churn environment - autoscaling groups cycling instances, ephemeral EKS nodes. Scope the recorder to the resource types you actually need evidence for rather than recording everything by default.

Chapter 55

Least privilege at scale

Three layers: SCPs as inescapable guardrails, permission boundaries for safe delegation, and generated policies from CloudTrail rather than hand-written guesses.

The cultural point first, because it determines whether any of this works: start from a working broad policy and tighten from real usage data. Teams that try to write a perfect least-privilege policy up front either block delivery for days or, far more commonly, attach a managed FullAccess policy that is never revisited.

Broad then tighten
THE WORKFLOW THAT WORKS
  1. deploy with a deliberately broad policy in a NON-PRODUCTION environment
  2. run realistic traffic for one to two weeks
  3. GENERATE the tightened policy from CloudTrail history
       aws accessanalyzer start-policy-generation \
         --policy-generation-details principalArn=arn:aws:iam::1234:role/worker \
         --cloud-trail-details 'trails=[{cloudTrailArn=...}],startTime=...'
  4. review it, apply it
  5. UNUSED-ACCESS findings keep it tight thereafter
       aws accessanalyzer create-analyzer --type ACCOUNT_UNUSED_ACCESS \
         --configuration '{"unusedAccessConfiguration":{"unusedAccessAge":90}}'

THE WORKFLOW THAT DOES NOT
  "write least privilege first" -> blocked for days -> attach *FullAccess
  -> never revisited
Delegation
PERMISSION BOUNDARY - safe self-service role creation
# Developers may create roles, ONLY with this boundary attached:
{ "Effect": "Allow", "Action": "iam:CreateRole", "Resource": "*",
  "Condition": { "StringEquals": {
      "iam:PermissionsBoundary": "arn:aws:iam::1234:policy/dev-boundary" } } }
# The boundary caps whatever they write:
{ "Effect": "Allow",
  "Action": ["s3:*","dynamodb:*","sqs:*","logs:*","secretsmanager:GetSecretValue"],
  "Resource": "*" }        # no iam:*, no kms key admin, no ec2
Principle

For a role that legitimately needs broad permissions, do not pretend it can be narrow - constrain it another way. Scope by resource and condition (specific ARNs, a source VPC endpoint, MFA, session tags), and monitor its use. A broad role that is watched is far safer than a narrow one nobody understands.

VIII. Scale, Resilience and Cost
Chapter 56

RTO, RPO and the DR patterns

How long you may be down, and how much data you may lose. Independent numbers, derived from the business, and they select the pattern between them.

The four
PATTERN              RTO        RPO        COST      PROVEN BY
backup + restore     hours      backup     lowest    a restore test
                                interval
pilot light          10-45 min  seconds    low       partly - the COMPUTE path
                     (promote +                      is UNPROVEN until the day
                      scale up)                      you need it
warm standby         minutes    seconds    moderate  CONTINUOUSLY - it is running
active-active        ~0         ~0         ~2x +     continuously
                                           real
                                           complexity

For a payments API, warm standby is usually the right answer, and the argument is about proof rather than speed: a scaled-down but functioning copy is continuously exercised, so you know it works. Pilot light's compute path is a hypothesis until the incident.

Concretely
A WARM STANDBY THAT MEETS RTO 15 MIN / RPO 5 SEC
  data      Aurora Global Database, writer eu-west-1, secondary eu-west-2
            replication lag typically <1s - MEASURE IT, alarm above 3s
            unplanned promotion ~60s
  compute   same IaC both regions, standby service at 20-30% capacity, WARM
  routing   Route 53 failover, deep health checks from >=3 regions
            + client-side retry (DNS caching makes the OBSERVED RTO longer)
  secrets   Secrets Manager cross-region replication
  KEYS      KMS MULTI-REGION KEYS - without these the standby decrypts NOTHING,
            and it is not fixable during the incident
  images    ECR cross-region replication
Principle

Derive the numbers from the business, in money and obligation - what an hour of downtime costs, what a regulator requires, what the contract commits to. Engineers picking them unilaterally is how you end up over-engineering or under-protecting.

Chapter 57

Multi-region, honestly

Stateless is easy; the data is hard. Synchronous cross-region replication is not viable, so you are choosing between async with a lag, last-writer-wins, or regional partitioning.

Three shapes
EASY - capacity and configuration
  images, IaC, compute, secrets, DNS routing

HARD - the data. Cross-region sync replication adds 10-150ms to EVERY write.

OPTION                    CONSISTENCY           SAFE FOR
active-passive            single writer,        ledgers, anything consistent
(Aurora Global DB)        async replica
active-active             LAST WRITER WINS      sessions, caches, preferences
(DynamoDB Global Tables)  (conflicts silently   NOT ledgers - it silently
                           discard a write)     discards a financial write
regional partitioning     single writer PER     anything, and it aligns with
(home region per record)  record                data residency

What does not fail over

  • SQS queues are regional — messages in the dead region are stranded until it returns.
  • ElastiCache is regional — you fail over into a cold cache, and the resulting miss storm can overwhelm the newly promoted database. A very common way a technically successful failover still causes an outage.
  • EventBridge schedules exist per region — both firing is a double-execution bug, neither firing is a silent gap. Take a cross-region lock rather than relying on configuration.
  • In-flight Step Functions executions do not migrate.
  • WebSocket and streaming connections all drop.
The honest part
AND STATE WHAT IS LOST ON UNPLANNED FAILOVER
  transactions committed in the primary but not yet replicated are GONE.
  Sub-second window, but not zero. In payments that is not an abstraction.
  THEREFORE the design must make REPLAY SAFE:
    idempotency key on every write
    an outbox that can be re-driven
    reconciliation against the scheme's or partner's records after failover
Principle

Complexity has its own failure rate. A badly-run multi-region system is frequently LESS available than a well-run single-region one, because you have doubled the surface of deploys, migrations, secrets and incidents. Recommend single-region multi-AZ plus a rehearsed warm standby unless a regulator, a contract, or a global user base says otherwise.

Chapter 58

Backups you can restore

A snapshot shares the blast radius of what it protects. A backup lives under separate control — different account, immutable — and its recovery time is a measurement, not an estimate.

The distinction
SNAPSHOT   same account, often same region. Fast. SHARES THE BLAST RADIUS -
           an attacker or a mistaken admin deletes the resource AND its
           snapshots in the same breath.
BACKUP     different ACCOUNT, ideally different region, independent
           permissions, IMMUTABLE.

AWS BACKUP - plan + tag-based selection, so a NEW resource tagged
             Backup=daily is covered automatically, not by someone remembering
  CopyActions -> cross-region AND cross-account
  VAULT LOCK in COMPLIANCE mode -> nobody can delete early. Not an admin.
              Not root. Not AWS. Irreversible after a 3-day grace period.
              This is the specific control against ransomware.
The test
PROVING RESTORE - a backup never restored has an UNKNOWN success rate
                  and an UNKNOWN RTO. You have a hypothesis, not a backup.

RestoreTestingPlan:
  ScheduleExpression: "cron(0 4 ? * SUN *)"      # weekly
  # restores into an ISOLATED account, validates, records duration, tears down

VALIDATION MUST BE REAL
  row counts within the replication window
  checksum of a sampled key range
  a canary query returning a KNOWN value
  the APPLICATION can connect and read  (not just: the instance exists)

RECORD THE ELAPSED TIME. That is your MEASURED RTO.
Re-run often enough that it reflects CURRENT data volume.
Principle

Test the restore of the thing you would actually NEED. Restoring a database is common practice; restoring the IAM roles, KMS keys, parameter values and networking it needs in order to be USABLE is what people discover is missing during the real event.

Chapter 59

Chaos and game days

Controlled failure experiments with a stop condition. The stop condition is what separates chaos engineering from an outage.

AWS Fault Injection Service runs experiments against real infrastructure — terminating instances, stopping tasks, injecting CPU pressure, adding network latency, throttling API calls, forcing an RDS failover — scoped by IAM and targeted by tag. A CloudWatch alarm as a stop condition aborts the experiment automatically if real users are affected.

What to inject first
ORDER, each with a HYPOTHESIS STATED BEFORE RUNNING

1  kill one task / pod            H: zero client errors
   -> fails? a graceful shutdown or deregistration-delay problem.
      Every later experiment is meaningless until this passes.

2  force an RDS failover          H: application recovers within 120s
   -> THE MOST VALUABLE NUMBER: the gap between "AWS recovered" and
      "WE recovered". This is where JVM DNS caching and a stale connection
      pool turn a 90-second failover into a ten-minute outage.

3  inject 200ms downstream latency H: the circuit breaker opens BEFORE the
                                      caller times out
   -> looking for a badly NESTED timeout budget: caller waits 10s, you wait
      30s, so you burn threads producing responses nobody will read

4  throttle an AWS API            H: retries back off with jitter and a cap
   -> unjittered retries turn a brief throttle into a sustained one

5  evacuate an AZ                 H: serve at reduced capacity, no errors
   -> did topology spread ACTUALLY spread you? is n-1 capacity real?
Principle

Always start with a hypothesis you EXPECT TO HOLD. The purpose is building evidence that the system behaves as designed; when it does not, you have found something real. An experiment you already expect to fail is not an experiment - fix the known problem first.

Interview angle

Announce it, run it in business hours, and treat a real incident during a game day as a SUCCESS - you found a genuine failure under controlled conditions with the right people watching. The only bad outcome is not having a stop condition.

Chapter 60

Cost

Five engineering levers, in order of size — and Savings Plans come last, because they discount whatever you are running.

The order
LEVER                                TYPICAL SAVING     EFFORT
1 right-size + Graviton (arm64)      20-40% compute     low
2 spot for stateless and batch       70-90% on that     medium
3 storage lifecycle + debris         often large        low
4 DATA TRANSFER architecture         10-20% of TOTAL    low  <- most missed
5 observability hygiene              varies             low

THEN, and only then: Savings Plans and Reserved Instances.
They discount whatever you are running, so committing BEFORE right-sizing
locks in the waste for one to three years.
Where the money goes
LEVER 4 IN DETAIL - invisible until you read the bill BY USAGE TYPE
  NAT Gateway: hourly + PER-GB PROCESSED
    every ECR pull, S3 write and Secrets call from a private subnet
    FIX: gateway endpoints for S3 + DynamoDB (FREE), interface endpoints for
         ECR (api, dkr, AND the S3 gateway), Secrets Manager, Logs, STS
  Cross-AZ: billed BOTH directions
    12 services x 3 AZs -> ~2/3 of internal calls cross a zone
    FIX: topology-aware routing (prefer same-zone), replicas still spread

FIND IT
  Cost Explorer -> group by USAGE TYPE, not by service
    EU-NatGateway-Bytes | EU-DataTransfer-Regional-Bytes | TimedStorage-ByteHrs

LEVER 3 - the ones that accrue silently
  unattached EBS volumes and orphaned snapshots
  CloudWatch log groups with NO retention (the default is never expire)
  incomplete multipart uploads (billed forever, invisible in a listing)

Estimate from units rather than from a price list — requests per day, tokens or gigabytes, instance-hours — because units are stable, they survive price changes, and they tell you which lever matters. Then sanity-check the total as cost per user per month, which is the number a director will actually ask for.

Principle

Make cost visible to the teams who create it: mandatory tagging enforced at admission, a monthly per-team report, and a budget they own. Attribution changes behaviour far more reliably than a policy does, and far more reliably than finance raising it quarterly in aggregate.

60 chapters · AWS Ground Up

Back to the ledger →

Made By Jatin Bhargava · 2026