A system design handbook · first principles to interview depth

System Design Ground Up

System design from the ground up — starting with the numbers and principles everything else rests on, then the building blocks, the data layer, distributed systems, and how to run a design round. Written to be read in order if you are new to it, and to be jumped into once you are not.

Practice

Build one, then break it

Reading a design and defending one are different skills. These simulators let you draw an architecture and push load through it until something gives — the quickest way to make the numbers in this handbook mean something.

Build an architecture on a canvas, simulate production traffic through it, and get scored the way an interviewer would.

by vijaygupta18Open →

Drag the load up and watch latency, queues and errors respond in real time. Build a system, load it until it breaks, and see why.

by xevrionOpen →
I. Foundations
Chapter 01

What system design actually is

System design is choosing where to put state, how components fail, and what you are willing to trade — everything else is vocabulary.

If you have only written application code, system design can feel like a different discipline with its own jargon. It is not. It is the same engineering judgement applied one level up: instead of choosing a data structure inside a process, you are choosing where data lives across machines, and instead of handling an exception you are deciding what happens when an entire component disappears.

Almost every system, however complicated it looks in a diagram, is doing four things: accepting requests, storing state, moving work around, and serving reads. The boxes people draw — load balancers, caches, queues, replicas, shards — exist to make one of those four survive more traffic, more data, or more failure than a single machine can.

Why it is genuinely hard

On one machine, things are pleasant: memory is reliable, function calls always return, and time moves forward. Spread the same program across machines and each of those stops being true. A call can time out and you cannot tell whether it succeeded. Two machines disagree about what time it is. A machine can be alive but unreachable, which is indistinguishable from dead. Every hard idea later in this book — consensus, idempotency, CAP, sagas — is a response to one of those facts.

What a design interview is actually measuring

This matters because candidates often prepare for the wrong thing. The interviewer is not checking whether you have memorised an architecture. They are checking five things:

What is being scored
  • Do you ask before you build? Requirements first, scope stated, assumptions said out loud.
  • Do you quantify? A design with no numbers is an opinion. Numbers turn "we'll need a cache" into "at 700 QPS with a 100:1 read ratio, a cache removes 99% of database load".
  • Do you go deep somewhere? Naming twelve components is a tour. Picking one and getting concrete — actual fields, an actual partition key — is engineering.
  • Do you know what your design costs? Every choice trades something. Saying what you gave up is the difference between a senior answer and a confident one.
  • Do you drive? The round is a conversation you are expected to lead, not a quiz you answer.

Notice that only one of those is about knowledge. That is genuinely good news if you are starting out: the vocabulary in this book takes a few weeks to learn, and the habits above can be practised from day one.

Interview angle

If you are new to this, the single most useful reframe is that there is no correct answer. There are defensible answers and indefensible ones. An interviewer who pushes back is usually not correcting you — they are checking whether you know why you chose what you chose. "I picked Postgres because the access patterns are relational and I need transactions; I'd revisit that if writes exceeded what one primary can take" is a complete answer even if they would have chosen differently.

Chapter 02

The numbers you must know by heart

A handful of latency and capacity numbers turn hand-waving into arithmetic — memorise these before anything else.

This is the highest-return chapter in the book relative to its length. Almost every design argument reduces to comparing two numbers, and if you do not have them, you cannot make the argument. If you learn nothing else this week, learn the ladder.

The latency ladder

Orders of magnitude
L1 cache reference               ~1 ns
Branch mispredict                ~3 ns
L2 cache reference               ~4 ns
Mutex lock/unlock               ~17 ns
Main memory reference          ~100 ns          <- 100x slower than L1
Compress 1 KB                   ~2 us
Send 1 KB over 1 Gbps network   ~10 us
SSD random read                ~100 us          <- 1,000x slower than memory
Read 1 MB sequentially from SSD ~200 us
Round trip within a datacentre  ~500 us  (0.5 ms)
Read 1 MB sequentially from HDD  ~2 ms
Disk seek (spinning)            ~10 ms
Round trip India <-> US        ~150 ms          <- 300,000x slower than memory

Do not memorise these to two significant figures — memorise the gaps. Memory is ~100× slower than L1. SSD is ~1,000× slower than memory. A cross-continent round trip is ~300× slower than a same-datacentre one. Those ratios are what let you say, instantly, that a cache hit and a cross-region call differ by five orders of magnitude — which is the justification for most caching and most data-locality decisions you will ever make.

Capacity numbers

Rules of thumb
seconds in a day             ~86,400   (call it 100,000 for mental arithmetic)
seconds in a month           ~2.5 million
1 million requests/day       ~12 QPS average
1 billion requests/day       ~12,000 QPS average

peak multiplier              2-3x the daily average for consumer traffic
                             10-50x if there is a scheduled event

one modern server            ~10,000s of simple QPS; ~1,000s with real work
one Postgres primary         ~10,000s of simple reads/s; far fewer complex writes
one Redis node               ~100,000 ops/s
one Kafka partition          ~10s of MB/s

Storage numbers

Sizing
1 char / 1 byte     |  a typical DB row with indexes: ~1 KB is a safe default
a UUID              16 bytes binary, 36 as text (never store it as text)
a timestamp          8 bytes
a small JSON blob    ~1 KB
a photo              ~200 KB - 2 MB
a minute of video    ~10-50 MB depending on quality

1 million rows x 1 KB   =   1 GB
1 billion rows x 1 KB   =   1 TB
Principle

Round aggressively to one significant figure. You are deciding "one box or a hundred", and that conclusion survives a 2× error. Precision beyond one digit is false confidence and wastes interview time — say "about 700 QPS", never "694.4 QPS".

Interview angle

Interviewers rarely ask you to recite these, but they notice immediately when you use them. Saying "a same-datacentre round trip is half a millisecond, so ten sequential internal calls costs 5 ms before we do any work — that is why I would parallelise these" is a small sentence that signals a great deal.

Chapter 03

Estimation, worked step by step

Four quantities — QPS, peak QPS, storage and bandwidth — take three minutes to derive and determine most of the design.

Estimation intimidates beginners because it looks like a maths test. It is not; it is four divisions, and the point is not accuracy but scale. You are answering one question: is this a single-server problem, a single-database problem, or a distributed problem?

The recipe

Recipe
1. USERS      daily active users
2. ACTIVITY   requests per user per day
3. QPS        (1) x (2) / 86,400
4. PEAK       QPS x 2-3   (say the multiplier out loud)
5. STORAGE    writes/day x bytes/write x retention x replication x index overhead
6. BANDWIDTH  QPS x average payload size

Worked example: a photo-sharing service

Say 10 million daily active users, each opening the app twice and each session loading 20 photos, with 5% of users uploading one photo a day at 1 MB.

Worked
READS
  10M users x 2 sessions x 20 photos = 400M photo views/day
  400M / 86,400                      = ~4,600 QPS average
  x3 peak                            = ~14,000 QPS peak

WRITES
  10M x 5%                           = 500,000 uploads/day
  500,000 / 86,400                   = ~6 writes/s average, ~18 peak

  -> READ:WRITE RATIO is about 800:1   <- THIS is the number that matters

STORAGE
  500,000 uploads x 1 MB             = 500 GB/day
  x 365                              = ~180 TB/year of original photos
  + thumbnails (say 3 sizes, ~10%)   = ~200 TB/year
  x 3 replication                    = ~600 TB/year of actual disk

BANDWIDTH (serving)
  4,600 QPS x 200 KB (a resized photo) = ~900 MB/s  = ~7 Gbps sustained
  -> this is the real cost driver. It is a CDN problem, not a database problem.

What the numbers just told you

  • 800:1 reads to writes → cache aggressively; read replicas will help, write scaling will not be the problem.
  • 6 writes/second → the metadata write path is trivial. A single Postgres handles this comfortably.
  • 200 TB/year of blobs → photos must go in object storage, never in the database.
  • 7 Gbps of egress → a CDN is not an optimisation, it is the architecture; and bandwidth, not storage, is where the money goes.

That is four divisions, and it has already determined the shape of the entire system. This is why estimation comes before drawing boxes rather than after.

Principle

Always ask for — or state — the read:write ratio before choosing a datastore. It drives more decisions than any other single number: caching strategy, replication, whether you shard for writes or add replicas for reads, and often the choice of database itself.

Interview angle

State your assumptions out loud as you go: "I'm assuming 3× peak, and that a photo averages 1 MB." That does two things — it invites the interviewer to correct an assumption that matters to them, and it makes your arithmetic auditable. An unstated assumption that turns out wrong looks like an error; a stated one looks like a decision.

Chapter 04

Latency, throughput, and why averages lie

Throughput is how much work per second; latency is how long one piece takes — and you must quote latency as a percentile, never a mean.

Throughput is capacity: requests per second, messages per second, megabytes per second. Latency is the time one request takes end to end. They are related but not the same, and systems routinely trade one for the other — batching raises throughput and raises latency; adding a cache lowers latency and raises effective throughput.

Percentiles, not averages

An average latency of 100 ms sounds fine and tells you almost nothing. If 95% of requests take 50 ms and 5% take 1,050 ms, the average is 100 ms — and one user in twenty is having a terrible time.

Percentiles
p50 (median)  half of requests are faster than this
p95           1 in 20 requests is slower
p99           1 in 100 requests is slower
p99.9         1 in 1,000

why the tail matters more than you think:
  a page that makes 20 backend calls hits p95 at least once
  with probability  1 - 0.95^20  =  64%
  -> most page loads experience your p95. The tail IS the typical experience.

That last calculation is worth internalising. In a system where one user action fans out into many internal calls, the tail latency of your components becomes the median latency of your product. This is why large systems obsess over p99 rather than average.

Little's Law

One formula, used constantly: L = λ × W — the average number of items in a system equals the arrival rate times the average time each spends there.

Little's Law
200 requests/second x 0.05 s each = 10 concurrent requests in flight
  -> so a thread pool of ~10-20 is right, NOT 200

run it backwards to find a ceiling:
  20 DB connections / 0.010 s per query = 2,000 QPS maximum
  -> beyond that, requests queue no matter how many app threads you add

Why latency explodes near full utilisation

Queueing delay grows roughly as 1/(1−ρ) where ρ is utilisation. This is the single most important non-obvious fact about capacity.

The cliff
utilisation 50%  ->  wait ~= 1x the service time
            80%  ->  ~5x
            90%  ->  ~10x
            95%  ->  ~20x        <- the cliff

a system "efficiently" running at 85% CPU is one traffic bump from collapse
Principle

Target 50–70% utilisation for latency-sensitive services, and size for peak plus the loss of one instance or availability zone. Batch systems can safely run much hotter, because queueing delay does not hurt them. Running "efficiently" at 90% is how a small spike becomes an outage.

Interview angle

This also explains a phenomenon that otherwise looks unrelated: a garbage-collection pause or a brief network blip causes a latency spike far larger than the pause itself, because the queue built during it takes much longer to drain at high utilisation. If someone asks why a 200 ms GC pause produced a 3-second p99 spike, this is the answer.

Chapter 05

Availability, SLOs and error budgets

Availability multiplies across dependencies, redundancy adds nines only if failures are independent, and an error budget turns reliability into a decision you can make with arithmetic.

The arithmetic

Serial dependencies multiply. If your service calls three others, each 99.9% available, your ceiling is 0.999³ = 99.7% — about two hours of downtime a month rather than 43 minutes. Every dependency you add lowers your maximum achievable availability, and that is a genuine architectural argument against chatty service chains.

Redundancy adds nines — but only if failures are independent. Two instances at 99.9% give 1 − (0.001)² = 99.9999%. In reality, two instances in the same rack, the same availability zone, behind the same load balancer, running the same bad deploy are not independent. Correlated failure is why real systems never achieve the theoretical number, and saying so is what separates understanding the formula from reciting it.

Nines
availability   downtime/month    downtime/year
99%            7.2 hours         3.65 days
99.9%          43 minutes        8.8 hours
99.99%         4.3 minutes       53 minutes
99.999%        26 seconds        5.3 minutes

serial:    0.999 x 0.999 x 0.999 = 99.7%
parallel:  1 - (0.001)^2         = 99.9999%   (IF independent)

SLI, SLO, SLA — in that order

  • SLI — the indicator, something you measure. "The proportion of requests served in under 300 ms without a 5xx."
  • SLO — the objective, a target you choose. "99.9% of requests meet the SLI, measured over 30 days." Deliberately stricter than any contract.
  • SLA — the agreement, a promise to customers with financial consequences. Looser than the SLO, so you have margin to react before you breach it.

The error budget

This is the idea that makes reliability operational rather than decorative. An SLO of 99.9% over 30 days permits 43 minutes of failure. That is not a failure to be ashamed of — it is a budget you are entitled to spend.

Principle

Budget intact → ship faster, take risks, deploy more often. Budget burned → freeze feature work and spend on reliability until it recovers. This converts an unwinnable argument ("we need to move faster" vs "we need to be more reliable") into arithmetic that both sides can read off the same dashboard.

Interview angle

Two things interviewers listen for. First, that you pick the lowest SLO the business actually needs — each additional nine costs roughly an order of magnitude more, and 99.999% for an internal reporting tool is waste. Second, that you alert on SLO burn rate rather than on causes: paging someone for high CPU wakes them for a condition users may not notice, while "at this rate we exhaust the month's budget in six hours" is always worth waking for.

Chapter 06

Twelve principles to design by

A dozen durable heuristics that survive across every system you will design — worth returning to whenever a decision feels arbitrary.

Techniques date quickly; principles do not. If you are starting out, these are worth more than any specific architecture, because they tell you what to do when you meet a problem this book does not cover.

Principle
  • 1. Start with the simplest thing that could work. A monolith on one big machine with one database serves more traffic than most people believe. Distribute when a specific constraint forces you to, and be able to name that constraint.
  • 2. Quantify before you architect. Four divisions determine whether this is a one-box problem or a hundred-box problem. Designing before estimating means designing for an imagined scale.
  • 3. Know your read:write ratio. It drives caching, replication, sharding and datastore choice more than any other number.
  • 4. Make the common case fast and the rare case correct. Optimise the 99% path; make the 1% path safe rather than fast.
  • 5. Everything fails; design for it explicitly. For every dependency, state its timeout, its retry policy, and what your system does when it is gone. A dependency with no stated failure behaviour is an outage waiting to be discovered.
  • 6. Push invariants into the layer that can actually enforce them. If two people must not book the same seat, that guarantee belongs in a database constraint — not in application code, and not in a distributed lock that a process pause can lose.
  • 7. Prefer idempotency over coordination. Making an operation safe to repeat is almost always cheaper and more robust than making it happen exactly once.
  • 8. Bound every queue, pool and cache. Unbounded anything converts a visible error into invisible memory growth, and then into a total collapse instead of a partial one.
  • 9. Make state the exception. Stateless components scale, restart and deploy trivially. Concentrate state deliberately in as few places as possible, and treat those places as the hard part.
  • 10. Derived data must be rebuildable. Caches, search indexes, projections and aggregates should all be reconstructible from a source of truth. If you cannot rebuild it, it is not derived data — it is unreplicated primary data.
  • 11. Design the migration, not just the end state. Any change to a running system must be deployable incrementally, with every intermediate state working and reversible.
  • 12. Optimise for the ability to change your mind. You will be wrong about something. Prefer decisions that are cheap to reverse, and spend your irreversibility budget carefully — a data model and a public API are much harder to undo than a service boundary.

How to use these in a round

They are most useful when you are stuck or when the interviewer pushes on a choice. "I'd put that constraint in the database rather than the application, because application-level checks can't survive a process pause" is principle 6 doing the work. "I'd rather make this idempotent than coordinate it" is principle 7. Reaching for a principle out loud is what a design discussion between two engineers sounds like.

Interview angle

If you want a single sentence to carry into every design round, make it this: name the trade. Every choice buys something and costs something, and the entire difference between a mid-level and a senior answer is whether the cost gets said out loud without being asked for.

Chapter 07

What happens when you type a URL

The whole request path, hop by hop — and the characteristic failure of each hop, which is where the interesting questions live.

This is a classic opening question, and it is really an invitation to show how much of the stack you can see. The route matters less than the fact that you attach a failure mode and a mitigation to each hop.

The path
browser
  |  1. DNS resolution        name -> IP (cached at OS, browser, resolver)
  v
  |  2. TCP handshake         1 round trip
  |  3. TLS handshake         1-2 round trips (0 with resumption / TLS 1.3)
  v
CDN edge                      static assets served here; may terminate TLS
  |  4. cache miss -> origin
  v
load balancer                 health-checked routing, connection draining
  |
  v
API gateway                   authN, rate limiting, routing
  |
  v
application service           your code
  |  \
  |   -> cache (Redis)         hit: ~1 ms
  v
database                      miss: ~5-50 ms
  |
  v
response travels back, browser parses HTML, fetches assets, renders

Failure and mitigation at each hop

  • DNS — stale records after a failover, bounded by TTL. Mitigate: short TTLs on failover-critical records; anycast so the IP itself moves.
  • TLS — extra round trips, worst on high-latency mobile. Mitigate: TLS 1.3, session resumption, terminate at a nearby edge.
  • CDN — serving stale HTML you cannot recall. Mitigate: hashed asset filenames with long cache lifetimes; no-cache on HTML.
  • Load balancer — sending traffic to an instance that is warming up or shutting down. Mitigate: readiness checks distinct from liveness, plus connection draining.
  • Gateway — becoming a bottleneck or accumulating business logic. Mitigate: keep it thin; it does auth, routing and rate limiting only.
  • Service — thread-pool or connection-pool exhaustion, which looks like slowness rather than failure. Mitigate: bounded pools, timeouts, a separate pool per dependency.
  • Cache — a stampede when a hot key expires; a cache outage becoming a database outage. Mitigate: single-flight loading, TTL jitter, a circuit breaker.
  • Database — slow queries, lock contention, replica lag. Mitigate: indexes, query timeouts, read replicas.
Principle

Every hop needs a timeout, and the budget must shrink down the chain. If the client waits 3 s, the gateway should wait 2.5 s and the service 2 s. Otherwise the client has given up while every layer below still holds threads and connections for a response nobody will read — which under load is how one slow dependency saturates the whole chain.

Interview angle

A good follow-up to volunteer: DNS TTL and connection pools are the two hops that surprise people in real incidents, because both are invisible in normal operation. A 24-hour TTL means a failover nobody can complete; an exhausted pool looks exactly like a slow application while the database sits idle.

Chapter 08

Scaling: vertical, horizontal, and what actually breaks

Vertical scaling is a bigger machine and is often the right answer; horizontal scaling requires statelessness and moves the bottleneck rather than removing it.

Vertical first, and without embarrassment

Buying a bigger machine requires no code changes, no distributed-systems problems, and no new failure modes. Modern hardware is remarkable — a single server with 128 cores and a terabyte of RAM handles workloads people reflexively assume need a cluster. Its limits are a hard ceiling on machine size, cost that grows superlinearly, and — decisively — it remains a single point of failure.

Saying "I would scale vertically first and buy six months" is a legitimate senior answer, provided you can name the point at which it stops working.

Horizontal requires statelessness

Before you can add a second instance, no request may depend on in-memory state left behind by a previous one. Everything a later request needs must live somewhere shared:

Preconditions
[ ] no in-memory session          -> Redis, or a signed token
[ ] no local file writes          -> object storage
[ ] no in-process scheduler       -> leader election, or a lease with a TTL
[ ] no in-memory cache as truth   -> shared cache, or accept per-node staleness
[ ] no local rate-limit counters  -> a shared counter
[ ] graceful shutdown + draining  -> so deploys do not drop in-flight requests

Sticky sessions are a workaround, not a design

Routing a user to the same instance so in-memory session works breaks load balancing (one instance runs hot while others idle), makes deploys disruptive (draining an instance logs those users out), defeats autoscaling, and turns a single node failure into user-visible session loss. Externalise the session instead.

What horizontal scaling does not fix

This is the part that separates a complete answer. Once the application tier is stateless and replicated, you have moved the bottleneck, not removed it. Ten app servers hammering one database primary is not a scaled system — it is a scaled queue in front of an unscaled resource. The shared dependencies are now the constraint: the database, the cache, and the load balancer itself.

Principle

Scale in this order, and stop as soon as the problem is solved: (1) vertical, (2) caching to remove reads entirely, (3) read replicas, (4) functional partitioning by domain, (5) sharding. Sharding is last because it permanently costs you cross-shard joins and transactions.

Interview angle

A useful thing to raise unprompted: fifty application servers each holding a twenty-connection pool is a thousand database connections, which is far more than a Postgres primary wants. Connection pooling — and often a pooler like PgBouncer in front — becomes a real constraint the moment you scale the stateless tier, and it catches people who only thought about the app layer.

II. The Building Blocks
Chapter 09

Load balancers

A load balancer spreads traffic across instances and removes unhealthy ones — the interesting parts are which layer it works at, how it picks, and how it behaves during a deploy.

Layer 4 versus Layer 7

L4 operates at the transport layer: it forwards TCP connections using IP and port, never looks at the payload, and is therefore extremely fast and protocol-agnostic. L7 terminates the connection and understands HTTP, so it can route by path, host or header, terminate TLS, retry idempotent requests, rewrite, compress and enforce rate limits. You pay latency and CPU for that visibility.

Modern systems usually run both: L4 at the edge for raw throughput and DDoS absorption, L7 inside for routing and policy.

Algorithms, and when each is right

  • Round-robin — the simple default. Assumes uniform requests and uniform servers, and is wrong when either assumption fails.
  • Least connections — better when request durations vary, because it naturally stops piling onto a slow instance. Usually the better default for real services.
  • Weighted — for heterogeneous instance sizes, or for gradually shifting traffic during a canary.
  • Hash / consistent hash by key — routes the same key to the same instance. What you want in front of a cache tier so you do not duplicate cache entries on every node.
  • Least response time — combines latency and connection count; good, but be aware it can amplify a temporarily fast-failing instance (it answers quickly with errors, so it attracts more traffic — hence outlier detection).

Health checks: liveness versus readiness

This distinction causes real outages, so be precise. A liveness check asks "is this process broken beyond recovery" — failing it means restart me. A readiness check asks "should I receive traffic right now" — failing it means remove me from the pool but leave me running.

An instance that is warming up, has briefly lost its database connection, or is shutting down should fail readiness only. Put a database check in your liveness probe and a database blip causes every instance to fail liveness simultaneously, the orchestrator restarts the whole fleet, and you now have a cold fleet on top of a database problem.

Connection draining

Draining
deploy sequence with zero dropped requests:
  1. new instance starts, FAILS readiness while warming (JIT, caches, pools)
  2. passes readiness      -> LB adds it to the pool
  3. old instance told to stop:
       a. starts failing readiness  -> LB stops sending NEW requests
       b. keeps serving in-flight   -> drain window (e.g. 30s)
       c. process exits
  -> zero dropped requests
Principle

Readiness gates traffic; liveness gates restarts. Never check a downstream dependency in a liveness probe — doing so makes your availability the product of everyone else's, and turns a partial degradation into a restart loop.

Interview angle

Volunteer that the load balancer is itself a single point of failure, so it is deployed redundantly behind DNS failover or anycast. Also mention outlier detection — automatically ejecting an instance whose error rate or latency deviates from the pool — because health checks only catch total failure, not an instance that is answering slowly or wrongly.

Chapter 10

Caching: the patterns

Caching trades correctness for speed; the pattern you choose determines what kind of wrongness you are accepting.

A cache is a copy of data kept closer to where it is used. That means there are now two copies, which means they can disagree — so every cache is a consistency decision, not a free optimisation. The question is never "should we cache" but "what staleness can we tolerate, and what do we do when the copy is wrong".

Where caches live

Cache layers
browser cache        per user, free, uncontrollable once shipped
CDN edge             per region, huge, purge is slow (seconds-minutes)
API gateway          shared, close to the edge
in-process (L1)      nanoseconds, per-instance, independently stale
shared cache (Redis) ~1 ms, consistent across instances, one more dependency
database buffer pool the database's own cache - already caching for you

The patterns

Cache-aside (lazy loading). The application checks the cache; on a miss it loads from the database and populates. Simplest and by far the most common. Failure modes: a cold cache means every request hits the database, and there is a race where a read repopulates a stale value just after a concurrent write.

Read-through. Same shape, but the cache library does the loading. Centralises the single-flight logic so you write it once rather than at every call site.

Write-through. Write to cache and database together, so the cache is never stale. Costs write latency on every write, and you cache data that may never be read.

Write-behind (write-back). Write to cache and flush asynchronously. Very fast writes, but you can lose acknowledged data if the cache node dies before the flush. Unacceptable for a ledger; fine for a view counter.

Refresh-ahead. Proactively reload entries before they expire, removing the latency spike at expiry — at the cost of refreshing things nobody asked for.

Pseudocode
cache-aside (the default):
  value = cache.get(key)
  if value is null:
      value = db.load(key)
      cache.set(key, value, ttl + jitter)
  return value

Eviction

LRU is the sensible default. LFU is better when popularity is stable but adapts poorly to shifting workloads. TTL bounds staleness rather than size — and you almost always want both a size bound and a TTL. Modern caches such as Caffeine use W-TinyLFU, which admits a new entry only if it is likely more valuable than the one it would evict; that resists the classic LRU weakness where a single large scan flushes the entire hot working set.

Principle

Always add jitter to TTLs. Entries populated together with identical TTLs expire together, which manufactures a stampede on a schedule. ttl = base + random(0, base × 0.1) costs one line and removes an entire class of incident.

Interview angle

Be ready to name the staleness you are accepting, in user-visible terms: "for up to five minutes a user could see a price that has just changed; that was acceptable because the value is advisory and confirmed at checkout." An answer that only names the speedup is a mid-level answer. Naming the wrongness you accepted is the senior half.

Chapter 11

Caching: the failure modes

Four named failures — stampede, hot keys, penetration, and the cache outage that becomes a database outage — each with a standard mitigation.

1. Cache stampede (thundering herd)

A popular key expires and every concurrent request misses simultaneously, so N requests all issue the same expensive query. The database, sized for the cached load, collapses — and because the cache is now empty and the database is slow, it cannot recover on its own.

  • Single-flight / request coalescing — only the first miss loads; the others wait for that result. computeIfAbsent in-process, or a short-lived lock key in Redis across nodes.
  • Probabilistic early expiry — each reader independently decides, with rising probability as the TTL approaches, to refresh early, so refreshes spread out instead of synchronising.
  • Stale-while-revalidate — serve the expired value immediately and refresh in the background. Removes the latency spike entirely and is usually the best answer when slight staleness is tolerable.

2. Hot keys

One key receives a disproportionate share of traffic — a celebrity profile, a flash-sale product — and saturates the single shard that owns it. Consistent hashing does not help: it distributes keys evenly, not traffic.

  • Key splitting — write the value to key#0..key#N and have readers pick a random suffix.
  • A local L1 cache in each application process, so most reads never leave the box.
  • Dedicated replicas for known-hot keys.

3. Cache penetration

Repeated requests for keys that do not exist — a scanner walking random ids, or a legitimate lookup of a deleted record. Every one misses the cache and hits the database, so the cache provides no protection at all.

  • Negative caching — store a tombstone with a short TTL (shorter than for real values, so a newly created entity appears quickly).
  • A Bloom filter of existing keys in front, which answers "definitely not present" cheaply and never has false negatives.
  • Rate limiting on 404s, since a burst of misses from one client is a distinctive signature.

4. The cache outage that becomes a database outage

This is the one people forget. If the cache is absorbing 99% of reads and it disappears, the database instantly receives 100× its normal load. A cache is an optimisation right up until the moment your database cannot survive without it — at which point it is a dependency you have not acknowledged.

Principle

Defend the database as if the cache will vanish: single-flight so N misses produce one load, a small in-process L1 to absorb the first seconds, load shedding and a circuit breaker so the database survives to recover, and — ideally — capacity planning that knows what happens at a 0% hit rate.

Interview angle

A sharp follow-up is how do you invalidate an in-process cache across fifty instances? Three answers with honest trade-offs: a short TTL (1–5 s) — simple, bounded staleness, no coordination, and usually the right choice; pub/sub invalidation — near-immediate but fire-and-forget, so a disconnected instance keeps stale data unless a TTL backs it up; or versioned keys, which sidestep invalidation entirely at the cost of garbage.

Chapter 12

CDN and the edge

A CDN moves bytes closer to users, which fixes the one thing you cannot optimise away — the speed of light.

A cross-continent round trip is roughly 150 ms and no amount of server optimisation changes it. A CDN puts copies of your content in points of presence near users, so the round trip becomes 10–30 ms. For any content-heavy or globally distributed product, this is the single largest latency win available.

Push versus pull

A pull CDN fetches from your origin on the first miss in each region and caches it — simple and self-maintaining, at the cost of a slow first request per region. A push CDN has you upload in advance — better for large, predictable assets such as video, and for content you know will be requested.

An origin shield is a mid-tier cache between the edges and your origin: without it, a global cold start sends hundreds of simultaneous requests to your origin, one per edge location.

Cache-Control, per content type

Headers
/static/app.9f2a1c.js   public, max-age=31536000, immutable
/index.html             no-cache                    <- store, but ALWAYS revalidate
/api/me                 private, no-store           <- never shared-cached
/api/products           public, max-age=60, stale-while-revalidate=300
  • Hashed asset filenames mean a new build is a new URL, so the old one can be cached forever. immutable additionally stops revalidation on an explicit reload.
  • no-cache does not mean "do not cache" — it means store it but revalidate before every use. no-store is the one that forbids storing.
  • private stops shared caches holding one user's response and serving it to another — a real data-leak class of bug.
  • stale-while-revalidate serves the cached copy instantly while refreshing behind it, so the user never waits for a revalidation.

ETags

The server sends ETag: "abc"; the client's next request carries If-None-Match; if unchanged the server replies 304 Not Modified with no body. It saves bandwidth but still costs a full round trip — which is exactly why max-age with a versioned URL beats ETag whenever you can version.

Principle

Version your assets, never your HTML. A wrong Cache-Control on HTML is an outage you cannot roll back: every client that received it will not check back until the TTL expires, and purging the CDN does not help because the copies are in users' browsers.

Interview angle

That asymmetry is the memorable point of this chapter and worth volunteering. It is also why deploy processes treat HTML and assets completely differently, and why "just purge the CDN" is not a complete recovery plan.

Chapter 13

Queues, pub/sub and logs

Three different shapes: a queue distributes work, pub/sub fans out notifications, and a log retains an ordered history that many consumers can replay independently.

Why put a queue in at all

Three reasons, and being able to name which one applies is the point. Decoupling — the producer does not need the consumer to be available. Buffering — a traffic spike becomes a longer queue rather than dropped requests. Levelling — a slow consumer processes at its own rate instead of forcing the producer to wait.

The cost is that everything becomes asynchronous, which means the caller no longer learns the outcome — so you need a way to report completion, and the system now has intermediate states that must be modelled.

The three shapes

Shapes
QUEUE (SQS, RabbitMQ)
  competing consumers; each message delivered to ONE consumer
  message is DELETED after acknowledgement
  per-message ack, retry, visibility timeout, priority
  -> work distribution

PUB/SUB (SNS, Google Pub/Sub)
  each message fans out to EVERY subscriber
  typically no retention
  -> notification

LOG (Kafka, Pulsar)
  messages RETAINED on disk for a configured period
  consumers track their own OFFSET
  many independent consumer groups read the same stream at their own pace
  any of them can REWIND and replay
  -> integration backbone

That replay capability is the defining difference and the reason Kafka became an integration backbone rather than just a queue: a new consumer can be added later and read all of history, and a bug can be fixed and the data reprocessed.

Delivery semantics

  • At-most-once — acknowledge before processing. Crash in between and the message is gone. Acceptable only for lossy telemetry.
  • At-least-once — acknowledge after processing. Crash after processing but before acknowledging and it is redelivered, so duplicates are guaranteed to happen eventually. This is the default in essentially every real system.
  • Exactly-once — as a delivery guarantee over an unreliable network, impossible: the acknowledgement itself can be lost, and the sender cannot distinguish a lost message from a lost ack.
Principle

What is achievable is exactly-once effect: at-least-once delivery plus an idempotent consumer. Give every message a stable id, record processed ids, and make the handler safe to run twice. "We are exactly-once because we use Kafka" is the answer that fails.

Interview angle

When would you choose a queue over a log? When you need per-message acknowledgement, per-message retry with visibility timeouts, and priority — SQS-style semantics. A log replays by offset, which makes "retry just this one message in five minutes" awkward, and is exactly why Kafka deployments end up adding retry topics and dead-letter topics.

Chapter 14

Kafka in depth

Kafka is a partitioned, replicated, append-only log — and almost every design decision reduces to choosing the partition key.

The model

A topic is split into partitions; each partition is an ordered, append-only file on disk. Producers append; consumers read from an offset they control. Partitions are the unit of both parallelism and ordering.

Structure
topic "payments"
  partition 0:  [msg][msg][msg][msg]...     offset 0,1,2,3...
  partition 1:  [msg][msg][msg]...
  partition 2:  [msg][msg][msg][msg][msg]...

ordering is guaranteed WITHIN a partition only
the partition key decides which partition a message lands in
  key = hash(key) % partitions   |   null key = round-robin (NO ordering)

The partition key is the design decision

Keying by account_id means one account's events are strictly ordered while different accounts process in parallel. Keying by nothing gives you throughput and no ordering. Keying by something with skew — a single huge tenant — gives you a hot partition that one consumer must handle alone. Almost every Kafka question is really about this choice.

Consumer groups

Within a group, each partition is consumed by exactly one consumer. So your parallelism ceiling is the partition count — adding consumers beyond that leaves them idle. Different groups consume the same topic independently, which is what makes one stream serve many purposes.

Durability and delivery settings

Settings
PRODUCER
  acks=all + min.insync.replicas=2   a write is acked only once on 2+ replicas
  enable.idempotence=true            dedupes producer retries (default on modern clients)
  linger.ms / batch.size             trade a little latency for a lot of throughput

CONSUMER
  enable.auto.commit=false           commit AFTER processing -> at-least-once
  max.poll.records                   keep a poll cycle under max.poll.interval.ms
  isolation.level=read_committed     with transactional producers

Auto-commit is the silent data-loss setting. It commits on a timer, so a crash between the commit and the processing loses messages with no error anywhere.

Rebalancing, and the classic incident

A rebalance happens when a consumer joins, leaves, or misses max.poll.interval.ms. The classic production incident: processing is slow, a consumer exceeds the interval, the broker assumes it died, the group rebalances, the work is reassigned — and it happens again, in a loop, while lag grows. Use cooperative-sticky assignment to avoid the stop-the-world variant, and keep per-poll work bounded.

Principle

Consumer lag is the primary SLI. Alert on lag and on its rate of growth, because growth predicts a breach long before latency alarms fire. And know the one unrecoverable failure: if lag exceeds retention, those messages are gone permanently.

Interview angle

Two follow-ups to have ready. How do you scale consumption? Add consumers up to the partition count, then add partitions — but note that adding partitions changes key-to-partition mapping and therefore breaks ordering for in-flight keys, so it is a planned migration rather than a scaling reflex. What about a poison message? Bounded retries with backoff, then a dead-letter topic, committing the offset so the partition advances — otherwise one bad message blocks everything behind it forever.

Chapter 15

Rate limiting

Five algorithms with different burst behaviour and memory cost; the sliding-window counter is the practical default and the token bucket is right when bursts are legitimate.

Rate limiting protects a system from being overwhelmed — by abuse, by a buggy client, or simply by success. It is also the clearest small example of an engineering trade with a numeric answer, which is why it comes up so often.

The algorithms

Comparison
                    memory/key   burst behaviour
fixed window        O(1)         2x burst at the window boundary
sliding window log  O(requests)  exact, no burst - but unusable memory at scale
sliding counter     O(1)         ~no burst, small bounded approximation error
token bucket        O(1)         allows bursts UP TO the bucket capacity
leaky bucket        O(queue)     no burst; output perfectly smoothed

Fixed window keeps one counter per key per minute. Trivially cheap, and its flaw is arithmetic: 1,000 requests at 11:59:59 and another 1,000 at 12:00:00 is 2,000 in one second while never breaching "1,000 per minute".

Sliding window log stores every request timestamp and counts those inside the window. Exact, and O(requests) memory per key — 10,000 keys at 1,000 requests each is ten million timestamps, so it is mostly a teaching example.

Sliding window counter keeps the current and previous window counts and interpolates. O(1) memory, no boundary burst, small bounded error. This is the industry default.

The formulas
count = current_window + previous_window * (1 - elapsed_fraction_of_window)

token bucket:
  tokens = min(capacity, tokens + elapsed_seconds * refill_rate)
  if tokens < cost: reject
  else: tokens -= cost; allow

Token bucket refills at a fixed rate up to a cap; each request takes a token. It deliberately permits bursts up to the capacity, which is the right model when bursts are legitimate — a client that has been idle should be allowed to catch up. Leaky bucket is the mirror image: requests queue and drain at a constant rate, so output is perfectly smooth and bursts are delayed rather than allowed.

Making it distributed

With several gateway nodes the counter must be shared, or a client with N nodes gets N times the limit. Use Redis with a Lua script so the read-modify-write is atomic in one round trip — otherwise two nodes both read 999 and both allow. Compute tokens lazily from elapsed time rather than running refill jobs.

Principle

Fail open or fail closed is a business decision, and you must state which. Guarding a fragile backend from abuse: fail closed. Preventing accidental over-use of a robust service: fail open, because taking the whole API down to enforce a rate limit is the worse outcome.

Interview angle

Always return 429 with Retry-After and X-RateLimit-Remaining, so a well-written client can pace itself rather than hammering and backing off blindly. And mention the latency-conscious variant: check a local in-process counter first and sync with Redis periodically, trading exactness for removing a network hop from every request.

Chapter 16

Consistent hashing

Modulo hashing remaps nearly every key when you add a node; consistent hashing remaps only about K/N — and virtual nodes make the distribution even.

The problem it solves

The obvious way to spread keys across N nodes is hash(key) % N. It distributes evenly and it has one fatal property: changing N remaps almost every key. Go from 4 nodes to 5 and roughly 80% of keys move to a different node. For a cache that means a near-total miss storm and a database that suddenly receives full traffic; for a sharded store it means moving nearly all the data.

The mechanism

Map the output of the hash function onto a conceptual ring, from 0 to 2³²−1. Hash each node onto the ring too. A key belongs to the first node encountered walking clockwise from the key's position.

The ring
          0/2^32
            |
     nodeC  |   nodeA
        \   |   /
         \  |  /        a key hashes to a point on the ring
          \ | /         -> walk CLOCKWISE -> the first node owns it
           \|/
       -----+-----
           /|\
          / | \
            |
          nodeB

add a node    -> only the keys between it and its predecessor move  (~K/N)
modulo hashing -> nearly EVERY key moves                             (~K)

Virtual nodes

The naive ring has two weaknesses: with few physical nodes, random placement produces very uneven arc lengths, so one node may own three times its share; and when a node leaves, all its keys land on a single successor, which may then be overwhelmed.

Virtual nodes fix both. Give each physical node many ring positions — typically 100–200 — and the distribution converges to uniform by the law of large numbers. Removal now spreads keys across many successors rather than dumping them on one. It also lets you weight heterogeneous machines by giving bigger ones more virtual nodes.

With vnodes
ring position = hash(node_id + ":" + vnode_index)     # 100-200 vnodes per node
key owner     = first vnode clockwise from hash(key)

Where it is used

Cache client sharding (Memcached), Cassandra and DynamoDB partitioning, Redis Cluster's slot mapping (a variant), and load balancers doing session or cache affinity.

Principle

Consistent hashing distributes keys evenly, not traffic. A single hot key still lands on one node and saturates it, and no amount of ring tuning helps — that needs key splitting or a local cache. Being able to say what a technique does not solve is worth as much as knowing what it does.

Interview angle

Two refinements worth naming: bounded-load consistent hashing caps how much above average any node may take, spilling overflow to the next node; and rendezvous (highest random weight) hashing achieves the same minimal-disruption property with much simpler code and no ring, at O(N) per lookup — which is fine for small N.

Chapter 17

Object storage

Blobs go in object storage, metadata goes in a database, and bytes should never traverse your API servers.

The structural decision

Object storage (S3 and equivalents) holds immutable blobs addressed by key, with essentially unlimited capacity, very high durability, and low cost. A database holds the metadata: key, size, checksum, content type, owner, version, state. That split is the first thing to say.

Never store large blobs in a relational database. It destroys backup and restore times, bloats replication, and evicts useful pages from the buffer cache. The database should hold a reference, not the bytes.

Presigned URLs

The most important operational decision: bytes must not pass through your application servers. The client asks your API to authorise an upload; you return a short-lived signed URL; the client uploads directly to storage; storage notifies you or the client confirms.

Flow
1. POST /uploads         -> app creates a metadata row (state=pending),
                            returns an uploadId + presigned URLs per part
2. PUT part 1..N         -> client uploads DIRECTLY to storage, in parallel
3. POST /uploads/{id}/done -> app verifies, completes, state=available

without this, a 1 GB upload occupies an application thread for minutes

Scope presigned URLs tightly — short expiry, one key, one method, a content-length range — because the signature is a bearer capability and those constraints are your only control.

Multipart and resumable uploads

Split large files client-side into parts (5–100 MB), upload each independently with its own checksum, retry only failed parts, and issue a final complete call listing the parts. Resumability follows for free: ask the server which parts it already has and skip them. Add a lifecycle rule to abort incomplete multipart uploads, or they silently accumulate cost.

Deduplication and durability

Content addressing — hash the content and use the hash as the key — gives deduplication for free, with a reference count on the blob and metadata rows pointing at it. Note the trade-off worth raising: dedup across users saves enormous storage but leaks information, because an attacker can test whether a file exists by observing that its upload completed instantly.

For durability, replication (three copies) is simple and fast to reconstruct at 3× storage; erasure coding splits into k data and m parity fragments for comparable durability at roughly 1.5× storage, at the cost of reconstruction CPU. Use replication for hot data and erasure coding for cold, tiered by lifecycle policy.

Principle

Create the metadata row first, before the bytes exist, and mark it available only once the upload is durable. Otherwise a client crash leaves orphaned bytes nobody can find, delete or bill for — and a reader can get a 404 for an object your API claims exists.

Interview angle

Object storage is eventually consistent for some operations historically (listings especially), and objects are immutable — you replace rather than edit. Both facts shape designs: versioned keys are natural, in-place mutation is not, and a listing is never a reliable source of truth for "does this exist".

Chapter 18

Gateways, BFFs and service meshes

Three components that are constantly confused — separate them by who owns them and which concern they serve.

API gateway — the edge

Owned by the platform team. It handles cross-cutting concerns that must happen once, at the boundary: TLS termination, authentication, coarse rate limiting, routing, request logging, IP allow-listing.

The critical rule is no business logic. A gateway that starts aggregating and transforming domain data becomes a distributed monolith that every team must change together — and a single point of failure that everyone depends on and nobody owns.

BFF — backend for frontend

One per client type, owned by that client's team. Its job is aggregation and shaping: a mobile app on a slow connection wants one call returning exactly the fields it renders, not seven calls returning full entities.

The real benefit is organisational: because the web and mobile clients have genuinely different needs, giving each its own BFF avoids the compromise API that serves neither well — and lets the client team change their own contract without cross-team coordination.

Service mesh — service to service

A sidecar proxy alongside each service, handling internal concerns uniformly: mTLS, retries, timeouts, circuit breaking, traffic splitting for canaries, and consistent telemetry. Its value is that these behaviours become language-agnostic and configurable without every team implementing them.

The edge
                 +-- web BFF    --+
internet -> gateway |               |--> services (mesh: mTLS, retries, breakers)
  TLS, authN,    +-- mobile BFF --+
  rate limit,    |
  routing        +-- partner API (versioned, documented, stricter quotas)

gateway : platform-owned, NO business logic
BFF     : client-team-owned, aggregation + shaping, changes with its client
mesh    : service-to-service, sidecar, language-agnostic
partner : external contract - you cannot redeploy their clients

The partner API is different

External consumers cannot be redeployed by you, so a partner API needs a stable versioned contract, thorough documentation, stricter quotas, and a real deprecation policy. Never let partners consume a BFF — a BFF's contract is designed to change with its client.

Principle

Decide by asking: does it apply to every client identically and must happen before anything else? → gateway. Is it about what one client needs to render? → BFF. Is it about how services talk to each other? → mesh.

Interview angle

Be willing to say when you would skip these. Below roughly a dozen services a mesh is usually not worth its operational complexity — libraries with sane defaults give you most of it. And with one client, a BFF is just another deployable to own. Recommending fewer components when the scale does not justify them is a strong signal, not a weak one.

III. The Data Layer
Chapter 19

Choosing a datastore

Drive the choice from access patterns, consistency needs and relationship cardinality — and make Postgres the default you have to argue your way out of.

Datastore choice is where beginners feel most pressure to name something impressive, and where experienced engineers are most boring on purpose. The honest position: a relational database is the correct default, and moving off it should be justified by a specific constraint you can name.

The decision order

Principle
  • 1. What are the access patterns? Point lookup by key, range scan, ad-hoc filtering, graph traversal, full-text search, aggregation over history — each favours a different structure.
  • 2. What consistency does it need? Linearizable, read-your-writes, or eventual — per operation, not globally.
  • 3. What is the read:write ratio and the absolute volume?
  • 4. Do you need transactions across entities?
  • 5. Only now pick the store — and say what you are giving up.

Relational versus NoSQL, honestly

Relational when the data is genuinely relational, you need transactions across entities, you cannot predict every query in advance, and consistency matters more than raw write throughput. Which is most systems.

NoSQL when you know every access pattern up front, you need write throughput or multi-region writes beyond what one primary can take, or the shape genuinely does not fit rows. The mental shift is that you model the query, not the entity — you denormalise deliberately and accept duplicated writes.

Worked examples

Choosing
user profiles      -> Postgres. Low volume, relational, needs consistency,
                      and you WILL want ad-hoc queries later.

audit log          -> append-only, huge, immutable, queried by time range:
                      time-partitioned Postgres, or Cassandra; cold data to
                      object storage. Never mixed into the OLTP tables.

session store      -> Redis. Ephemeral, key-value, needs TTL and sub-ms reads;
                      losing it is an inconvenience, not data loss.

social graph       -> traversal-heavy ("friends of friends"), where recursive
                      SQL joins get expensive: a graph DB - or, in practice,
                      an adjacency table plus heavy caching, because most teams
                      never actually need Neo4j.

transaction ledger -> Postgres, emphatically. ACID, double-entry rows that must
                      balance, unique constraints for idempotency, full audit.
                      This is the case where eventual consistency is simply WRONG.

search             -> Elasticsearch, fed FROM the database, never the source
                      of truth.

time series/metrics-> a purpose-built TSDB. Relational rows per data point are
                      orders of magnitude too expensive.

When does Postgres genuinely stop being enough?

  • Write throughput beyond what one primary can absorb (note: read scaling is solved much earlier and more cheaply with replicas and caching).
  • A genuine need for multi-region active-active writes.
  • A working set far larger than can be indexed economically.
Interview angle

Most real systems use several stores — Postgres as the system of record, Redis for sessions and caching, Elasticsearch for search, object storage for blobs. The discipline that keeps that sane is one source of truth per fact, with everything else derived and rebuildable. If two stores both claim to own the same fact, you have a reconciliation problem forever.

Chapter 20

Indexes and how they actually work

An index is a sorted structure that turns a scan into a seek — and understanding B-trees explains exactly which queries it can and cannot accelerate.

The B-tree

A B-tree (specifically a B+tree in most databases) is a balanced tree whose leaves hold sorted key values with pointers to rows, and whose leaves are linked together. Depth is typically 3–4 levels even for hundreds of millions of rows, so a lookup is a handful of page reads.

B+tree
                [ 50 | 100 ]                 root
               /     |      \
     [10|30]      [60|80]     [120|150]        internal
     /  |  \      /  |  \      /   |   \
   ...leaves: sorted keys -> row pointers, linked left-to-right...

-> point lookup:  O(log n), ~3-4 page reads
-> RANGE scan:    find the start, then walk the linked leaves. Very cheap.
-> ORDERED read:  free - the leaves are already in order

That structure explains what an index can do: equality, range, and ordering on a prefix of the key. And what it cannot: a function of the column (WHERE lower(email) = ? needs an expression index) or a leading wildcard (LIKE '%foo' has no prefix to seek to).

Composite indexes and the leftmost-prefix rule

An index on (a, b, c) is sorted by a, then b within equal a, then c. So it serves queries filtering on a, a,b, or a,b,c — but not on b alone. That is the leftmost-prefix rule, and it is the single most common indexing mistake.

Column order
INDEX (status, created_at)

WHERE status = 'PENDING' ORDER BY created_at      -> PERFECT (seek + ordered walk)
WHERE status = 'PENDING' AND created_at > ?      -> perfect
WHERE created_at > ?                             -> USELESS (no leading column)

rule: EQUALITY columns first, then the RANGE or SORT column last

Selectivity

An index is only used if it is selective enough. If one value covers 60% of the rows, the planner will correctly prefer a sequential scan — because random-access lookups for most of the table cost more than reading it in order. That is why an index on a boolean flag usually does nothing.

The fix for a skewed column is a partial index: CREATE INDEX ... WHERE status = 'pending'. It is a fraction of the size, stays in cache, and costs writes only for matching rows — ideal for a job-queue table where 99% of rows are complete.

Covering indexes

If the index contains every column the query needs, the database answers from the index alone — an index-only scan — and never touches the heap. Adding one or two payload columns via INCLUDE can turn a two-step lookup into a single read.

Covering
CREATE INDEX ON orders (account_id, created_at DESC) INCLUDE (total, status);
Principle

Every index costs write throughput, WAL volume and buffer-cache space. A table with a dozen indexes can be write-bound, and a rarely-used index evicts pages a hot one needs. Before adding one, check whether an existing index can be extended instead.

Interview angle

A good follow-up: why did adding an index make things slower? Usually writes — every insert and update maintains every index. Occasionally the planner now chooses a worse plan because statistics are stale. And on a large live table, always CREATE INDEX CONCURRENTLY, or the build takes a lock that stalls writes.

Chapter 21

Reading a query plan

EXPLAIN shows intent; EXPLAIN ANALYZE shows reality — and the first thing to check is estimated versus actual rows, not the scan type.

Being able to diagnose a slow query from a plan is one of the most practically useful skills in this book, and it is a rehearsable procedure rather than an art.

The procedure

Procedure
1. FIND IT      SELECT query, calls, mean_exec_time, total_exec_time
                  FROM pg_stat_statements ORDER BY total_exec_time DESC;
                -> sort by TOTAL time, not mean: a 5ms query run a million times
                   consumes more capacity than a 2s query run twice

2. EXPLAIN IT   EXPLAIN (ANALYZE, BUFFERS) SELECT ...;
                -> ANALYZE actually RUNS it; BUFFERS shows cache hits vs disk reads

3. READ IT      inside-out: the innermost nodes execute first

4. CHECK ROWS   estimated vs ACTUAL - a large divergence means stale statistics,
                and every downstream join choice is then wrong.  Run ANALYZE first.

5. THEN index   only after the statistics are trustworthy

What to look for

  • Estimated vs actual rows wildly different — stale statistics. Fix that before anything else, because the planner is choosing joins based on a fiction.
  • Seq Scan on a large table — a missing or unusable index. But note a seq scan on a small table is correct and fast; do not "fix" it.
  • Nested Loop with a high outer row count — an O(n×m) join. Often a missing index on the inner side.
  • Sort ... external merge Disk: — the sort spilled to disk. Either raise work_mem or add an index that provides the order for free.
  • Low buffer hit ratio — going to disk. Either the working set exceeds memory, or the access pattern is random (the UUIDv4 problem).
  • Rows Removed by Filter: large — you are reading many rows only to discard them; the index is not selective enough.

The N+1 problem

One query for the parents, then one query per parent for a related collection. 200 orders becomes 201 round trips, each paying network latency. It is invisible in a plan because each individual query is fast — you find it by counting queries, not by reading plans. Tools: ORM statistics, a datasource proxy asserting query counts in tests, or an APM trace showing 201 spans.

Principle

Measure, do not guess. The order is always: find the actual statement by total time → get a real plan with ANALYZE → check statistics → then change something → then re-measure under comparable production traffic. Adding an index because a query "looks slow" is how tables accumulate a dozen unused indexes.

Interview angle

Interviewers frequently ask you to walk through diagnosing a slow endpoint. The strong answer names the tools (pg_stat_statements, EXPLAIN ANALYZE, the APM), names the signal you looked for, and names what you rejected — "I considered caching but the data changes per request, so I fixed the index instead". Method beats the specific fix.

Chapter 22

Pagination

Offset pagination degrades linearly with depth and is unstable under concurrent writes; keyset pagination is O(1) per page and stable.

Pagination looks trivial and is one of the most common sources of production slowness, so it is worth understanding properly.

Why OFFSET is a trap

Offset
SELECT * FROM orders ORDER BY created_at DESC OFFSET 100000 LIMIT 10;

-> the database must still READ and DISCARD 100,000 rows to reach row 100,001
-> cost grows LINEARLY with page depth
-> and it is UNSTABLE: if a row is inserted while the user pages,
   everything shifts and they see a duplicate or miss a row entirely

Keyset (cursor) pagination

Carry the sort key of the last row seen and use a row-value comparison. With a matching index the database seeks directly to the position — O(1) per page regardless of depth — and it is stable under concurrent writes.

Keyset
SELECT * FROM orders
 WHERE (created_at, id) < (:last_created_at, :last_id)
 ORDER BY created_at DESC, id DESC
 LIMIT 10;

index: (created_at DESC, id DESC)

The unique tie-breaker is not optional. created_at is not unique, so without id in the comparison, rows sharing a timestamp straddle the page boundary and get skipped or duplicated — a bug that only appears with real data.

What you give up

Keyset pagination cannot jump to "page 5000", and it does not give you a total count. That is why UIs using it show infinite scroll or next/previous rather than numbered pages — and that is a product conversation you should raise, not a limitation you hide.

On total counts: an exact count on a large filtered table is itself a full scan. The options are an approximate count from table statistics, a capped count ("1,000+"), or dropping it. Most products that thought they needed deep random access actually needed better search and filters.

Principle

Default to keyset pagination for any list that can grow. Use offset only for small bounded sets, and if you must support arbitrary page jumps, cap the maximum page so a crawler cannot ask for page 500,000.

Interview angle

This is also an API design point: a cursor should be opaque to the client — base64-encode the sort key rather than exposing raw column values — so you can change the sort implementation later without breaking every consumer.

Chapter 23

Transactions, ACID and isolation levels

Each isolation level prevents a specific named anomaly — and two of the most dangerous anomalies are not in the standard table.

ACID

  • Atomicity — all or nothing. A transaction either commits entirely or leaves no trace.
  • Consistency — the database enforces its declared constraints. This is the least interesting letter: it only means foreign keys, unique and check constraints hold. What a valid state is remains your job.
  • Isolation — concurrent transactions do not interfere in ways that break correctness. This is where all the complexity lives.
  • Durability — committed means it survives a crash, via the write-ahead log.

The levels, by the anomaly each prevents

Isolation
level             dirty  non-repeat  phantom  lost-update  write-skew
Read Uncommitted   yes      yes        yes       yes          yes
Read Committed     no       yes        yes       yes*         yes    <- PG default
Repeatable Read    no       no         yes**     no           yes
Serializable       no       no         no        no           no

 * preventable with SELECT FOR UPDATE / an atomic UPDATE / a version column
** Postgres's Repeatable Read (snapshot isolation) DOES prevent phantoms
  • Dirty read — seeing another transaction's uncommitted data.
  • Non-repeatable read — reading the same row twice and getting different values, because another transaction committed in between.
  • Phantom — a range query returns new rows on re-execution.

The two that are not in the table

Lost update. Two transactions read a balance of 100, each adds 50, and one write vanishes. The cause is read-modify-write in application code. Fixes: an atomic UPDATE ... SET x = x + 50 so the read and write are one statement; SELECT FOR UPDATE to lock the row; or optimistic locking with a version column and a retry.

Write skew. Two transactions each read a set, each verify an invariant, and each write something that is individually fine but together violates it. The classic example: two doctors both go off-call because each sees the other still on. Snapshot isolation does not prevent write skew — only Serializable or explicit locking does, and since Postgres defaults to Read Committed, most systems are exposed to it without knowing.

The fix
-- lost update, prevented in one statement
UPDATE accounts SET balance = balance - :amt
 WHERE id = :id AND balance >= :amt;
-- 0 rows affected -> insufficient funds. No read-then-write anywhere.
Principle

Never read into the application, compute, and write back without either an atomic statement, a row lock, or a version check. That pattern is the single most common source of silent data corruption in web applications, and it passes every single-user test.

Interview angle

Why is Serializable not the default? Cost. Postgres implements it with Serializable Snapshot Isolation, which detects conflicts and aborts transactions — so the application must handle serialization failures and retry. Most workloads do not need it, and paying for it everywhere is wasteful; the usual approach is Read Committed plus explicit locking or constraints where an invariant actually matters.

Chapter 24

MVCC and locking

MVCC lets readers and writers not block each other by keeping multiple row versions — at the cost of bloat, which is why long transactions are dangerous.

How MVCC works

Under multi-version concurrency control, an UPDATE does not modify a row in place — it writes a new version of the row, tagged with the transaction that created it, and marks the old version as deleted by that transaction. Every transaction sees a snapshot: the set of row versions that were committed when it began.

The consequence is the property that makes it worth the complexity: readers never block writers and writers never block readers. A long analytical read does not stop writes from proceeding, and a write does not stop concurrent reads from seeing a consistent (older) view.

Versions
row versions for id=1:
  v1  (created by txn 100, deleted by txn 140)   <- txn 120 sees THIS
  v2  (created by txn 140, still live)           <- txn 150 sees this

each transaction reads the version visible at its snapshot

The cost: bloat

Old versions must survive while any transaction might still need them. So a long-running transaction prevents cleanup of every dead row created since it started — not just rows it touched. Tables and indexes grow, scans get slower, and the effect persists long after the transaction ends because the space must be reclaimed.

In Postgres, the cleanup process is VACUUM (usually autovacuum). Watching for long-running transactions and for autovacuum falling behind is routine operational work, and it is why "never hold a transaction open across a remote call or user think-time" is a hard rule rather than a style preference.

Locking

Optimistic locking assumes conflicts are rare: add a version column, and every update includes WHERE version = ? and increments it. If another transaction got there first, zero rows update and you detect the conflict. No database locks are held, so it scales well — and the caller must be prepared to retry or show a conflict.

Pessimistic locking assumes conflicts are likely: take a lock up front with SELECT ... FOR UPDATE, blocking other writers until you commit. Correct for genuinely contended rows — seat inventory, account balances — at the cost of held locks, reduced concurrency, and deadlock risk if lock order is inconsistent.

Locking
-- pessimistic, with a timeout, and rows locked in a DETERMINISTIC order
SET LOCAL lock_timeout = '3s';
SELECT * FROM seats WHERE id IN (:ids) ORDER BY id FOR UPDATE;
-- ordering by id prevents deadlock between two users selecting overlapping seats
Principle

Choose by contention and the cost of failure. A user editing a profile: optimistic, and show "someone else changed this". Two people buying the last seat: pessimistic, or a constraint that makes overselling impossible. Always set a lock timeout, or one stuck transaction stalls the system.

Interview angle

A follow-up worth being ready for: how do you retry an optimistic failure? Catch it outside the transaction, reload fresh state, reapply the change, and retry with a small bounded number of attempts and jitter. Retrying inside the failed transaction cannot work — it is already marked rollback-only.

Chapter 25

Replication

Replication buys read capacity and failover; the lag it introduces breaks user experience in three specific, named ways.

Leader-follower

All writes go to one leader, which streams its change log to followers that serve reads. This scales reads, provides failover candidates, and keeps a single writer so there are no write conflicts to resolve.

Modes
synchronous       ack only after a follower has it
                  -> no data loss on leader failure
                  -> write latency includes the SLOWEST follower; if it stalls,
                     writes stall

asynchronous      ack immediately
                  -> fast
                  -> a leader crash LOSES anything not yet shipped

semi-synchronous  ONE follower synchronous, the rest async
                  -> bounded data loss, bounded latency  <- the usual production answer

Replication lag, and the three ways it breaks users

Lag is the gap between leader and follower. It is not merely a performance metric — it is your data-loss window (RPO) in an asynchronous failover, and it produces three distinct user-visible bugs.

  • Read-your-writes violated. A user posts a comment, the next read hits a lagging replica, and their own comment is missing — so they assume it failed and post again. Now you have duplicates and an annoyed user.
  • Monotonic reads violated. Two consecutive reads hit replicas with different lag, so time appears to move backwards: a counter decreases, a message disappears and reappears.
  • Consistent prefix violated. With partitioned replication, a reply arrives before the message it answers, so a conversation reads out of order.

The fixes

Mitigations
read-your-writes  -> route that user's reads to the LEADER for N seconds after a write
                  -> or pass an LSN token the replica must have applied
monotonic reads   -> pin the session to ONE replica (sticky by user hash)
consistent prefix -> keep causally related writes in the same partition

Note that sticky routing gives monotonic reads but not read-your-writes — the pinned replica may still be behind the leader. The two need different fixes, and conflating them is a common error.

Failover and split-brain

Promoting a follower under asynchronous replication loses the un-shipped writes. Worse, if the old leader returns without knowing it was demoted, you get split-brain — two leaders accepting conflicting writes. The defence is a consensus-based election plus fencing tokens: a monotonically increasing epoch number that storage checks, so the deposed leader's writes are rejected.

Principle

Your replication lag is your RPO. If lag is five seconds, an unplanned failover loses five seconds of writes. Monitor lag as a first-class metric and know that number, because it is the answer to "how much data can we lose".

Interview angle

When do read replicas not help? When the load is write-heavy (replicas perform the same writes), when queries need the very latest data, or when the problem is a slow query rather than volume — a bad query is equally slow on a replica. Saying that unprompted stops "add a read replica" being a reflex.

Chapter 26

Partitioning and sharding

Sharding is the last resort because the shard key permanently determines which queries stay cheap and which become impossible.

Try everything else first

Before sharding: archive cold data to object storage, partition by time within one instance, add read replicas, cache aggressively, and scale vertically. Sharding is last because it costs you cross-shard joins and cross-shard transactions permanently — that is not a temporary cost you pay during migration.

Strategies

  • Range — shard by key ranges (A–F, G–M…). Simple, supports range scans, and produces hotspots on skewed keys or on time-ordered keys where all writes hit the newest shard.
  • Hash — hash the key and modulo. Distributes evenly, destroys range queries, and rebalancing moves nearly everything unless you use consistent hashing.
  • Directory-based — an explicit lookup table mapping key to shard. The most flexible, allows targeted rebalancing and per-tenant placement, at the cost of one extra hop and a service that must be highly available.

The shard key is the whole design

Choose it so that the dominant access pattern stays on one shard. For a transactions table, account_id is natural: "all transactions for this account" is the common query and stays single-shard, and per-account transactions remain atomic.

Choosing
shard by account_id     -> account history = single-shard read.  GOOD
shard by transaction_id -> every account's history is scattered across every
                           shard; the common query becomes scatter-gather.  BAD
shard by time           -> tempting for append-only data, but ALL writes hit the
                           newest shard while the rest sit idle.  HOTSPOT

What becomes expensive afterwards

  • Cross-shard transactions — a transfer between accounts on different shards can no longer be one database transaction, so you need a saga with an explicit pending state.
  • Joins across shards — either denormalise, or fan out and join in the application.
  • Aggregate reporting — becomes scatter-gather across every shard, which is why analytics usually moves to a separate warehouse.
  • Global uniqueness — a unique constraint spanning shards must be enforced by a separate service or by making the key itself globally unique.
  • Secondary-key lookups — finding a row by anything other than the shard key means querying every shard, unless you maintain a secondary index mapping that key to a shard.

Resharding with zero downtime

Migration
1. dual-write      the app writes to old AND new (new is not read yet)
2. backfill        batch copy history: throttled, resumable, idempotent
3. verify          row counts + checksums per shard; reconcile any drift
4. cut over reads  1% -> 10% -> 50% -> 100% behind a flag, watching error rate
5. stop old writes, then drop

keep the rollback available until step 4 has been clean for days
Principle

Pick the shard key for the query you cannot afford to make slow. You can add caches and replicas later; you cannot change the shard key later without a full migration. It is the most expensive decision in the data layer to reverse.

Interview angle

A good thing to volunteer: ids must not come from a per-shard sequence, or rebalancing produces collisions. Use UUIDv7 or a Snowflake-style generator so ids are unique and time-ordered independent of placement — and remember UUIDv4 is time-unordered, which destroys B-tree insert locality on every shard.

Chapter 27

NoSQL families

Four families with different data models — and in every one you model the query rather than the entity.

Key-value

Redis, DynamoDB, Memcached. A dictionary at scale: get and put by key, sometimes with TTL. Extremely fast, trivially partitionable. No querying by value — if you need "all users in Mumbai", you must have built that key yourself. Ideal for sessions, caches, feature flags, rate-limit counters and materialised lookups.

Document

MongoDB, DynamoDB, Couchbase. Stores self-contained JSON-like documents with flexible schemas, and supports querying inside the document with secondary indexes. Good when the aggregate is naturally self-contained — a product with its variants and attributes — and when the shape varies between records. Weak when you need joins across documents or transactions spanning several.

The honest caveat: "flexible schema" means the schema moved from the database into your application code, where it is unenforced. Documents written two years ago have a different shape, and every reader must handle both.

Wide-column

Cassandra, HBase, ScyllaDB. Rows keyed by a partition key plus clustering columns that determine sort order within the partition. Built for very high write throughput and multi-region writes with no single leader, and tuned per-query consistency (quorum reads and writes).

Wide-column
Cassandra's model forces query-first design:
  PRIMARY KEY ((account_id), created_at)
              ^partition     ^clustering (sort order within the partition)

-> "transactions for this account, newest first" is a single fast read
-> "all transactions over 1000 across all accounts" is NOT SUPPORTED
   unless you built a table for it

no joins. no ad-hoc queries. one table PER QUERY, with duplicated writes.

Graph

Neo4j, Neptune. Nodes and edges as first-class citizens, so multi-hop traversals ("friends of friends who work at X") are cheap where recursive SQL joins get expensive. Genuinely the right tool for fraud rings, recommendations over a social graph, and permission hierarchies — and genuinely rare, because most teams' "graph" is two hops deep and an adjacency table with caching handles it.

What you give up

Principle

Moving to NoSQL trades query flexibility for scale and write throughput. You must know your access patterns in advance, because a new access pattern later means a new table, a backfill and dual writes. That cost is paid at design time and it is the main reason to stay relational until something forces you off.

Interview angle

When asked "SQL or NoSQL", the strong answer names the specific constraint that would move you: multi-region active-active writes, write volume beyond one primary, or a data shape that genuinely does not fit rows. "NoSQL scales better" is not an answer — Postgres scales further than most people assume, and the real difference is what you are allowed to ask it.

Chapter 28

Storage engines: B-tree versus LSM

B-trees update in place and favour reads; LSM trees append sequentially and favour writes, moving the cost into background compaction.

B-tree

Updates are made in place: find the leaf page, modify it, and make it durable via the write-ahead log. Reads are excellent — a point lookup is a handful of page reads and a range scan follows the linked leaves. Used by Postgres, MySQL/InnoDB, and most relational databases.

The cost is that writes are random IO: each one touches a specific page, plus the WAL write. Write amplification is roughly the page size divided by the row size — updating a 100-byte row may write an 8 KB page — and writes do not batch well.

LSM tree

Writes go to an in-memory memtable and a sequential commit log. When the memtable fills, it is flushed to an immutable, sorted file on disk called an SSTable. So every write is sequential, which is dramatically faster on both spinning disks and SSDs and batches naturally. Used by Cassandra, RocksDB, LevelDB, ScyllaDB.

LSM
WRITE  -> memtable (in memory) + sequential commit log
           |  memtable full
           v
         SSTable (immutable, sorted, on disk)
         SSTable
         SSTable ...      <- reads may have to check SEVERAL

READ   -> memtable, then each SSTable newest-first
          bloom filter per SSTable answers "definitely not here" cheaply

COMPACTION (background) merges SSTables, discards superseded rows,
           bounds the number of files a read must touch

The amplification trade

Trade
                   read amplification   write amplification
B-tree             LOW                  HIGH (random IO, page-sized writes)
LSM                HIGHER (N SSTables)  LOW at write time...
                                        ...but compaction pays it later,
                                        in background IO

That last line is the operationally important part: an LSM store is fast until compaction falls behind. Then SSTable count grows, reads probe more files, latency climbs, and disk usage grows because superseded data has not been merged away. It presents as a gradual degradation rather than a failure, which makes it easy to miss until it is severe.

Principle

Choose by workload shape, not by fashion: read-heavy with ad-hoc queries and transactions → B-tree (relational). Write-heavy with known access patterns and a need for multi-region writes → LSM (Cassandra-family).

Interview angle

Bloom filters are worth being able to explain here, because they are what makes LSM reads viable: a compact bit array that answers "definitely not present" or "probably present", with false positives but never false negatives. Each SSTable carries one, so a read skips most files without touching disk.

Chapter 29

Search and the inverted index

Search is a different system because it needs a different index — term to documents, not document to terms.

Why LIKE '%term%' does not scale

A relational index is sorted by value, so it can seek to a prefix. A leading wildcard has no prefix, so the database must scan every row and run a substring match. At a hundred million documents that is not slow, it is impossible.

The inverted index

Invert the mapping: instead of document → terms, store term → posting list of document ids (with positions and frequencies). A query for two terms then intersects two posting lists rather than scanning the corpus.

Inverted index
docs:  1 "fast payment gateway"   2 "payment ledger"   3 "gateway timeout"

index: payment -> [1, 2]
       gateway -> [1, 3]
       ledger  -> [2]
       fast    -> [1]

query "payment gateway" -> intersect [1,2] and [1,3] -> doc 1 -> then RANK

Analysis

Text is processed by an analyzer at both index time and query time, and the two must match or nothing hits. Tokenize, lowercase, strip punctuation, remove stop words, then stem or lemmatize so "running" matches "run". Language matters here, and a mismatched analyzer is the most common cause of "why doesn't this match?".

Ranking

BM25 is the modern default, the successor to TF-IDF: term frequency with saturation (the tenth occurrence adds less than the second), inverse document frequency (rare terms matter more), and length normalisation so a long document does not win by repetition. Real systems then layer business signals on top — recency, popularity, click-through — and increasingly blend a vector/semantic score with the lexical one.

Elasticsearch operationally

  • An index is split into primary shards, fixed at creation — you cannot change the count without reindexing. This is the single most important operational fact.
  • A search fans out to one copy of every shard, each returns its top K, and a coordinating node merges. So over-sharding hurts: querying 200 shards costs 200 round trips and 200 result merges.
  • It is near-real-time, not real-time: documents become searchable only after a refresh, by default once a second. A write-then-immediately-search flow will miss.
  • It should not be your source of truth — index from your database via CDC or an outbox so you can always rebuild.
Principle

Reindexing is routine, not an emergency — analyzers change, mappings change, shard counts change. Design for it: applications query an alias, never a concrete index name, so you build a new index and flip the alias atomically with instant rollback.

Interview angle

For typeahead, do not use the full search path — the latency budget is far tighter and you need prefix matching rather than relevance ranking. Use a completion suggester (an in-memory finite state transducer) or a prefix trie. Conflating search and autocomplete is a common design mistake.

Chapter 30

CDC and the OLTP/OLAP split

Keep analytics off your production database, and move data out with change data capture rather than polling.

Why the split exists

OLTP is optimised for many small, indexed, latency-sensitive transactions: row-oriented, heavily indexed, tuned for concurrency. OLAP is optimised for scanning and aggregating huge volumes: column-oriented, compressed, tuned for throughput.

They are genuinely different systems. Running an analyst's GROUP BY over three years of transactions on your production primary competes for the same buffer cache, CPU and locks as the checkout path — and column-oriented storage can be ten to a hundred times faster for that query anyway.

Why the obvious solutions fail

  • A read replica helps with load, but the schema and storage format are still wrong for scans — and a long analytical query on a Postgres replica delays WAL replay, causing replication lag or query cancellation.
  • Nightly SELECT * exports put heavy periodic load on the primary, miss deletes entirely, and give data that is up to a day stale.
  • Polling by updated_at adds query load, misses deletes, misses intermediate updates between polls, and has ordering problems under concurrent writes.

Change data capture

CDC reads the database's write-ahead log — the same stream replication uses — via a tool like Debezium, and emits an event per row change into Kafka. Three properties make it superior to every alternative above:

Principle
  • It adds no query load on the primary — it is reading the log the database already writes.
  • It captures deletes and every intermediate update, which polling structurally cannot.
  • It preserves commit order, so downstream consumers see a consistent sequence.
CDC
Postgres WAL --> Debezium --> Kafka topic per table --> consumers
                                                          |- warehouse (upsert by PK)
                                                          |- search index
                                                          |- cache invalidation
                                                          |- stream processing

one pipeline feeds every derived system, and all of them are REBUILDABLE
by replaying from the log

The operational realities

  • The replication slot must be consumed, or the primary retains WAL indefinitely and fills its disk — an unconsumed slot is a genuine way to take down a database.
  • Schema changes need a schema registry with compatibility rules.
  • Initial snapshots of large tables need care and throttling.
  • The pipeline is at-least-once, so consumers must be idempotent — for a warehouse that means upsert by primary key, which is naturally idempotent.
Interview angle

If you also need near-real-time aggregates rather than just a warehouse, add a stream processor — Kafka Streams or Flink — consuming the same CDC topics to maintain windowed aggregates in a low-latency store. That is the standard shape for live dashboards without touching the primary, and it reuses the pipeline you already built.

IV. Distributed Systems
Chapter 31

Why distributed systems are hard

Every hard idea in this part is a response to one fact: over a network, you cannot tell the difference between slow, dead, and unreachable.

On a single machine, calls return, memory is reliable, and time moves forward. Distribute the same program and each of those stops holding. It is worth being explicit about what breaks, because every technique in the next eight chapters exists to work around one of these.

The fallacies

The classic list of things people implicitly assume and that are all false: the network is reliable, latency is zero, bandwidth is infinite, the network is secure, topology does not change, there is one administrator, transport cost is zero, the network is homogeneous. You do not need to recite them, but each one, violated, is a real class of bug.

The central problem

A timeout is not an answer. You called a service and got nothing back. Did it never arrive? Did it execute and the reply was lost? Is it still running and about to execute? You cannot distinguish these, and no amount of engineering makes you able to.

The ambiguity
you send a charge request -> TIMEOUT

possible realities:
  1. the request never arrived         -> retrying is correct
  2. it executed, the reply was lost   -> retrying DOUBLE-CHARGES
  3. it is still executing             -> retrying double-charges
  4. it failed after partial work      -> retrying may compound it

you cannot tell which. This single fact motivates idempotency,
reconciliation, sagas, and most of what follows.

The other three

  • Partial failure. Some components work and others do not. There is no global "the system is down" — there is a rolling set of things that are and are not currently working, and your code must behave sensibly in every combination.
  • No global clock. Two machines disagree about the time by milliseconds or worse, and clocks can jump backwards when NTP corrects them. So "which of these two events happened first" is not answerable by comparing timestamps.
  • No global state. Any view of the system is a view of the past, gathered at different moments from different places. By the time you have assembled it, it is already wrong.
Principle

Design so that the ambiguous case is safe rather than resolved. You cannot make a timeout unambiguous, so make retrying harmless (idempotency), make the intermediate state explicit (a pending status), and have a slower process that establishes truth eventually (reconciliation). Trying to eliminate the ambiguity is how people end up with distributed locks that do not work.

Interview angle

This chapter is worth internalising before CAP or consensus, because those are answers and this is the question. When an interviewer asks "what happens if this call times out?", the answer they are listening for is not a mechanism — it is whether you know that success and failure are not the only two outcomes.

Chapter 32

CAP, PACELC and what they really say

CAP only constrains you during a partition, and PACELC adds the trade you actually make every day: latency versus consistency.

State CAP precisely

During a network partition, you must choose between consistency and availability. That is all it says. It is not "pick two of three" — partitions are a fact of distributed systems, not an option you can decline. So the real choice is CP or AP, and only while partitioned.

A system that is "CA" is either a single node or a system that has not been partitioned yet.

The choice
partition happens: node A cannot reach node B

CP: refuse to serve rather than risk divergence
    -> the minority side becomes unavailable
    -> Postgres with synchronous replication, etcd, ZooKeeper, HBase

AP: keep serving, reconcile later
    -> both sides accept writes; you now have conflicts to resolve
    -> Cassandra (tunable), DynamoDB (eventual mode), DNS

PACELC — the more useful framing

If Partitioned, choose Availability or Consistency; Else, choose Latency or Consistency. The second half is what you actually trade every day: even with a perfectly healthy network, waiting for a quorum or a cross-region acknowledgement costs latency.

PACELC
Postgres, synchronous replication   PC / EC   consistent in both cases, slower
DynamoDB, eventual reads            PA / EL   available and fast, may be stale
Cassandra with QUORUM               PA / EC   tunable per query

Where the choice appears in real settings

This is not abstract — it is a dial in your configuration.

The dials
Postgres    synchronous_commit = on | remote_write | off
Kafka       acks=all + min.insync.replicas=2   vs   acks=1
Dynamo-ish  W + R > N  (quorum)  vs  W=1, R=1  (eventual)
MongoDB     writeConcern majority + readConcern majority
Principle

Choose per operation, not per system. In the same product, a payment write should be linearizable, a user reading their own profile needs read-your-writes, and a recommendations panel is happily eventual. Applying one consistency level globally either costs you latency you did not need to spend, or correctness you could not afford to lose.

Interview angle

The most common CAP mistake in interviews is treating it as a system-level label — "we'll use an AP database". Push past that: name the specific operation, name the failure the user would see if it were inconsistent, and choose accordingly. That is the answer that shows you have made this decision rather than read about it.

Chapter 33

Consistency models

Learn these by the symptom a user sees when each is violated — the vocabulary alone is cheap.

There is a hierarchy of guarantees, from strongest and most expensive to weakest and cheapest. What makes them memorable is not the definitions but the bugs they produce.

The models, and their violation symptoms

  • Linearizable (strong). Every read returns the most recent committed write, as if there were one copy. Violation: two people both successfully book the last seat. Systems: single-primary Postgres, etcd, Spanner.
  • Sequential. All operations appear in some total order consistent with each client's own order, but not necessarily real time. Rarely chosen explicitly.
  • Causal. Operations that are causally related appear in order everywhere; concurrent ones may be seen in different orders. Violation: a reply appears above the message it answers — the classic comment-thread bug.
  • Read-your-writes. You always see your own writes. Violation: a user posts a comment, the next read hits a lagging replica, their comment is missing, so they post it again.
  • Monotonic reads. You never see time move backwards. Violation: a counter decreases; a message disappears and reappears on refresh.
  • Eventual. Replicas converge if writes stop, with no ordering guarantee meanwhile. Fine for a follower count; catastrophic for a balance. Systems: DNS, S3 listings.
Diagnosis
symptom                                    -> model violated
"my own comment vanished on refresh"       -> read-your-writes
"the count went backwards"                 -> monotonic reads
"the reply is above the message"           -> causal
"both users got the last seat"             -> linearizability
"the follower count is 2s behind"          -> eventual, and that is FINE

How to get read-your-writes without going fully strong

This is the practical part, and the one that shows you have shipped it.

Techniques
1. route that user's reads to the PRIMARY for N seconds after they write
2. pin the session to one replica (gives monotonic reads, NOT read-your-writes)
3. return the write's log sequence number (LSN) to the client, and require the
   replica to have applied at least that LSN before serving
   -> the most precise, and the one that scales
Principle

Pick the weakest model that avoids a user-visible bug. Stronger consistency always costs latency, availability, or both, and paying for guarantees nobody needs is a real and common waste.

Interview angle

Causal consistency has a cheap partial solution worth knowing: keeping causally related writes in the same partition gives you causal order within that partition for free. That is why partitioning a conversation by thread id is such a common design — it buys ordering without any coordination.

Chapter 34

Idempotency

Since duplicates are guaranteed, the goal is not to prevent them but to make them harmless — and that single shift solves most distributed-systems problems.

An operation is idempotent if performing it twice has the same effect as performing it once. Given that at-least-once delivery is the only achievable guarantee and that timeouts are ambiguous, idempotency is the single most valuable property you can design in.

Which operations are naturally idempotent

Which is which
idempotent by nature:
  SET balance = 500          absolute assignment
  DELETE WHERE id = 5        deleting twice = deleting once
  PUT /users/5 {...}         full replacement
  upsert by primary key

NOT idempotent:
  balance = balance + 50     relative change
  INSERT INTO transactions   creates a new row each time
  POST /payments             creates a new resource
  "send an email"            cannot be un-sent

Idempotency keys

For operations that are not naturally idempotent, the client supplies a stable key per logical attempt, and reuses it on every retry of that attempt. The server records the key and the resulting response.

The critical detail is atomicity: recording the key and applying the effect must happen in one transaction, with a unique constraint on the key doing the work. Record first and apply second, and a crash in between loses the effect permanently; apply first and record second, and a crash duplicates it.

The pattern
BEGIN;
  INSERT INTO processed(idempotency_key) VALUES (?);   -- UNIQUE constraint
  UPDATE balances SET amount = amount + ? WHERE id = ?;
COMMIT;
-- duplicate key -> unique violation -> skip, return the STORED response

for an API, also store the request HASH:
  same key + same body      -> return the stored response
  same key + DIFFERENT body -> 422. That is a client bug, not a retry.

Dedup windows

The processed-keys store must be bounded or it grows forever. Size the retention to cover your maximum redelivery window — broker retention plus retry topics plus a margin, typically days — and partition by time so you drop old partitions rather than running deletes.

State the trade honestly: too short a window and a late redelivery duplicates; too long and the store becomes a cost and a bottleneck.

Principle

Prefer idempotency over coordination. Making an operation safe to repeat is almost always cheaper, simpler and more robust than building machinery to ensure it happens exactly once — and unlike that machinery, it still works when the machinery fails.

Interview angle

For effects you do not control — charging a card, sending an email — the idempotency key must be passed to the downstream provider, which every serious payment API accepts. If a provider does not support one, you need a local pending state plus reconciliation against their records, because you cannot make their side idempotent from outside.

Chapter 35

The dual-write problem and the outbox

You cannot atomically write to a database and publish to a broker — the outbox pattern turns two writes into one local transaction plus a relay.

The problem

An extremely common piece of code: save an order to the database, then publish an OrderPlaced event. It looks fine and it is broken in both orderings.

Dual write
save to DB, then publish:
  DB commits, publish FAILS  -> state exists with NO event
                                downstream systems never learn about it

publish, then save to DB:
  publish succeeds, DB rolls back -> an event for something that NEVER HAPPENED
                                     downstream acts on a phantom order

and you cannot wrap them in one transaction: they are different systems

Both failure modes are silent and both corrupt downstream state. Retrying does not help, because the failure is in the gap between the two systems, not in either one.

The transactional outbox

Write the domain row and an event row into the same database, in the same local transaction. Now there is only one write, so it is atomic by construction. A separate relay then reads unsent outbox rows and publishes them.

Outbox
-- one local transaction: atomic, no dual write
BEGIN;
  INSERT INTO orders(id, ...) VALUES (...);
  INSERT INTO outbox(id, topic, payload, created_at) VALUES (...);
COMMIT;

-- relay (separate process):
--   read unsent outbox rows in order -> publish -> mark sent
--   crash after publish, before mark? -> republished -> DUPLICATE
--   which is fine, because consumers are idempotent

Note what this buys and what it does not. It guarantees at-least-once publication of exactly the events whose transactions committed. It does not give exactly-once — the relay can publish twice — which is why the consumer must be idempotent. The whole design assumes the previous chapter.

Polling relay versus CDC

The relay can poll the outbox table, or you can use CDC to read the write-ahead log directly (Debezium). CDC is better: it adds no query load, it captures commit order naturally, and there is no polling interval to tune. The cost is operating Debezium and coupling to the database's replication mechanism.

The inbox, for the receiving side

The mirror pattern: a consumer records processed event ids in an inbox table, in the same transaction as the effect. That is exactly the idempotent-consumer pattern from the previous chapter, and outbox plus inbox together give you reliable messaging over unreliable delivery.

Principle

Whenever a design says "save it and then publish an event", say the words dual write and reach for the outbox. It is one of the highest-signal patterns to name in an interview because it shows you have hit the failure rather than read the happy path.

Interview angle

Watch the outbox table's growth: mark rows sent and delete or partition them, or it becomes the largest table in your database. And keep the relay single-writer or ordered per key, because publishing outbox rows out of order reintroduces the ordering problem the transaction just solved.

Chapter 36

Distributed transactions: 2PC and sagas

Two-phase commit gives real atomicity and blocks on coordinator failure; sagas give you forward progress with compensations that are not rollbacks.

Two-phase commit

A coordinator asks every participant to prepare (do the work, hold locks, promise you can commit), and if all vote yes it tells them to commit. It gives genuine atomicity across services — and it is rarely the right answer.

  • It blocks. If the coordinator dies after the prepare phase, participants sit holding locks with no way to decide, indefinitely.
  • It couples availability to the least available participant.
  • It holds database locks across network calls, which destroys throughput.
  • It does not scale, and it requires participants that support the protocol.

Legitimate uses exist — within a single database, or between two resources managed by one transaction manager (a database and a broker via XA). Across services over a network, essentially never.

Sagas

Replace atomicity with a sequence of local transactions, each with a compensating action that semantically undoes it. No global lock, no coordinator holding everything hostage.

The price is that the system is never atomic: there are real intermediate states where the payment succeeded and the inventory has not yet been reserved, and every consumer of that data must understand them.

Saga
order flow:
  reserve payment -> reserve inventory -> create shipment

if inventory fails:
  compensate payment (void or refund)

STATE MACHINE (orchestrated):
  PENDING -> PAYMENT_RESERVED -> INVENTORY_RESERVED -> SHIPPED
                 |                     |
                 v                     v
            VOID_PAYMENT        REFUND + RELEASE_INVENTORY

Choreography versus orchestration

Choreography: each service listens for events and emits its own. Loosely coupled, no central component — but the flow exists nowhere in the code, so understanding or debugging it means tracing events across services, and accidental cycles are easy to create.

Orchestration: a coordinator explicitly drives the steps as a state machine. The flow is in one readable place, easy to monitor, and easy to attach timeouts and retries to. The cost is a component that must itself be highly available and must not accumulate business logic.

For an order flow of any complexity, orchestration is usually the better answer, and the reason to give is debuggability and explicit timeouts.

Principle

Compensations are not rollbacks. They are new forward transactions with their own semantics and their own failure modes. A refund is visible on the customer's statement; an email cannot be unsent; a dispatched shipment can only be recalled. Some steps are genuinely irreversible — so order the saga to put reversible and risky steps first and irreversible ones last.

Interview angle

Two follow-ups to have ready. What if a compensation fails? Retry with backoff, then escalate to a human via a dead-letter queue — there is no automatic escape, which is another argument for ordering. How do you make the intermediate states safe? Model them explicitly in the domain; an order in PAYMENT_RESERVED is a real state with defined semantics for the UI and for reporting. Pretending the operation is atomic is what produces the confusing bugs.

Chapter 37

Consensus and leader election

Consensus is how nodes agree despite failures — you use a consensus system, you never implement one.

Consensus underpins anything that requires exactly one authoritative answer: who is the leader, what is the current configuration, who holds this lock. State the practical position first — use etcd, ZooKeeper or a managed equivalent; implementing consensus correctly is a research-grade problem.

Raft, in the amount of detail an interview wants

Nodes are follower, candidate or leader. Time is divided into terms, each with at most one leader. A follower that hears nothing from a leader for a randomized election timeout becomes a candidate, increments the term, and requests votes; a node votes once per term, so a candidate winning a majority becomes leader. The randomized timeouts are what prevent perpetual split votes.

Raft
follower --(election timeout)--> candidate --(majority votes)--> leader

log replication:
  all writes go to the LEADER
  leader appends to its log, replicates to followers
  once a MAJORITY have acknowledged -> the entry is COMMITTED -> applied

quorum:  3 nodes tolerate 1 failure
         5 nodes tolerate 2
         6 nodes ALSO tolerate only 2  -> use ODD counts

Safety comes from the election restriction: a candidate cannot win unless its log is at least as up to date as the voter's, which guarantees a committed entry can never be lost by a leader change. That single rule is the heart of Raft's correctness, and mentioning it shows you have read past the diagram.

What these systems are actually used for

Uses
config and feature data     small, critical, rarely changing
service discovery           which instances exist right now
distributed locks + leases  "only one worker should do this"
leader election             for singleton jobs and schedulers
cluster membership          Kubernetes stores ALL its state in etcd
                            Kafka's KRaft mode replaced ZooKeeper with Raft

The limits

  • Consensus requires a majority to be reachable, so it sacrifices availability under partition — it is CP. A minority partition becomes read-only or unavailable, deliberately.
  • Every write costs at least one round trip to a majority, so throughput is bounded and latency tracks your slowest quorum member.
  • Never put high-volume application data in it. It is for small, critical, rarely-changing state.
Principle

A distributed lock is not sufficient for correctness even with consensus, because the holder can be paused (GC, VM suspend) past its lease expiry while another node legitimately acquires it. If correctness depends on mutual exclusion, you need a fencing token — a monotonically increasing number the protected resource checks and rejects if stale.

Interview angle

That fencing-token point is one of the most reliable ways to distinguish someone who has read about distributed locks from someone who has thought about them. The short version: locks are a performance optimisation to reduce contention; the invariant must be enforced by the resource itself.

Chapter 38

Clocks, ordering and causality

Wall-clock timestamps cannot order events across machines — use sequence numbers within a partition, and logical clocks when you need causality.

Why you cannot trust timestamps

Machine clocks drift, and NTP corrects them by jumping — sometimes backwards. So two events timestamped on different machines cannot be reliably ordered, and even on one machine a timestamp can go backwards between two consecutive operations.

Clock skew
machine A: event at 10:00:00.100  (clock is 40ms fast)
machine B: event at 10:00:00.090  (clock is correct)

-> B's timestamp is earlier, but A's event may have happened first
-> "last write wins" by timestamp can silently discard the newer write

also: use a MONOTONIC clock for measuring durations (System.nanoTime),
      never a wall clock - which can jump backwards mid-measurement

Logical clocks

Lamport timestamps are a counter each node increments on every event and includes in every message; a receiver sets its counter to max(local, received) + 1. This gives a total order consistent with causality — if A caused B then A's timestamp is lower — but the converse does not hold, so it cannot tell you whether two events were concurrent.

Vector clocks carry one counter per node, so they can distinguish "A happened before B" from "A and B were concurrent". That is what lets a system detect a genuine conflict rather than silently discarding one side. The cost is metadata that grows with the number of nodes.

What to actually do

Principle
  • Within a partition, use a monotonic sequence number assigned by a single authority. This is the practical answer for chat messages, ledger entries and event streams — it gives a total order where it matters and needs no coordination beyond the partition.
  • Across partitions, do not promise ordering unless you genuinely need it. Keeping causally related writes in one partition is far cheaper than any cross-partition ordering scheme.
  • Never resolve conflicts by wall-clock timestamp unless losing data is acceptable — last-writer-wins silently discards a concurrent write with no record that it happened.
  • For durations, use a monotonic clock. Wall clocks jump; monotonic clocks do not.

The exception worth knowing

Google's Spanner uses TrueTime: GPS and atomic clocks give it a bounded uncertainty interval, and it simply waits out the uncertainty before committing. That buys externally consistent global transactions — at the cost of specialised hardware in every datacentre, which is why nobody else does it.

Interview angle

A neat way to demonstrate this understanding: when someone proposes ordering by created_at, ask what happens when two events share a millisecond, and what happens when the clock is corrected backwards. Both are real, and the fix — a sequence number with a unique tie-breaker — is the same fix as keyset pagination needs.

Chapter 39

Multi-region

Most systems do not need multi-region, and the ones that do usually need it for latency or compliance rather than availability.

Be honest about this one: multi-AZ within a single region already survives a datacentre failure and costs a fraction as much. Multi-region is justified by user latency (users on another continent), data residency (a legal requirement), or a disaster-recovery target that a single region cannot meet — not by a vague desire for resilience.

Active-passive

  • One region serves traffic, the other replicates asynchronously and stands ready.
  • Simple, cheap, no conflict resolution.
  • Failover is a real operation with real risk, and your RPO equals the replication lag.
  • The failure mode to name: a standby you have never failed over to does not work. Certificates expire, secrets never got replicated, capacity was never provisioned.

Active-active

Both regions accept writes. Excellent latency and true regional resilience — and it forces you to solve write conflicts, which is the hard part.

  • Last-writer-wins — simple, and it silently discards data. Unacceptable for anything transactional.
  • CRDTs — converge without coordination, but only for data types that fit: counters, sets, sequences.
  • Application-level merge — correct and expensive to build.
  • Or: avoid conflicts entirely, which is the next section.

Region-pinned users — usually the right answer

The middle path most teams should choose, and the one worth naming explicitly. Each user's writes are routed to their home region, so there are never concurrent conflicting writes for a given entity, with asynchronous cross-region replication for disaster recovery and cross-region reads. You get active-active's latency and utilisation without the conflict problem.

Data residency

GDPR and similar regimes can make this mandatory rather than optional: EU users' personal data must stay in the EU. That means partitioning by jurisdiction, keeping PII in a region-local store with only non-personal references crossing borders, and being able to prove it. Compliance frequently dictates the topology before performance does.

Principle

Define RTO and RPO before choosing an architecture, because they determine it. RPO of zero requires synchronous cross-region replication and its 100 ms+ write latency. RPO of five minutes allows asynchronous. RTO of minutes needs automated, health-checked failover; RTO of hours can be a runbook. Then test the failover on a schedule — an untested failover is a plan, not a capability.

Interview angle

The thing that breaks first when you go multi-region is the database. Application servers are easy to run in two places; a single-writer database is not, and moving to multi-master introduces exactly the conflict resolution you were trying to avoid. That is why region-pinning is such a common resolution — it keeps a single writer per entity.

V. Making It Reliable
Chapter 40

Timeouts, retries and backoff

Nothing else works without timeouts, and retries without jitter turn a brownout into an outage.

Timeouts first

Every network call needs one. A call with no timeout is a resource leak waiting for a slow dependency — the thread, the connection and the memory are held indefinitely, and under load that exhausts your pool while the CPU sits idle.

The crucial refinement is that the timeout budget must shrink down the chain.

Budget
client        3000 ms
 gateway      2500 ms
  service     2000 ms
   cache        50 ms   (fail open -> fall through to the DB)
   db          800 ms   (also enforce statement_timeout server-side)
   downstream  600 ms

if budgets do NOT shrink: the client gives up at 3s while every layer below
still holds threads and connections for its own longer timeout -> saturation

Retries — narrowly

Retry only idempotent operations, only on transient failures (timeouts, connection resets, 5xx, 429), and with a hard cap on attempts. Never retry a 4xx — a validation error or a declined card will fail identically every time, so retrying wastes capacity and can trip the provider's abuse controls.

Backoff and jitter

Exponential backoff spaces retries out. Jitter — randomising the delay — is what stops every client retrying at the same instant. Without it, a brief failure produces a synchronized retry storm that keeps a recovering service down.

Backoff
delay = min(cap, base * 2^attempt) * random(0.5, 1.5)   // full jitter

base = 100ms, cap = 30s, max 3 attempts

without jitter: 1000 clients all retry at exactly t+200ms, t+400ms, t+800ms
                -> the recovering service is hit by 1000 simultaneous requests

Retry amplification

The multiplication people miss: 3 retries at the gateway × 3 at the service = 9 requests for one user action. Layered retries turn a struggling dependency into a dead one.

Principle

Retry at one layer only, and make it the layer that knows whether the operation is safe to repeat. Every other layer fails fast and propagates. If you cannot decide which layer, retry closest to the failure and nowhere else.

Interview angle

A subtle addition worth mentioning: retries should also respect a total time budget, not just an attempt count. Three attempts with backoff can exceed the caller's timeout, so the retries complete into a void — burning capacity for a response nobody is waiting for. Check remaining budget before each attempt.

Chapter 41

Circuit breakers and bulkheads

A circuit breaker stops you calling something that is clearly broken; a bulkhead stops one broken thing consuming all your capacity.

Circuit breaker

Three states. Closed — normal, calls pass through while failures are counted. Open — fail immediately without calling, for a cooldown period. Half-open — let a trial request or two through; if they succeed, close; if not, open again.

States
        failure rate > threshold
CLOSED ------------------------> OPEN
   ^                              |  after cooldown
   |  trial succeeds              v
   +---------- HALF-OPEN <--------+
                  |  trial fails
                  +------------> OPEN

It buys two things: your threads are not held waiting on a dead dependency, and the failing service gets breathing room to recover instead of being hammered while it restarts.

Trip it on an error rate or slow-call rate over a rolling window with a minimum call volume — not on a single failure, and not on a raw count. Slow calls matter as much as errors: a dependency answering in ten seconds is failing even though it returns 200.

Bulkheads

Named after ship compartments: isolate resources so one flooded compartment does not sink the vessel. In practice that means a separate connection pool or semaphore per dependency, so a slow payment provider cannot consume every thread and take down your health check.

Isolation
without bulkheads:  one shared pool of 200 threads
  payment provider slows to 10s -> all 200 threads end up waiting on it
  -> your health endpoint, your login, everything stops responding

with bulkheads:
  payments        max 20 concurrent
  recommendations max 10
  avatars         max  5
  -> the payment provider degrades PAYMENTS only

Graceful degradation

The product half: for each dependency, decide in advance what happens when it is unavailable. Recommendations → show a popularity-based list. Avatar service → show initials. Payment provider → fail the request honestly.

Principle

A fallback must never fabricate data that looks authoritative. Degrading a balance to a cached value without saying so is worse than an error — the user acts on information they believe is current. Degrade to something visibly approximate, or fail.

Interview angle

Where not to put a circuit breaker: on your own primary database. If the database is down, the correct response is usually to fail the request, not to serve a fallback that hides data loss. Breakers belong on dependencies that can fail independently and for which a degraded answer is genuinely acceptable.

Chapter 42

Backpressure and load shedding

Without backpressure, overload becomes invisible memory growth and then a total collapse — bounded queues convert that into a visible, survivable error.

What backpressure is

The ability of a slow consumer to make a fast producer slow down. Without it, the producer keeps accepting work the system cannot complete and the excess accumulates somewhere — a queue, a thread pool, a heap — until something breaks.

The failure mode, precisely

This is worth being able to narrate, because the shape is counter-intuitive and it is why people are surprised by outages.

Anatomy
1. load rises above capacity
2. the queue absorbs it -> latency creeps up, but NO ERRORS
   -> dashboards look fine. Nobody is paged.
3. the queue keeps growing (memory, or broker lag)
4. latency exceeds EVERY upstream timeout AT ONCE
5. every caller times out and RETRIES
6. the retry storm triples load on a system already failing
7. total collapse

queueing converts an ERROR into LATENCY, which is why the cliff is invisible
until you are over it

The mechanisms, per boundary

Boundaries
client -> service     concurrency limit; 429/503 + Retry-After; load shedding
service -> service    bounded connection pool + semaphore + timeout
service -> queue      BOUNDED queue + rejection policy
                      (CallerRuns is elegant: the producer does the work itself,
                       so it slows down automatically)
queue -> consumer     PULL-based (Kafka) -> the imbalance shows as LAG,
                      which is visible, rather than as memory, which is not

Load shedding

Backpressure's more aggressive sibling: when you are overloaded, deliberately reject some work so the rest succeeds. The key refinement is to shed by priority — drop low-value traffic (bulk exports, analytics, retries of retries) before user-facing requests, and drop before doing expensive work rather than after.

Note the difference from rate limiting: rate limiting is a policy about fairness (this client may send N per minute); load shedding is a reaction to your own health (I am overloaded, so I am dropping the least important work regardless of who sent it).

Principle

Bound every queue, pool and buffer. An unbounded queue converts a visible error into invisible memory growth, and a partial failure into a total one. A fast, explicit rejection is more useful than a slow success nobody is waiting for — and it lets the caller's circuit breaker do its job.

Interview angle

Kafka is a good illustration of doing this well: the producer is not blocked, but the imbalance is recorded durably and visibly as consumer lag rather than consuming memory. That is why lag is the metric to alert on — it is the backpressure signal made observable.

Chapter 43

Observability

Three signals with three different jobs — and the mistake is making one do another's work.

The three signals

  • Metrics — cheap aggregated numbers over time. Good for dashboards and alerts; useless for asking "what happened to this request".
  • Logs — detailed per-event records. Good for forensics, expensive at volume, and near-useless without structure.
  • Traces — one request's spans tied together across services. The only signal that answers "which hop was slow".

You need all three. The common failure is trying to make metrics answer per-request questions (which is what causes cardinality explosions) or making logs answer aggregate questions (which is slow and expensive).

What to measure

Frameworks
RED - for request-driven services (the caller's view)
  Rate      requests per second
  Errors    error rate
  Duration  latency distribution (p50/p95/p99)

USE - for resources (CPU, pools, queues, disks)
  Utilisation  how busy
  Saturation   how much queued/waiting   <- the LEADING indicator, usually omitted
  Errors

Saturation is the one people leave out and the one that predicts trouble. A connection pool at 100% utilisation with a growing wait queue tells you about the latency spike before it happens; CPU alone does not.

Structured logging and correlation

Log JSON with consistent field names so you can query rather than regex. And put a correlation/trace id in every line, propagated through every hop — including across queues, where it must travel in message headers or the trace breaks at every asynchronous boundary. The most common investigative query is "show me everything for this request", and it only works if the id was propagated.

Cardinality is what kills metric systems

Every unique label combination is a separate time series with its own index entry and memory. Putting a user id, request id or raw URL in a label multiplies series into the millions and takes the metrics backend down — usually caused by one well-meaning deploy.

Labels
good:  {endpoint="/orders/{id}", status="500", region="ap-south-1"}   bounded
fatal: {user_id="...", request_id="...", url="/orders/12345"}        unbounded

high-cardinality detail belongs in TRACES and LOGS, which are built for it
Principle

Alert on symptoms, not causes. High CPU wakes someone for a condition users may not notice. "Error rate exceeds the SLO burn threshold" wakes them only when it matters. Cause-based alerts multiply as the system grows and train people to ignore the pager.

Interview angle

The cheapest useful observability to add first, if you have none: structured logs with a propagated correlation id, plus RED metrics per endpoint. That combination answers most incidents. Full distributed tracing is more valuable but more work, so it usually comes second — and alert on saturation as soon as you have it.

Chapter 44

Alerting and on-call

Alert on what users experience, at a burn rate that justifies waking someone — every other alert trains people to ignore the pager.

Symptom, not cause

A cause-based alert ("CPU above 80%", "disk 70% full") fires for conditions users may not notice, and the number of possible causes grows without bound as the system grows. A symptom-based alert ("error rate is breaching the SLO", "checkout latency p99 above target") fires only when something that matters is actually wrong.

The practical test: if this fires at 3am and everything the user does still works, it should not have been a page. It might still be a ticket.

Burn-rate alerting

The refinement that makes SLOs operational. Rather than alerting when the budget is exhausted (too late) or on every error (too noisy), alert on how fast you are consuming the error budget.

Burn rate
budget for 99.9% over 30 days = 43 minutes

burn rate 1x   = you will exactly exhaust the budget in 30 days
burn rate 14.4x = you will exhaust it in ~2 days     -> page
burn rate 6x    = exhausted in ~5 days               -> page (slower window)
burn rate 1x    = on track                           -> ticket, not a page

MULTI-WINDOW: require both a short window (fast outages) and a long window
(slow bleeds) to fire, so a 2-minute blip does not page anyone

The multi-window, multi-burn-rate approach is what catches both a total outage in five minutes and a slow degradation over two days, without paging for transient noise.

What makes a good alert

Principle
  • Actionable — there is something the person woken can actually do. If the answer is "wait", it is not a page.
  • Attributable — it points at a service and ideally a likely cause, rather than "something is wrong somewhere".
  • Documented — it links to a runbook saying what to check first. An alert with no runbook is a puzzle handed to someone half asleep.
  • Rare — if a page fires several times a week and is usually nothing, people stop reading them. Alert fatigue is a real operational failure and it is caused by well-meaning over-alerting.

On-call as a design constraint

Worth saying in an interview because it signals experience: the design determines the on-call burden. A system with unbounded queues, no circuit breakers and cause-based alerting generates pages; one with backpressure, graceful degradation and symptom-based alerting mostly does not. If a service pages every week, the fix is usually architectural rather than a better runbook.

Interview angle

A strong question to ask an interviewer, and a good one to be able to answer: how many pages per week does this team get, and who fixes what they find? A team that pages nightly and files tickets nobody picks up has an unaddressed reliability debt — and the answer tells you a great deal about what working there is like.

Chapter 45

Deploys and zero-downtime migrations

Code rolls back; data does not — which is why every schema change must be expand-contract.

Deployment strategies

  • Rolling — replace instances gradually. The default and cheapest, but old and new run simultaneously, so both versions must be compatible with each other and with the schema.
  • Blue/green — run two full environments and switch traffic at once. Instant rollback, double the infrastructure, and the database is still shared.
  • Canary — send a small percentage to the new version and watch error rate and latency before proceeding. The safest for risky changes and the one to name — with automated rollback on a metric threshold, because a canary a human must watch will eventually ship unattended at 2am.

Feature flags decouple deploy from release

Code ships dark, is enabled for internal users, then a percentage, then everyone — and disabling is instant with no deploy. That turns a rollback from a fifteen-minute pipeline into a config change, which materially changes how much risk you can take.

The discipline: every flag needs an owner and an expiry, or you accumulate hundreds and the combinatorial state space becomes untestable.

Expand-contract, the only safe schema migration

The reason this pattern exists: during a rolling deploy, the old application version will run against the new schema. So every step must be independently deployable and backward compatible.

Expand-contract
release 1: ADD COLUMN new_email TEXT NULL          -- expand (additive, safe)
release 2: write BOTH old_email and new_email      -- dual write
           backfill in batches: throttled, resumable, idempotent
release 3: read from new_email, verify parity       -- switch reads
release 4: stop writing old_email
release 5: DROP COLUMN old_email                    -- contract

NEVER rename in one step. A rename is a delete plus a create from the old
code's point of view, and it fails instantly.

Migrations that lock

The operations that cause outages on large tables, and their safe forms:

Danger
ADD COLUMN with a DEFAULT     -> rewrote the whole table on older Postgres
                                 (modern versions handle it; know your version)
CREATE INDEX                  -> takes a write lock
                                 -> use CREATE INDEX CONCURRENTLY
ADD NOT NULL                  -> full table scan to validate
                                 -> add the constraint NOT VALID, then VALIDATE
long UPDATE over 10M rows     -> one enormous lock and no way to interrupt
                                 -> batch it, in application code or a job
Principle

Migrations are forward-only and additive within a release. Never edit an applied migration — add a new one. And keep long backfills out of the migration itself, so a deploy is not blocked behind an hour of data movement.

Interview angle

How do you verify a canary automatically? Compare error rate, latency percentiles and one or two business metrics between the canary and the baseline over a fixed window, and roll back if any breach a threshold. Compare against the baseline rather than an absolute number, so normal traffic variation does not trigger false rollbacks.

Chapter 46

Capacity planning and autoscaling

Size for peak plus the loss of one zone, target 50–70% utilisation, and scale on a leading indicator rather than CPU.

Planning from the SLO

Work out peak QPS (average × peak multiplier, plus known events), the per-request cost in CPU and memory, and how many instances that needs with headroom. Headroom is the part people skip: you size for peak plus the loss of one instance or availability zone, plus the queueing cliff from Chapter 4.

Find the per-request cost by load-testing a single instance to saturation and dividing — against realistic data volumes, because a query that is fast on 10k rows and slow on 10M invalidates the whole model.

Why CPU is usually the wrong autoscaling metric

  • For an IO-bound service, threads are blocked waiting rather than computing, so CPU stays low while latency climbs and the queue grows — you scale far too late, or never.
  • CPU lags: by the time the average crosses a threshold, requests have already been queueing.
  • In containers, cgroup throttling makes CPU readings misleading.
Signals
scale on:                    because
  in-flight concurrency        leading indicator; works for IO-bound work
  queue depth / consumer lag   directly measures "falling behind"
  p99 latency vs the SLO       measures what users actually experience
  CPU                          ONLY when the work is genuinely CPU-bound

Scale-up lag is often longer than the spike

Image pull, container start, JVM warmup, cache fill, connection pool establishment — that can be one to three minutes, which is frequently longer than the traffic spike you are scaling for. So reactive autoscaling alone arrives after the incident.

Combine it with scheduled or predictive scaling for known patterns (market open, a marketing send, month-end), keep a warm buffer, and make scale-down slow and hysteretic so the system does not flap — removing capacity, seeing latency rise, and adding cold instances back.

Principle

Autoscaling the stateless tier just moves the bottleneck. Fifty new application instances each open connections and can exhaust the database's connection limit or overload the primary. Connection pool sizing and database capacity must be part of the plan, or scaling makes the outage arrive faster.

Interview angle

Target 50–70% utilisation for latency-sensitive services. Batch systems can safely run much hotter, because queueing delay does not hurt them. If someone proposes running at 90% for efficiency, the counter-argument is Chapter 4's cliff: at 90% you are already waiting roughly ten times the service time, and there is no room for a deploy, a GC pause, or losing one instance.

Chapter 47

Incidents and postmortems

Mitigate before diagnosing, and write blameless postmortems that produce a small number of tracked actions.

During an incident

Principle
  • Mitigate first, diagnose second. Roll back, disable the feature flag, shed load, fail over. Understanding why can wait; stopping the bleeding cannot. This is the instinct that most distinguishes experienced responders.
  • One incident commander. Someone coordinates and communicates; everyone else investigates. Without this, five people debug the same thing and nobody talks to stakeholders.
  • Communicate on a schedule, even when there is nothing new — silence is read as chaos, both internally and by customers.
  • Keep a timeline as you go. Reconstructing it afterwards from memory and Slack is unreliable, and the timeline is most of the postmortem.

The blameless postmortem

"Blameless" does not mean nobody is accountable — it means the analysis looks for why the system allowed the mistake rather than who made it. The reason is practical rather than sentimental: if people are blamed, they stop reporting near-misses, and you lose the cheapest source of information you have.

Structure
a postmortem contains:
  TIMELINE       what happened, when, in UTC, including what people BELIEVED
  IMPACT         who was affected, how many, for how long, and the business cost
  ROOT CAUSE     usually several contributing causes, not one
  DETECTION      how did we find out? how LONG did that take? (often the biggest gap)
  ACTIONS        a SMALL number of concrete, owned, dated items

not: "engineer X deployed a bad config"
but: "a config change with no validation reached production, and the canary
      did not cover this code path"

The two metrics worth tracking

Time to detect and time to mitigate. Teams obsess over root cause and under-invest in detection, but an incident found by a customer forty minutes in has already done most of its damage. If time to detect is consistently long, the fix is monitoring, not more careful engineering.

Action items

The most common postmortem failure is producing twenty action items and completing three. Produce three to five, each with a named owner and a date, tracked in the normal backlog. Prefer actions that remove a class of failure (add validation, add a canary check) over ones that fix this instance ("be more careful").

Interview angle

The strongest thing to say about incidents in an interview is that the error budget connects them to planning: if incidents are consuming the budget, reliability work is prioritised automatically rather than argued for. That closes the loop between Chapter 5's arithmetic and the daily reality of running a system, and it is what makes reliability a shared engineering decision rather than an ops complaint.

VI. Architecture
Chapter 48

Monolith, modular monolith, microservices

Microservices solve an organisational problem at a technical cost — start with a modular monolith and extract when a specific force demands it.

What each actually is

  • Monolith — one deployable, one database, one codebase. Simple to develop, test, deploy and debug. Scales further than people assume. Its real problems appear at organisational scale: merge conflicts, coupled release trains, and a codebase where nobody knows the boundaries.
  • Modular monolith — one deployable, but with enforced internal boundaries: modules own their data, communicate through published interfaces, and cannot reach into each other's internals. You get most of the design discipline of services with none of the network.
  • Microservices — many independently deployable services, each owning its data. Independent scaling and deployment, technology freedom, and a fault boundary — at the cost of network calls, distributed transactions, distributed debugging, and a great deal of operational machinery.

The honest trade

Microservices are an organisational solution. Their central benefit is that teams can deploy independently. If you do not have multiple teams whose release cadences genuinely conflict, you are paying the cost without collecting the benefit.

The cost
what you PAY for microservices, always:
  network calls where function calls used to be (latency + failure)
  distributed transactions -> sagas, compensations, intermediate states
  distributed debugging    -> you now NEED tracing to answer basic questions
  availability multiplies  -> three services at 99.9% = 99.7%
  operational surface      -> CI/CD, monitoring, on-call, per service
  data duplication and eventual consistency between services

When extraction is genuinely justified

Principle
  • A module has a materially different scaling profile — it needs 50 instances while everything else needs 3.
  • A module has a different deploy cadence — it changes daily while the rest changes monthly.
  • A separate team owns it and coordination is the bottleneck.
  • A different compliance boundary applies — PCI, or data residency.
  • A genuinely different technology is required — an ML serving path in Python.

"Microservices are best practice" is not on that list, and saying so is a stronger answer than proposing a service mesh.

The migration path

The strangler pattern: put a facade in front of the monolith, extract one bounded context at a time behind it, migrate its data, and cut traffic over incrementally. Never a rewrite — the monolith's behaviour is not fully documented anywhere except in the monolith.

Interview angle

Extraction from a well-modularised monolith is straightforward, because the seams already exist. Recombining prematurely split services is not. That asymmetry is the whole argument for starting modular and extracting later: you can always split a boundary you drew in code, but merging two deployed services with separate databases is a project.

Chapter 49

Service boundaries and bounded contexts

A boundary should own its data, match a business capability, and let a typical feature ship from one place.

Bounded contexts

The core idea from domain-driven design: a bounded context is a region where a term has one unambiguous meaning. "Customer" means something different to Billing (a payment method and an address) than to Support (a ticket history) than to Marketing (a segment).

Forcing one shared Customer model across all three produces a bloated entity that every team must coordinate on — the classic distributed monolith. Each context should own its own model and integrate through explicit contracts. Duplicating the fields you need is correct here; it is not denormalisation, it is each context holding its own view.

Tests for a good boundary

Checklist
[ ] owns its data EXCLUSIVELY - nobody else reads its tables
[ ] cohesive around a business CAPABILITY, not a technical layer
[ ] a typical feature ships from ONE service
[ ] can be deployed without coordinating with another team
[ ] its interface is stable enough to publish

Boundaries drawn along technical layers — a "database service", a "validation service" — fail the second and third tests immediately, because every feature needs a change in all of them.

Three signs the split was wrong

Principle
  • Most features require coordinated changes across two or three services. The boundary was drawn through the middle of a capability.
  • Routine operations need a saga or two-phase commit. If an ordinary business action spans services, the invariant it protects lives in one place and the split cut through it.
  • Chatty synchronous chains — A calls B calls C on every request. That multiplies latency, couples availability, and means you have distributed a function call.

A fourth worth naming: a shared database. If two services write the same tables, they are one service with two deployables and all the costs of both.

How contexts share data

Through explicit contracts: synchronous APIs for queries you need immediately, or events for propagating state changes so each context maintains its own read model. The second is usually better — it removes a runtime dependency and lets each context keep exactly the shape it needs.

Interview angle

Conway's Law is worth naming: system architecture tends to mirror the communication structure of the organisation that builds it. That cuts both ways — if your team structure does not match your intended boundaries, the boundaries will erode; and deliberately organising teams around the architecture you want (the "inverse Conway manoeuvre") is a real technique.

Chapter 50

Event-driven architecture, CQRS and event sourcing

Three separable ideas that get conflated — you can adopt any one without the others, and usually should.

Event-driven

Services communicate by publishing and consuming events rather than calling each other. The benefits are real: the producer does not need the consumer to be available, new consumers can be added without changing the producer, and load is naturally buffered.

The distinction worth knowing: event notification ("order 123 changed", forcing the consumer to call back for details — chatty but loosely coupled) versus event-carried state transfer (the event carries the data the consumer needs — no callback, at the cost of a larger contract and duplicated data).

CQRS

Command Query Responsibility Segregation: separate the write model from the read model. Writes go through commands into a normalised model that enforces invariants; reads come from one or more projections shaped for specific queries.

The motivation is that the shape which protects invariants and the shape which answers queries fast are usually different. CQRS does not require event sourcing — a read replica with denormalised views is CQRS, and that is often the right dose.

Event sourcing

The events are the source of truth, and current state is a fold over them. You never store "balance = 500"; you store every credit and debit and derive the balance.

For an audit-critical system that is powerful: you get complete history for free, you can answer "what did we believe on this date", and you can build a new projection retroactively over data you already have — because you kept the facts rather than just the conclusions.

Event sourcing
command -> validate against the aggregate -> append EVENT (immutable, forever)
        -> projections fold events into read models

current state = fold(events)      balance is DERIVED, never stored

The three costs, which is where the honest answer lives

Principle
  • Eventual consistency between write and read models. A user submits a form and immediately queries — the projection has not caught up, so their own change is missing. Design for it: return the command result directly, apply it optimistically, or expose the projection's version so the client can wait.
  • Schema evolution of stored events, forever. Events are immutable and kept indefinitely, so version one of your event will still be in the store in five years. You cannot migrate them like a table; you need upcasters that transform old versions on read, and every one is permanent code. This is the cost people most underestimate.
  • Replay time grows without bound. Rebuilding a projection means folding millions of events. The mitigation is snapshots — periodically persist aggregate state at a version and replay only from there — which is then another artefact to version and invalidate.
Interview angle

Use event sourcing where the history is the product — ledgers, audit, regulatory systems, anything where "how did we get here" is a question people ask. Use plain CQRS for most read-scaling problems. Adopting event sourcing because it sounds sophisticated is one of the most expensive architectural mistakes available, because the costs are permanent and arrive later.

Chapter 51

API design

Design the contract as something you will have to keep for years — additive change is free, breaking change is not.

Versioning

URI versioning (/v1/orders) is the pragmatic default: visible, cacheable, trivially routable, easy to read in logs. Header or media-type versioning is purer and harder to debug. The more important discipline is to avoid needing a new version — additive, backward-compatible changes (new optional fields, new endpoints) cost nothing, and consumers must ignore unknown fields.

Pagination, errors, idempotency

Essentials
PAGINATION  cursor-based by default (Chapter 22), limit CAPPED server-side
            an uncapped page size is a denial-of-service vector

ERRORS      RFC 9457 application/problem+json:
            type, title, status, detail, instance
            + a STABLE machine-readable `code` and a list of field errors
            + a traceId
            return ALL validation failures at once, never just the first
            NEVER leak stack traces, SQL, or internal hostnames

IDEMPOTENCY GET/PUT/DELETE are idempotent by definition.
            POST is not -> any POST that creates or moves money takes an
            Idempotency-Key header (Chapter 34)

REST, gRPC or GraphQL

  • REST — the default for public and cross-team APIs. Universally understood, cacheable, debuggable with curl.
  • gRPC — for internal service-to-service calls: a schema, code generation, streaming, and lower overhead. Poor browser story, harder to debug by hand.
  • GraphQL — when many diverse clients need different shapes of the same graph and over-fetching is a real problem. The costs are caching complexity, query-cost control (a malicious query can be very expensive), and the N+1 problem moving into your resolvers.

HATEOAS, honestly

The theory is that responses carry links driving discoverable state transitions. In practice almost no client is written to follow them, so it adds payload and complexity for little benefit. Include links where they genuinely help — pagination, the next action in a workflow — and skip the rest. Being able to say that with a reason is better than reciting the Richardson maturity model.

Principle

Design for the consumer you cannot redeploy. Internal APIs can change with a coordinated release; public and partner APIs cannot. That single distinction determines how strictly you version, how long you support old versions, and how much you invest in the contract up front.

Interview angle

Deprecation is a process, not an announcement: publish it, add Deprecation and Sunset headers, monitor which clients still call it, contact them, and only then remove. Removing on a schedule without measuring usage is how you break a customer you did not know existed.

Chapter 52

Authentication and authorization

Sessions are revocable and need a lookup; JWTs scale statelessly and cannot be revoked — the standard resolution is short access tokens plus a revocable refresh token.

The core trade

Sessions vs JWT
SERVER-SIDE SESSIONS
  token is an opaque id; all state lives server-side
  + revocation is INSTANT; permission changes apply immediately
  - a lookup on every request; needs a highly available shared store

JWT (self-contained)
  signed claims, verifiable with no lookup
  + scales statelessly across services
  - NOT revocable before expiry; permission changes are invisible until refresh

The hybrid nearly everyone lands on: short-lived access JWTs (5–15 minutes) plus a long-lived, revocable refresh token stored server-side with rotation and reuse detection. Stateless verification on the hot path, and a genuine revocation point.

OAuth2 and OIDC

Authorization code flow with PKCE is the standard for web and mobile. The client redirects to the authorization server with a code_challenge; the user authenticates there; the server redirects back with a short-lived code; the client exchanges that code plus the original verifier for tokens. PKCE exists so an intercepted code is useless without the verifier — which is why the implicit flow is deprecated.

OIDC layers identity on top: an id_token about the user alongside the access_token about permissions. That is what enables SSO — one identity provider, many products, one place for MFA and password policy.

Validating a token properly

JWT validation
verify the SIGNATURE against the JWKS (select the key by `kid`)
check iss, aud, exp, nbf
REJECT alg: none, and PIN the expected algorithm
  -> accepting whatever the header claims is the classic algorithm-confusion
     vulnerability (signing with HMAC using the public RSA key as the secret)

RBAC versus ABAC

Roles are simple and auditable but explode combinatorially — once you need "admin, but only for their own region, and only during business hours", you are inventing a role per combination, and past roughly fifty roles it is unmaintainable.

ABAC evaluates a policy over attributes of the subject, resource, action and environment, which expresses those rules directly. The usual architecture is a policy engine (OPA, Cedar) with policies as versioned, testable code, evaluated locally in each service from a distributed bundle — so authorization is not a network hop.

Principle

Put coarse claims in the token (roles, tenant) and evaluate fine-grained decisions at request time against current policy. Embedding detailed permissions makes them stale for the token's lifetime, which is exactly the wrong property for a permission you just revoked.

Interview angle

Be ready for the blast-radius question: if a signing key leaks, an attacker can mint tokens for any user until it is rotated and every issued token expires. That is the argument for short access-token lifetimes, key rotation via JWKS with overlapping keys, and keeping keys in a KMS or HSM rather than in configuration.

Chapter 53

Multi-tenancy

Three isolation models on a cost/isolation spectrum — and tenant scoping must be structural, never by convention.

The three models

  • Silo — a separate database, often separate infrastructure, per tenant. Strongest isolation, easiest compliance story, per-tenant backup and restore, no noisy-neighbour risk. Operationally expensive: migrations across a thousand databases, poor resource utilisation, slow onboarding.
  • Bridge — a schema per tenant in a shared database. Better isolation than pooling, cheaper than silo; schema count becomes a migration problem in the thousands.
  • Pool — all tenants share tables with a tenant_id column. Cheapest and most efficient, trivial onboarding, one migration — and one query missing a tenant filter is a cross-tenant data leak.

Most platforms run a hybrid: pool for the long tail of small tenants, silo for large or regulated customers who pay for it. Design the data-access layer so a tenant can be migrated between models without application changes.

Scoping must be structural

"Remember to add WHERE tenant_id = ?" fails eventually — one forgotten filter in one query is a breach. Enforce it so that an unscoped query is impossible.

RLS
-- Postgres row-level security: the database enforces it, not the developer
ALTER TABLE orders ENABLE ROW LEVEL SECURITY;
CREATE POLICY tenant_isolation ON orders
  USING (tenant_id = current_setting('app.tenant_id')::uuid);

-- set per request/connection, from the VERIFIED token - never from a parameter
SET LOCAL app.tenant_id = '...';

The tenant id must come from the authenticated token, resolved server-side. Taking it from a header, path or request body lets a client set it — which is the most direct possible cross-tenant data access, and it has shipped in real products.

Noisy neighbours

Limits
gateway   per-tenant rate limits
service   per-tenant concurrency cap (semaphore or pool partition)
database  statement_timeout + per-tenant query cost budget
workers   separate queues for large tenants (bulkhead applied to customers)
Principle

Track per-tenant usage and cost from day one. Without it you cannot tell who is expensive, cannot price accurately, and cannot make the case to move a tenant to dedicated infrastructure. Retrofitting per-tenant attribution onto a pooled system is painful.

Interview angle

The hardest operational problem in the pool model is per-tenant restore and deletion. Restoring one tenant to a point in time from a shared table means extracting and replaying their rows; a contractual deletion request means proving their data is gone from backups too. Customers will ask about both, so have an answer.

VII. The Design Round
Chapter 54

The 45-minute framework

Five phases with minute markers — the structure matters more than any individual piece of knowledge, because it is what lets you drive.

The most common failure in a design round is not ignorance. It is spending twenty-five minutes on requirements and never reaching a design, or drawing boxes at minute two and having nothing to say at minute thirty. A time-box fixes both.

Time-box
 0-5   REQUIREMENTS  functional + non-functional; state what is OUT of scope
                       ask the read:write ratio
 5-10  ESTIMATES      DAU -> QPS -> peak -> storage -> bandwidth, 1 sig fig
                      WRITE THE NUMBERS WHERE THEY CAN SEE THEM
10-20  HIGH LEVEL     components + data flow, coarse
                      then ASK: "where would you like me to go deep?"
20-40  DEEP DIVE      the data model with real fields, the partition key and WHY,
                      indexes, the API contract, failure behaviour
40-45  TRADEOFFS      top 3 risks + what you would build first + what you'd validate

Phase 1 — requirements

Functional: what must it do, for whom. Non-functional: scale, latency target, consistency needs, availability target. Then explicitly scope things out — "I will not cover analytics or the admin interface unless you want me to." Scoping out is a positive signal; it shows you know the problem is bigger than the time.

Phase 2 — estimates

Four divisions (Chapter 3). The point is not precision, it is establishing scale. Put the numbers on the board — if the interviewer has to ask "how much data is that?", you left points behind.

Phase 3 — high level, then ask

Draw the components and the data flow, coarsely. Then say which part you find most interesting and ask where they want to go deep. That question is itself scored, and it stops you spending twenty minutes on the component they do not care about.

Phase 4 — the deep dive, where the round is won

Be concrete enough to be wrong. Actual field names. An actual partition key with a justification. An actual index. An actual failure behaviour. If everything you said would be true of any system, you have not gone deep — and vague breadth loses to concrete depth every time.

Phase 5 — the highest-leverage 90 seconds

Close with your top three risks and what you would build first. Almost no candidate volunteers this, and it is the clearest available demonstration of engineering judgement rather than recall.

Principle

Drive the conversation. Count, afterwards, how many times the interviewer had to prompt you with "what about…". Each one is a point you left on the table, because the round is testing whether you can lead a design discussion — not whether you can answer questions about one.

Interview angle

If you go blank at the start, begin with requirements out loud — it is the one phase that needs no knowledge, only questions. Asking three clarifying questions gets you talking and buys the minute you need to settle.

Chapter 55

A design worked end to end

The whole framework applied to one prompt — a URL shortener — so you can see the shape of a complete answer.

Read this once for the rhythm rather than the content. The value is seeing how the phases connect: how the estimates determine the architecture, and how the deep dive follows from the numbers.

0–5 · Requirements

Requirements
FUNCTIONAL      shorten a long URL -> short code; redirect; optional custom alias;
                optional expiry; click analytics
NON-FUNCTIONAL  extremely read-heavy (~100:1); redirects must be fast (tens of ms);
                high availability - a dead shortener breaks every link ever shared;
                links are immutable once created
OUT OF SCOPE    user accounts, spam detection, the admin UI  (say this out loud)

5–10 · Estimates

Estimates
100M new URLs/year  -> 100M / 3.15e7 s  = ~3 writes/second
100:1 read ratio    -> ~300 reads/second average, ~1,000 at peak
storage: 100M rows x ~500 bytes = 50 GB/year   (small!)
key space: base62, 7 chars = 62^7 = 3.5 trillion   (ample)

=> THE NUMBERS SAY: this is a CACHING and AVAILABILITY problem,
   not a storage or write-throughput problem.

10–20 · High level

Architecture
POST /shorten                     GET /aX9k2Bq
     |                                 |
     v                                 v
  API service                     CDN / edge cache (optional)
     |                                 |
     v                                 v
  key generator                   API service
     |                                 |
     v                                 v
  Postgres (code -> long URL)     Redis  --miss-->  Postgres
                                       |
                                       v
                                 302 + async click event -> Kafka -> analytics

Then: "the interesting decisions here are key generation and the redirect path — where would you like to go deep?"

20–40 · Deep dive: key generation

Compare the options rather than asserting one, because the comparison is the content.

Key generation
hash + truncate (MD5 -> 7 chars)   collisions -> detect and retry loop
random                              same problem
base62 of a counter                 collision-free by construction, shortest keys
                                    BUT a single counter is a bottleneck, and
                                    sequential codes are ENUMERABLE
distributed id (Snowflake) + scramble  no coordination on the hot path
PRE-GENERATED KEY POOL              <- my choice
  a background job fills a table of unused keys
  creation = one atomic claim: no collisions, no coordination, non-sequential

20–40 · Deep dive: the read path

Read path
GET /aX9k2Bq
  -> Redis GET short:aX9k2Bq
       HIT  -> 302 + async click event
       MISS -> Postgres lookup -> populate Redis -> 302
  -> not found -> 404 (cache the negative briefly: stops scan abuse)

why this caches almost perfectly: entries are IMMUTABLE after creation,
so there is no invalidation problem at all - just a long TTL.

301 vs 302:  301 is permanent and browsers cache it -> fastest, but you STOP
             seeing the clicks.  302 -> every click reaches you.
             CHOOSE 302 because analytics is a requirement.  <- name the trade

Schema: links(code PK, long_url, created_at, expires_at, owner_id). Custom aliases rely on a unique constraint and handling the violation — never a read-then-write check, which races.

40–45 · Tradeoffs and risks

Principle
  • Risk 1 — Redis outage. Every redirect falls through to Postgres. At ~1,000 QPS that is survivable with an index and connection pooling, which is worth stating: the cache is an optimisation, not a dependency. I would add a small in-process L1 so a blip is invisible.
  • Risk 2 — enumeration and abuse. Non-sequential keys from the pool, plus rate limiting on 404s, since a scanner produces a distinctive burst of misses.
  • Risk 3 — links are forever. A 301 already cached in a user's browser cannot be recalled, so deletion is best-effort for those clients. Another argument for 302.
  • What I would build first: the redirect path with the cache, because it is 99% of traffic and everything else can be added behind it.
Interview angle

Notice what made this a complete answer: the estimates determined the architecture ("the numbers say this is a caching problem"), the deep dive compared alternatives rather than asserting one, one decision (301 vs 302) was explicitly a trade, and it closed with risks unprompted. That shape transfers to any prompt.

Chapter 56

The recurring patterns

A dozen solutions that appear across almost every design — recognise them and most prompts become assembly rather than invention.

Once you have seen enough systems, you notice the same answers recurring. Knowing the catalogue is what lets you move quickly through the familiar parts of a design and spend your time on what is actually novel.

Principle
  • Cache-aside + TTL + jitter — the default read optimisation. Reach for it whenever reads dominate and bounded staleness is acceptable.
  • Read replicas — scale reads without touching the write path; accept replication lag and handle read-your-writes.
  • Shard by the entity that owns the access pattern — so the common query stays single-shard.
  • Idempotency key + unique constraint — makes any operation safe to retry. The single most reusable pattern in this book.
  • Transactional outbox — whenever you must change state and publish an event atomically.
  • Saga with an explicit state machine — for a multi-step operation across services, with compensations and a pending state modelled in the domain.
  • Bounded queue + rejection — converts overload into a visible error instead of invisible memory growth.
  • Circuit breaker + bulkhead + timeout budget — the standard resilience trio around any external dependency.
  • Lease with expiry + fencing token — for "only one worker should do this", where correctness matters.
  • Precompute on write vs compute on read — the fan-out decision. Push for normal cases, pull for the extreme ones, hybrid for both (Chapter on feeds).
  • Two-tier storage: hot in a database, cold in object storage — with time-based partitions so expiry is a drop, not a delete.
  • Fast approximate path + slow exact path — a streaming aggregate for dashboards, a batch recomputation for billing, reconciled daily.

How to use the catalogue

When a prompt arrives, most of it will map onto these. "Design a payment system" is idempotency keys, an outbox, a saga, a ledger, and reconciliation. "Design a feed" is fan-out, caching, and pagination. "Design a rate limiter" is one algorithm plus atomicity.

That is not cheating — it is what experience is. The interviewer is not testing whether you can invent consistent hashing on the spot; they are testing whether you recognise which problem you have and can apply the right shape, then reason about the parts that are genuinely specific to this system.

Interview angle

The corollary is where to spend your remaining time: on the part of the prompt the catalogue does not cover. If a design is entirely standard patterns and you have twenty minutes left, ask what makes this system hard — the interviewer usually has something specific in mind, and asking gets you there faster than guessing.

Chapter 57

What each classic prompt is really testing

Every well-known prompt exists to probe one or two specific ideas — know which, and you know where to spend your twenty minutes.

The prompts
URL SHORTENER          key generation, caching, 301 vs 302 analytics trade
RATE LIMITER           algorithm choice + ATOMICITY across nodes + fail open/closed
NEWS FEED              fan-out on write vs read, and the CELEBRITY problem
CHAT                   connection registry, per-conversation ordering, offline sync
NOTIFICATIONS          fan-out, per-channel isolation, dedup, quiet hours
TYPEAHEAD              a DIFFERENT problem from search: prefix structures, latency
WEB CRAWLER            the frontier: politeness + priority, and dedup at scale
FILE STORAGE / DROPBOX chunking, dedup, delta sync, presigned URLs, conflicts
VIDEO PIPELINE         transcode fan-out, segments + manifest, adaptive bitrate
RIDE HAILING           geospatial indexing + the double-assignment race
TICKET BOOKING         CONTENTION: atomic claim, holds with TTL, no overselling
PAYMENTS / LEDGER      double-entry, idempotency, the timeout-but-charged case
AD CLICK AGGREGATION   fast approximate path vs exact batch path, watermarks
DISTRIBUTED CACHE      placement, replication, eviction, and COLD START
METRICS BACKEND        time-series compression, rollups, and CARDINALITY
SEARCH                 inverted index, analysis, ranking, near-real-time indexing
JOB SCHEDULER          bucketing, leases, at-least-once + idempotent handlers
MULTI-TENANT SAAS      isolation model + structural tenant scoping + noisy neighbours

How to use this

When you recognise the prompt, you know where the interviewer wants to go — so you can steer the deep dive there rather than discovering it at minute thirty-five. If you are asked to design a ticket booking system and spend twenty minutes on caching, you have spent the round on the part nobody cares about.

When you do not recognise it

Fall back on the method, which does not change. Requirements, estimates, and then ask what is stored, what is queried, what the read:write ratio is, what must be consistent and what can be eventual. Almost every unfamiliar system decomposes into ingestion, storage, indexing, a serving path and a background pipeline — say that out loud and the design assembles itself.

And say when you are outside your experience: "I have not built a transcoding pipeline, so I will reason from the constraints — the work is CPU-heavy and embarrassingly parallel, the output is immutable and segment-based, so I want a queue, a worker fleet and object storage." That is a strong answer. Silently pretending familiarity and getting a detail wrong is a weak one.

Principle

Prepare two prompts adjacent to each target company's domain. Interviewers frequently pick prompts near what they build, because that is what they can evaluate deeply — and being fluent in the vocabulary of their problem is a real advantage that costs an hour to acquire.

Interview angle

Notice how many of these reduce to a single hard idea: booking is contention, payments is the ambiguous timeout, feeds are fan-out, metrics is cardinality. If you can name the hard idea in the first two minutes, you have already shown the interviewer that you know what kind of problem you are looking at.

Chapter 58

The one-page cheat sheet

Everything worth memorising, on one page — write it out from memory and whatever you cannot recall is what to review.

Concepts you can derive under pressure; numbers and lists you cannot. This page holds the second kind. Reproduce it from memory rather than reading it — the retrieval attempt is the exercise.

Cheat sheet
LATENCY LADDER
  L1 ~1ns | RAM ~100ns | SSD random ~100us | same-DC RTT ~0.5ms | cross-cont ~150ms

ESTIMATION
  QPS = DAU x req_per_user / 86,400        peak = 2-3x
  storage = rows/day x size x retention x replication(3) x index(1.3)
  ALWAYS ask the read:write ratio first

AVAILABILITY
  serial multiplies: 0.999^3 = 99.7%
  99%=7.2h/mo | 99.9%=43m | 99.99%=4.3m | 99.999%=26s

QUEUEING
  L = lambda x W        wait multiplier ~ 1/(1-rho)
  rho 0.5->2x  0.8->5x  0.9->10x  0.95->20x   target 50-70%

CONSISTENCY (by the symptom of violation)
  linearizable  "both got the last seat"
  read-your-writes  "my own comment vanished"
  monotonic reads   "the count went backwards"
  causal            "the reply is above the message"
  eventual          fine for feeds and counts

ISOLATION LEVELS (by the anomaly prevented)
  RC: no dirty read | RR: no non-repeatable | SER: no phantom/write-skew
  NOT in the table: lost update, WRITE SKEW (snapshot isolation does NOT prevent)

DELIVERY
  at-most-once | at-least-once (the real default) | exactly-once DELIVERY = impossible
  achievable: at-least-once + IDEMPOTENT consumer = exactly-once EFFECT

CACHING
  cache-aside | read-through | write-through | write-behind | refresh-ahead
  failures: stampede | hot key | penetration | cache outage -> DB outage
  always: TTL + JITTER, single-flight, negative caching

RATE LIMITING
  fixed window (2x boundary burst) | sliding log (memory) |
  SLIDING COUNTER (default) | token bucket (bursts OK) | leaky bucket (smooth)

RESILIENCE
  timeout budget SHRINKS down the chain
  retry: idempotent only, exponential + JITTER, capped, ONE layer only
  circuit breaker: closed -> open -> half-open, on error/slow RATE
  bulkhead: a pool per dependency | shed by priority | bound every queue

PAGINATION
  offset: O(depth), unstable | keyset: O(1), stable, needs a unique tie-breaker

THE FRAMEWORK
  5 requirements | 5 estimates | 10 high level (then ASK) | 20 deep dive | 5 risks
  close with: top 3 risks + what you'd build first + what you'd validate
Principle

If it does not fit on one page, you have not compressed enough. The constraint is what forces you to separate what you must recall from what you can derive — and that separation is exactly what you need under pressure.

Interview angle

A design round is a conversation between two engineers about a problem, not an examination. Ask questions, state assumptions, quantify, name your trades, and say when you do not know something and reason from constraints anyway. Candidates who engage rather than perform do better — and if you have read this far, you have the material to engage with.

58 chapters · System Design Ground Up

Back to the ledger →

Made By Jatin Bhargava · 2026