The problem containers solve
Not 'it works on my machine' — that is the marketing. The real problem is that deploying software meant coupling an application to a specific machine's configuration, and containers break that coupling.
Before containers, deploying a Java service meant a machine with the right JDK version, the right system libraries, the right file layout, the right environment variables, and a configuration management tool trying to keep dozens of machines identical. Every one of those was a place where production could differ from staging.
A container packages the application with its filesystem — the JRE, the libraries, the layout — into a single immutable artefact. What you tested is bit-for-bit what runs, because it is the same artefact. That is the actual value, and everything else follows from it.
What you also get
- Density. No guest operating system, so hundreds of containers per host rather than tens of VMs.
- Speed. Milliseconds to start, which is what makes autoscaling and rolling deploys practical.
- A uniform interface. Every service starts the same way, logs the same way and is health-checked the same way, regardless of language — which is what makes an orchestrator possible at all.
A container is an immutable artefact plus runtime configuration. The moment you build a different image per environment, you have lost the property that made it worth doing - because the thing you tested is no longer the thing you shipped.
Asked why containers, the weak answer is dependency isolation. The strong answer is that the deployable artefact becomes immutable and identical across environments, and that a uniform lifecycle interface is what an orchestrator needs to exist.
Namespaces
A container is a normal Linux process. Namespaces restrict what it can see — and that restriction is most of what makes it feel like a separate machine.
There is no container object in the Linux kernel. There are processes, and there are namespaces that scope which instances of a global resource a process can observe. Docker creates a process with several new namespaces at once, and the result behaves like an isolated system.
NAMESPACE WHAT IT HIDES WHY IT MATTERS
pid other processes the app sees itself as PID 1
net interfaces, routes, ports two containers can both bind 8080
mnt the filesystem tree the IMAGE becomes the root filesystem
uts hostname, domain name each container has its own hostname
ipc shared memory, semaphores no accidental IPC between containers
user uid/gid mapping root INSIDE, unprivileged OUTSIDE
cgroup the cgroup hierarchy cannot see the host's limits
time clock offsets rarely used
$ docker run --rm -it alpine sh
/ # ps aux # only our own processes <- pid namespace
1 root sh
/ # hostname # not the host's <- uts namespace
3f2a91c4b0deThe user namespace is the one worth understanding for security: it maps uid 0 inside the container to an unprivileged uid on the host, so a process that thinks it is root has no host privileges. It is not enabled by default in every runtime, which is why runAsNonRoot remains an important control.
The key sentence for an interview: namespaces restrict what a process can SEE, cgroups restrict what it can CONSUME. Being able to separate those two cleanly signals you understand the mechanism rather than the abstraction.
cgroups
Control groups bound what a process can consume. Memory is incompressible so exceeding it kills you; CPU is compressible so exceeding it merely slows you — and that asymmetry explains almost every resource incident.
cgroups account for and limit CPU time, memory, block IO, PIDs and network. The container runtime creates a cgroup per container and writes the limits into it. Reading them directly is often the fastest way to answer what is this container actually allowed.
/ # cat /sys/fs/cgroup/memory.max # bytes
268435456
/ # cat /sys/fs/cgroup/cpu.max # quota period (microseconds)
50000 100000 # 0.5 CPU
MEMORY - incompressible
exceed the limit -> the kernel OOM-kills the process IMMEDIATELY
no warning, no signal you can handle, exit code 137
because the kernel cannot give you "less memory"
CPU - compressible
exceed the quota -> every thread is FROZEN until the next 100ms period
no error, no restart, nothing in the logs
symptom is LATENCY, visible only in
container_cpu_cfs_throttled_seconds_totalThat asymmetry is worth committing to memory because it determines how you set limits. For memory there is no benefit to bursting above your request and real risk in it, so setting request equal to limit is usually correct. For CPU, a tight limit buys predictability at the cost of tail latency — and for latency-sensitive services many teams now omit the CPU limit deliberately.
For a JVM, the container limit covers the WHOLE process: heap plus metaspace plus code cache plus thread stacks plus direct buffers. Overflow of the non-heap portion is what produces the confusing incident where a pod is OOM-killed while the heap graph shows plenty of headroom.
Images and layers
An image is a stack of content-addressed filesystem diffs plus configuration metadata. Layers are additive only, which is why deleting a file does not remove it.
Each build instruction that changes the filesystem produces a layer: a tarball of what changed, identified by the SHA-256 of its content. The image manifest lists those layer digests plus a config blob holding the entrypoint, environment, exposed ports and default user.
At runtime the layers are union-mounted into a single filesystem view with a thin writable layer on top. Writes go to that layer and are discarded when the container is removed — which is why a container filesystem is ephemeral and why volumes exist.
Additive only
Deleting a file in a later layer writes a whiteout marker. The bytes remain in the earlier layer and are still shipped, still pullable, still extractable by anyone with the image. This is the mechanism behind most credentials-in-image leaks, and it is why RUN wget secret && make && rm secret as three instructions ships the secret forever.
$ docker history --no-trunc myapp:1.4.2
IMAGE CREATED BY SIZE
sha256:.. COPY target/app.jar /app/app.jar 62MB
sha256:.. RUN apk add --no-cache curl 3.2MB
sha256:.. ENV JAVA_TOOL_OPTIONS=... 0B
<missing> /bin/sh -c #(nop) ADD file:... in / 7.6MB
# Layers are SHARED between images built on the same base.
# Standardising the base image across services cuts registry storage,
# network transfer, and cold-node pull time.Separate the vocabulary precisely: an IMAGE is immutable layers plus metadata; a CONTAINER is a running process with those layers mounted plus a writable layer. People use them interchangeably and it causes confusion about what persists.
The container lifecycle, and PID 1
Your process becomes PID 1, which has special kernel semantics. Get that wrong and your application ignores SIGTERM and is killed after the grace period, losing in-flight work on every deploy.
PID 1 is the init process. The kernel treats it specially: it does not get default signal handlers, so a signal with no explicit handler is simply discarded. A process that never installed a SIGTERM handler will therefore ignore it entirely, run until the grace period expires, and then be SIGKILLed.
PID 1 is also responsible for reaping orphaned child processes. A container that spawns children and does not reap them accumulates zombies, which in a long-running container can exhaust the PID limit.
# WRONG - shell form: /bin/sh is PID 1 and may not forward signals
ENTRYPOINT java -jar /app/app.jar
# RIGHT - exec form: the JVM is PID 1 and receives SIGTERM
ENTRYPOINT ["java","-jar","/app/app.jar"]
# Verify it, before you need it to work:
$ docker run -d --name t myapp && docker stop -t 30 t
$ docker inspect t --format '{{.State.ExitCode}}'
0 # clean shutdown. 137 means it was SIGKILLed after the grace period.
# Spring Boot side:
server.shutdown: graceful
spring.lifecycle.timeout-per-shutdown-phase: 25sThe lifecycle itself: docker stop sends SIGTERM, waits the grace period, then sends SIGKILL. Kubernetes does the same, with a preStop hook running before the SIGTERM. Everything about graceful shutdown depends on the signal actually arriving at a process that handles it.
Use an init like tini, or handle SIGTERM explicitly. A service that loses in-flight requests on every deploy usually has this bug, and it presents as mysterious 5xx spikes correlated with rollouts.
Container networking on one host
Each container gets its own network namespace and a virtual ethernet pair into a host bridge. Port publishing is NAT. DNS by container name only exists on user-defined networks.
On the default bridge, the runtime creates a veth pair: one end inside the container as eth0, the other attached to a Linux bridge on the host. The bridge hands out addresses from a private subnet, typically 172.17.0.0/16. Outbound traffic is source-NATed to the host address; -p 8080:8080 installs a DNAT rule forwarding a host port inward.
The default bridge deliberately has no DNS. On a user-defined network the runtime runs an embedded resolver at 127.0.0.11, resolving container names and network aliases to current addresses — which is why Compose services address each other by service name, and why you never hardcode a container IP.
host
+-- docker0 (bridge, 172.17.0.1)
+-- veth --- eth0 (172.17.0.2) container A
+-- veth --- eth0 (172.17.0.3) container B
# Both containers can bind port 8080: separate net namespaces, separate
# port spaces. There is no conflict until you publish to the same host port.
# User-defined network -> embedded DNS
services:
api: { build: ., ports: ["8080:8080"], depends_on: [db] }
db: { image: postgres:16 }
# api connects to jdbc:postgresql://db:5432/app <- resolved by 127.0.0.11Across hosts this model breaks, because 172.17.0.2 exists on every machine and means something different on each. Docker's answer is an overlay network with VXLAN encapsulation. Kubernetes takes a stronger position, which is the subject of a later chapter: every pod gets an address routable from every other pod, with no NAT between them.
--network host drops the network namespace entirely — the container uses the host's interfaces and ports. It removes a small amount of overhead and is right for infrastructure agents that must see host traffic. For application workloads it is a bad default: you lose port isolation and weaken the boundary.
Compose, and the limits of one machine
Compose is a local development and integration-test tool. It is not an orchestrator, and conflating the two is a junior signal.
What Compose gives you is genuinely valuable: the real Postgres, the real Kafka, the real network naming, on one machine, from one file, with one command. That makes integration tests run against real dependencies rather than mocks, which catches a class of bug nothing else does.
What it does not give you
- Rescheduling. If the machine dies, everything dies.
- Health-driven replacement.
depends_onwaits for start, not for ready — a healthcheck condition helps and is still single-host. - Horizontal scale across machines.
- Rolling updates with traffic management.
- Declarative reconciliation — Compose applies a change; it does not continuously enforce a desired state.
Those five gaps are, almost exactly, the list of things Kubernetes exists to provide. Which is a useful way to introduce it: an orchestrator is what you need once a single machine is no longer an acceptable failure domain.
services:
api:
build: .
ports: ["8080:8080"]
environment: [SPRING_PROFILES_ACTIVE=local]
depends_on:
db: { condition: service_healthy } # waits for READY, not just started
db:
image: postgres:16
environment: [POSTGRES_PASSWORD=dev]
healthcheck:
test: ["CMD-SHELL","pg_isready -U postgres"]
interval: 3s
retries: 10Use Compose in CI for integration tests against real dependencies. Testcontainers is the same idea driven from the test code, and for a JVM service it is usually the better fit because the lifecycle is owned by the test rather than by a separate file.
Anatomy of a Dockerfile
Every instruction either changes the filesystem (producing a layer) or changes metadata (producing none). Knowing which is which is most of what you need to write a good one.
FROM the base layers you inherit -> layers
RUN execute a command, keep the result -> LAYER
COPY copy from build context or a stage -> LAYER
ADD like COPY, plus tar extraction and URL fetch -> LAYER (avoid)
WORKDIR set the working directory -> metadata
ENV environment variables -> metadata (VISIBLE)
ARG build-time variable -> metadata (VISIBLE)
USER the user the process runs as -> metadata
EXPOSE documentation only, publishes nothing -> metadata
ENTRYPOINT the executable -> metadata
CMD default arguments to the entrypoint -> metadata
LABEL metadata for tooling and traceability -> metadataAlways COPY, never ADD. ADD auto-extracts local tar archives and can fetch remote URLs without checksum verification — both surprising, and the second is a supply chain risk. Use an explicit RUN curl with a checksum when you genuinely need a download.
ENTRYPOINT in exec form, so your process is PID 1 and receives signals. EXPOSE publishes nothing — it is documentation that tooling may read. And never put a secret in ENV or ARG: both are visible in docker inspect and in the image history to anyone who can pull it.
LABEL org.opencontainers.image.source="https://github.com/acme/payments-api"
LABEL org.opencontainers.image.revision="$GIT_SHA"
LABEL org.opencontainers.image.created="$BUILD_TIME"
# -> a running container can be traced back to a commit:
# docker inspect --format '{{index .Config.Labels "..revision"}}'A .dockerignore is not optional. Without it the whole working directory becomes the build context - .git, target/, local .env files, IDE state - which is slow to transfer and is how local credentials end up in images.
The build cache
Layers are cached by content, and a cache miss invalidates that layer and everything after it. Cache is a prefix property, which is the single fact that determines instruction order.
For each instruction the builder checks whether it already has a layer for that instruction against that parent. If yes it reuses it and moves on. If no, it builds — and every subsequent instruction must also rebuild, because their parent has changed.
So the ordering rule follows directly: rarely-changing things first, frequently-changing things last. Dependencies change weekly; your source changes every commit. Copy the dependency descriptor and resolve dependencies in their own layer, then copy source.
# BAD - every commit re-resolves the entire dependency tree
COPY . /src # any file change busts everything below
RUN mvn -q package # 4 minutes, every time
# GOOD - dependencies cached in their own layer
COPY pom.xml .
RUN mvn -q dependency:go-offline # cached until pom.xml changes
COPY src ./src # changes every commit
RUN mvn -q package -o -DskipTests # 20 seconds
# CI runs on fresh runners, so the local cache is always cold.
# Use BuildKit with an EXTERNAL cache or none of this helps:
docker buildx build \
--cache-from type=registry,ref=$ECR/payments-api:buildcache \
--cache-to type=registry,ref=$ECR/payments-api:buildcache,mode=max \
-t $IMG --push .BuildKit also supports cache mounts, which persist a directory across builds without putting it in a layer — ideal for a package manager cache. The dependency artefacts are reused, and they never bloat the image.
Teams often conclude that layer ordering does not matter, because their CI is cold on every build. The ordering is still right; the missing piece is an external cache. Fixing both together is what turns a four-minute build into forty seconds.
Multi-stage builds
Build in one stage, ship from another. The compiler, the source, the package cache and any build-time credentials stay behind in a stage that is never pushed.
A multi-stage build declares several FROM instructions. Only the last one produces the shipped image; earlier stages are used for their outputs. COPY --from=build pulls an artefact across, leaving everything else behind.
The benefits compound: a much smaller image, a far smaller CVE surface (no compiler, no package manager, no shell in a distroless final stage), and — importantly — the build becomes reproducible in CI rather than depending on a jar someone built locally.
FROM maven:3.9-eclipse-temurin-21 AS build
WORKDIR /src
COPY pom.xml .
RUN --mount=type=cache,target=/root/.m2 mvn -q dependency:go-offline
COPY src ./src
RUN --mount=type=cache,target=/root/.m2 mvn -q package -DskipTests
FROM eclipse-temurin:21-jre-alpine
RUN addgroup -S -g 10001 app && adduser -S -u 10001 -G app app
COPY --from=build --chown=10001:10001 /src/target/app.jar /app/app.jar
USER 10001:10001
ENTRYPOINT ["java","-jar","/app/app.jar"]
# The JDK, Maven, the source and the ~/.m2 cache are ALL left behind.
# ~800MB -> ~230MB, and no compiler in production.Stages are also useful for testing and scanning inside the build: a stage that runs the test suite, a stage that produces an SBOM. And you can target a specific stage with --target, which is how you get a debug image with a shell from the same Dockerfile.
Any secret used at build time - a private repository token, an npm credential - must never appear in a layer. Use BuildKit secret mounts, which make the value available during a RUN without recording it.
Choosing a base image
Smaller is better up to a point. The real axes are attack surface, debuggability, and libc — and for a JVM the libc choice has caused real production surprises.
BASE SIZE SHELL PKG MGR libc NOTES
eclipse-temurin:21 ~450MB yes yes glibc full JDK, do not ship
eclipse-temurin:21-jre ~270MB yes yes glibc sensible default
...-jre-alpine ~180MB yes yes MUSL smaller; see caveat
gcr.io/distroless/java21 ~190MB NO NO glibc smallest attack surface
custom jlink runtime ~90MB no no glibc maximum effortDistroless has no shell and no package manager, which is a genuine security improvement — an attacker with code execution has almost nothing to work with. The cost is that you cannot exec into it, which you work around with ephemeral debug containers rather than by abandoning it.
Alpine uses musl rather than glibc. For pure Java this is usually fine, and it has historically caused subtle issues with native libraries, DNS resolution behaviour and some profiling agents. Test rather than assume, and be aware that if you hit a strange native-library problem, the libc is the first thing to check.
- Pin the base image by DIGEST, not by tag, so builds are reproducible.
- Standardise one base across services - shared cached layers cut pull time on cold nodes more than shaving 50MB off one image.
- Let a bot raise base image bumps as reviewable pull requests.
- Rescan published images continuously: a clean image today is not clean next month.
A reasonable size target for a Spring Boot service is 200-300MB. Chasing much smaller with a custom jlink runtime is possible and rarely repays the maintenance - the honest answer in an interview is that pull time matters and the last 50MB usually does not.
Containerising a JVM service
Multi-stage, JRE base, non-root numeric UID, layered jar, and MaxRAMPercentage rather than a hardcoded heap size. The last one is where most JVM container incidents come from.
Layered jars
Spring Boot can split its fat jar into dependency, loader, snapshot-dependency and application layers. Copying those as separate instructions means a code-only change re-ships a few hundred kilobytes rather than the whole 60MB jar — a real saving on a rolling deploy across many nodes.
Memory
Modern JVMs read the cgroup limit, so a hardcoded -Xmx is wrong the moment the limit changes. Use MaxRAMPercentage — and set it around 70, not 90, because the container limit covers the whole process.
FROM maven:3.9-eclipse-temurin-21@sha256:<digest> AS build
WORKDIR /src
COPY pom.xml .
RUN --mount=type=cache,target=/root/.m2 mvn -q dependency:go-offline
COPY src ./src
RUN --mount=type=cache,target=/root/.m2 mvn -q package -DskipTests \
&& java -Djarmode=layertools -jar target/app.jar extract --destination /layers
FROM eclipse-temurin:21-jre-alpine@sha256:<digest>
RUN addgroup -S -g 10001 app && adduser -S -u 10001 -G app app
WORKDIR /app
COPY --from=build --chown=10001:10001 /layers/dependencies/ ./
COPY --from=build --chown=10001:10001 /layers/spring-boot-loader/ ./
COPY --from=build --chown=10001:10001 /layers/snapshot-dependencies/ ./
COPY --from=build --chown=10001:10001 /layers/application/ ./
USER 10001:10001
ENV JAVA_TOOL_OPTIONS="-XX:MaxRAMPercentage=70 -XX:+ExitOnOutOfMemoryError \
-XX:MaxDirectMemorySize=64m -XX:MaxMetaspaceSize=128m"
ENTRYPOINT ["java","org.springframework.boot.loader.launch.JarLauncher"]Inside a 1Gi limit, MaxRAMPercentage=90:
heap ~921 MB
metaspace ~80 MB
JIT code cache ~50 MB
thread stacks (200 x 1MB) ~200 MB <- GROWS with the thread pool
direct buffers / GC ~60 MB
--------
~1311 MB > 1Gi -> OOMKilled
...with the heap graph showing 40% free. That is the confusing incident.ExitOnOutOfMemoryError matters more than it looks: a JVM that has thrown OOM is in an undefined state - some threads dead, others serving broken results, and the liveness probe may still pass. Exiting turns an ambiguous zombie into a clean restart.
Image hygiene: size, secrets and provenance
Prove the image is production-ready before it reaches a cluster: reproducible build, scanned, behaves under a memory limit, shuts down cleanly, probes mean something, and no secrets in any layer.
# 1. reproducible build in CI, not on a laptop
docker buildx build --provenance=true -t $IMG --push .
# 2. scan with a POLICY, not a report nobody reads
trivy image --exit-code 1 --severity CRITICAL,HIGH --ignorefile .trivyignore $IMG
# 3. behaves under a memory limit (the step teams skip)
docker run -d --name t --memory=512m --cpus=1 $IMG
hey -z 60s -c 50 http://localhost:8080/api/health # REAL load, not one curl
docker inspect t --format '{{.State.OOMKilled}}' # must be false
# 4. graceful shutdown within the grace period
docker stop -t 30 t
docker inspect t --format '{{.State.ExitCode}}' # 0, not 137
# 5. probes actually distinguish alive from ready
docker network disconnect bridge postgres
curl -sf localhost:9090/actuator/health/liveness # must STILL be 200
curl -sf localhost:9090/actuator/health/readiness # must be 503
# 6. no secrets anywhere in the history
docker history --no-trunc $IMG | grep -iE 'password|secret|token' && exit 1Step five is worth dwelling on. If liveness fails when a dependency is down, you have built a system that restarts its entire fleet during any database blip. Catching that on your machine is trivial; discovering it during an incident is not.
One image, promoted unchanged from dev to production. Configuration comes from the environment at runtime. The moment you build a per-environment image, the thing you tested is not the thing you shipped and your promotion pipeline is verifying the wrong artefact.
Registries, tags and digests
A tag is a mutable pointer; a digest is immutable. Pin digests in production, or two pods from the same Deployment can be running different code while every piece of evidence says they match.
Nothing prevents someone pushing different content to v1.4.2 tomorrow. So a pod created today and a pod created next week from the same manifest can differ. Your rollout looks complete and consistent; it is not — and debugging that is miserable because every observation says the versions are identical.
The pattern: CI builds, pushes with a semantic tag and records the digest, and a bot opens a pull request updating the digest in the manifest repository. The bump becomes a reviewable, revertible commit rather than an invisible side effect.
# CI
docker buildx build -t $ECR/payments-api:1.4.2 --push .
DIGEST=$(crane digest $ECR/payments-api:1.4.2)
cosign sign --yes $ECR/payments-api@$DIGEST # keyless, OIDC identity
# Manifest references the DIGEST
image: 1234.dkr.ecr.eu-west-1.amazonaws.com/payments-api@sha256:9f2c...
# Registry settings that matter
imageTagMutability: IMMUTABLE # a tag can never be overwritten
scanOnPush: true # plus CONTINUOUS rescanning
lifecyclePolicy: expire untagged after 7d, keep last 30 taggedAround that sits the rest of the supply chain: scan in CI and continuously afterwards, generate an SBOM so you can answer are we affected by this CVE in minutes rather than days, sign images, and — the step that makes the rest meaningful — verify signatures at admission. Signing without verification proves nothing.
imagePullPolicy: Always is not a substitute for digests; it is nearly the opposite. It guarantees you re-resolve the tag on every pod start, which makes drift MORE likely - a pod restarting at 3am silently picks up whatever the tag points to now.
Kubernetes is a control loop
You declare desired state; controllers continuously compare reality against it and act. That one idea explains the entire object model, and most questions answer themselves once you have it.
Nothing in Kubernetes is imperative. You do not tell it to start three pods — you record that three pods should exist, and a controller notices that they do not and creates them. If one dies, the same controller notices again. There is no event you missed and no command that failed to take effect; there is only a gap between declared and actual, being continuously closed.
Level-triggered, not edge-triggered
This is the property that makes the system robust. An edge-triggered system reacts to events and breaks when an event is lost. A level-triggered system looks at current state and acts, so a missed event is self-correcting on the next resync. It is also why I applied it and nothing happened usually means wait a moment and read the events.
for ever:
desired = read_spec() # what you declared
actual = observe_world() # what exists
if actual != desired:
act_to_close_the_gap()
Deployment reconciles -> ReplicaSets
ReplicaSet reconciles -> Pods
scheduler assigns pods to nodes
kubelet runs containers, reports status
endpoints keeps Services pointing at READY pods
CRD + operator = the SAME pattern, extended to your own objects- Nothing happens instantly - everything converges. Debugging is reading the gap.
- Health is defined by YOUR probes, so the cluster is exactly as smart as they are.
- Requests and limits are how you talk to the scheduler.
- Once you have the loop, the object model stops being arbitrary.
Asked to explain Kubernetes to a colleague starting from zero, this is the answer - not a list of objects. Someone with the control-loop model can reason about a controller they have never seen; someone who memorised YAML cannot.
The control plane
API server, etcd, scheduler, controller manager. Everything goes through the API server, and everything is stored in etcd — which is why a control plane outage is a freeze rather than an outage.
- API server — the only component that talks to etcd. Authenticates, authorises, runs admission control, validates, persists. Every other component is a client of it.
- etcd — a consistent key-value store holding all cluster state. Needs quorum, so an odd number of members, and it is the most operationally sensitive component in a self-managed cluster.
- Scheduler — watches for pods with no node assigned, filters nodes by predicates, scores the survivors, and writes the binding. It makes a decision; it does not start anything.
- Controller manager — a bundle of controllers: Deployment, ReplicaSet, node lifecycle, endpoints, service accounts, and more.
What happens when it is unavailable
Running pods keep running — the kubelet supervises containers locally and restarts them per the restart policy without needing the API server. kube-proxy has already programmed the node's routing rules, so existing Services keep load balancing. Traffic flows.
What stops is everything requiring a decision: no new scheduling, no rescheduling after a node failure, no rollouts, no scaling, no endpoint updates — so a pod that dies stays in the routing rules and a slice of traffic is blackholed. And every kubectl command fails, so you are also blind.
Never put the Kubernetes API in your application's request path. A service that calls the API server per request has a hard dependency on the control plane, which turns a freeze into a genuine outage. Use an informer with a local cache that survives disconnection.
The node
kubelet, container runtime, kube-proxy. The kubelet is the agent that actually runs things — it watches for pods assigned to its node and makes them exist.
- kubelet — watches the API server for pods bound to this node, pulls images, starts containers via the runtime, runs probes, reports status, and enforces eviction under resource pressure.
- Container runtime — containerd or CRI-O, speaking the Container Runtime Interface. Docker Engine is no longer used directly by Kubernetes.
- kube-proxy — programs iptables or IPVS rules so Service virtual IPs resolve to pod IPs. Increasingly replaced by eBPF dataplanes.
- CNI plugin — sets up pod networking when a pod is created.
Allocatable is not capacity
A node's capacity is its hardware. Its allocatable is what remains after kubelet reservations, system reservations and the eviction threshold. A 16Gi node typically offers around 14.5Gi to pods, so a pod requesting 15Gi will never schedule on it however large the node looks — a genuinely common source of permanent Pending.
$ kubectl describe node ip-10-20-16-33
Capacity:
cpu: 4 memory: 16305672Ki pods: 58
Allocatable:
cpu: 3920m memory: 14892424Ki pods: 58 <- what you can actually use
# Eviction: under memory pressure the KUBELET reclaims by deleting pods,
# in QoS order (BestEffort first, then Burstable exceeding requests).
# That is different from an OOM kill, which is the KERNEL enforcing one
# container's cgroup limit. Different cause, different fix.On EKS the pod-per-node limit is driven by ENIs and secondary IP addresses, not by CPU or memory - a small instance may cap at 8 or 17 pods regardless of how much it could otherwise run. Prefix delegation raises the ceiling and is a standard tuning step.
Pods
One or more containers sharing a network namespace and a lifecycle, always co-scheduled. The default is one container; a second needs to justify itself.
A pod exists because some processes must be co-located and share a network identity. If the scheduler could put a sidecar proxy on a different node from the application it fronts, it would not be a sidecar. Making the pod the atom lets Kubernetes express these processes are one deployable thing without a second concept.
Containers in a pod share the network namespace (one IP, one port space, mutual localhost), IPC, and any volumes you mount into both. They do not share a filesystem otherwise.
When a second container is justified
The test: would I ever want a different replica count for it? If yes, it belongs in its own pod. Legitimate cases are narrow — a log shipper reading a shared volume, a mesh proxy intercepting the network namespace, a config reloader watching a mounted file, a short-lived credential helper.
spec:
initContainers:
- name: wait-for-db # runs to completion, THEN app containers start
image: busybox:1.36
command: ['sh','-c','until nc -z postgres 5432; do sleep 2; done']
- name: otel-agent # NATIVE SIDECAR (1.29+)
image: otel/collector:0.96
restartPolicy: Always # <- makes it a sidecar: starts BEFORE the app,
# and is terminated AFTER it
containers:
- name: api
image: payments-api@sha256:...
# Native sidecars fixed a long-standing bug where a mesh proxy died before
# the app finished draining, cutting in-flight requests on every shutdown.A database migration in an init container works with one replica and breaks at scale: three replicas run three migrations concurrently. Use a separate Job or a sync hook that runs once before the rollout.
ReplicaSets and Deployments
A Deployment owns ReplicaSets; a ReplicaSet owns pods. The Deployment never touches pods directly, and that indirection is what makes rollback trivial.
A ReplicaSet has one job: keep N pods matching a template alive. A Deployment manages ReplicaSets: on an update it computes a hash of the new pod template, creates a ReplicaSet for it, and shifts replicas from old to new.
Old ReplicaSets are retained at zero replicas — bounded by revisionHistoryLimit — and that retained set is your rollback history. kubectl rollout undo simply scales the previous ReplicaSet back up, which is why rollback takes seconds rather than a redeploy.
$ kubectl get rs --sort-by=.metadata.creationTimestamp
NAME DESIRED CURRENT READY AGE
payments-api-6d4f9c 0 0 0 12d <- retained history
payments-api-7b8a2d 0 0 0 3d
payments-api-9c1e5f 4 4 4 20m <- current
$ kubectl rollout history deploy/payments-api
$ kubectl rollout undo deploy/payments-api --to-revision=7
# kubectl rollout restart is not a special mechanism: it patches an annotation
# on the pod template, which changes the hash, which creates a new ReplicaSet.Two other controllers complete the picture: a DaemonSet runs one pod per node and adds one automatically when a node joins, which is how log shippers and CNI agents get everywhere; and a StatefulSet adds identity and per-replica storage, covered later.
A useful diagnostic habit: when a rollout misbehaves, look at the ReplicaSets rather than the pods. The counts across old and new tell you immediately whether the rollout is progressing, stalled, or complete.
Rolling updates
maxSurge and maxUnavailable bound the dance. The readiness probe is the actual safety mechanism — without it, Kubernetes cheerfully replaces every healthy pod with a broken one.
maxSurge is how many pods may exist above the desired count — extra capacity, extra cost, faster rollout. maxUnavailable is how many of the desired count may be missing — how much capacity you will give up mid-rollout. Both default to 25%.
The pairing that matters for a production API is maxUnavailable: 0 with maxSurge: 1: you never dip below full capacity, at the cost of one extra pod's resources and a slower rollout. It is the wrong setting for a cluster with no spare room, where the rollout simply blocks.
Readiness is the safety mechanism
The controller only proceeds when new pods report Ready. If they never do, the rollout stalls with the old pods still serving — which is exactly the behaviour you want from a failed deploy. Without a readiness probe, a pod is Ready as soon as its container starts, so the controller replaces the whole fleet at full speed with something broken.
spec:
replicas: 4
strategy:
type: RollingUpdate
rollingUpdate: { maxSurge: 1, maxUnavailable: 0 }
minReadySeconds: 15 # Ready for 15s before counting
# catches crash-on-first-request
progressDeadlineSeconds: 300 # marks a stalled rollout FAILED
# (it does NOT roll back automatically)
$ kubectl rollout status deploy/payments-api
Waiting for deployment rollout to finish: 1 of 4 updated replicas are available
# <- the HEALTHY failure mode. Old pods still serving. Investigate; do not force.A rolling update means two versions serve simultaneously. That is a hard constraint on your changes: the API must be backward compatible, and any schema change must be expand-contract so both versions work against the same schema.
Labels, selectors and ownership
Labels are how everything finds everything. Selectors are queries over them, and the ownership chain is what makes cascading deletion work.
Kubernetes has almost no hard references between objects. A Service does not list its pods — it has a label selector, and the endpoint controller continuously finds pods matching it. A ReplicaSet does not list its pods either. This is what lets pods come and go without anything being reconfigured.
metadata:
labels:
app.kubernetes.io/name: payments-api # the application
app.kubernetes.io/instance: payments-api # this deployment of it
app.kubernetes.io/version: "1.4.2"
app.kubernetes.io/component: api
team: payments # for cost attribution
tier: api # for NetworkPolicy selectors
# Service finds pods by QUERY, not by list:
spec:
selector: { app.kubernetes.io/name: payments-api }
$ kubectl get pods -l 'tier=api,team=payments'
$ kubectl get pods -l 'app.kubernetes.io/name in (payments-api,settlement-api)'Annotations are the other half: arbitrary metadata not used for selection. Tooling reads them — ingress controller configuration, cert-manager issuers, Prometheus scrape hints, checksums that force a rollout when a ConfigMap changes.
ownerReferences link a child to its parent, and are how cascading deletion works: delete a Deployment and garbage collection removes its ReplicaSets and their pods. A pod with no owner — a bare pod — is never recreated and is destroyed by a node drain, which is why bare pods are a bad idea outside debugging.
A Deployment's selector is IMMUTABLE after creation. Changing it means deleting and recreating the Deployment, which is a genuine outage on a live service - so establish label conventions before you have fifty services using them inconsistently.
Namespaces and multi-tenancy
A namespace scopes names and is where RBAC and quota attach. It is soft multi-tenancy — it does not isolate the network, the kernel, or the node.
Namespaces give you three real things: a naming scope, an attachment point for RBAC, and an attachment point for resource quotas. That is enough to separate trusted teams within one organisation, and it is what most clusters need.
kind: ResourceQuota
spec:
hard:
requests.cpu: "40"
requests.memory: 80Gi
limits.memory: 80Gi
pods: "120"
count/services.loadbalancers: "2" # each one is a real, billed LB
# SIDE EFFECT: once CPU/memory quota is set, a pod WITHOUT requests and limits
# is REJECTED at admission. A useful forcing function.
---
kind: LimitRange
spec:
limits:
- type: Container
default: { cpu: "500m", memory: 512Mi } # applied if omitted
defaultRequest: { cpu: "100m", memory: 256Mi }
max: { cpu: "4", memory: 8Gi } # nobody grabs a whole nodeWhat a namespace does NOT isolate
- The network. Every pod can reach every pod in every namespace until you add a default-deny NetworkPolicy — and a CNI that enforces it.
- The kernel. Pods from different namespaces share nodes and therefore share a kernel; a container escape crosses namespaces trivially.
- Node resources beyond cgroups. Disk IO, network bandwidth and page cache are shared, so a noisy neighbour still hurts.
- Cluster-scoped objects. CRDs, ClusterRoles, StorageClasses, admission webhooks and nodes are all shared.
- The control plane. One API server and one CoreDNS serve everyone.
Namespaces separate TEAMS. Clusters separate FAILURE DOMAINS. Splitting clusters per team multiplies operational cost to solve a problem namespaces already solve; splitting per region or per environment is separating things that genuinely need separating.
The network model
Every pod gets an IP routable from every other pod, on any node, with no NAT between them. That mandate is what makes the rest of the networking simple.
Kubernetes does not implement networking — it specifies requirements and delegates. The requirements are strict and worth stating precisely, because a great deal follows from them.
- Every pod has its own IP address.
- Pods can reach all other pods on any node without NAT.
- Agents on a node (kubelet, system daemons) can reach all pods on that node.
- Pods see their own IP as the same address others use to reach them.
That last one matters more than it sounds. In the Docker single-host model, a container's own view of its address differs from how it is reached, which breaks anything that registers its own address — service discovery, clustered software, some RPC frameworks. Kubernetes removes the discrepancy.
The absence of NAT between pods is the other big simplification: source addresses are preserved for pod-to-pod traffic, so network policy and audit logging work on real identities rather than on translated addresses.
The model says nothing about isolation. Flat and routable means ANY pod can reach ANY pod by default, across namespaces. Isolation is opt-in via NetworkPolicy, which is a common surprise for people expecting namespaces to be a network boundary.
CNI, and the AWS VPC CNI
A CNI plugin implements the network model. On EKS the default gives every pod a real VPC address, which has consequences for subnet sizing and pod density that catch people out.
When the kubelet creates a pod it calls a Container Network Interface plugin to set up the network namespace: create the interface, assign an address, configure routes. Different plugins solve it differently.
- Calico — routes pod traffic natively where the underlying network allows, or uses an overlay. Enforces NetworkPolicy.
- Cilium — eBPF-based, can replace kube-proxy entirely, adds identity-aware and DNS-aware policy plus transparent encryption.
- AWS VPC CNI — assigns each pod a real VPC IP address from your subnets, so pods are first-class citizens of the VPC.
What the VPC CNI implies
Two consequences. Subnet sizing is driven by pod count, not node count — a /24 holds about 251 pods, not 251 nodes, and IP exhaustion presents as pods stuck Pending with no obvious resource shortage. And pod density per node is capped by ENIs: each instance type supports a fixed number of network interfaces and secondary addresses, so a small instance may max out at 8 or 17 pods regardless of its CPU and memory.
# Pods per node with the VPC CNI, by default:
# (ENIs x (IPs per ENI - 1)) + 2
# m5.large -> (3 x (10-1)) + 2 = 29
# t3.small -> (3 x (4-1)) + 2 = 11 <- CPU is irrelevant here
# PREFIX DELEGATION - allocate /28 prefixes instead of single IPs
kubectl set env ds aws-node -n kube-system ENABLE_PREFIX_DELEGATION=true
# m5.large -> ~110 pods
# And note: the VPC CNI alone does NOT enforce NetworkPolicy unless you
# explicitly enable network policy support, or run Calico/Cilium alongside.
# Policies that exist but are not enforced are worse than none - they LOOK
# like control.Verify NetworkPolicy enforcement with an actual connectivity test, not by reading YAML. It is the one security control whose absence is completely invisible in the manifests.
Services
A stable name and virtual IP in front of a changing set of READY pods. Only Ready pods are in the endpoint list, and that readiness gate is the whole traffic-management mechanism.
A Service has a label selector, and a separate controller maintains EndpointSlices listing the addresses of pods that match it and are Ready. Traffic goes only to that list — which is why a readiness probe gates traffic, why a rollout is safe, and why a pod can be Running and receiving nothing.
ClusterIP internal virtual IP, cluster-only (the default)
NodePort + opens the same port on EVERY node
LoadBalancer + provisions a cloud load balancer
ExternalName just a CNAME - no proxying at all
Headless (clusterIP: None)
no virtual IP; DNS returns the POD IPs directly
used by StatefulSets so peers can address each other individually,
and by gRPC clients that want to balance connections themselvesOn AWS the load balancer controller can operate in instance mode (targets are nodes, traffic hops through kube-proxy) or IP mode (targets are pod IPs directly, since the VPC CNI makes them routable). IP mode removes a hop and gives the load balancer accurate health checking of the actual pod.
externalTrafficPolicy is worth knowing: with Cluster, a node receiving external traffic may forward to a pod elsewhere — even distribution, extra hop, and the client source IP is lost to SNAT. With Local, only pods on that node receive it: source IP preserved, no extra hop, uneven if pods are unevenly spread.
A pod showing 1/1 Running but receiving no traffic is almost always a failing readiness probe. Check EndpointSlices before checking anything in the application - if the pod is not listed, no amount of application debugging will help.
How load balancing actually works
There is no proxy. kube-proxy programs the kernel on every node, and it balances PER CONNECTION — which skews badly for gRPC and HTTP keep-alive.
A ClusterIP is not a real interface and nothing listens on it. kube-proxy watches Services and EndpointSlices and writes rules into the kernel on every node: in iptables mode, a chain of DNAT rules with probabilistic matching; in IPVS mode, a proper in-kernel load balancer that scales better past a few thousand services.
So load balancing happens on the calling node, in the kernel, before the packet leaves. There is no central component and therefore no single bottleneck — but also no L7 awareness.
The per-connection problem
iptables picks a backend at random per connection. With HTTP keep-alive, and especially with gRPC where a single HTTP/2 connection multiplexes every request, a client pins to one backend for the connection's lifetime. Traffic skews badly, and adding replicas does not help — the existing connections do not move.
SYMPTOM: 6 replicas, one at 80% CPU and five nearly idle. Scaling out
changes nothing.
FIXES
shorter connection lifetimes crude, but works
client-side load balancing gRPC + a headless Service, so the client
sees pod IPs and balances per request
a proxy that balances per request ingress, or a service mesh
Cilium eBPF dataplane replaces kube-proxy, better balancing
This is one of the most common real reasons teams adopt a service mesh.Being able to explain WHY gRPC traffic skews in Kubernetes - per-connection balancing plus HTTP/2 multiplexing - is a strong signal. It is a specific, mechanical explanation for a symptom that otherwise looks like a mystery.
DNS
CoreDNS resolves Service names to ClusterIPs. The search path in resolv.conf is convenient and is the source of two recurring surprises.
Every Service gets a DNS record at <service>.<namespace>.svc.cluster.local. The kubelet writes CoreDNS's address into each pod's /etc/resolv.conf along with a search path, which is why postgres resolves from the same namespace and postgres.payments is needed from another.
/etc/resolv.conf inside a pod in namespace 'payments':
nameserver 172.20.0.10
search payments.svc.cluster.local svc.cluster.local cluster.local eu-west-1.compute.internal
options ndots:5
SURPRISE 1 - ndots:5
any name with FEWER than 5 dots is tried against every search domain FIRST.
"api.example.com" (2 dots) generates:
api.example.com.payments.svc.cluster.local NXDOMAIN
api.example.com.svc.cluster.local NXDOMAIN
api.example.com.cluster.local NXDOMAIN
api.example.com.eu-west-1.compute.internal NXDOMAIN
api.example.com. <- finally
Five lookups for every external call. Fix with a trailing dot, or by
lowering ndots in dnsConfig.
SURPRISE 2 - headless Services return POD IPs, not a virtual IP,
so a client caching the first result talks to one pod forever.CoreDNS is a normal Deployment in kube-system, so it can be under-provisioned, and DNS latency shows up as mysterious application slowness. NodeLocal DNSCache runs a caching resolver on each node, which cuts latency and load substantially and is a standard addition to a busy cluster.
When you enable a default-deny egress NetworkPolicy, you MUST explicitly allow egress to kube-dns on UDP and TCP 53. Forgetting it breaks every hostname lookup in the namespace, and the symptom looks like total application failure rather than a policy problem.
Ingress and the Gateway API
An Ingress object does nothing by itself — a controller watches it and programs a real proxy. Its API is deliberately minimal, which is why everything real ended up in vendor annotations.
Ingress describes host and path routing to Services, plus TLS. A controller — ingress-nginx, Traefik, or the AWS Load Balancer Controller — reads those objects and configures actual infrastructure. You can have Ingress objects and no controller, in which case nothing routes and nothing tells you why.
TLS terminates at the controller, or at the cloud load balancer in front of it, with certificates held in a Secret. cert-manager is the standard companion: it watches Ingresses, requests certificates, completes the challenge and renews automatically.
Why the Gateway API exists
Ingress covers host and path routing and nothing else, so every real feature — rate limiting, timeouts, rewriting, auth, canary weights, sticky sessions — arrived as vendor-specific annotations. Your manifests become non-portable strings that no schema validates and no reviewer can check. That is the actual pain, not any missing capability.
# Ingress: features live in unvalidated vendor strings
annotations:
nginx.ingress.kubernetes.io/canary: "true"
nginx.ingress.kubernetes.io/canary-weight: "10" # a string
nginx.ingress.kubernetes.io/proxy-read-timeout: "300" # a string
# Gateway API: split by OWNERSHIP, with typed fields
kind: Gateway # platform team owns this
spec:
gatewayClassName: alb
listeners:
- { name: https, port: 443, protocol: HTTPS,
tls: { certificateRefs: [{ name: api-tls }] },
allowedRoutes: { namespaces: { from: Selector, selector: {...} } } }
---
kind: HTTPRoute # application team owns this
spec:
parentRefs: [{ name: prod-gateway }]
rules:
- matches: [{ path: { type: PathPrefix, value: /v1 } }]
backendRefs: # canary as a TYPED field
- { name: api-stable, port: 80, weight: 90 }
- { name: api-canary, port: 80, weight: 10 }Run two ingress controllers - one internet-facing, one internal, selected by ingressClassName. That gives a structural guarantee that an internal service cannot be exposed by a mistaken annotation, which is stronger than any review process.
Graceful shutdown and the endpoint race
Deleting a pod removes it from endpoints and sends SIGTERM concurrently, with no ordering. Traffic is still routed to a closing listener — which is why pods need a preStop hook that deliberately does nothing.
On delete, two things start at the same time. The endpoint controller removes the pod from EndpointSlices, and that removal must propagate to kube-proxy on every node, to the ingress controller, and to any cloud load balancer target group — which takes hundreds of milliseconds to seconds. Meanwhile the kubelet sends SIGTERM and your application begins shutting down.
The result is a window in which nodes that have not yet converged are still sending new connections to a pod that has closed its listener. Connection refused, 502 at the ingress, and nothing in the application logs — because from the application's point of view it shut down correctly.
WITHOUT preStop
t=0.000 delete: endpoint removal starts AND SIGTERM sent (concurrent)
t=0.010 app closes listener, begins draining
t=0.400 kube-proxy on node-7 finally updates its rules
^^^ ~390ms of NEW connections to a closed listener -> 502s
WITH preStop
t=0.000 delete: endpoint removal starts propagating
t=0.000 preStop begins: sleep 10 <- app STILL SERVING NORMALLY
t=0.400 every kube-proxy and ingress has converged
t=10.0 preStop returns -> SIGTERM -> app drains in-flight requests
t=22.0 exits cleanly
spec:
terminationGracePeriodSeconds: 45 # > sleep + drain + margin
containers:
- name: api
lifecycle:
preStop: { exec: { command: ["sh","-c","sleep 10"] } }It feels wrong to add a deliberate delay to a shutdown path, and it is the standard, correct pattern. Note the inversion: the fix is for the application to keep serving longer, not to shut down faster — shutting down faster widens the window of refused connections.
Verify it rather than assuming: run a load test while deleting pods and increase the sleep until the non-2xx count is zero. Propagation time grows with cluster size, so a value that worked at ten nodes may not at two hundred.
Probes
Three probes, three different questions. Readiness gates traffic, liveness gates restarts, startup suspends both while the application boots.
- Readiness — should this pod receive traffic right now? Fail it and the pod is removed from EndpointSlices but keeps running. Reversible: pass again and traffic returns.
- Liveness — is this process irrecoverably broken? Fail it past the threshold and the kubelet kills and restarts the container. Destructive and irreversible.
- Startup — has the application finished booting? While it is failing, the other two are suspended entirely.
The startup probe exists for slow starters, and a Spring Boot service taking 40–90 seconds to build its context is exactly the case. Without one you must set the liveness initialDelaySeconds to cover the worst-case boot, which means a genuinely hung process goes undetected for that whole window. With one you get patience at startup and a tight threshold afterwards.
startupProbe: # patience, only while booting
httpGet: { path: /actuator/health/liveness, port: 9090 }
failureThreshold: 30
periodSeconds: 5 # up to 150s to start, then hands over
livenessProbe: # "is this JVM alive" - nothing else
httpGet: { path: /actuator/health/liveness, port: 9090 }
periodSeconds: 10
failureThreshold: 3 # tight, because startup is handled above
readinessProbe: # "can I serve a real request"
httpGet: { path: /actuator/health/readiness, port: 9090 }
periodSeconds: 5
failureThreshold: 2 # react fast; removal is harmless
# Probe types: httpGet (usual) | tcpSocket (weak - a hung JVM still accepts)
# exec (forks a process every check - expensive at short intervals)
# grpc (standard gRPC health protocol)Expose probes on a separate management port that is never added to a Service. The kubelet reaches it from the node; nothing outside the pod network can. That is structural protection for /actuator rather than a URL rule someone can misconfigure.
The probe mistakes that cause outages
A liveness probe that checks the database will restart your entire fleet during a database blip. It is the single most damaging misconfiguration in Kubernetes.
It looks responsible: if we cannot reach the database we are not healthy. Trace what happens. The database has a 30-second blip or a failover. Every replica's liveness probe fails simultaneously, because they all share the dependency. The kubelet on every node kills every container at once. Your entire fleet is now cold-starting into a database that is still recovering, connection storms make the recovery worse, and a brief degradation becomes a multi-minute total outage with no obvious cause.
# DANGEROUS
livenessProbe:
httpGet: { path: /actuator/health, port: 9090 } # includes db, redis, kafka
# CORRECT - Spring Boot puts the right things in each group
management:
endpoint.health.probes.enabled: true
endpoint.health.group:
liveness: { include: livenessState } # NO dependencies
readiness: { include: readinessState, db, redis } # dependencies live HERE
server.port: 9090
# RULE OF THUMB
# liveness fails -> would a RESTART fix it? (deadlock, hung event loop)
# readiness fails -> would WAITING fix it? (dependency down, still warming)The other common errors
- Liveness and readiness pointing at the same deep endpoint — the same bug in a different shape.
- Thresholds too tight for a GC pause, so a stop-the-world pause is read as a hung process.
- The probe sharing the exhausted thread pool it is meant to be checking, so overload becomes restarts. A separate management connector fixes it.
- A readiness probe checking transitive dependencies — a failure three services away removes your whole fleet from load balancing. Prefer a circuit breaker that degrades the response; being partially useful beats being absent.
Exit 137 is SIGKILL. Check kubectl describe pod: OOMKilled true means the memory limit; a 'failed liveness probe' event means the probe. Same exit code, opposite fixes - and conflating them is how people spend an afternoon raising a memory limit that was never the problem.
Requests and limits
Requests are what the scheduler reserves; limits are what the runtime enforces. Two different systems reading two different numbers, and confusing them causes most resource incidents.
The request is a reservation. The sum of requests on a node cannot exceed its allocatable capacity, so requests decide placement and bin-packing. A pod requesting 2 CPU and using 200 millicores reserves ten times what it needs — nothing errors, and the cluster silently runs several times the nodes it should.
The limit is enforcement, applied by cgroups at runtime, and the two resources behave completely differently at the limit.
MEMORY overrun - instant, loud, obvious
$ kubectl describe pod api-7d9f
Last State: Terminated
Reason: OOMKilled <- kernel SIGKILL, no grace, exit 137
CPU overrun - silent, slow, invisible in logs
$ rate(container_cpu_cfs_throttled_seconds_total[5m]) > 0
# every thread FROZEN until the next 100ms period
# p99 20ms -> 120ms. No errors. Nothing restarts.
# This is the metric teams do not have on their dashboard.How to choose the numbers
From observed usage, not guesses. Run VPA in recommendation mode (updateMode: Off) over a representative window that includes your peak — a week covering a month-end batch, not a quiet Tuesday — and take the p95. Guessing produces both failure modes at once: over-request and waste half the cluster, under-request and get evictions and OOM kills.
Memory: set request EQUAL to limit. It is incompressible, so there is no benefit to bursting and real risk in it, and equal values give Guaranteed QoS. CPU: set the request from p95, and consider omitting the limit for latency-sensitive services - a deliberate current position, because throttling costs p99 with no reliability gain.
QoS and eviction
QoS is derived from what you set, not declared. It determines eviction order under node pressure, and BestEffort pods go first.
GUARANTEED every container has requests == limits for BOTH cpu and memory
evicted LAST, best CPU scheduling
BURSTABLE requests set, limits higher or absent
evicted after BestEffort, in order of how far above request
BESTEFFORT nothing set at all
evicted FIRST - and this is why a manifest with no resources
is a review blockerEviction versus OOM kill
They are different mechanisms with different fixes and they are routinely confused. An OOM kill is the kernel enforcing one container's cgroup limit; the container usually restarts in place, and the fix is a higher limit or lower usage. An eviction is the kubelet reclaiming node resources under pressure by deleting whole pods and rescheduling them elsewhere; the fix is a less oversubscribed node or correct requests.
The kubelet evicts on memory pressure, disk pressure and PID pressure, using configured thresholds with a grace period. Ephemeral storage counts: a service writing logs to its container filesystem can fill the node and be evicted for disk pressure, which is a genuinely confusing incident because nothing crashed.
$ kubectl get events --field-selector reason=Evicted
The node was low on resource: memory. Container api was using 1.4Gi,
which exceeds its request of 512Mi.
# Set ephemeral-storage requests and limits too - they are quota-able
resources:
requests: { cpu: 500m, memory: 1Gi, ephemeral-storage: 1Gi }
limits: { memory: 1Gi, ephemeral-storage: 2Gi }PriorityClass interacts with this: preemption evicts LOWER-priority pods to make room for higher-priority ones, independently of QoS. Batch workloads with a low priority class are the standard way to give online services guaranteed headroom.
Sizing a JVM container
The container limit covers the whole process, not the heap. Non-heap overflow is what produces the incident where a pod is OOM-killed with the heap graph showing plenty free.
Modern JVMs are container-aware and read the cgroup limit, so a hardcoded -Xmx is wrong the moment the limit changes. Use MaxRAMPercentage — and the value matters more than people expect.
Inside a 1Gi limit, MaxRAMPercentage=90:
heap ~921 MB
metaspace ~80 MB class metadata
JIT code cache ~50 MB compiled methods
thread stacks (200 x 1MB) ~200 MB GROWS with the thread pool
direct byte buffers ~40 MB Netty, NIO, some drivers
GC structures ~20 MB
--------
~1311 MB > 1Gi -> OOMKilled
...while the heap graph shows 40% free. That is the confusing incident.
WITH HEADROOM
ENV JAVA_TOOL_OPTIONS="-XX:MaxRAMPercentage=70 \
-XX:MaxMetaspaceSize=128m \
-XX:MaxDirectMemorySize=64m \
-XX:+ExitOnOutOfMemoryError \
-XX:+HeapDumpOnOutOfMemoryError \
-XX:HeapDumpPath=/dump"Bound the parts that grow. Thread stacks scale with your pool sizes, so an unbounded servlet thread pool is an unbounded memory commitment. Direct memory is used by Netty and NIO and is invisible in heap metrics unless you look for it.
CPU
The JVM sizes several things from the available processor count — GC threads, the common ForkJoinPool, JIT compiler threads. A CPU limit below 1 core rounds to one available processor and can serialise GC, which is why very small CPU limits produce disproportionate latency. And CFS throttling freezes GC threads too, so a stop-the-world pause that should take 20ms can take hundreds.
Set the memory request equal to the limit and size it from observed PEAK plus non-heap overhead. Then verify by running the image under that limit with realistic load - the check takes ten minutes and prevents the most common JVM-on-Kubernetes incident.
ConfigMaps and Secrets
Same API, different intent. The distinction that matters operationally is env vars versus volume mounts: env vars are frozen at process start, volumes refresh.
ENV VARS read ONCE at process start. Changing the ConfigMap does
NOTHING until the pod restarts. Also leak into crash dumps,
child processes, and /actuator/env.
VOLUME MOUNT the kubelet refreshes the files within ~60-90 seconds,
so a process watching the file can reload live.
VOLUME + subPath NEVER updates. A silent trap that catches everyone once.A useful pattern that avoids the whole question: annotate the pod template with a hash of the configuration. Changing the config changes the template hash, which triggers a normal rolling update — so the change is visible in the deployment history, auditable, and rolled out safely rather than mutating a running fleet silently.
spec:
template:
metadata:
annotations:
checksum/config: "{{ sha256sum .Files.Get 'application.yaml' }}"
spec:
containers:
- name: api
volumeMounts:
- { name: config, mountPath: /config, readOnly: true }
volumes:
- name: config
configMap: { name: api-config } # NO subPathBoth are capped at roughly 1MB because they live in etcd, which is not a file store. Teams hit it with TLS bundles, certificate chains and seed data — and the answer is a real store (an S3 object fetched by an init container, EFS, or a CSI driver), not splitting the data across ten ConfigMaps.
Most teams should not chase live config reload. A rollout takes seconds, appears in the deployment history, and avoids a class of bug where half your pods have the new config and half do not. Reserve live reload for things that genuinely cannot restart.
Secrets, done properly
A Kubernetes Secret is base64, not encryption. What protects it is etcd encryption, RBAC and audit — and the better answer is to keep the secret outside the cluster entirely.
Base64 is an encoding. Anyone with get on Secrets in that namespace reads the plaintext with one command. So the protection is entirely external to the object: encryption at rest in etcd (on EKS, envelope encryption with a KMS key, which you must enable), RBAC restricting who can read them, audit logging on Secret access, and automountServiceAccountToken: false so a token is not sitting in every pod.
The permissions that are secretly secret-read
pods/execandpods/attach— shell in and read the mounts and environment.pods: create— mount any Secret in the namespace into a container you control.pods/log— if the application ever logs its configuration.- Any write to a Deployment or Job — it can mount whatever it likes.
An RBAC review that greps for the word secrets misses all four. Use a tool that reasons about escalation paths.
# The better answer: the secret's home is AWS Secrets Manager.
# Nothing is ever written to a Kubernetes Secret object at all.
apiVersion: secrets-store.csi.x-k8s.io/v1
kind: SecretProviderClass
metadata: { name: payments-secrets }
spec:
provider: aws
parameters:
objects: |
- objectName: "prod/payments/db"
objectType: "secretsmanager"
jmesPath: [{ path: "password", objectAlias: "db-password" }]
---
spec:
serviceAccountName: payments-api # IRSA -> no bootstrap credential
volumes:
- name: secrets
csi:
driver: secrets-store.csi.k8s.io
readOnly: true
volumeAttributes: { secretProviderClass: payments-secrets }For GitOps, never commit a plaintext secret - git history is permanent. Use External Secrets or the CSI driver so the repository holds only a REFERENCE, and SOPS or Sealed Secrets for the few values that genuinely must live in the repo.
The non-Kubernetes leak that actually happens: a Spring Boot /actuator/env endpoint exposed without authentication prints your entire configuration including credentials. Keep actuator on a separate management port and disable the sensitive endpoints.
How the scheduler decides
Filter, then score, then bind. It makes one decision per pod and writes it down — the kubelet does everything after that.
The scheduler watches for pods with no node assigned. For each, it filters nodes by predicates that must hold — enough allocatable resources, matching node selector and affinity, tolerating any taints, volume topology satisfiable, no port conflicts. Then it scores the survivors by priorities — spreading across nodes, image locality, balanced resource allocation — and binds the pod to the winner.
Two things follow. The scheduler only considers requests, not actual usage: a node running at 90% CPU with low requests looks empty to it. And the decision is made once — Kubernetes does not reschedule a running pod because a better node appeared. Rebalancing requires eviction, which is what the descheduler and Karpenter's consolidation do.
$ kubectl describe pod stuck-7d9f | grep -A15 Events
Warning FailedScheduling 0/14 nodes are available:
6 Insufficient cpu,
4 node(s) had untolerated taint {workload: batch},
4 node(s) had volume node affinity conflict.
# It counts how many nodes failed EACH predicate. That message is usually
# the entire diagnosis - skipping past it to guess is the most common
# time-waster in this scenario.A scheduling decision is not a guarantee of resources - it is a reservation based on requests. If every pod on a node bursts above its request simultaneously, the node is oversubscribed and the kubelet starts evicting. That gap between reservation and reality is where capacity planning lives.
Affinity, taints and topology spread
Five mechanisms answering different questions. For a normal stateless service you need one of them, and reaching for the rest is how pods end up permanently Pending.
nodeSelector exact label match. Simplest, least expressive.
node affinity operators (In, NotIn, Exists, Gt), required or PREFERRED
pod (anti-)affinity place relative to OTHER PODS, via a topologyKey
required anti-affinity is EXPENSIVE at scale
taints + tolerations the INVERSE: a taint on a NODE repels pods
a toleration PERMITS - it does not ATTRACT
topology spread even distribution, with maxSkew and a hard/soft switchThe taint relationship is the one people muddle. Reserving a GPU or spot node pool takes two things: a taint so ordinary pods stay off, and a node selector or affinity so your pods actually land there. A toleration alone means your pod is allowed on the tainted node and equally allowed everywhere else.
# The 90% case: spread across zones, tolerate imbalance
topologySpreadConstraints:
- maxSkew: 1
topologyKey: topology.kubernetes.io/zone
whenUnsatisfiable: ScheduleAnyway # <- availability BEATS symmetry
labelSelector: { matchLabels: { app: payments-api } }
# Reserving a pool: taint AND selector
tolerations:
- { key: workload, operator: Equal, value: batch, effect: NoSchedule }
nodeSelector: { workload: batch } # without this it may land anywhere
# The classic self-inflicted outage
affinity:
podAntiAffinity:
requiredDuringSchedulingIgnoredDuringExecution: # HARD
- topologyKey: kubernetes.io/hostname
labelSelector: { matchLabels: { app: payments-api } }
# replicas: 10, nodes: 6 -> 4 pods Pending FOREVER.
# The scheduler is correct; the constraint is impossible.Use ScheduleAnyway rather than DoNotSchedule for stateless services: when a zone is short of capacity you get an uneven deployment rather than Pending pods. Reserve hard constraints for quorum systems, where scheduling two members together silently removes the fault tolerance you deployed three for.
Why pods sit Pending
Six causes, and the scheduler names which one in its event. Read it literally before doing anything else.
1 REQUESTS > LARGEST ALLOCATABLE
32Gi requested, node offers 30Gi allocatable
-> no autoscaler can fix this. The pod is unschedulable by construction.
2 UNSATISFIABLE AFFINITY / TOPOLOGY SPREAD
hard anti-affinity with replicas > nodes; DoNotSchedule with no room
3 TAINT WITH NO TOLERATION
the pod cannot reach the pool it was meant for
4 VOLUME IN THE WRONG ZONE
"volume node affinity conflict"
-> EBS is AZ-scoped. Fix with StorageClass WaitForFirstConsumer.
5 NAMESPACE RESOURCEQUOTA EXHAUSTED
the cluster looks empty; the NAMESPACE is full
$ kubectl describe quota -n payments
6 IP EXHAUSTION (EKS / VPC CNI)
every POD takes a subnet address; the /24 is full
-> secondary CIDR, or ENABLE_PREFIX_DELEGATIONA seventh that is not really a cause but presents as one: allocatable is not capacity. A 16Gi node offers around 14.5Gi to pods after reservations, so a pod requesting 15Gi never schedules on it however large the node looks.
And if the autoscaler should have helped
Check its logs, because it explains itself. Common reasons it did not add a node: an EC2 service quota, no spot capacity in the requested instance pools, subnet IP exhaustion, or a NodePool CPU limit already reached.
kubectl logs -n kube-system deploy/cluster-autoscaler | tail -30
kubectl logs -n karpenter deploy/karpenter | grep -i "no instance type"
# And the whole-namespace clue: if only ONE namespace is stuck while others
# are fine, check the quota first. Cluster-wide exhaustion would affect
# everyone.The distinction that determines your action: is it a QUANTITY problem or a CONSTRAINT problem? Not enough resources is fixed by adding nodes. An unsatisfiable affinity or a taint is binary - adding nodes does nothing, and the manifest must change.
Horizontal autoscaling
The HPA computes a replica count from a ratio and damps the result. It needs a metrics source, and it cannot create nodes.
The control loop runs roughly every 15 seconds and computes desiredReplicas = ceil(currentReplicas × currentMetric / targetMetric). A tolerance band suppresses small deviations, and stabilisation windows damp oscillation — 0 seconds to scale up, 300 by default to scale down, so it reacts fast to load and slowly to its absence.
apiVersion: autoscaling/v2
kind: HorizontalPodAutoscaler
spec:
scaleTargetRef: { kind: Deployment, name: payments-api }
minReplicas: 3
maxReplicas: 30
metrics:
- type: Pods
pods:
metric: { name: http_server_requests_active }
target: { type: AverageValue, averageValue: "25" }
behavior:
scaleUp:
stabilizationWindowSeconds: 0
policies: [{ type: Percent, value: 100, periodSeconds: 30 }] # can double
scaleDown:
stabilizationWindowSeconds: 600 # slow, cautious
policies: [{ type: Percent, value: 10, periodSeconds: 60 }]Metric sources: metrics-server provides CPU and memory only and is not installed by default everywhere. Anything else needs an adapter implementing the custom or external metrics API — Prometheus Adapter, or KEDA.
It cannot create nodes
The HPA changes a replica count. If the cluster has no room, the new pods sit Pending until a node autoscaler provisions capacity. So end-to-end scale-up is HPA reaction plus node provisioning plus image pull plus JVM start — one to five minutes in practice, which is often too slow for a sharp spike.
For a spike you know the time of, PRE-SCALE on a schedule. Reactive autoscaling always lags because provisioning takes minutes; scaling in advance is simply correct and cheaper than carrying peak capacity all day.
Scaling on the right metric
CPU is the wrong metric for an IO-bound service. Under overload it sits at 25% while latency climbs, so the HPA never fires.
A typical Spring Boot service spends its time blocked — waiting on the database, on a downstream HTTP call, on Kafka. Under increasing load the thread pool fills, requests queue and latency climbs, but the CPU is idle while threads are parked. The pathological picture is p99 at four seconds, queue depth rising, users timing out, and CPU comfortably below a 70% target.
It is not misconfigured; it is measuring the wrong thing. The exhausted resource is concurrency, not CPU.
SCALE ON WHAT ACTUALLY SATURATES, best first:
in-flight requests / thread pool utilisation the thing running out
requests per second per pod once you know a pod's capacity
QUEUE DEPTH cleanest of all for consumers -
it measures unprocessed work
directly, not a proxy
# KEDA for queue-driven workloads - and it scales to ZERO, which HPA cannot
kind: ScaledObject
spec:
scaleTargetRef: { name: settlement-worker }
minReplicaCount: 0
maxReplicaCount: 50
triggers:
- type: aws-sqs-queue
metadata:
queueURL: https://sqs.../settlements
queueLength: "20" # 20 messages per podTwo interactions to know. HPA and VPA conflict when both act on CPU or memory — VPA changes the request, which is the denominator in the HPA's ratio — so run VPA in recommendation mode when an HPA is active. And oscillation usually means a scale-down window that is too short combined with a metric that falls as soon as capacity is added.
Queue depth is the cleanest autoscaling signal there is, because it is a direct measurement of work not yet done rather than a symptom of work in progress. If a workload can be expressed as a queue, scale it on the queue.
Cluster autoscaling and Karpenter
Cluster Autoscaler scales predefined node groups. Karpenter provisions right-sized nodes from the pending pods and repacks continuously — materially better bin-packing and faster.
CLUSTER AUTOSCALER KARPENTER
scales node GROUPS (ASGs) provisions INDIVIDUAL nodes from pending pods
shapes predefined by you picks instance types from EC2 directly
~2-4 min to schedulable <60s
removes near-EMPTY nodes CONSOLIDATES: repacks onto fewer/cheaper nodes
spot per group spot diversified across many pools natively
portable across clouds AWS-focusedapiVersion: karpenter.sh/v1
kind: NodePool
spec:
template:
spec:
requirements:
- { key: kubernetes.io/arch, operator: In, values: ["arm64"] }
- { key: karpenter.sh/capacity-type, operator: In, values: ["spot","on-demand"] }
- { key: karpenter.k8s.aws/instance-family, operator: In,
values: ["m7g","c7g","r7g"] } # diversify -> spot resilience
disruption:
consolidationPolicy: WhenEmptyOrUnderutilized # <- the cost lever
consolidateAfter: 1m
limits: { cpu: "2000" } # a bug cannot provision
# the whole regionWhy nodes never scale down
Both autoscalers can only remove a node they can drain, and several things block that. A PodDisruptionBudget that cannot be satisfied. A pod with emptyDir local storage, which blocks eviction by default. A bare pod with no controller. A kube-system pod without the safe-to-evict annotation. So utilisation looks fine on paper and the cluster never shrinks — and the autoscaler logs name the blocking pod explicitly.
Do not run Cluster Autoscaler and Karpenter over the same capacity. They fight: one provisions, the other decides the node is underutilised and removes it. Migrate by letting Karpenter own new capacity while the old node groups scale down to a static baseline.
Disruption: budgets, drains and spot
A PodDisruptionBudget bounds voluntary disruption only. Too loose and a drain takes your service down; too strict and the drain hangs forever.
Voluntary disruption is anything initiated through the eviction API: kubectl drain, node upgrades, autoscaler consolidation. Involuntary is a node crashing, an OOM kill, a spot reclamation — and a PDB does nothing about those. So a budget is your safety net during maintenance, not during failure; for failure you need replicas spread across zones.
kind: PodDisruptionBudget
spec:
maxUnavailable: 1 # correct for most services
selector: { matchLabels: { app: payments-api } }
# maxUnavailable: 0 -> the drain HANGS FOREVER. The upgrade stalls and
# nothing looks "broken", which is worse.
# no PDB at all -> a drain can evict every replica at once.
# minAvailable: 2 with replicas: 2 -> also unsatisfiable. Check the arithmetic.Spot
Spot instances are up to 70–90% cheaper and can be reclaimed with a two-minute warning. That is enough time only if your pods actually shut down within it — which brings you back to preStop hooks and a terminationGracePeriodSeconds comfortably under 120. A workload with a 300-second grace period will simply be killed.
- Diversify across many instance families, sizes and zones, so one capacity pool draining does not take you out.
- Baseline on on-demand, burst on spot, so a reclamation degrades headroom rather than the service.
- Handle the notice — Karpenter consumes interruption events and cordons and drains automatically.
- Checkpoint batch work, so an interruption costs one item rather than the run.
- Never put quorum members or long non-checkpointable work on spot.
The subtlety: the reclaim itself is involuntary, but the DRAIN triggered by the two-minute warning goes through the eviction API and therefore respects the PDB - which can slow it past the window. Keep budgets permissive enough to drain quickly.
Priority, preemption and headroom
PriorityClasses let important pods evict less important ones. Combined with deliberately idle pause pods, they give you instant capacity while the autoscaler catches up.
Every pod has a priority, defaulting to zero. When a high-priority pod cannot schedule, the scheduler looks for lower-priority pods it could evict to make room, and preempts them. That is the mechanism that guarantees online services capacity when batch work is competing for it.
kind: PriorityClass
metadata: { name: online-critical }
value: 1000
---
kind: PriorityClass
metadata: { name: batch-low }
value: 100 # preempted to make room for online pods
---
kind: PriorityClass
metadata: { name: overprovisioning }
value: -10 # NEGATIVE: preempted by anything real
globalDefault: falseThe pause-pod trick
End-to-end scale-up takes one to five minutes: HPA reaction, node provisioning, image pull, JVM start. For a sharp spike that is too slow. Run a Deployment of pause pods with a negative priority class that reserve capacity and do nothing. When real pods need room, they are preempted instantly and the capacity is available immediately, while the autoscaler provisions replacements in the background.
kind: Deployment
metadata: { name: overprovisioning }
spec:
replicas: 6
template:
spec:
priorityClassName: overprovisioning
containers:
- name: pause
image: registry.k8s.io/pause:3.9
resources: { requests: { cpu: "1", memory: "2Gi" } }
# Cost: some deliberately idle nodes.
# Benefit: instant capacity. The standard answer to "autoscaling is too slow
# for our spike" that does not involve carrying peak capacity all day.Headroom has three named components and they should be quantified separately: enough to lose one AZ (~33% across three), enough for rolling-update surge, and enough to cover autoscaler lag. Reporting utilisation without them turns a reliability decision into a cost argument you will lose.
StatefulSets
Stable ordinal identity, one retained volume per replica, ordered updates. Those three are exactly what clustered software needs and what a Deployment cannot give it.
Pods are named by ordinal — redis-0, redis-1 — and that name survives rescheduling to another node. With a headless Service, each pod gets its own DNS record, which is how peers find each other. A Deployment gives random suffixes that change on every restart, so there is no stable address to put in a cluster configuration.
A volumeClaimTemplate creates one PersistentVolumeClaim per replica, bound to that ordinal and retained when the pod is deleted or rescheduled. redis-1 always gets data-redis-1.
WHAT BREAKS WITH A DEPLOYMENT AND ONE PVC
EBS is ReadWriteOnce -> pod 1 mounts it, pods 2 and 3 sit Pending forever
Shared filesystem -> all three mount it and write over each other.
WORSE, because it appears to work.
Pod names change on restart -> peers cannot find each other.
WHAT THE STATEFULSET GIVES YOU
redis-0 -> pvc data-redis-0 -> redis-0.redis.ns.svc.cluster.local
redis-1 -> pvc data-redis-1 -> redis-1.redis.ns.svc.cluster.local
rescheduled to another node -> SAME name, SAME volumeOrdered updates, and the probe that carries them
Rolling updates go one pod at a time in reverse ordinal order, waiting for each to become Ready before proceeding. That ordering is what keeps a quorum system above a majority — provided Ready means what you need it to mean. A readiness probe that only checks the port passes on a node still loading its dataset, so the controller moves on and you lose quorum during a routine upgrade.
readinessProbe:
exec:
command: ["sh","-c","redis-cli ping | grep -q PONG && \
redis-cli info replication | grep -q 'master_link_status:up'"]
# "joined the cluster and caught up", not "the port is open"
updateStrategy:
rollingUpdate: { partition: 2 } # canary ONLY redis-2, verify, then lowerDeleting a StatefulSet does NOT delete its PVCs. That is deliberate protection against a typo and it is also a silent cost leak - orphaned volumes accrue charges indefinitely. Set persistentVolumeClaimRetentionPolicy consciously either way.
Persistent volumes
A PVC is a request, a PV is the storage, a StorageClass provisions one dynamically. Access modes are a property of the technology, not a preference.
PersistentVolumeClaim the REQUEST: size, access mode, class
PersistentVolume the actual storage
StorageClass a template: which CSI driver, what parameters,
reclaim policy, binding mode
ACCESS MODES - determined by the storage technology
ReadWriteOnce one NODE may mount read-write <- EBS
ReadWriteOncePod exactly one POD
ReadOnlyMany many nodes, read-only
ReadWriteMany many nodes, read-write <- EFS, not EBSThe consequence people hit: a Deployment with three replicas and one EBS-backed PVC. All three reference the same claim; the first mounts it and the others fail with a multi-attach error. The correct shapes are a StatefulSet with a volume claim template, EFS if replicas genuinely need shared read-write, or — most often — no volume at all, because the state belongs in a database.
reclaimPolicy: Delete destroys the volume when the PVC goes.
Convenient, and a data-loss footgun if someone
deletes a namespace.
reclaimPolicy: Retain keeps it. Manual cleanup, and orphaned volumes
silently accrue cost.
allowVolumeExpansion: true set it IN ADVANCE - you cannot usefully add it
to already-provisioned volumes.
You can grow. You cannot shrink.EFS earns its cost when you genuinely need ReadWriteMany or storage that is not AZ-pinned. It is significantly more expensive per GB, higher and more variable latency, and NFS semantics that many applications handle badly - file locking especially. Never put a database on it.
Storage classes and the availability zone problem
An EBS volume lives in exactly one availability zone and cannot be mounted from another. WaitForFirstConsumer exists to stop that constraining your scheduling.
With the older Immediate binding mode, the volume is provisioned as soon as the PVC is created — before the scheduler has picked a node. It lands in an arbitrary zone, and the resulting PV carries a node affinity pinning it there. Now the volume constrains the pod: if that zone has no capacity, the pod sits Pending, and the error message is volume node affinity conflict.
WaitForFirstConsumer inverts the dependency. Provisioning is deferred until the scheduler has chosen a node, and the volume is created in that node's zone. Scheduling determines storage rather than storage constraining scheduling. Make it your default.
kind: StorageClass
metadata: { name: gp3-wait }
provisioner: ebs.csi.aws.com
volumeBindingMode: WaitForFirstConsumer # <- make this the default
reclaimPolicy: Delete # Retain for anything precious
allowVolumeExpansion: true # you WILL need to grow it
parameters:
type: gp3
iops: "3000"
throughput: "125"
encrypted: "true"
kmsKeyId: arn:aws:kms:...The residual limit
Once bound, the pod is pinned to that zone for its lifetime. Lose the zone and the pod cannot be rescheduled elsewhere with its data. That is not a Kubernetes flaw; it is what block storage means. Real availability for stateful workloads comes from application-level replication across zones — three replicas in three zones, each with its own local volume — not from expecting a volume to follow a pod.
Every stateful workload needs an explicit answer to: what happens when this pod's availability zone is lost? If the answer is 'the volume is there and the pod cannot reach it', you have a single-zone service regardless of how many replicas you run.
Jobs, CronJobs and DaemonSets
Run-to-completion work, scheduled work, and per-node work. The failure modes are stacking, unbounded accumulation, and forgetting that DaemonSets consume every node's capacity.
JOB completions | parallelism | backoffLimit | activeDeadlineSeconds
completionMode: Indexed gives each pod an ordinal ->
static partitioning without a queue
restartPolicy MUST be Never or OnFailure, never Always
(Never keeps the logs of each failed attempt - usually what you want)
CRONJOB creates Jobs on a schedule. A controller over Jobs, not a
separate mechanism.
DAEMONSET one pod per node, automatically on new nodes.
Log shippers, CNI agents, node exporters, CSI node plugins.
-> every node must have room for your DaemonSets BEFORE it has
room for workloads. This bites when right-sizing small instances.The CronJob settings that matter
spec:
schedule: "*/5 * * * *"
timeZone: "Europe/London" # otherwise UTC - and DST WILL surprise you
concurrencyPolicy: Forbid # <- STACKING FIX
startingDeadlineSeconds: 300 # <- FLOOD FIX (and avoids the >100-missed
# dead-stop where scheduling gives up)
successfulJobsHistoryLimit: 1 # <- ACCUMULATION FIX
failedJobsHistoryLimit: 3
jobTemplate:
spec:
activeDeadlineSeconds: 240 # hard stop BELOW the schedule interval
ttlSecondsAfterFinished: 600Without Forbid, a run that overruns its schedule produces a second concurrent run, then a third, compounding until the quota or the database gives out. Without ttlSecondsAfterFinished, a job every five minutes leaves 288 Job objects a day in etcd, and after a month the namespace is unusable.
Alert on ABSENCE, not on failure. The dangerous case is that nothing runs at all, which produces no failure to alert on. Use a rule on the last-successful-time metric, or a dead-man's switch where the job pings a heartbeat and you alert when the ping stops.
CRDs and operators
A CRD extends the API with your own object type. An operator is a controller that reconciles it — level-triggered, idempotent and interruptible, or it will corrupt things enthusiastically.
A CustomResourceDefinition registers a new type, so kubectl get settlementjobs works, RBAC applies, and an OpenAPI schema validates it. On its own it is inert — a place to put data.
An operator is what gives it meaning: a controller watching that type and running a reconcile loop that drives real state toward the spec. It encodes the operational knowledge a human would otherwise apply from a runbook.
THREE PROPERTIES A RECONCILE LOOP MUST HAVE
LEVEL-TRIGGERED compare CURRENT to DESIRED and act.
Not "react to an event" - a missed event must self-correct
on the next resync.
IDEMPOTENT reconciling twice does the same as reconciling once.
CreateOrUpdate, never a blind Create.
INTERRUPTIBLE the operator pod WILL be killed mid-reconcile.
State lives in the CLUSTER (status subresource, finalizers),
never in the operator's memory.When to write one
The test: is there a runbook of conditional steps a human follows at 3am? A database needing leader election, failover, backup and version upgrade — yes. Provisioning a cloud resource whose lifecycle is tied to a namespace — yes. Anything a Helm chart plus a Job can do — no. Anything where the real motivation is a nicer YAML interface — that is a templating problem.
An operator is PRIVILEGED software running permanently in your cluster, with broad RBAC and the ability to delete production data repeatedly. Prefer a mature community operator over writing your own, and treat writing one as a serious commitment rather than a weekend project.
Finalizers are how cleanup outside the cluster happens: deletion sets a timestamp instead of removing the object, the operator does its external cleanup, then removes the finalizer. They are also a classic way to wedge a namespace - if the operator is gone, the finalizer is never removed and nothing can be deleted.
Backups and cluster recovery
Three layers: infrastructure as code rebuilds the cluster, git plus GitOps restores the workloads, and Velero or database-native backups restore the data. Velero alone is not a recovery plan.
Velero snapshots API objects in selected namespaces plus volume snapshots via CSI, into object storage, and — the property that makes it a backup rather than a copy — it can restore into a different cluster. That is also the only meaningful test of it.
What it does not cover
- It should not be your source of truth. With GitOps, the desired state is in git with history and review. Restoring objects from a backup is a convenience; the authoritative path is to point the GitOps controller at the repository.
- It is not application-consistent by default. A CSI snapshot is crash-consistent, so restoring a database mid-write is a power-cut recovery. Use pre- and post-snapshot hooks, or better, the database's own mechanism.
- It does not cover cluster-level configuration — node groups, IAM roles and trust relationships, VPC, add-on versions. A restore into an empty account restores nothing, because there is no cluster to restore into.
LAYER 1 Terraform / CDK rebuild cluster + node pools + IAM + add-ons
LAYER 2 git + Argo CD reconcile ALL workloads (the real source of truth)
LAYER 3 Velero / DB-native restore the DATA
Recovery from total cluster loss, IF REHEARSED:
terraform apply ~10-20 min
bootstrap add-ons (in the IaC)
Argo reconciles workloads ~5-10 min
volume restore size-dependent
shift DNS
-> 30-60 minutes. UNBOUNDED if never rehearsed.
Velero's best everyday role is narrower and very valuable:
recovering ONE accidentally deleted namespace, in minutes.Test restores monthly, into a DIFFERENT cluster, with an automated integrity check, and record the elapsed time. That number is your measured RTO. An untested backup is a hypothesis with an unknown success rate.
RBAC
Role and ClusterRole grant verbs on resources; bindings attach them to subjects. Purely additive, no deny rules — and several permissions are secretly equivalent to reading every secret.
Role verbs on resources, in ONE namespace
ClusterRole cluster-wide, and the only way to grant access to
cluster-scoped resources (nodes, PVs, CRDs)
RoleBinding attaches either kind, scoped to ONE namespace
ClusterRoleBinding attaches cluster-wide
The useful subtlety: a RoleBinding can reference a ClusterRole, granting
those permissions in that one namespace. Define a permission SET once,
bind it per namespace, rather than duplicating a Role twelve times.
Subjects: User and Group (from the authenticator - OIDC claims, or on EKS
an IAM identity mapping) and ServiceAccount (a real object).
Effective permission = the UNION of everything bound to you. No deny.Permissions that are effectively secret-read
pods/exec,pods/attach— shell in and read the mounts and environment. Also lets you act as that workload.pods: create— mount any Secret in the namespace into a container you control.pods/log— if the application ever logs its configuration.serviceaccounts/token: create— mint a token for a more privileged identity.- Any write to a Deployment or Job — it can mount whatever it likes.
An audit that greps for the word secrets misses every one of those. Use a tool that reasons about escalation paths, and check specific subjects with kubectl auth can-i rather than reading bindings.
Never bind cluster-admin because something did not work. It is never reverted, and it makes every other control decorative. Debug with 'kubectl auth can-i --list' and grant the specific missing verb.
Workload identity
Pods authenticate as a ServiceAccount with a short-lived projected token. On EKS that token is exchanged for an IAM role, so no static AWS credential exists anywhere.
Every pod runs as a ServiceAccount and receives a projected, audience-scoped, short-lived token — a signed JWT rotated automatically, not the old permanent secret. That token authenticates it to the Kubernetes API.
To reach AWS, IRSA registers the cluster's OIDC provider in IAM; an IAM role's trust policy accepts AssumeRoleWithWebIdentity from that provider, conditioned on a specific ServiceAccount; and the SDK inside the pod finds the projected token, calls STS, and gets temporary credentials that refresh themselves. EKS Pod Identity is the newer, simpler variant — an agent on the node handles the exchange, with no OIDC provider to manage.
kind: ServiceAccount
metadata:
name: settlement-worker
namespace: payments
annotations:
eks.amazonaws.com/role-arn: arn:aws:iam::1234:role/settlement-worker
---
# IAM trust policy - note the CONDITION
{ "Effect": "Allow",
"Principal": { "Federated": "arn:aws:iam::1234:oidc-provider/oidc.eks..." },
"Action": "sts:AssumeRoleWithWebIdentity",
"Condition": { "StringEquals": {
"oidc.eks...:aud": "sts.amazonaws.com",
"oidc.eks...:sub": "system:serviceaccount:payments:settlement-worker" }}}
# omit the sub condition and ANY pod in the cluster can assume this role
---
# And for everything else:
kind: ServiceAccount
metadata: { name: default, namespace: payments }
automountServiceAccountToken: false # no API credential in pods that
# never call the APIThe single highest-value hardening step on EKS is unrelated to any of this: set the instance metadata hop limit to 1, so a container cannot reach IMDS and steal the NODE's IAM role. Combined with IRSA, every pod then has only its own narrow permissions.
NetworkPolicy
Additive allow rules with a per-pod switch: once a pod is selected by any policy for a direction, everything not allowed is denied. Start with default-deny — and remember DNS.
By default every pod can reach every pod in every namespace. NetworkPolicy changes that, and the semantics are worth stating precisely: the moment a pod is selected by any policy for a direction, everything not explicitly allowed in that direction is denied. There is no ordering, no priority, no deny rule — effective permission is the union of allows.
# 1. DEFAULT DENY - both directions, all pods
kind: NetworkPolicy
metadata: { name: default-deny, namespace: payments }
spec:
podSelector: {}
policyTypes: [Ingress, Egress]
---
# 2. THE ONE EVERYONE FORGETS - without this, ALL DNS fails
kind: NetworkPolicy
metadata: { name: allow-dns, namespace: payments }
spec:
podSelector: {}
policyTypes: [Egress]
egress:
- to:
- namespaceSelector: { matchLabels: { kubernetes.io/metadata.name: kube-system } }
podSelector: { matchLabels: { k8s-app: kube-dns } }
ports: [{ protocol: UDP, port: 53 }, { protocol: TCP, port: 53 }]
# SELECTOR SEMANTICS - getting this backwards opens more than you meant
from:
- namespaceSelector: {...}
podSelector: {...} # AND: those pods IN those namespaces
from:
- namespaceSelector: {...}
- podSelector: {...} # OR: any pod in those namespaces, OR those podsWhat it cannot do
- No L7 rules. It is IP, port and protocol — allow GET /health but not POST /transfer is out of scope.
- No identity beyond labels and CIDRs, and labels are mutable by anyone who can edit a pod spec.
- No encryption.
- No egress by DNS name, which is a real gap since allowing an external API means allowing a set of addresses that change.
Verify enforcement with a connectivity test. On EKS the VPC CNI does not enforce NetworkPolicy unless explicitly enabled - and policies that exist but are not enforced are worse than none, because they look like control.
Pod security and securityContext
Label namespaces with the restricted profile, then set five fields on every workload. Each prevents a specific escalation step, and the JVM needs one accommodation.
Pod Security admission is built in and free: label a namespace enforce: restricted and it blocks privileged containers, host namespaces, hostPath mounts, running as root and most capabilities. Use warn or audit mode first to see what would break.
securityContext: # pod level
runAsNonRoot: true
runAsUser: 10001 # explicit UID, verifiable at admission
runAsGroup: 10001
fsGroup: 10001
seccompProfile: { type: RuntimeDefault }
containers:
- securityContext: # container level
allowPrivilegeEscalation: false
readOnlyRootFilesystem: true
capabilities: { drop: ["ALL"] }
WHAT EACH PREVENTS
runAsNonRoot a container escape lands as root on the host
readOnlyRootFilesystem an attacker drops a binary and persists
allowPrivilegeEscalation=false a setuid binary gaining privileges
drop ALL capabilities the default set is far more than anything needs
seccompProfile blocks most syscalls container escapes depend onThe JVM caveat
readOnlyRootFilesystem breaks a JVM unless you give it a writable temp directory. The JVM writes hsperfdata for tooling, Tomcat needs a work directory, and anything using temp files needs somewhere to put them. Mount an emptyDir at /tmp and set java.io.tmpdir, or disable perf data. Knowing this specific failure is a good signal you have actually applied it to a Java service.
env:
- { name: JAVA_TOOL_OPTIONS, value: "-Djava.io.tmpdir=/tmp -XX:MaxRAMPercentage=70" }
volumeMounts:
- { name: tmp, mountPath: /tmp }
volumes:
- { name: tmp, emptyDir: { sizeLimit: 256Mi } }Migration order: run in warn mode, collect the list of what would break, fix the images (non-root user, writable mounts, ports above 1024), then enforce. Going straight to enforce breaks deploys and makes the whole effort unpopular.
Admission control and policy
Admission runs after authorisation and before persistence — the last place to reject an object and the only place to change it. Enforce your rules here, not in documentation.
Mutating webhooks run first and can modify the object: injecting a sidecar, defaulting labels, adding a securityContext. Validating webhooks run after all mutations and can only accept or reject, so they see the final object. Kubernetes also ships validating admission policies in-tree, written in CEL, which need no webhook pod at all.
POLICIES WORTH ENFORCING IN A PAYMENTS CLUSTER
images only from our ECR, pinned by DIGEST -> no unreviewed images, no drift
a valid signature from our CI identity -> makes signing MEAN something
resource requests and limits required -> no BestEffort, no guesswork
no privileged, hostNetwork, hostPID, hostPath -> container escape vectors
runAsNonRoot + readOnlyRootFilesystem
a team / cost-centre label on every workload -> unglamorous, and the only
way cost attribution worksThe operational trap
A webhook sits on the critical path of every matching API write. With failurePolicy: Fail, if the webhook pod is down, unreachable or its certificate has expired, every matching create and update in the cluster is rejected — including, potentially, the deployment that would fix it. Teams have locked themselves out of their own clusters this way.
- Always exempt
kube-systemand the policy controller's own namespace. - Scope the rules narrowly to specific resources and operations rather than matching everything.
- Run multiple replicas with a PDB and a short
timeoutSeconds. - Roll new policies out in audit mode first, measure what they would have blocked, then enforce.
A rule in a wiki is advisory. A rule at admission is a rule. If you find yourself writing the same review comment repeatedly, that comment should become a policy.
Observability
stdout to node files tailed by a DaemonSet; metrics pulled by Prometheus; traces from an injected agent with tail sampling. And one trace id in every log line.
Container logs are a runtime feature: whatever the process writes to stdout and stderr is captured to a file on the node. kubectl logs reads that file and serves only the current container instance plus one previous — so a pod that has crash-looped six times has lost the earlier evidence, and a deleted pod's logs are gone entirely.
app -> stdout -> /var/log/pods/... -> Fluent Bit DaemonSet -> backend
WHY A DAEMONSET: the files are NODE-local. One shipper per node.
WHY NOT A FILE INSIDE THE CONTAINER:
1. invisible - the shipper tails /var/log/pods, not your filesystem
2. fills ephemeral storage -> the pod is EVICTED for disk pressure
(a confusing incident: nothing crashed, pods just vanish)
3. gone on restart, exactly when you need it
# Structured JSON, one line per event, and DROP DEBUG AT THE SHIPPER
# (never pay to ingest what you will not read)
logging.structured.format.console: ecsMetrics are pull-based: Prometheus scrapes each pod's endpoint, discovered through the API. Keep label cardinality bounded — a label carrying a user id or request id will kill a Prometheus long before it tells you anything. And metrics-server is not monitoring: it provides current CPU and memory for kubectl top and the HPA, with no history.
Traces come from the OpenTelemetry agent, injected by the operator with a single pod annotation so twelve services get tracing with no code change. The collector does the expensive thinking: scrubbing PII attributes and tail sampling — keep 100% of errors and slow traces plus a small baseline, or tracing costs more than the services being traced.
Put the trace id in every log line. Then a slow span leads to exactly the log lines emitted during it, and an error log leads to the full trace. That one field is worth more than any single signal on its own.
Debugging a cluster
Read the events, read the exit code, and use --previous. Those three cover most incidents, and skipping them to guess is the most common way an investigation takes an hour instead of five minutes.
CrashLoopBackOff - the kubelet has restarted it repeatedly and is backing off.
The cause is elsewhere.
$ kubectl describe pod api-7d9f
Last State: Terminated
Reason: OOMKilled -> memory limit too low
Exit Code: 137
Events:
FailedMount secret "payments-db" not found -> wrong namespace
$ kubectl logs api-7d9f --previous # <- THE STEP PEOPLE SKIP
# the CURRENT container may be 3 seconds old with no output.
# --previous shows the instance that actually died, with the stack trace.
EXIT CODE MEANING LOOK AT
137 SIGKILL OOMKilled: true? -> memory limit
else -> liveness probe killed it
143 SIGTERM, handled normal shutdown - why was it told to?
1 application error logs --previous
0 + restart exited SUCCESSFULLY wrong command/args, or a batch job
deployed as a DeploymentPENDING never an application problem. Read the FailedScheduling event.
RUNNING, 0/1 readiness failing. Port-forward and read the response body.
502 AT INGRESS between ingress and pod. Check the controller log for the
upstream error, then compare Endpoints against expected pods.
A subset failing points at a partial rollout.
$ kubectl debug -it api-7d9f --image=busybox --target=api --share-processes
# ephemeral container sharing the pod's namespaces - the answer to
# "the image is distroless and I cannot exec into it"
$ kubectl get events -n payments --sort-by=.lastTimestamp | tail -20
# and always ask: WHAT CHANGED? A pod that ran for a month and now fails
# points at something external - a rotated secret, a dependency, a node.At 3am: restore service, then understand it. Rolling back is fast, safe (the previous ReplicaSet is retained) and eliminates one variable in about a minute. Capture a describe and the logs first, because that evidence disappears with the pods.
Upgrades, GitOps and running it
Upgrade control plane, add-ons, then nodes — never skipping a minor. Deploy through git rather than kubectl. And accept that the add-on matrix is where the real work is.
Upgrades
ORDER control plane -> add-ons -> node groups (kubelet may be BEHIND the
API server, never ahead)
THE CHECK THAT MATTERS MOST: removed APIs
$ pluto detect-all-in-cluster
$ kubent
# also check HELM RELEASE HISTORY - an old release secret can carry a
# removed apiVersion and break the next upgrade
PDBs ARE LOAD-BEARING IN BOTH DIRECTIONS
none -> a drain evicts every replica at once
maxUnavailable:0-> the drain HANGS FOREVER and nothing looks broken
DRAIN SURPRISES
emptyDir volumes block eviction (annotate safe-to-evict)
bare pods are evicted and NEVER rescheduled
replicas: 1 has no safe drain - scale to 2 first
a webhook with failurePolicy: Fail, whose pods you just drained,
rejects EVERY write including the ones needed to recoverGitOps
Push-based CI holds cluster credentials, so CI becomes a highly privileged system and a manual kubectl change silently diverges forever. Pull-based inverts it: CI writes desired state to a repository and an in-cluster controller reconciles. CI needs no cluster credentials, drift is corrected automatically, and git history is the deployment record — with author, reviewer, timestamp and diff, which is stronger evidence than any API log because it shows intent and approval.
kind: Application # Argo CD
spec:
source: { repoURL: ..., targetRevision: main, path: services/payments/overlays/prod }
destination: { server: https://kubernetes.default.svc, namespace: payments }
syncPolicy:
automated:
prune: true # removed from git -> deleted from the cluster
selfHeal: true # manual kubectl change -> REVERTED
# Two independent status axes:
# Synced / OutOfSync does live state match git?
# Healthy / Degraded is the workload actually working?
# OutOfSync + Healthy = someone changed the cluster by hand
# Synced + Degraded = git is right and the app is broken -> your bugDeployment permission becomes MERGE permission on a repository, governed by branch protection and review rules you already have. That moves access control from cluster RBAC - which is awkward to reason about - to code review, which everyone understands.
Upgrade every four to six months and stay in the supported window. Falling behind is the expensive path: skipped deprecations accumulate and each upgrade gets larger. Frequent small upgrades are dramatically less risky than infrequent large ones.