Kubernetes gotchas we keep hitting (and the fixes)

This is the list I wish I'd had when we started running real workloads. Nothing here is exotic — that's the point. These are the mistakes that cost us days, that everyone swears won't happen to them, and that are almost always boring to fix.

The DNS timeout that wasn't

Intermittent DNS timeouts, only in production, only on some nodes. We tuned timeouts, then resolv.conf, then gave up and bought more retries. The actual cause was the host node's nscd being out of date after a package upgrade, poisoning answers for a while and then dropping them.

Turns out "DNS problem in the cluster" is usually a DNS problem somewhere, and it's not always under the cluster.

The readiness probe race

Traffic started 502ing right after deploys, for a few seconds, reliably. The app's readiness probe passed as soon as the HTTP server was up — but the worker pool that actually handles requests takes longer to warm. The kubelet marked it ready, the Service sent traffic, and the app dropped it.

# the fix: make readiness mean "ready", not "listening"
readinessProbe:
  httpGet:
    path: /-/ready   # becomes functional only after warmup
  initialDelaySeconds: 5

Readiness probes should answer the question "can this pod serve traffic right now?", not "is the process alive?". Those are two different probes.

Request limits that were quotas

We set a CPU request equal to the limit because it seemed tidy. Then we added a second container to the pod. The scheduler, correctly, made room by… evicting nodes during spikes, because the first container had been stranded by its own generous request.

Requests and limits are different tools. A limit caps blast radius; a request is a promise to the scheduler. Treating them as one knob makes small surprises into big ones.

Image pull policy at rest

AKA the "it worked on my machine" of cluster deploys. The manifest didn't set imagePullPolicy, and the running container had the old image because the tag was the same and no one updated. The cluster was never wrong — it applied the policy it was given.

Pin tags to immutable versions, or set Always, or both. Silent image drift is the kind of bug that survives a restart.

The rule underneath all of these

Every item above was a failure to be explicit. Kubernetes does what you tell it, not what you mean — and the gap between those two is where the interesting bugs live. When in doubt, state the expectation in the manifest.