Production Kubernetes Operations — Resources, Probes, and Rollouts

Kubernetes will run almost anything you give it. Whether that thing survives a node going away, a rollout, or a memory spike is decided by roughly six settings, and the defaults for most of them are wrong for production.

Requests and limits are two different things

This is the most consequential misunderstanding in Kubernetes operations.

• Requests are used for scheduling. The scheduler places a pod on a node with enough unreserved capacity to satisfy its requests. Requests reserve capacity.
• Limits are enforced at runtime. Exceeding a CPU limit throttles the process; exceeding a memory limit kills it.

The gap between them defines a workload’s QoS class, and QoS class decides who dies first when a node runs out of memory:

Requests and limits	QoS class	Eviction order
Not set	BestEffort	Killed first
Set, and different	Burstable	Killed second, worst offender first
Set, and identical	Guaranteed	Killed last

For anything that matters, set both. For anything critical, set them equal.

Three probes, three jobs

They are frequently configured identically, which defeats the purpose of having three.

startupProbe:            # "has it finished starting?"
  httpGet: { path: /healthz, port: 8080 }
  failureThreshold: 30
  periodSeconds: 10      # up to 5 minutes, then give up

readinessProbe:          # "should it receive traffic right now?"
  httpGet: { path: /ready, port: 8080 }
  periodSeconds: 5
  failureThreshold: 3

livenessProbe:           # "is it wedged and in need of a restart?"
  httpGet: { path: /healthz, port: 8080 }
  periodSeconds: 10
  failureThreshold: 3

startupProbe suspends the other two until the application has started. It exists so you can allow a slow start without also allowing a hung process to run for minutes before being noticed. Anything that loads a model, warms a cache, or runs migrations needs one.

readinessProbe controls Service endpoint membership. Failing it removes the pod from load balancing without restarting it — which is exactly right for a pod waiting on a dependency. It should check whether this pod can serve, not whether the whole system is healthy.

livenessProbe restarts the container when it fails. It should be cheap, local, and check only that the process is responsive. A liveness probe that checks a database will restart every pod in the deployment when the database has a bad minute, turning a degradation into an outage.

Rollouts

strategy:
  type: RollingUpdate
  rollingUpdate:
    maxUnavailable: 0     # never drop below the desired count
    maxSurge: 1           # add one, then remove one
minReadySeconds: 10       # a pod must stay ready before counting as ready
progressDeadlineSeconds: 600

maxUnavailable: 0 costs one pod’s worth of extra capacity during a rollout and guarantees you never serve from fewer replicas than intended. For most production services that is the right trade.

minReadySeconds prevents a pod that passes readiness once and then falls over from being counted as a successful step. Without it, a broken rollout can proceed to completion because each new pod was briefly ready.

Graceful shutdown needs cooperation between the pod and the service:

terminationGracePeriodSeconds: 60
lifecycle:
  preStop:
    exec:
      command: ["sleep", "5"]

The preStop sleep exists because endpoint removal and SIGTERM happen concurrently. Without a short pause, the process starts shutting down while load balancers are still sending it requests. Five seconds of doing nothing eliminates a whole class of dropped-connection reports during deploys.

Disruption budgets

A PodDisruptionBudget constrains voluntary disruptions — node drains, cluster upgrades, autoscaler scale-down. It does not protect against a node failing.

apiVersion: policy/v1
kind: PodDisruptionBudget
spec:
  minAvailable: 2
  selector:
    matchLabels: { app: api }

Without one, a routine node drain during a cluster upgrade can evict every replica of a service at once, because nothing told the eviction API not to.

Spreading

By default the scheduler can place every replica on one node. A three-replica deployment on one node has the availability of a single instance and the cost of three.

topologySpreadConstraints:
  - maxSkew: 1
    topologyKey: topology.kubernetes.io/zone
    whenUnsatisfiable: ScheduleAnyway
    labelSelector:
      matchLabels: { app: api }
  - maxSkew: 1
    topologyKey: kubernetes.io/hostname
    whenUnsatisfiable: DoNotSchedule
    labelSelector:
      matchLabels: { app: api }

DoNotSchedule on hostname enforces one replica per node. ScheduleAnyway on zone prefers even zone distribution but does not block scheduling when a zone is unavailable — which is what you want during the incident where a zone is unavailable.

Reading a cluster during an incident

# What is the cluster complaining about, most recent last?
kubectl get events -A --sort-by=.lastTimestamp | tail -40

# Which pods are not healthy?
kubectl get pods -A --field-selector=status.phase!=Running

# What is actually allocated on this node, versus what is requested?
kubectl describe node <node> | sed -n '/Allocated resources/,/Events/p'

# Is anything being evicted, and why?
kubectl get events -A --field-selector reason=Evicted

# Rollout stuck? This tells you which replica set is not progressing
kubectl rollout status deployment/<name> --timeout=30s
kubectl describe deployment <name> | sed -n '/Conditions/,/Events/p'

kubectl describe node showing allocated requests near 100% while actual usage is low means the cluster is full of over-requested workloads. That is a capacity problem you solve by fixing requests, not by adding nodes.

Namespace guardrails

apiVersion: v1
kind: LimitRange
metadata: { name: defaults }
spec:
  limits:
    - type: Container
      default: { memory: 512Mi }
      defaultRequest: { cpu: 100m, memory: 256Mi }
      max: { memory: 8Gi }

A LimitRange gives every container in the namespace a default request, which stops unconfigured workloads landing in BestEffort and being killed first. A ResourceQuota on top prevents one namespace consuming the cluster.

These are the highest-leverage settings in a multi-team cluster, because they set a floor without requiring every team to get it right.

Checklist

• Memory requests and limits set on every container; equal for critical workloads
• CPU requests set; CPU limits omitted unless there is a specific reason
• startupProbe sized to real startup time
• readinessProbe checks only this pod’s ability to serve
• livenessProbe is local and cheap, and checks no dependency
• maxUnavailable: 0 and a non-zero minReadySeconds for production services
• terminationGracePeriodSeconds longer than the longest request, with a preStop pause
• PodDisruptionBudget with at least one pod of headroom
• Topology spread across nodes, and across zones where they exist
• LimitRange defaults in every namespace