Diagnosing Kubernetes Workload Failures — Pending, CrashLoop, and Evicted

Three commands answer most of these, and the order matters:

kubectl describe pod <pod> | sed -n '/Events:/,$p'
kubectl logs <pod> --previous --tail=100
kubectl get events -n <ns> --sort-by=.lastTimestamp | tail -30

Events tell you what the control plane decided and why. --previous logs tell you what the dead container said. Status alone tells you almost nothing.

Pending — it will not schedule

The pod exists and no node has accepted it. describe names the reason.

$ kubectl describe pod api-7d9f -n prod | sed -n ‘/Events:/,$p’Events: Type Reason Message –– —— —–– Warning FailedScheduling 0/12 nodes are available: 3 Insufficient memory, 2 node(s) had untolerated taint {gpu: true}, 7 node(s) didn’t match Pod’s node affinity/selector.

That message is a complete answer if you read it as arithmetic: the numbers add up to the cluster size, and each names the predicate that rejected those nodes.

Insufficient <resource> — no node has enough unreserved capacity for the request. Note that this is about requests, not usage. A cluster can be 20% utilised and still unable to schedule, because requests reserve capacity whether or not it is used.

# Requested versus capacity, per node
kubectl describe node <node> | sed -n '/Allocated resources/,/Events/p'

Untolerated taint — the nodes are reserved and your pod lacks the toleration. Intentional most of the time.

Didn’t match node affinity/selector — your selector matches no node. Check the labels actually present:

kubectl get nodes --show-labels

Insufficient nvidia.com/gpu on a cluster with free GPUs is a special case worth knowing: it usually means the device plugin is not advertising the resource, not that GPUs are in use.

kubectl get nodes -o custom-columns='NODE:.metadata.name,GPU:.status.allocatable.nvidia\.com/gpu'
# Empty column → device plugin problem, not a capacity problem

Volume node affinity conflict — the pod needs a volume that exists in one zone, and the nodes that could take the pod are in another. Common with zonal block storage.

CrashLoopBackOff — it will not stay up

CrashLoopBackOff is not a cause. It is Kubernetes telling you it has given up restarting quickly. The cause is in the previous container.

kubectl logs <pod> --previous --tail=100
kubectl get pod <pod> -o jsonpath='{.status.containerStatuses[*].lastState.terminated}'

The terminated object gives you exitCode, reason, and finishedAt together, which is usually enough to classify it immediately:

• reason: OOMKilled — memory limit too low. If it happens during startup the limit is below the initialisation requirement; if under load, below the working set.
• exitCode: 1 with application logs — an application failure. Missing configuration, an unreachable dependency, a bad migration.
• exitCode: 0 — the process completed. Either it is not staying in the foreground, or it is a Job’s pod being treated as a long-running service.
• exitCode: 137 without OOMKilled — it ignored SIGTERM and was killed after the grace period.
• Logs end mid-startup with no error — a probe is killing it before it becomes ready. Add a startupProbe.

That last one is the most misdiagnosed failure in Kubernetes. The logs look like a normal startup that simply stops, there is no error anywhere, and the application is completely fine.

Container startup failures covers the image-level causes in more depth.

Evicted — the node pushed it out

Eviction is a node-level decision under resource pressure, not a scheduling decision.

kubectl get events -A --field-selector reason=Evicted --sort-by=.lastTimestamp
kubectl describe node <node> | grep -A5 Conditions

Look at the node conditions. MemoryPressure, DiskPressure, or PIDPressure being True identifies which resource ran out. The kubelet then evicts pods in QoS order — BestEffort first, then Burstable in order of how far they exceed their requests.

The most common underlying cause is DiskPressure from image and log accumulation rather than from application data. ephemeral-storage requests and limits are the control for that, and almost nobody sets them.

Pods with no resource requests are BestEffort and are always killed first. If your workload is being evicted repeatedly, setting requests is usually the fix — it moves it out of the queue.

ImagePullBackOff — it cannot get the image

kubectl describe pod <pod> | grep -A5 'Failed'

The message distinguishes the causes clearly:

• not found — wrong name or tag; check for a typo or a tag that was never pushed
• unauthorized / denied — missing or wrong imagePullSecrets, or the secret is in the wrong namespace. Pull secrets are namespaced and do not follow the pod
• no such host / timeout — the node cannot reach the registry. Network or DNS, not Kubernetes
• manifest unknown for a digest — the image was garbage collected from the registry

The path through

Pod is not working.

├─ Pending
│   └─ describe → FailedScheduling message
│       ├─ Insufficient <resource> → requests vs allocatable
│       ├─ untolerated taint       → intentional, or add toleration
│       ├─ affinity/selector       → check node labels
│       └─ volume node affinity    → zone mismatch
│
├─ ContainerCreating (stuck)
│   └─ describe → volume mount or secret/configmap not found
│
├─ ImagePullBackOff
│   └─ describe → name, auth, or reachability
│
├─ CrashLoopBackOff
│   └─ lastState.terminated
│       ├─ OOMKilled     → memory limit
│       ├─ exit 1        → application failure, read --previous logs
│       ├─ exit 0        → not staying in foreground
│       ├─ exit 137      → ignored SIGTERM
│       └─ logs stop mid-startup, no error → probe killing it
│
└─ Evicted
    └─ node conditions → which pressure, then QoS class of the victim

Confirming a fix

# Restart count stops climbing
kubectl get pod <pod> -o jsonpath='{.status.containerStatuses[*].restartCount}'

# Rollout completes rather than timing out
kubectl rollout status deployment/<name> --timeout=120s

# No new events for this workload
kubectl get events -n <ns> --field-selector involvedObject.name=<pod> --sort-by=.lastTimestamp

Running is not confirmation. A pod can be Running and restarting every ninety seconds. The restart count holding steady under normal load is the signal.