Triaging a Pod in CrashLoopBackOff

Purpose

Identify why a pod is repeatedly crashing and restore it to a stable running state.

Impact

High. Capacity is reduced for the affected workload and a rollout will not complete. If every replica is affected, the service is down.

Preconditions

• kubectl access with permission to read pods, events, and logs in the namespace
• Permission to edit the workload if remediation requires a change

Safety considerations

Every command in the diagnosis section is read-only. Do not delete the pod before capturing its previous logs — deleting it discards the evidence you need.

Initial checks

NS=<namespace>; POD=<pod>

# Capture the evidence before doing anything else
kubectl logs "$POD" -n "$NS" --previous --tail=200 > /tmp/crashloop-previous.log
kubectl describe pod "$POD" -n "$NS" > /tmp/crashloop-describe.txt

# The single most informative field
kubectl get pod "$POD" -n "$NS" -o jsonpath='{.status.containerStatuses[*].lastState.terminated}' | jq .

That last command returns exitCode, reason, startedAt, and finishedAt together. Almost every case is classified from it.

Decision tree

lastState.terminated.reason / exitCode

├─ reason: OOMKilled          → memory limit. Diagnosis A.
├─ exitCode: 1 + app errors   → application failure. Diagnosis B.
├─ exitCode: 0                → not staying in foreground. Diagnosis C.
├─ exitCode: 137, not OOM     → ignored SIGTERM. Diagnosis D.
├─ exitCode: 126 / 127        → image problem. Diagnosis E.
└─ logs end mid-startup, no error, no useful exit code
                              → probe killing it. Diagnosis F.

Diagnosis A — OOMKilled

The container exceeded its memory limit.

kubectl get pod "$POD" -n "$NS" -o jsonpath='{.spec.containers[*].resources}' | jq .

# How much was it actually using before it died?
kubectl top pod "$POD" -n "$NS" --containers 2>/dev/null

Compare finishedAt to startedAt. Death within seconds of start means the limit is below the initialisation requirement. Death after minutes or hours under load means the limit is below the working set.

Check whether the runtime is container-aware. A JVM or similar that sizes its heap from host memory rather than the cgroup limit will consistently exceed a limit it cannot see.

Diagnosis B — application error

grep -iE 'error|fatal|exception|refused|denied|timeout' /tmp/crashloop-previous.log | head -30

This is an application failure, not a platform one. The usual causes are a missing environment variable, an unreachable dependency, a bad configuration value, or a failed migration.

Confirm the configuration the pod actually received:

kubectl get pod "$POD" -n "$NS" -o jsonpath='{.spec.containers[*].env}' | jq .
kubectl get configmap,secret -n "$NS"

A ConfigMap or Secret that was updated after the pod started will not be reflected in environment variables — those are set at container start.

Diagnosis C — exit code 0

The process completed successfully and the container ended. Either the command daemonises, or a Job’s pod is being run as a Deployment.

kubectl get pod "$POD" -n "$NS" -o jsonpath='{.spec.containers[*].command} {.spec.containers[*].args}'

See container startup failures for the image side of this.

Diagnosis D — exit 137 without OOMKilled

The container was sent SIGTERM, did not exit within the grace period, and was killed.

Usually shell-form CMD or ENTRYPOINT: the shell is PID 1 and does not forward the signal.

kubectl get pod "$POD" -n "$NS" -o jsonpath='{.spec.containers[*].command}'
# A shell string rather than an array is the tell

Diagnosis E — exit 126 or 127

The entrypoint is not executable, not present, or missing a shared library.

kubectl describe pod "$POD" -n "$NS" | grep -A5 'Last State'

This is an image problem and needs a rebuild, not a manifest change.

Diagnosis F — killed by a probe

The signature: logs show a normal startup sequence that stops partway with no error, and restarts occur at a regular interval matching the probe configuration.

kubectl get pod "$POD" -n "$NS" -o jsonpath='{.spec.containers[*].livenessProbe}' | jq .
kubectl get pod "$POD" -n "$NS" -o jsonpath='{.spec.containers[*].startupProbe}' | jq .

# Events will show Unhealthy entries before each kill
kubectl get events -n "$NS" --field-selector involvedObject.name="$POD" \
  --sort-by=.lastTimestamp | tail -20

Unhealthy events followed by Killing confirms it.

Remediation

A — memory limit

kubectl set resources deployment/<name> -n "$NS" \
  --limits=memory=2Gi --requests=memory=2Gi

Set request and limit equal for a workload that must not be evicted. Base the number on observed usage plus headroom, not on a guess.

B — application error

Fix the configuration or restore the dependency. If a recent deploy introduced it, roll back first and diagnose afterwards:

kubectl rollout undo deployment/<name> -n "$NS"

C — foreground

Fix the image so the process runs in the foreground, or change the workload type to Job if it is genuinely meant to run to completion.

D — signal handling

Rebuild the image with exec-form ENTRYPOINT. As a stopgap, raising terminationGracePeriodSeconds reduces the impact but does not fix it — the process still never receives SIGTERM.

F — probe

startupProbe:
  httpGet: { path: /healthz, port: 8080 }
  failureThreshold: 30
  periodSeconds: 10        # allows 5 minutes to start

kubectl patch deployment <name> -n "$NS" --type=strategic -p '
spec:
  template:
    spec:
      containers:
        - name: <container>
          startupProbe:
            httpGet: { path: /healthz, port: 8080 }
            failureThreshold: 30
            periodSeconds: 10'

Add a startupProbe rather than loosening the liveness probe. Loosening liveness leaves you unable to detect a genuinely hung process later.

Validation

# 1. Rollout completes rather than timing out
kubectl rollout status deployment/<name> -n "$NS" --timeout=180s

# 2. Restart count stops increasing -- check twice, minutes apart
kubectl get pods -n "$NS" -l app=<label> \
  -o custom-columns='POD:.metadata.name,RESTARTS:.status.containerStatuses[0].restartCount'
sleep 300
kubectl get pods -n "$NS" -l app=<label> \
  -o custom-columns='POD:.metadata.name,RESTARTS:.status.containerStatuses[0].restartCount'

# 3. Endpoints match the expected replica count
kubectl get endpoints <service> -n "$NS" -o jsonpath='{.subsets[*].addresses[*].ip}' | wc -w

# 4. No new Unhealthy or Killing events
kubectl get events -n "$NS" --field-selector involvedObject.name="$POD" | tail

Rollback

Every remediation here is reversible.

# Revert the most recent change to the workload
kubectl rollout undo deployment/<name> -n "$NS"

# Or to a specific known-good revision
kubectl rollout history deployment/<name> -n "$NS"
kubectl rollout undo deployment/<name> -n "$NS" --to-revision=<n>

Capture the current spec before any change so you can restore it exactly:

kubectl get deployment <name> -n "$NS" -o yaml > /tmp/deployment-before.yaml

Escalation

Escalate to the application team for Diagnosis B and E — those require a code or image change.

Escalate to platform engineering if:

• Pods across multiple unrelated workloads begin crash-looping at the same time, which points to a node or cluster-level cause rather than an application one
• Memory limits are correct and OOM kills continue, which may indicate node memory pressure or a cgroup accounting problem
• Restarts continue after the specific cause has been addressed and validated

Include the captured --previous logs, the describe output, and the lastState.terminated object.