Lab: A Deployment That Will Not Stay Up

The situation

At 14:05 a deployment of payments-api went out. It contained a configuration change enabling a new fraud-scoring model. The image tag also changed, as it does on every deploy.

By 14:12 the service was down. Every replica is crash-looping.

The application team says the change is trivial and the same image works in staging.

What you have

NAME READY STATUS RESTARTS AGEpayments-api-6b4d9c8f7-2xk9p 0/1 CrashLoopBackOff 7 (42s ago) 11mpayments-api-6b4d9c8f7-8vn2q 0/1 CrashLoopBackOff 7 (31s ago) 11mpayments-api-6b4d9c8f7-mq4rt 0/1 CrashLoopBackOff 7 (55s ago) 11m

Previous container logs

{“ts”:“2026-08-24T14:11:02Z”,“level”:“info”,“msg”:“starting payments-api v2.14.0”}{“ts”:“2026-08-24T14:11:02Z”,“level”:“info”,“msg”:“config loaded”,“source”:“/etc/config”}{“ts”:“2026-08-24T14:11:03Z”,“level”:“info”,“msg”:“database pool established”,“size”:20}{“ts”:“2026-08-24T14:11:03Z”,“level”:“info”,“msg”:“cache connected”}{“ts”:“2026-08-24T14:11:04Z”,“level”:“info”,“msg”:“loading fraud model”,“file”:“fraud-v4.bin”}(no further output)

No error. No stack trace. The log just stops.

Terminated state

$ kubectl get pod payments-api-6b4d9c8f7-2xk9p -n payments \ -o jsonpath=‘{.status.containerStatuses[*].lastState.terminated}’ | jq .{ “exitCode”: 137, “reason”: “Error”, “startedAt”: “2026-08-24T14:11:02Z”, “finishedAt”: “2026-08-24T14:11:32Z”}

Events

Type Reason Age Message–– —— –– —––Normal Pulled 11m Container image already presentNormal Created 11m Created container payments-apiNormal Started 11m Started container payments-apiWarning Unhealthy 10m (x3 over 11m) Liveness probe failed: Get “http://10.42.3.17:8080/healthz”: dial tcp: connect: connection refusedNormal Killing 10m Container failed liveness probe, will be restarted

The workload spec

livenessProbe:
  httpGet:
    path: /healthz
    port: 8080
  initialDelaySeconds: 10
  periodSeconds: 10
  failureThreshold: 3

readinessProbe:
  httpGet:
    path: /healthz
    port: 8080
  initialDelaySeconds: 5
  periodSeconds: 5

resources:
  requests: { cpu: 500m, memory: 1Gi }
  limits:   { cpu: 2,    memory: 2Gi }

Stop and work it

Write down:

1. What is exitCode: 137 with reason: Error and no OOMKilled telling you?
2. Do the timestamps support or contradict an application crash?
3. What single fact in the evidence identifies the cause?

---

Working the evidence

Exit code 137 without OOMKilled

137 is 128 + 9: killed by SIGKILL. The container did not exit on its own — something killed it.

If it had been the memory limit, reason would be OOMKilled. It is Error, so the kill came from somewhere else.

That single field eliminates “the application crashed”. You are now looking for what did the killing.

The timestamps

startedAt:   14:11:02
finishedAt:  14:11:32
                  ──── exactly 30 seconds

Thirty seconds, consistently, across every restart. Application crashes are rarely that punctual. A fixed interval points at something on a timer.

Compare it to the liveness probe:

initialDelaySeconds: 10     probing starts at t+10
periodSeconds: 10           checks at t+10, t+20, t+30
failureThreshold: 3         three failures → kill at t+30

The probe configuration predicts a kill at exactly thirty seconds. The container died at exactly thirty seconds.

The last log line

{"msg":"loading fraud model","file":"fraud-v4.bin"}

The last thing the application did was start loading a model file — and then nothing. Not a crash: it was still working. It had not yet reached the point of binding its HTTP listener, which is why the probe got connection refused rather than an error response.

connection refused is the important detail. It means nothing was listening on the port. An application that had started and then broken would return a 500 or time out. Refused means the listener does not exist yet.

Why staging worked

The new fraud model is larger than the previous one. In staging, with a smaller dataset and a warm local cache, it loads in under ten seconds. In production it takes longer than thirty.

The application is entirely healthy. It is being killed for not finishing its startup within a window that was sized for the previous model.

The fix

Add a startupProbe. It suspends the liveness and readiness probes until the application has started, then hands over.

startupProbe:
  httpGet:
    path: /healthz
    port: 8080
  failureThreshold: 30
  periodSeconds: 10          # allows up to 5 minutes to start

livenessProbe:
  httpGet:
    path: /healthz
    port: 8080
  periodSeconds: 10          # no initialDelaySeconds needed now
  failureThreshold: 3

readinessProbe:
  httpGet:
    path: /ready
    port: 8080
  periodSeconds: 5

Why not just raise initialDelaySeconds?

It would stop the crash loop, and it is the wrong fix.

initialDelaySeconds: 120 on the liveness probe means that for the whole two minutes, a genuinely hung process is also not detected. You have traded away your ability to notice a real hang in order to accommodate a slow start.

A startupProbe separates the two concerns: generous while starting, strict once started. That is exactly what the three-probe design is for.

Validation

# 1. Rollout completes
kubectl rollout status deployment/payments-api -n payments --timeout=300s

# 2. Restart count stops moving -- check, wait, check again
kubectl get pods -n payments -l app=payments-api \
  -o custom-columns='POD:.metadata.name,RESTARTS:.status.containerStatuses[0].restartCount'
sleep 300
kubectl get pods -n payments -l app=payments-api \
  -o custom-columns='POD:.metadata.name,RESTARTS:.status.containerStatuses[0].restartCount'

# 3. Endpoints match replica count
kubectl get endpoints payments-api -n payments -o jsonpath='{.subsets[*].addresses[*].ip}' | wc -w

# 4. No new Unhealthy events
kubectl get events -n payments --field-selector reason=Unhealthy --sort-by=.lastTimestamp | tail

Then measure the actual startup time, so the failureThreshold is set from data rather than guessed:

kubectl logs <pod> -n payments | head -1     # first log line timestamp
# compare to the first successful readiness probe

What to take away

lastState.terminated before anything else. Exit code and reason together classify most failures in one command. 137 with OOMKilled and 137 without are different incidents.

Regular timing means something on a timer. Consistent intervals between restarts point at probes, not at application logic.

connection refused means not started. Not broken — not yet listening. That distinction separates “still starting” from “started and failed”.

Slow startup is a configuration problem, not an application problem. When startup time changes, the probe configuration has to change with it. This is worth checking on any deploy that changes what an application loads at boot.