Zero-Downtime Deployments with Kubernetes
Understand readiness, surge capacity, graceful shutdown, and telemetry without treating rolling updates as an availability guarantee.
Diagnose Kubernetes CrashLoopBackOff by reading Waiting state, exit codes, kubectl logs --previous, and env config—then fix the cause instead of restarting the Pod.
Muted looping animation of an illustrative kubectl investigation: CrashLoopBackOff with a high restart count, describe output, previous logs showing a missing DATABASE_URL, a Secret-backed fix, and a healthy rollout.
CrashLoopBackOff is a Waiting-state reason shown while kubelet delays the next restart after a container has exited repeatedly. It is a symptom of a restart loop, not the root cause.
A Pod can look “stuck” in CrashLoopBackOff while the real problem is elsewhere: a missing environment variable, a bad command, an application panic, an OOM kill, or a probe that keeps killing a still-starting process. The waiting reason tells you Kubernetes is applying restart back-off. Your job is to find why the container keeps exiting or being restarted.
This guide walks an illustrative api Deployment through diagnosis and repair. Pod names, restart counts, and log lines are educational example data—not a personal incident, client story, or production metric claim.
Deployment Pods use restartPolicy: Always. When a container exits, kubelet starts it again according to that policy. After repeated failures, kubelet waits between attempts with an exponential back-off.
Documented defaults many operators still reason about are an initial delay around 10s, doubling on each failure, capped at five minutes, and a reset after the container has run successfully for about 10 minutes. Timing is not universal: Kubernetes version, kubelet configuration, and feature gates (including tunable or reduced CrashLoopBackOff curves in newer releases) can change the schedule. Treat the exact seconds as cluster-specific, not as a contract.
Illustrative flow: the application exits, kubelet restarts it according to restartPolicy, then waits during exponential back-off while the container Waiting reason is CrashLoopBackOff. CrashLoopBackOff is the waiting symptom, not the root cause.
Illustrative scenario. A Deployment named api shows a Pod with STATUS CrashLoopBackOff and a high restart count (for example 47). Watching the Pod, you may see brief Running periods with READY 0/1, then Error, then CrashLoopBackOff again as the restart count climbs.
The animation above shows that loop, then the investigation: describe output, previous logs, a missing DATABASE_URL, a Secret-backed fix, and a new Ready Pod with restart count 0. Use it as a map of the workflow, not as evidence from a specific cluster.
Start with the list view, then describe the failing Pod.
kubectl get pods -l app=api -o wide
kubectl get pods -l app=api -wIllustrative output:
NAME READY STATUS RESTARTS AGE
api-7d9f8c6b5-xk2nq 0/1 CrashLoopBackOff 47 (3m12s ago) 4hConfirm the owner and image before changing anything:
kubectl get deploy api -o wide
kubectl get rs -l app=apikubectl describe pod is where Waiting reason, last termination, and events come together.
kubectl describe pod api-7d9f8c6b5-xk2nqLook for a container status shaped like this illustrative fragment:
State: Waiting
Reason: CrashLoopBackOff
Last State: Terminated
Reason: Error
Exit Code: 1
Started: ...
Finished: ...
Ready: False
Restart Count: 48And an event such as:
Warning BackOff kubelet Back-off restarting failed container api in pod api-7d9f8c6b5-xk2nq_...Interpret the fields separately:
| Signal | What it tells you |
|---|---|
| Waiting / CrashLoopBackOff | kubelet is delaying the next start |
| Last State Terminated | the previous container run ended |
| Exit Code | how the process ended (for example 1 = application error; 137 often OOM-related) |
| Events | probe failures, image pulls, back-off, scheduling |
Current logs may be empty if the new attempt has not produced output yet. Use the previous container:
kubectl logs api-7d9f8c6b5-xk2nq --previous
# Multi-container Pods:
kubectl logs api-7d9f8c6b5-xk2nq -c api --previousIllustrative previous logs for this scenario:
ERROR required environment variable DATABASE_URL is not set
FATAL config validation failed, exiting (code 1)That points to an application exit after the container started, not a failure to create the container.
Compare what the app needs with what the Pod actually received.
kubectl set env deployment/api --listIllustrative list (missing DATABASE_URL):
# deployments/api, container api
LOG_LEVEL=info
PORT=8080You can also inspect the rendered Pod spec:
kubectl get pod api-7d9f8c6b5-xk2nq \
-o jsonpath='{range .spec.containers[0].env[*]}{.name}{"\n"}{end}'In this illustrative scenario, DATABASE_URL is absent. The container still starts; the process validates config and exits with code 1. That path produces CrashLoopBackOff.
Contrast that with a non-optional secretKeyRef to a missing Secret or key: kubelet cannot create the container, and you typically see CreateContainerConfigError instead of an application crash loop.
The fix belongs on the Deployment (or its GitOps source), using an existing Secret. Here the Secret api-db already has key url.
# Illustrative container env fragment added to deployment/api
env:
- name: LOG_LEVEL
value: info
- name: PORT
value: "8080"
- name: DATABASE_URL
valueFrom:
secretKeyRef:
name: api-db
key: urlApply the reviewed manifest, then wait:
kubectl apply -f deploy.yaml
kubectl rollout status deployment/apiPreferred for reviewed environments: add the env entry in the Deployment manifest (or Kustomize/Helm/GitOps source), apply or reconcile, then wait for the rollout.
env:
- name: DATABASE_URL
valueFrom:
secretKeyRef:
name: api-db
key: urlPods only pick up env changes when they are recreated by a rollout. Deleting the Pod without fixing the Deployment leaves the next Pod with the same missing variable.
Confirm the new ReplicaSet Pod is Ready and stable:
kubectl rollout status deployment/api
kubectl get pods -l app=api
kubectl get pod -l app=api -o jsonpath='{range .items[*]}{.metadata.name}{" "}{.status.containerStatuses[0].restartCount}{"\n"}{end}'
kubectl logs deploy/api --tail=50Illustrative healthy result:
NAME READY STATUS RESTARTS AGE
api-6b8c4d9f7-p8m2n 1/1 Running 0 45sA new Pod starts with restart count 0. That does not prove the old Pod was “healed”; it proves the replacement was created from a corrected template. If you only delete the crashing Pod and leave the Deployment unchanged, the next Pod can crash again immediately.
CrashLoopBackOff has many upstream causes. After you read exit code, events, and previous logs, map the evidence:
| Pattern | Typical signals | First checks |
|---|---|---|
| OOM | Exit code 137, Reason OOMKilled | Memory limits vs working set; leaks; oversized JVM/heap |
| Application exception | Non-zero exit, stack traces in --previous logs | Config, migrations, dependency URLs, fatal assert paths |
| Liveness / startup probe | Events show probe failures; kubelet restarts the container | Probe path/port, initialDelay/thresholds, slow startup |
| Bad command / args / entrypoint | Immediate exit, often code 1 or 127 | Container command/args, image ENTRYPOINT, shell vs exec form |
| Wrong or incomplete config | Missing files/env, auth failures at boot | ConfigMaps, Secrets, volume mounts, optional vs required refs |
Probes deserve special care: a failing liveness or startup probe causes kubelet to restart the container even when the main process has not self-exited yet. Events usually mention the probe. That is still a restart loop; the Waiting reason can still surface as CrashLoopBackOff after repeated failures.
kubectl get pods and describe. Note Waiting reason, Last State, exit code, and events.
CreateContainerConfigError usually means a missing Secret/ConfigMap/key. CrashLoopBackOff means the container started, then exited.
kubectl logs --previous (and -c for multi-container Pods) shows why the last process exited.
Check env, command/args, probes, resources, and image. A missing DATABASE_URL is an application exit; a broken secretKeyRef is a config error.
Change the Deployment (or GitOps source), wait for rollout, confirm Ready 1/1 and restart count 0. Do not treat delete/restart as the fix.
Accessible step flow for CrashLoopBackOff investigation. Use evidence from describe, events, and previous logs before changing the workload.
Use the tree as a discipline check: status → startability → previous logs → config/probes/resources → durable fix → verify Ready and restart count.
secretKeyRef / configMapKeyRef over copied plaintext values; keep key names aligned with what the app reads.kubectl rollout status (or equivalent) and confirm the new Pod’s env and logs.No. It means kubelet is waiting during restart back-off after repeated container failures. The cause is in termination state, events, and logs.
kubectl logs empty?The current container attempt may not have logged yet, or it crashed before writing. Use kubectl logs --previous and specify -c when the Pod has multiple containers.
Only if the new Pod template already contains the correction. Restarting a Deployment that still omits DATABASE_URL recreates the same failure.
CrashLoopBackOff usually means the container started and then failed or was restarted. CreateContainerConfigError usually means kubelet could not create the container—commonly a missing non-optional Secret, ConfigMap, or key.
No. Many clusters still behave like the classic 10s → double → cap at 5 minutes curve with a ~10 minute successful-run reset, but kubelet version and configuration can change that. Confirm against your cluster’s Kubernetes docs and kubelet settings.
Need help troubleshooting Kubernetes workloads or improving platform reliability? Explore Cloud9 Solutions.
Understand readiness, surge capacity, graceful shutdown, and telemetry without treating rolling updates as an availability guarantee.
When hosts and probes look fine but users still fail — separating ALB health checks, Kubernetes readiness, and dependency health from real traffic paths.
How HPA, resource requests, and cluster capacity interact—and why a target utilization is not a capacity plan.