Kubernetes

Kubernetes CrashLoopBackOff: How to Diagnose and Fix Repeated Pod Restarts

Diagnose Kubernetes CrashLoopBackOff by reading Waiting state, exit codes, kubectl logs --previous, and env config—then fix the cause instead of restarting the Pod.

Illustrative CrashLoopBackOff investigation: missing DATABASE_URL, previous logs, Secret mapping, and a healthy rollout. Educational example data — not a client incident.

Muted looping animation of an illustrative kubectl investigation: CrashLoopBackOff with a high restart count, describe output, previous logs showing a missing DATABASE_URL, a Secret-backed fix, and a healthy rollout.

What CrashLoopBackOff means

CrashLoopBackOff is a Waiting-state reason shown while kubelet delays the next restart after a container has exited repeatedly. It is a symptom of a restart loop, not the root cause.

A Pod can look “stuck” in CrashLoopBackOff while the real problem is elsewhere: a missing environment variable, a bad command, an application panic, an OOM kill, or a probe that keeps killing a still-starting process. The waiting reason tells you Kubernetes is applying restart back-off. Your job is to find why the container keeps exiting or being restarted.

This guide walks an illustrative api Deployment through diagnosis and repair. Pod names, restart counts, and log lines are educational example data—not a personal incident, client story, or production metric claim.

How Kubernetes handles container restarts and back-off

Deployment Pods use restartPolicy: Always. When a container exits, kubelet starts it again according to that policy. After repeated failures, kubelet waits between attempts with an exponential back-off.

Documented defaults many operators still reason about are an initial delay around 10s, doubling on each failure, capped at five minutes, and a reset after the container has run successfully for about 10 minutes. Timing is not universal: Kubernetes version, kubelet configuration, and feature gates (including tunable or reduced CrashLoopBackOff curves in newer releases) can change the schedule. Treat the exact seconds as cluster-specific, not as a contract.

ILLUSTRATIVECrashLoopBackOff loop
App exitsexit code / logskubelet restartrestartPolicyBack-off waitCrashLoopBackOffStartLoop until the underlying failure is fixed

Illustrative flow: the application exits, kubelet restarts it according to restartPolicy, then waits during exponential back-off while the container Waiting reason is CrashLoopBackOff. CrashLoopBackOff is the waiting symptom, not the root cause.

Example scenario: 47 restarts

Illustrative scenario. A Deployment named api shows a Pod with STATUS CrashLoopBackOff and a high restart count (for example 47). Watching the Pod, you may see brief Running periods with READY 0/1, then Error, then CrashLoopBackOff again as the restart count climbs.

The animation above shows that loop, then the investigation: describe output, previous logs, a missing DATABASE_URL, a Secret-backed fix, and a new Ready Pod with restart count 0. Use it as a map of the workflow, not as evidence from a specific cluster.

Inspecting Pod status with kubectl

Start with the list view, then describe the failing Pod.

BASH · kubectl-get-pods.sh
kubectl get pods -l app=api -o wide
kubectl get pods -l app=api -w

Illustrative output:

TEXT · illustrative-get-pods.txt
NAME                   READY   STATUS             RESTARTS         AGE
api-7d9f8c6b5-xk2nq    0/1     CrashLoopBackOff   47 (3m12s ago)   4h

Confirm the owner and image before changing anything:

BASH · kubectl-get-deploy.sh
kubectl get deploy api -o wide
kubectl get rs -l app=api

Reading events and termination info

kubectl describe pod is where Waiting reason, last termination, and events come together.

BASH · kubectl-describe-pod.sh
kubectl describe pod api-7d9f8c6b5-xk2nq

Look for a container status shaped like this illustrative fragment:

TEXT · illustrative-describe.txt
State:          Waiting
  Reason:       CrashLoopBackOff
Last State:     Terminated
  Reason:       Error
  Exit Code:    1
  Started:      ...
  Finished:     ...
Ready:          False
Restart Count:  48

And an event such as:

TEXT · illustrative-backoff-event.txt
Warning  BackOff  kubelet  Back-off restarting failed container api in pod api-7d9f8c6b5-xk2nq_...

Interpret the fields separately:

SignalWhat it tells you
Waiting / CrashLoopBackOffkubelet is delaying the next start
Last State Terminatedthe previous container run ended
Exit Codehow the process ended (for example 1 = application error; 137 often OOM-related)
Eventsprobe failures, image pulls, back-off, scheduling

Previous container logs

Current logs may be empty if the new attempt has not produced output yet. Use the previous container:

BASH · kubectl-logs-previous.sh
kubectl logs api-7d9f8c6b5-xk2nq --previous
# Multi-container Pods:
kubectl logs api-7d9f8c6b5-xk2nq -c api --previous

Illustrative previous logs for this scenario:

TEXT · illustrative-previous-logs.txt
ERROR required environment variable DATABASE_URL is not set
FATAL config validation failed, exiting (code 1)

That points to an application exit after the container started, not a failure to create the container.

Finding the missing DATABASE_URL

Compare what the app needs with what the Pod actually received.

BASH · kubectl-set-env-list.sh
kubectl set env deployment/api --list

Illustrative list (missing DATABASE_URL):

TEXT · illustrative-env-list.txt
# deployments/api, container api
LOG_LEVEL=info
PORT=8080

You can also inspect the rendered Pod spec:

BASH · kubectl-jsonpath-env.sh
kubectl get pod api-7d9f8c6b5-xk2nq \
  -o jsonpath='{range .spec.containers[0].env[*]}{.name}{"\n"}{end}'

In this illustrative scenario, DATABASE_URL is absent. The container still starts; the process validates config and exits with code 1. That path produces CrashLoopBackOff.

Contrast that with a non-optional secretKeyRef to a missing Secret or key: kubelet cannot create the container, and you typically see CreateContainerConfigError instead of an application crash loop.

Fixing the Deployment from an existing Secret

The fix belongs on the Deployment (or its GitOps source), using an existing Secret. Here the Secret api-db already has key url.

YAML · deploy-env-diff.yaml
# Illustrative container env fragment added to deployment/api
env:
  - name: LOG_LEVEL
    value: info
  - name: PORT
    value: "8080"
  - name: DATABASE_URL
    valueFrom:
      secretKeyRef:
        name: api-db
        key: url

Apply the reviewed manifest, then wait:

BASH · kubectl-apply-rollout.sh
kubectl apply -f deploy.yaml
kubectl rollout status deployment/api

Preferred for reviewed environments: add the env entry in the Deployment manifest (or Kustomize/Helm/GitOps source), apply or reconcile, then wait for the rollout.

env:
  - name: DATABASE_URL
    valueFrom:
      secretKeyRef:
        name: api-db
        key: url

Pods only pick up env changes when they are recreated by a rollout. Deleting the Pod without fixing the Deployment leaves the next Pod with the same missing variable.

Verifying rollout and readiness

Confirm the new ReplicaSet Pod is Ready and stable:

BASH · kubectl-verify.sh
kubectl rollout status deployment/api
kubectl get pods -l app=api
kubectl get pod -l app=api -o jsonpath='{range .items[*]}{.metadata.name}{" "}{.status.containerStatuses[0].restartCount}{"\n"}{end}'
kubectl logs deploy/api --tail=50

Illustrative healthy result:

TEXT · illustrative-healthy-pod.txt
NAME                   READY   STATUS    RESTARTS   AGE
api-6b8c4d9f7-p8m2n    1/1     Running   0          45s

A new Pod starts with restart count 0. That does not prove the old Pod was “healed”; it proves the replacement was created from a corrected template. If you only delete the crashing Pod and leave the Deployment unchanged, the next Pod can crash again immediately.

Other causes

CrashLoopBackOff has many upstream causes. After you read exit code, events, and previous logs, map the evidence:

PatternTypical signalsFirst checks
OOMExit code 137, Reason OOMKilledMemory limits vs working set; leaks; oversized JVM/heap
Application exceptionNon-zero exit, stack traces in --previous logsConfig, migrations, dependency URLs, fatal assert paths
Liveness / startup probeEvents show probe failures; kubelet restarts the containerProbe path/port, initialDelay/thresholds, slow startup
Bad command / args / entrypointImmediate exit, often code 1 or 127Container command/args, image ENTRYPOINT, shell vs exec form
Wrong or incomplete configMissing files/env, auth failures at bootConfigMaps, Secrets, volume mounts, optional vs required refs

Probes deserve special care: a failing liveness or startup probe causes kubelet to restart the container even when the main process has not self-exited yet. Events usually mention the probe. That is still a restart loop; the Waiting reason can still surface as CrashLoopBackOff after repeated failures.

Troubleshooting decision tree

ILLUSTRATIVETroubleshooting decision flow
  1. Read Pod status

    kubectl get pods and describe. Note Waiting reason, Last State, exit code, and events.

  2. Did the container start?

    CreateContainerConfigError usually means a missing Secret/ConfigMap/key. CrashLoopBackOff means the container started, then exited.

  3. Inspect previous logs

    kubectl logs --previous (and -c for multi-container Pods) shows why the last process exited.

  4. Compare expected config

    Check env, command/args, probes, resources, and image. A missing DATABASE_URL is an application exit; a broken secretKeyRef is a config error.

  5. Fix and roll out

    Change the Deployment (or GitOps source), wait for rollout, confirm Ready 1/1 and restart count 0. Do not treat delete/restart as the fix.

Accessible step flow for CrashLoopBackOff investigation. Use evidence from describe, events, and previous logs before changing the workload.

Use the tree as a discipline check: status → startability → previous logs → config/probes/resources → durable fix → verify Ready and restart count.

Production prevention checklist

  • Fail fast in pre-deploy checks when required Secret keys or ConfigMap keys are missing.
  • Keep required env vars declared in the same reviewed manifest/GitOps path as the Deployment.
  • Prefer secretKeyRef / configMapKeyRef over copied plaintext values; keep key names aligned with what the app reads.
  • Separate readiness from liveness; give slow starts a startup probe instead of aggressive liveness.
  • Set memory requests/limits from measured usage so OOM is visible and tunable.
  • Alert on rising restart counts and repeated BackOff events, not only on “Pod not Running”.
  • After config changes, wait on kubectl rollout status (or equivalent) and confirm the new Pod’s env and logs.
  • Do not run “delete pod” runbooks as the primary fix for CrashLoopBackOff.

FAQ

Is CrashLoopBackOff itself the error?

No. It means kubelet is waiting during restart back-off after repeated container failures. The cause is in termination state, events, and logs.

Why is kubectl logs empty?

The current container attempt may not have logged yet, or it crashed before writing. Use kubectl logs --previous and specify -c when the Pod has multiple containers.

Will restarting the Deployment fix it?

Only if the new Pod template already contains the correction. Restarting a Deployment that still omits DATABASE_URL recreates the same failure.

How is this different from CreateContainerConfigError?

CrashLoopBackOff usually means the container started and then failed or was restarted. CreateContainerConfigError usually means kubelet could not create the container—commonly a missing non-optional Secret, ConfigMap, or key.

Do back-off timers always start at 10 seconds?

No. Many clusters still behave like the classic 10s → double → cap at 5 minutes curve with a ~10 minute successful-run reset, but kubelet version and configuration can change that. Confirm against your cluster’s Kubernetes docs and kubelet settings.

Need help troubleshooting Kubernetes workloads or improving platform reliability? Explore Cloud9 Solutions.

Related technical guides

LET’S BUILD WHAT’S NEXT

Good engineering starts with a conversation.

Let’s connect